Shortformer: Better Language Modeling using Shorter Inputs
Open paper
Benchmark
Model
Rank
Results
language-modelling-on-wikitext-103
Staged Training
#26
Test perplexity: 17.56
Validation perplexity: 16.89
…
language-modelling-on-wikitext-103
Shortformer
#31
Test perplexity: 18.15
Validation perplexity: 17.47
…
Rank counts only results with a code link.