Improved Language Modeling by Decoding the Past

Benchmark Model Rank Results
language-modelling-on-penn-treebank-character-levelPast Decode Reg. + AWD-LSTM-MoS + dyn. eval.–Bit per Character (BPC): 1.169Number of params: 13.8M
language-modelling-on-penn-treebank-word-levelPast Decode Reg. + AWD-LSTM-MoS + dyn. eval.–Test perplexity: 47.3Validation perplexity: 48.0Params: 22M
language-modelling-on-wikitext-2Past Decode Reg. + AWD-LSTM-MoS + dyn. eval.–Test perplexity: 40.3Validation perplexity: 42.0Number of params: 35M