Improved Language Modeling by Decoding the Past

Benchmark Model Rank Results
language-modelling-on-penn-treebank-characterPast Decode Reg. + AWD-LSTM-MoS + dyn. eval.Bit per Character (BPC): 1.169Number of params: 13.8M
language-modelling-on-penn-treebank-wordPast Decode Reg. + AWD-LSTM-MoS + dyn. eval.Test perplexity: 47.3Validation perplexity: 48.0Params: 22M
language-modelling-on-wikitext-2Past Decode Reg. + AWD-LSTM-MoS + dyn. eval.Test perplexity: 40.3Validation perplexity: 42.0Number of params: 35M