An Analysis of Neural Language Modeling at Multiple Scales

Benchmark Model Rank Results
language-modelling-on-enwiki8AWD-LSTM (3 layers)#31Bit per Character (BPC): 1.232Number of params: 47M
language-modelling-on-hutter-prize3-layer AWD-LSTM#13Bit per Character (BPC): 1.232Number of params: 47M
language-modelling-on-penn-treebank-character3-layer AWD-LSTM#6Bit per Character (BPC): 1.175Number of params: 13.8M
language-modelling-on-penn-treebank-character6-layer QRNN#8Bit per Character (BPC): 1.187Number of params: 13.8M
language-modelling-on-wikitext-1034 layer QRNN#70Test perplexity: 33.0Validation perplexity: 32.0