Breaking the Softmax Bottleneck: A High-Rank RNN Language Model

Benchmark Model Rank Results
language-modelling-on-penn-treebank-wordAWD-LSTM-MoS + dynamic eval#10Test perplexity: 47.69Validation perplexity: 48.33Params: 22M
language-modelling-on-penn-treebank-wordAWD-LSTM-MoS#19Test perplexity: 54.44Validation perplexity: 56.54Params: 22M
language-modelling-on-wikitext-2AWD-LSTM-MoS + dynamic eval#15Test perplexity: 40.68Validation perplexity: 42.41
language-modelling-on-wikitext-2AWD-LSTM-MoS#25Test perplexity: 61.45Validation perplexity: 63.88