Dynamic Evaluation of Transformer Language Models

Benchmark Model Rank Results
language-modelling-on-enwiki8Transformer-XL (24 layers, RMS dynamic eval, decay)#2Bit per Character (BPC): 0.940Number of params: 277M
language-modelling-on-hutter-prizeTransformer-XL + RMS dynamic eval#1Bit per Character (BPC): 0.94Number of params: 277M
language-modelling-on-text8Transformer-XL + RMS dynamic eval + decay#2Bit per Character (BPC): 1.038Number of params: 277M
language-modelling-on-wikitext-103Transformer-XL (RMS dynamic eval)#13Test perplexity: 16.4Validation perplexity: 15.8
language-modelling-on-wikitext-103Transformer-XL (SGD dynamic eval)#19Test perplexity: 17.0Validation perplexity: 16.3