When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute

Benchmark Model Rank Results
language-modelling-on-enwiki8SRU++ Large#3Bit per Character (BPC): 0.95Number of params: 195M
language-modelling-on-enwiki8SRU++ Base#8Bit per Character (BPC): 0.97Number of params: 108M
language-modelling-on-one-billion-wordSRU++ Large#9PPL: 23.5Number of params: 465M
language-modelling-on-one-billion-wordSRU++#13PPL: 25.1Number of params: 328M
language-modelling-on-wikitext-103SRU++ Large#22Test perplexity: 17.1Validation perplexity: 16.4
language-modelling-on-wikitext-103SRU++ Base#34Test perplexity: 18.3Validation perplexity: 17.5