Pay Attention when Required

Benchmark Model Rank Results
language-modelling-on-text8PAR Transformer 24B#15Bit per Character (BPC): 1.18
language-modelling-on-wikitext-103PAR Transformer Large#39Test perplexity: 18.4
language-modelling-on-wikitext-103PAR Transformer Base#51Test perplexity: 22.7
sentiment-analysis-on-sst-2-binary-classificationPAR BERT Base#49Accuracy: 91.6