Pay Attention when Required

Benchmark Model Rank Results
language-modelling-on-text8PAR Transformer 24B#13Bit per Character (BPC): 1.18
language-modelling-on-wikitext-103PAR Transformer Large#35Test perplexity: 18.4
language-modelling-on-wikitext-103PAR Transformer Base#47Test perplexity: 22.7
sentiment-analysis-on-sst-2-binaryPAR BERT Base#46Accuracy: 91.6