Efficient Language Modeling with Sparse all-MLP

Benchmark Model Rank Results
common-sense-reasoning-on-recordBase Layers 10B (0-shot)EM: 60.7
common-sense-reasoning-on-recordGshard 9BEM: 72.4
common-sense-reasoning-on-recordHASH Layers 10B (0-shot)EM: 67.2
common-sense-reasoning-on-recordSwitch Transformer 9BEM: 79.9
common-sense-reasoning-on-recordsMLP – deterministic 9.4B (0-shot)EM: 73.4
common-sense-reasoning-on-winograndeBase Layers 10B (0-shot)Accuracy: 51
common-sense-reasoning-on-winograndeGshard 9B (0-shot)Accuracy: 51.1
common-sense-reasoning-on-winograndeHASH Layers 10B (0-shot)Accuracy: 51.7
common-sense-reasoning-on-winograndeSwitch Transformer 9B (0-shot)Accuracy: 53.4
common-sense-reasoning-on-winograndesMLP – deterministic 9.4B (0-shot)Accuracy: 54.3
question-answering-on-copaBase Layers 10B (0-shot)Accuracy: 63
question-answering-on-copaGshard 9BAccuracy: 76
question-answering-on-copaHASH Layers 10B (0-shot)Accuracy: 64
question-answering-on-copaSwitch Transformer 9BAccuracy: 75
question-answering-on-copasMLP – deterministic 9.4B (0-shot)Accuracy: 79
question-answering-on-piqaBase Layers 10B (0-shot)Accuracy: 63.8
question-answering-on-piqaGshard 9BAccuracy: 68.1
question-answering-on-piqaHASH Layers 10B (0-shot)Accuracy: 63.8
question-answering-on-piqasMLP - deterministic 9.4B (0-shot)Accuracy: 73
question-answering-on-storyclozeBase Layers 10B (0-shot)Accuracy: 61.4
question-answering-on-storyclozeGshard 9BAccuracy: 67.9
question-answering-on-storyclozeHASH Layers 10B (0-shot)Accuracy: 64.7
question-answering-on-storyclozeSwitch Transformer 9BAccuracy: 73.3
question-answering-on-storyclozesMLP – deterministic 9.4B (0-shot)Accuracy: 74.7
sentence-completion-on-hellaswagBase Layers 10B (0-shot)Accuracy: 30.2
sentence-completion-on-hellaswagGshard 9BAccuracy: 38
sentence-completion-on-hellaswagHASH Layers 10B (0-shot)Accuracy: 33
sentence-completion-on-hellaswagSwitch Transformer 9BAccuracy: 52.5
sentence-completion-on-hellaswagsMLP – deterministic 9.4B (0-shot)Accuracy: 54.5