Efficient Pre-Training with Token Superposition

Benchmark Model Rank Results
common-sense-reasoning-on-arc-easyMoE TST (10B A1B)#39Accuracy: 74.2
question-answering-on-boolqMoE TST (10B A1B)#70Accuracy: 69.4
question-answering-on-piqaMoE TST (10B A1B)#88Accuracy: 77.4
common-sense-reasoning-on-winograndeMoE TST (10B A1B)#102Accuracy: 63.0
sentence-completion-on-hellaswagMoE TST (10B A1B)#103Accuracy: 71.2