ST-MoE: Designing Stable and Transferable Sparse Expert Models

Open paper
Benchmark Model Rank Results
common-sense-reasoning-on-arc-challengeST-MoE-32B 269B (fine-tuned)#3Accuracy: 86.5
common-sense-reasoning-on-arc-challengeST-MoE-L 4.1B (fine-tuned)#17Accuracy: 56.9
common-sense-reasoning-on-arc-easyST-MoE-32B 269B (fine-tuned)#1Accuracy: 95.2
common-sense-reasoning-on-arc-easyST-MoE-L 4.1B (fine-tuned)#20Accuracy: 75.4
common-sense-reasoning-on-recordST-MoE-32B 269B (fine-tuned)#1EM: 95.1
common-sense-reasoning-on-recordST-MoE-L 4.1B (fine-tuned)#7EM: 88.9
common-sense-reasoning-on-winograndeST-MoE-32B 269B (fine-tuned)#1Accuracy: 96.1
common-sense-reasoning-on-winograndeST-MoE-L 4.1B (fine-tuned)#12Accuracy: 81.7
coreference-resolution-on-winograd-schemaST-MoE-32B 269B (fine-tuned)#3Accuracy: 96.6
coreference-resolution-on-winograd-schemaST-MoE-L 4.1B (fine-tuned)#6Accuracy: 93.3
natural-language-inference-on-commitmentbankST-MoE-L 4.1B (fine-tuned)#2Accuracy: 98.2
natural-language-inference-on-commitmentbankST-MoE-32B 269B (fine-tuned)#3Accuracy: 98
natural-language-inference-on-rteST-MoE-32B 269B (fine-tuned)#2Accuracy: 93.5%
natural-language-inference-on-rteST-MoE-L 4.1B (fine-tuned)#8Accuracy: 92.1%
question-answering-on-boolqST-MoE-32B 269B (fine-tuned)#2Accuracy: 92.4
question-answering-on-boolqST-MoE-L 4.1B (fine-tuned)#8Accuracy: 88.6
question-answering-on-copaST-MoE-32B 269B (fine-tuned)#2Accuracy: 99.2
question-answering-on-copaST-MoE-L 4.1B (fine-tuned)#13Accuracy: 91
question-answering-on-multircST-MoE-32B 269B (fine-tuned)#2F1: 89.6
question-answering-on-multircST-MoE-L 4.1B (fine-tuned)#6F1: 86
word-sense-disambiguation-on-words-in-contextST-MoE-32B 269B (fine-tuned)#3Accuracy: 77.7
word-sense-disambiguation-on-words-in-contextST-MoE-L 4.1B (fine-tuned)#8Accuracy: 74

Rank counts only results with a code link.