Distributionally Robust Mixture-of-Experts Training

Benchmark Model Rank Results
multiple-choice-question-answering-mcqa-on-arc-eDRMoET (1.7B active, 10.3B total, eta=0.01)#8Accuracy: 72.1
multiple-choice-question-answering-mcqa-on-arc-cDRMoET (1.7B active, 10.3B total, eta=0.01)#21Accuracy: 38.05
question-answering-on-piqaDRMoET (1.7B active, 10.3B total, eta=0.01)#92Accuracy: 76.66
common-sense-reasoning-on-winograndeDRMoET (1.7B active, 10.3B total, eta=0.01)#105Accuracy: 62.35
sentence-completion-on-hellaswagDRMoET (1.7B active, 10.3B total, eta=0.001)#181Accuracy: 49.96