Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts

Benchmark Model Rank Results
common-sense-reasoning-on-arc-easyUpcycled 32→64 (100% CPT)#32Accuracy: 76.0
question-answering-on-piqaUpcycled 32→64 (100% CPT)#87Accuracy: 77.4
sentence-completion-on-hellaswagUpcycled 32→64 (100% CPT)#122Accuracy: 67.3
arithmetic-reasoning-on-gsm8kUpcycled 32→64 (100% CPT)#241Accuracy: 48.3