Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks

Benchmark Model Rank Results
arithmetic-reasoning-on-gsm8kCamelidae-8×34B (5-shot)#56Accuracy: 78.3
arithmetic-reasoning-on-gsm8kQwen2idae-16x14B (5-shot)#57Accuracy: 77.8
code-generation-on-mbppQwen2idae-16x14B (4-shot)#54Accuracy: 48.6
code-generation-on-mbppCamelidae-8×34B (4-shot)#71Accuracy: 41.4
common-sense-reasoning-on-arc-challengeCamelidae-8×34B#10Accuracy: 65.2
common-sense-reasoning-on-arc-easyCamelidae-8×34B#6Accuracy: 86.2
common-sense-reasoning-on-winograndeCamelidae-8×34B#15Accuracy: 80.9
math-word-problem-solving-on-mathQwen2idae-16x14B (4-shot)#76Accuracy: 29.9
math-word-problem-solving-on-mathCamelidae-8×34B (4-shot)#86Accuracy: 22.6
question-answering-on-piqaCamelidae-8×34B#18Accuracy: 82.7
sentence-completion-on-hellaswagCamelidae-8×34B (10-shot)#27Accuracy: 83.2
sentence-completion-on-hellaswagQwen2idae-16x14B (10-shot)#30Accuracy: 82.3