FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach

Benchmark Model Rank Results
common-sense-reasoning-on-winograndeFineRMoE (Qwen2.5-7B)#54Accuracy: 75.69
code-generation-on-mbppFineRMoE (Qwen2.5-7B)#60Accuracy: 69.4
sentence-completion-on-hellaswagFineRMoE (Qwen2.5-7B)#77Accuracy: 79.51
arithmetic-reasoning-on-gsm8kFineRMoE (Qwen2.5-7B)#113Accuracy: 85.37