SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
Open paper
Benchmark
Model
Rank
Results
multi-task-language-understanding-on-mmlu-pro
SlimQwen (23A2B, NTP KD + LM)
–
Accuracy: 51.44
Rank counts only results with a code link.