Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts
Open paper
Benchmark
Model
Rank
Results
code-generation-on-humaneval
Foundation MoE (155B total, 17B active)
–
Pass@1: 56.3
Rank counts only results with a code link.