From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation

Open paper

Benchmark	Model	Rank	Results
mathematical-reasoning-on-math-500	SmolLM3-3B (PG/domain-response SGD)	–	Accuracy: 53.8