Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training

Open paper

Benchmark	Model	Rank	Results
question-answering-on-piqa	Qwen3 (1.7B) with DeMix (224 proxies)	#132	Accuracy: 72.94
common-sense-reasoning-on-winogrande	Qwen3 (1.7B) with DeMix (224 proxies)	#150	Accuracy: 55.43
arithmetic-reasoning-on-gsm8k	Qwen3 (1.7B) with DeMix (224 proxies)	#264	Accuracy: 20.98