Enabling Token-wise Importance Sampling in Off-Policy RLVR with Proximal KL Regularization

Benchmark Model Rank Results
mathematical-reasoning-on-minerva-mathProximal KL (Qwen3-4B-Base, off-policy, staleness 4, lr 5e-6)#22Accuracy: 40.44
mathematical-reasoning-on-olympiadbenchProximal KL (Qwen3-4B-Base, off-policy, staleness 16, lr 5e-6)#37Accuracy: 45.99
mathematical-reasoning-on-math-500Proximal KL (Qwen3-4B-Base, off-policy, staleness 4, lr 5e-6)#84Accuracy: 82.4
mathematical-reasoning-on-aime24Proximal KL (Qwen3-1.7B-Base, on-policy, lr 5e-6)#130Acc: 10.07
mathematical-reasoning-on-aime-2025Proximal KL (Qwen3-4B-Base, off-policy, staleness 4, lr 5e-6)#143Accuracy: 10.63