Learnable Multipliers: Freeing the Scale of Language Model Matrix Layers

Benchmark Model Rank Results
multi-task-language-understanding-on-mmlu-proFalcon-H1-0.5B with Learnable Multipliers (Adam)#133Accuracy: 19.92
sentence-completion-on-hellaswagFalcon-H1-0.5B with Learnable Multipliers (Muon)#179Accuracy: 50.56
arithmetic-reasoning-on-gsm8kFalcon-H1-0.5B with Learnable Multipliers (Muon)#229Accuracy: 50.63