Variable-Width Transformers

Open paper

Benchmark	Model	Rank	Results
reading-comprehension-on-race	3B/1B MoE > <former	#31	Accuracy: 34.3
question-answering-on-boolq	2B > <former	#96	Accuracy: 60.9
common-sense-reasoning-on-winogrande	2B > <former	#116	Accuracy: 60.2