Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

Open paper

Benchmark	Model	Rank	Results
common-sense-reasoning-on-arc-challenge	Pythia 12B (5-shot)	#36	Accuracy: 36.8
common-sense-reasoning-on-arc-challenge	Pythia 12B (0-shot)	#38	Accuracy: 31.8
common-sense-reasoning-on-arc-easy	Pythia 12B (5-shot)	#24	Accuracy: 71.5
common-sense-reasoning-on-arc-easy	Pythia 12B (0-shot)	#29	Accuracy: 70.2
common-sense-reasoning-on-winogrande	Pythia 12B (5-shot)	#39	Accuracy: 66.6
common-sense-reasoning-on-winogrande	Pythia 12B (0-shot)	#43	Accuracy: 63.9
common-sense-reasoning-on-winogrande	Pythia 6.9B (0-shot)	#45	Accuracy: 60.9
common-sense-reasoning-on-winogrande	Pythia 2.8B (0-shot)	#48	Accuracy: 59.4
coreference-resolution-on-winograd-schema	Pythia 12B (0-shot)	#55	Accuracy: 54.8
coreference-resolution-on-winograd-schema	Pythia 2.8B (0-shot)	#61	Accuracy: 38.5
coreference-resolution-on-winograd-schema	Pythia 6.9B (0-shot)	#63	Accuracy: 36.5
coreference-resolution-on-winograd-schema	Pythia 12B (5-shot)	#64	Accuracy: 36.5
language-modelling-on-lambada	Pythia 12B (0-shot)	#17	Accuracy: 70.46
language-modelling-on-lambada	Pythia 6.9B (0-shot)	#20	Accuracy: 67.28
language-modelling-on-lambada	Pythia 12B(Zero-Shot)	#28	Perplexity: 3.92
language-modelling-on-lambada	Pythia 6.9B(Zero-Shot)	#29	Perplexity: 4.45
question-answering-on-piqa	Pythia 12B (5-shot)	#43	Accuracy: 76.7
question-answering-on-piqa	Pythia 12B (0-shot)	#45	Accuracy: 76
question-answering-on-piqa	Pythia 6.9B (0-shot)	#48	Accuracy: 75.2
question-answering-on-piqa	Pythia 1B (5-shot)	#55	Accuracy: 70.4