Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

Open paper

Benchmark	Model	Rank	Results
language-modelling-on-lambada	Pythia 12B (0-shot)	#21	Accuracy: 70.46
language-modelling-on-lambada	Pythia 6.9B (0-shot)	#29	Accuracy: 67.28
language-modelling-on-lambada	Pythia 12B(Zero-Shot)	#68	Perplexity: 3.92
language-modelling-on-lambada	Pythia 6.9B(Zero-Shot)	#69	Perplexity: 4.45
common-sense-reasoning-on-arc-challenge	Pythia 12B (5-shot)	#37	Accuracy: 36.8
common-sense-reasoning-on-arc-challenge	Pythia 12B (0-shot)	#39	Accuracy: 31.8
common-sense-reasoning-on-arc-easy	Pythia 12B (5-shot)	#52	Accuracy: 71.5
common-sense-reasoning-on-arc-easy	Pythia 12B (0-shot)	#57	Accuracy: 70.2
coreference-resolution-on-winograd-schema-challenge	Pythia 12B (0-shot)	#55	Accuracy: 54.8
coreference-resolution-on-winograd-schema-challenge	Pythia 2.8B (0-shot)	#61	Accuracy: 38.5
coreference-resolution-on-winograd-schema-challenge	Pythia 6.9B (0-shot)	#63	Accuracy: 36.5
coreference-resolution-on-winograd-schema-challenge	Pythia 12B (5-shot)	#64	Accuracy: 36.5
common-sense-reasoning-on-winogrande	Pythia 12B (5-shot)	#91	Accuracy: 66.6
common-sense-reasoning-on-winogrande	Pythia 12B (0-shot)	#100	Accuracy: 63.9
common-sense-reasoning-on-winogrande	Pythia 6.9B (0-shot)	#111	Accuracy: 60.9
common-sense-reasoning-on-winogrande	Pythia 2.8B (0-shot)	#122	Accuracy: 59.4
question-answering-on-piqa	Pythia 12B (5-shot)	#97	Accuracy: 76.7
question-answering-on-piqa	Pythia 12B (0-shot)	#106	Accuracy: 76
question-answering-on-piqa	Pythia 6.9B (0-shot)	#113	Accuracy: 75.2
question-answering-on-piqa	Pythia 1B (5-shot)	#154	Accuracy: 70.4