Training Compute-Optimal Large Language Models

Benchmark Model Rank Results
common-sense-reasoning-on-big-benchChinchilla-70B (few-shot, k=5)#4Accuracy: 54.7
common-sense-reasoning-on-big-bench-causalChinchilla-70B (few-shot, k=5)#4Accuracy: 57.4
common-sense-reasoning-on-big-bench-dateChinchilla-70B (few-shot, k=5)#5Accuracy: 52.3
common-sense-reasoning-on-big-bench-knownChinchilla-70B (few-shot, k=5)#2Accuracy: 65.2
common-sense-reasoning-on-big-bench-sportsChinchilla-70B (few-shot, k=5)#4Accuracy: 71
common-sense-reasoning-on-big-bench-winowhyChinchilla-70B (few-shot, k=5)#2Accuracy: 62.5
common-sense-reasoning-on-winograndeChinchilla 70B (0-shot)#26Accuracy: 74.9
crass-ai-on-big-benchChinchilla-70B (few-shot, k=5)#1Accuracy: 75.0
language-modelling-on-lambadaChinchilla (Zero-Shot)#11Accuracy: 77.7
logical-reasoning-on-big-bench-formalChinchilla-70B (few-shot, k=5)#7Accuracy: 52.1
logical-reasoning-on-big-bench-logic-gridChinchilla-70B (few-shot, k=5)#1Accuracy: 44
logical-reasoning-on-big-bench-penguins-in-aChinchilla-70B (few-shot, k=5)#3Accuracy: 48.7
logical-reasoning-on-big-bench-reasoningChinchilla-70B (few-shot, k=5)#3Accuracy: 59.7
logical-reasoning-on-big-bench-strategyqaChinchilla-70B (few-shot, k=5)#2Accuracy: 68.3
logical-reasoning-on-big-bench-temporalChinchilla-70B (few-shot, k=5)#5Accuracy: 32.0
multi-task-language-understanding-on-mmluchatgpt/gpt3.5(20B)#9Average (%): 67.5
multiple-choice-question-answering-mcqa-on-27Chinchilla-70B (few-shot, k=5)#8Accuracy: 54.2
multiple-choice-question-answering-mcqa-on-28Chinchilla-70B (few-shot, k=5)#8Accuracy: 75.6
multiple-choice-question-answering-mcqa-on-29Chinchilla-70B (few-shot, k=5)#4Accuracy: 52.6
multiple-choice-question-answering-mcqa-on-30Chinchilla-70B (few-shot, k=5)#8Accuracy: 47.1
multiple-choice-question-answering-mcqa-on-31Chinchilla-70B (few-shot, k=5)#2Accuracy: 65.6
question-answering-on-boolqChinchilla 70B (0-shot)#19Accuracy: 83.7
question-answering-on-natural-questionsChinchilla (few-shot, k=64)#21EM: 35.5
question-answering-on-piqaChinchilla 70B (0-shot)#25Accuracy: 81.8
question-answering-on-social-iqaChinchilla (zero-shot)#19Accuracy: 51.3
sarcasm-detection-on-big-bench-snarksChinchilla-70B (few-shot, k=5)#7Accuracy: 58.6
sentence-completion-on-hellaswagChinchilla 70B (0-shot)#33Accuracy: 80.8
word-sense-disambiguation-on-big-benchChinchilla-70B (few-shot, k=5)#1Accuracy: 69.1