Galactica: A Large Language Model for Science

Open paper
Benchmark Model Rank Results
molecular-property-prediction-on-moleculenetUni-Mol#1AUC: 0.77
molecular-property-prediction-on-moleculenetGAL 30B#2AUC: 0.69
molecular-property-prediction-on-moleculenetGAL 6.7B#3AUC: 0.64
molecular-property-prediction-on-moleculenetGAL 1.3B#4AUC: 0.619
molecular-property-prediction-on-moleculenetGAL 125M#5AUC: 0.581
multiple-choice-question-answering-mcqa-on-mmlu-abstract-algebraGAL 30B (zero-shot)#1Accuracy: 33.3
multiple-choice-question-answering-mcqa-on-mmlu-abstract-algebraChinchilla (few-shot, k=5)#2Accuracy: 31
multiple-choice-question-answering-mcqa-on-mmlu-abstract-algebraGAL 120B (zero-shot)#3Accuracy: 27
multiple-choice-question-answering-mcqa-on-mmlu-abstract-algebraGopher (few-shot, k=5)#4Accuracy: 25
multiple-choice-question-answering-mcqa-on-mmlu-abstract-algebraOPT (few-shot, k=5)#5Accuracy: 21
multiple-choice-question-answering-mcqa-on-mmlu-astronomyChinchilla (few-shot, k=5)#1Accuracy: 73.0
multiple-choice-question-answering-mcqa-on-mmlu-astronomyGopher (few-shot, k=5)#2Accuracy: 65.8
multiple-choice-question-answering-mcqa-on-mmlu-astronomyGAL 120B (zero-shot)#3Accuracy: 65.1
multiple-choice-question-answering-mcqa-on-mmlu-astronomyBLOOM (few-shot, k=5)#4Accuracy: 25.7
multiple-choice-question-answering-mcqa-on-mmlu-astronomyOPT (few-shot, k=5)#5Accuracy: 23.0
multiple-choice-question-answering-mcqa-on-mmlu-college-chemistryChinchilla (few-shot, k=5)#1Accuracy: 51
multiple-choice-question-answering-mcqa-on-mmlu-college-chemistryGAL 120B (zero-shot)#2Accuracy: 46
multiple-choice-question-answering-mcqa-on-mmlu-college-chemistryGopher (few-shot, k=5)#3Accuracy: 45
multiple-choice-question-answering-mcqa-on-mmlu-college-chemistryOPT (few-shot, k=5)#4Accuracy: 30
multiple-choice-question-answering-mcqa-on-mmlu-college-chemistryBLOOM (few-shot, k=5)#5Accuracy: 19
multiple-choice-question-answering-mcqa-on-mmlu-college-computer-scienceChinchilla (few-shot, k=5)#1Accuracy: 51.0
multiple-choice-question-answering-mcqa-on-mmlu-college-computer-scienceGAL 120B (zero-shot)#2Accuracy: 49
multiple-choice-question-answering-mcqa-on-mmlu-college-computer-scienceOPT (few-shot, k=5)#3Accuracy: 17.0
multiple-choice-question-answering-mcqa-on-mmlu-college-computer-scienceBLOOM (few-shot, k=5)#4Accuracy: 6.0
multiple-choice-question-answering-mcqa-on-mmlu-college-mathematicsGAL 120B (zero-shot)#1Accuracy: 43
multiple-choice-question-answering-mcqa-on-mmlu-college-mathematicsGopher (few-shot, k=5)#2Accuracy: 37
multiple-choice-question-answering-mcqa-on-mmlu-college-mathematicsOPT (few-shot, k=5)#3Accuracy: 33
multiple-choice-question-answering-mcqa-on-mmlu-college-mathematicsChinchilla (few-shot, k=5)#4Accuracy: 32
multiple-choice-question-answering-mcqa-on-mmlu-college-mathematicsBLOOM (few-shot, k=5)#5Accuracy: 25
multiple-choice-question-answering-mcqa-on-mmlu-college-physicsChinchilla (few-shot, k=5)#1Accuracy: 46.1
multiple-choice-question-answering-mcqa-on-mmlu-college-physicsGAL 120B (zero-shot)#2Accuracy: 42.2
multiple-choice-question-answering-mcqa-on-mmlu-college-physicsGopher (few-shot, k=5)#3Accuracy: 34.3
multiple-choice-question-answering-mcqa-on-mmlu-college-physicsOPT (few-shot, k=5)#4Accuracy: 21.6
multiple-choice-question-answering-mcqa-on-mmlu-college-physicsBLOOM (few-shot, k=5)#5Accuracy: 18.6
multiple-choice-question-answering-mcqa-on-mmlu-econometricsGopher (few-shot, k=5)#1Accuracy: 43
multiple-choice-question-answering-mcqa-on-mmlu-econometricsGAL 120B (zero-shot)#2Accuracy: 42.1
multiple-choice-question-answering-mcqa-on-mmlu-econometricsChinchilla (few-shot, k=5)#3Accuracy: 38.6
multiple-choice-question-answering-mcqa-on-mmlu-econometricsBLOOM (few-shot, k=5)#4Accuracy: 23.7
multiple-choice-question-answering-mcqa-on-mmlu-econometricsOPT (few-shot, k=5)#5Accuracy: 21
multiple-choice-question-answering-mcqa-on-mmlu-electrical-engineerGAL 120B (zero-shot)#1Accuracy: 62.8
multiple-choice-question-answering-mcqa-on-mmlu-electrical-engineerChinchilla (few-shot, k=5)#2Accuracy: 62.1
multiple-choice-question-answering-mcqa-on-mmlu-electrical-engineerGopher (few-shot, k=5)#3Accuracy: 60
multiple-choice-question-answering-mcqa-on-mmlu-electrical-engineerOPT (few-shot, k=5)#4Accuracy: 36.6
multiple-choice-question-answering-mcqa-on-mmlu-electrical-engineerBLOOM (few-shot, k=5)#5Accuracy: 32.4
multiple-choice-question-answering-mcqa-on-mmlu-elementary-mathematicsChinchilla (few-shot, k=5)#1Accuracy: 41.5
multiple-choice-question-answering-mcqa-on-mmlu-elementary-mathematicsGAL 120B (zero-shot)#2Accuracy: 38.1
multiple-choice-question-answering-mcqa-on-mmlu-elementary-mathematicsGopher (few-shot, k=5)#3Accuracy: 33.6
multiple-choice-question-answering-mcqa-on-mmlu-elementary-mathematicsBLOOM (few-shot, k=5)#4Accuracy: 27.6
multiple-choice-question-answering-mcqa-on-mmlu-elementary-mathematicsOPT (few-shot, k=5)#5Accuracy: 25.7
multiple-choice-question-answering-mcqa-on-mmlu-formal-logicGopher (few-shot, k=5)#1Accuracy: 35.7
multiple-choice-question-answering-mcqa-on-mmlu-formal-logicChinchilla (few-shot, k=5)#2Accuracy: 33.3
multiple-choice-question-answering-mcqa-on-mmlu-formal-logicGAL 120B (zero-shot)#3Accuracy: 32.5
multiple-choice-question-answering-mcqa-on-mmlu-formal-logicOPT (few-shot, k=5)#4Accuracy: 29.4
multiple-choice-question-answering-mcqa-on-mmlu-formal-logicBLOOM (few-shot, k=5)#5Accuracy: 26.2
multiple-choice-question-answering-mcqa-on-mmlu-high-school-biologyChinchilla (few-shot, k=5)#1Accuracy: 80.3
multiple-choice-question-answering-mcqa-on-mmlu-high-school-biologyGopher (few-shot, k=5)#2Accuracy: 71.3
multiple-choice-question-answering-mcqa-on-mmlu-high-school-biologyGAL 120B (zero-shot)#3Accuracy: 69.4
multiple-choice-question-answering-mcqa-on-mmlu-high-school-biologyBLOOM (few-shot, k=5)#4Accuracy: 29.4
multiple-choice-question-answering-mcqa-on-mmlu-high-school-biologyOPT (few-shot, k=5)#5Accuracy: 27.7
multiple-choice-question-answering-mcqa-on-mmlu-high-school-chemistryChinchilla (few-shot, k=5)#1Accuracy: 58.1
multiple-choice-question-answering-mcqa-on-mmlu-high-school-chemistryGAL 120B (zero-shot)#2Accuracy: 47.8
multiple-choice-question-answering-mcqa-on-mmlu-high-school-chemistryBLOOM (few-shot, k=5)#3Accuracy: 23.2
multiple-choice-question-answering-mcqa-on-mmlu-high-school-chemistryOPT (few-shot, k=5)#4Accuracy: 21.7
multiple-choice-question-answering-mcqa-on-mmlu-high-school-computer-scienceGAL 120B (zero-shot)#1Accuracy: 70
multiple-choice-question-answering-mcqa-on-mmlu-high-school-computer-scienceChinchilla (few-shot, k=5)#2Accuracy: 58
multiple-choice-question-answering-mcqa-on-mmlu-high-school-computer-scienceGopher (few-shot, k=5)#3Accuracy: 54
multiple-choice-question-answering-mcqa-on-mmlu-high-school-computer-scienceOPT (few-shot, k=5)#4Accuracy: 30
multiple-choice-question-answering-mcqa-on-mmlu-high-school-computer-scienceBLOOM (few-shot, k=5)#5Accuracy: 25
multiple-choice-question-answering-mcqa-on-mmlu-high-school-mathematicsGAL 120B (zero-shot)#1Accuracy: 32.6
multiple-choice-question-answering-mcqa-on-mmlu-high-school-mathematicsChinchilla (few-shot, k=5)#2Accuracy: 31.9
multiple-choice-question-answering-mcqa-on-mmlu-high-school-mathematicsBLOOM (few-shot, k=5)#3Accuracy: 27
multiple-choice-question-answering-mcqa-on-mmlu-high-school-mathematicsOPT (few-shot, k=5)#4Accuracy: 24.4
multiple-choice-question-answering-mcqa-on-mmlu-high-school-mathematicsGopher (few-shot, k=5)#5Accuracy: 23.7
multiple-choice-question-answering-mcqa-on-mmlu-high-school-physicsChinchilla (few-shot, k=5)#1Accuracy: 36.4
multiple-choice-question-answering-mcqa-on-mmlu-high-school-physicsGAL 120B (zero-shot)#2Accuracy: 33.8
multiple-choice-question-answering-mcqa-on-mmlu-high-school-physicsOPT (few-shot, k=5)#3Accuracy: 29.8
multiple-choice-question-answering-mcqa-on-mmlu-high-school-physicsBLOOM (few-shot, k=5)#4Accuracy: 25.2
multiple-choice-question-answering-mcqa-on-mmlu-high-school-statisticsChinchilla (few-shot, k=5)#1Accuracy: 58.8
multiple-choice-question-answering-mcqa-on-mmlu-high-school-statisticsGopher (few-shot, k=5)#2Accuracy: 50
multiple-choice-question-answering-mcqa-on-mmlu-high-school-statisticsOPT (few-shot, k=5)#3Accuracy: 43.5
multiple-choice-question-answering-mcqa-on-mmlu-high-school-statisticsGAL 120B (zero-shot)#4Accuracy: 41.2
multiple-choice-question-answering-mcqa-on-mmlu-high-school-statisticsBLOOM (few-shot, k=5)#5Accuracy: 19.4
multiple-choice-question-answering-mcqa-on-mmlu-machine-learningChinchilla (few-shot, k=5)#1Accuracy: 41.1
multiple-choice-question-answering-mcqa-on-mmlu-machine-learningGAL 120B (zero-shot)#2Accuracy: 38.4
multiple-choice-question-answering-mcqa-on-mmlu-machine-learningOPT (few-shot, k=5)#3Accuracy: 28.6
multiple-choice-question-answering-mcqa-on-mmlu-machine-learningBLOOM (few-shot, k=5)#4Accuracy: 25
protein-function-prediction-on-caspsimseqGAL 120B#1ROUGE-L: 0.252
protein-function-prediction-on-caspsimseqGAL 30B#2ROUGE-L: 0.137
protein-function-prediction-on-caspsimseqGAL 6.7B#3ROUGE-L: 0.109
protein-function-prediction-on-caspsimseqGAL 1.3B#4ROUGE-L: 0.069
protein-function-prediction-on-caspsimseqGAL 125M#5ROUGE-L: 0.062
protein-function-prediction-on-paenseqGAL 120B#1ROUGE-L: 0.272
protein-function-prediction-on-paenseqGAL 30B#2ROUGE-L: 0.196
protein-function-prediction-on-paenseqGAL 6.7B#3ROUGE-L: 0.137
protein-function-prediction-on-paenseqGAL 1.3B#4ROUGE-L: 0.084
protein-function-prediction-on-paenseqGAL 125M#5ROUGE-L: 0.073
protein-function-prediction-on-uniprotseqGAL 120B#1ROUGE-L: 0.252
protein-function-prediction-on-uniprotseqGAL 30B#2ROUGE-L: 0.186
protein-function-prediction-on-uniprotseqGAL 6.7B#3ROUGE-L: 0.111
protein-function-prediction-on-uniprotseqGAL 1.3B#4ROUGE-L: 0.079
protein-function-prediction-on-uniprotseqGAL 125M#5ROUGE-L: 0.061
protein-structure-prediction-on-caspseqGAL 120B#1Validation perplexity: 17.26
protein-structure-prediction-on-caspseqGAL 30B#2Validation perplexity: 17.27
protein-structure-prediction-on-caspseqGAL 6.7B#3Validation perplexity: 17.29
protein-structure-prediction-on-caspseqGAL 1.3B#4Validation perplexity: 17.58
protein-structure-prediction-on-caspseqGAL 125M#5Validation perplexity: 20.62
protein-structure-prediction-on-caspsimseqGAL 120B#1Validation perplexity: 12.77
protein-structure-prediction-on-caspsimseqGAL 30B#2Validation perplexity: 15.42
protein-structure-prediction-on-caspsimseqGAL 6.7B#3Validation perplexity: 16.35
protein-structure-prediction-on-caspsimseqGAL 1.3B#4Validation perplexity: 17.04
protein-structure-prediction-on-caspsimseqGAL 125M#5Validation perplexity: 19.18
protein-structure-prediction-on-paenseqGAL 120B#1Validation perplexity: 3.14
protein-structure-prediction-on-paenseqGAL 30B#2Validation perplexity: 4.28
protein-structure-prediction-on-paenseqGAL 6.7B#3Validation perplexity: 7.76
protein-structure-prediction-on-paenseqGAL 1.3B#4Validation perplexity: 12.53
protein-structure-prediction-on-paenseqGAL 125M#5Validation perplexity: 16.35
protein-structure-prediction-on-uniprotseqGAL 120B#1Validation perplexity: 5.54
protein-structure-prediction-on-uniprotseqGAL 30B#2Validation perplexity: 8.23
protein-structure-prediction-on-uniprotseqGAL 6.7B#3Validation perplexity: 11.58
protein-structure-prediction-on-uniprotseqGAL 1.3B#4Validation perplexity: 15.82
protein-structure-prediction-on-uniprotseqGAL 125M#5Validation perplexity: 19.05
stereotypical-bias-analysis-on-crows-pairsGAL 120B#1Gender: 51.9Religion: 51.9Race/Color: 59.9Sexual Orientation: 77.4…
molecular-property-prediction-on-hiv-datasetUni-Mol#2AUC: 0.808
molecular-property-prediction-on-hiv-datasetGAL 30B#7AUC: 0.759
molecular-property-prediction-on-hiv-datasetGAL 120B#8AUC: 0.745
molecular-property-prediction-on-hiv-datasetGAL 1.3B#9AUC: 0.724
molecular-property-prediction-on-hiv-datasetGAL 6.7B#10AUC: 0.722
molecular-property-prediction-on-hiv-datasetGAL 125M#11AUC: 0.702
question-answering-on-bioasqGAL 120B (zero-shot)#2Accuracy: 94.3
question-answering-on-bioasqBLOOM (zero-shot)#4Accuracy: 91.4
question-answering-on-bioasqOPT (zero-shot)#6Accuracy: 81.4
word-sense-disambiguation-on-big-bench-anachronismsOPT 175B#3Accuracy: 49.1
word-sense-disambiguation-on-big-bench-anachronismsGAL 120B (few-shot, k=5)#4Accuracy: 48.7
word-sense-disambiguation-on-big-bench-anachronismsGAL 30B (few-shot, k=5)#5Accuracy: 47.0
word-sense-disambiguation-on-big-bench-anachronismsBLOOM 176B#6Accuracy: 1.3
multiple-choice-question-answering-mcqa-on-mmlu-college-biologyChinchilla (few-shot, k=5)#4Accuracy: 79.9
multiple-choice-question-answering-mcqa-on-mmlu-college-biologyGopher (few-shot, k=5)#5Accuracy: 70.8
multiple-choice-question-answering-mcqa-on-mmlu-college-biologyGAL 120B (zero-shot)#6Accuracy: 68.8
multiple-choice-question-answering-mcqa-on-mmlu-college-biologyOPT (few-shot, k=5)#7Accuracy: 30.6
multiple-choice-question-answering-mcqa-on-mmlu-college-biologyBLOOM (few-shot, k=5)#8Accuracy: 28.5
multiple-choice-question-answering-mcqa-on-mmlu-medical-geneticsGAL 30B (zero-shot)#4Accuracy: 70
multiple-choice-question-answering-mcqa-on-mmlu-medical-geneticsChinchilla (few-shot, k=5)#5Accuracy: 69
multiple-choice-question-answering-mcqa-on-mmlu-medical-geneticsGAL 120B (zero-shot)#6Accuracy: 68
multiple-choice-question-answering-mcqa-on-mmlu-medical-geneticsBLOOM (few-shot, k=5)#7Accuracy: 36
multiple-choice-question-answering-mcqa-on-mmlu-medical-geneticsOPT (few-shot, k=5)#8Accuracy: 35
molecular-property-prediction-on-tox21Uni-Mol#5ROC-AUC: 79.6
molecular-property-prediction-on-tox21GAL 120B#14ROC-AUC: 68.9
molecular-property-prediction-on-tox21GAL 30B#15ROC-AUC: 68.5
molecular-property-prediction-on-tox21GAL 6.7B#16ROC-AUC: 63.9
molecular-property-prediction-on-tox21GAL 1.3B#17ROC-AUC: 60.6
molecular-property-prediction-on-tox21GAL 125M#18ROC-AUC: 54.3
question-answering-on-truthfulqaGAL 120B#7MC1: 0.26
question-answering-on-truthfulqaGAL 30B#9MC1: 0.24
question-answering-on-truthfulqaOPT 175B#15MC1: 0.21
question-answering-on-truthfulqaGAL 125M#18MC1: 0.19
question-answering-on-truthfulqaGAL 1.3B#19MC1: 0.19
question-answering-on-truthfulqaGAL 6.7B#20MC1: 0.19
bias-detection-on-stereosetGAL 120B#8ICAT Score: 65.6LMS: 75SS: 56.2
bias-detection-on-stereosetGPT-3 (text-davinci-002)#10ICAT Score: 60.8LMS: 77.6SS: 60.8
bias-detection-on-stereosetOPT 175B#11ICAT Score: 60LMS: 74.8SS: 59.9
molecular-property-prediction-on-clintoxGAL 120B#8ROC-AUC: 82.6Molecules (M): 2
molecular-property-prediction-on-clintoxGAL 30B#9ROC-AUC: 82.2Molecules (M): 2
molecular-property-prediction-on-clintoxGAL 6.7B#12ROC-AUC: 78.4Molecules (M): 2
molecular-property-prediction-on-clintoxGAL 1.3B#16ROC-AUC: 58.9Molecules (M): 2
molecular-property-prediction-on-clintoxGAL 125M#18ROC-AUC: 51.8Molecules (M): 2
tdc-admet-benchmarking-group-on-tdcommonsGalactica-GAL-120B#8TDC.BBB_Martins: 0.661
tdc-admet-benchmarking-group-on-tdcommonsGalactica-GAL-1.3B#9TDC.BBB_Martins: 0.604
tdc-admet-benchmarking-group-on-tdcommonsGalactica-GAL-30B#10TDC.BBB_Martins: 0.596
tdc-admet-benchmarking-group-on-tdcommonsGalactica-GAL-6.7B#11TDC.BBB_Martins: 0.535
tdc-admet-benchmarking-group-on-tdcommonsGalactica-GAL-125M#12TDC.BBB_Martins: 0.393
common-sense-reasoning-on-arc-challengeGAL 120B (zero-shot)#9Accuracy: 67.9
common-sense-reasoning-on-arc-challengeGPT-3 (zero-shot)#24Accuracy: 51.4
common-sense-reasoning-on-arc-challengeBLOOM (few-shot, k=5)#38Accuracy: 32.9
common-sense-reasoning-on-arc-challengeOPT (few-shot, k=5)#40Accuracy: 31.1
molecular-property-prediction-on-siderGAL 120B#11ROC-AUC: 63.2
molecular-property-prediction-on-siderGAL 30B#13ROC-AUC: 61.3
molecular-property-prediction-on-siderGAL 125M#15ROC-AUC: 55.9
molecular-property-prediction-on-siderGAL 6.7B#16ROC-AUC: 55.9
molecular-property-prediction-on-siderGAL 1.3B#17ROC-AUC: 54.0
common-sense-reasoning-on-arc-easyGAL 120B (0-shot)#12Accuracy: 83.8
common-sense-reasoning-on-arc-easyGPT-3 (zero-shot)#65Accuracy: 68.8
common-sense-reasoning-on-arc-easyBLOOM (5-shot)#92Accuracy: 40.7
common-sense-reasoning-on-arc-easyOPT (5-shot)#95Accuracy: 37.4
question-answering-on-pubmedqaGAL 120B (zero-shot)#13Accuracy: 77.6
question-answering-on-pubmedqaBLOOM (zero-shot)#20Accuracy: 73.6
question-answering-on-pubmedqaOPT (zero-shot)#24Accuracy: 70.2
molecular-property-prediction-on-baceGAL 30B#15ROC-AUC: 72.7
molecular-property-prediction-on-baceGAL 120B#16ROC-AUC: 61.7
molecular-property-prediction-on-baceGAL 6.7B#17ROC-AUC: 58.4
molecular-property-prediction-on-baceGAL 1.3B#18ROC-AUC: 57.6
molecular-property-prediction-on-baceGAL 125M#19ROC-AUC: 56.1
molecular-property-prediction-on-bbbpUni-Mol#15ROC-AUC: 72.9
molecular-property-prediction-on-bbbpGAL 120B#23ROC-AUC: 66.1
molecular-property-prediction-on-bbbpGAL 1.3B#24ROC-AUC: 60.4
molecular-property-prediction-on-bbbpGAL 30B#25ROC-AUC: 59.6
molecular-property-prediction-on-bbbpGAL 6.7B#26ROC-AUC: 53.5
molecular-property-prediction-on-bbbpGAL 125M#27ROC-AUC: 39.3
question-answering-on-medqa-usmleGAL 120B (zero-shot)#18Accuracy: 44.4
question-answering-on-medqa-usmleBLOOM (few-shot, k=5)#23Accuracy: 23.3
question-answering-on-medqa-usmleOPT (few-shot, k=5)#24Accuracy: 22.8
multiple-choice-question-answering-mcqa-on-medmcqaGAL 120B (zero-shot)#21Dev Set (Acc-%): 0.529
multiple-choice-question-answering-mcqa-on-medmcqaBLOOM (few-shot, k=5)#25Dev Set (Acc-%): 0.325
multiple-choice-question-answering-mcqa-on-medmcqaOPT (few-shot, k=5)#26Dev Set (Acc-%): 0.296
multi-task-language-understanding-on-mmluGAL 120B (zero-shot)#62Average (%): 52.6
math-word-problem-solving-on-mathMinerva 540B (5-shot) mCoT#130Accuracy: 33.6Parameters (Billions): 540
math-word-problem-solving-on-mathGAL 120B (5-shot) mCoT#161Accuracy: 20.4Parameters (Billions): 120
math-word-problem-solving-on-mathGAL 120B <work>#166Accuracy: 16.6Parameters (Billions): 120
math-word-problem-solving-on-mathGAL 30B (5-shot) mCoT#173Accuracy: 12.7Parameters (Billions): 30
math-word-problem-solving-on-mathGAL 30B <work>#176Accuracy: 11.4Parameters (Billions): 30
math-word-problem-solving-on-mathPaLM 540B (5-shot) mCoT#181Accuracy: 8.8Parameters (Billions): 540
math-word-problem-solving-on-mathGPT-3 175B (8-shot)#193Accuracy: 5.2Parameters (Billions): 175

Rank counts only results with a code link.