Galactica: A Large Language Model for Science

Open paper
Benchmark Model Rank Results
bias-detection-on-stereoset-1GAL 120B#8ICAT Score: 65.6LMS: 75SS: 56.2
bias-detection-on-stereoset-1GPT-3 (text-davinci-002)#10ICAT Score: 60.8LMS: 77.6SS: 60.8
bias-detection-on-stereoset-1OPT 175B#11ICAT Score: 60LMS: 74.8SS: 59.9
common-sense-reasoning-on-arc-challengeGAL 120B (zero-shot)#9Accuracy: 67.9
common-sense-reasoning-on-arc-challengeGPT-3 (zero-shot)#23Accuracy: 51.4
common-sense-reasoning-on-arc-challengeBLOOM (few-shot, k=5)#37Accuracy: 32.9
common-sense-reasoning-on-arc-challengeOPT (few-shot, k=5)#39Accuracy: 31.1
common-sense-reasoning-on-arc-easyGAL 120B (0-shot)#8Accuracy: 83.8
common-sense-reasoning-on-arc-easyGPT-3 (zero-shot)#34Accuracy: 68.8
common-sense-reasoning-on-arc-easyBLOOM (5-shot)#37Accuracy: 40.7
common-sense-reasoning-on-arc-easyOPT (5-shot)#39Accuracy: 37.4
math-word-problem-solving-on-mathMinerva 540B (5-shot) mCoT#69Accuracy: 33.6Parameters (Billions): 540
math-word-problem-solving-on-mathGAL 120B (5-shot) mCoT#89Accuracy: 20.4Parameters (Billions): 120
math-word-problem-solving-on-mathGAL 120B <work>#93Accuracy: 16.6Parameters (Billions): 120
math-word-problem-solving-on-mathGAL 30B (5-shot) mCoT#98Accuracy: 12.7Parameters (Billions): 30
math-word-problem-solving-on-mathGAL 30B <work>#100Accuracy: 11.4Parameters (Billions): 30
math-word-problem-solving-on-mathPaLM 540B (5-shot) mCoT#104Accuracy: 8.8Parameters (Billions): 540
math-word-problem-solving-on-mathGPT-3 175B (8-shot)#115Accuracy: 5.2Parameters (Billions): 175
molecular-property-prediction-on-bace-1GAL 30B#15ROC-AUC: 72.7
molecular-property-prediction-on-bace-1GAL 120B#16ROC-AUC: 61.7
molecular-property-prediction-on-bace-1GAL 6.7B#17ROC-AUC: 58.4
molecular-property-prediction-on-bace-1GAL 1.3B#18ROC-AUC: 57.6
molecular-property-prediction-on-bace-1GAL 125M#19ROC-AUC: 56.1
molecular-property-prediction-on-bbbp-1Uni-Mol#15ROC-AUC: 72.9
molecular-property-prediction-on-bbbp-1GAL 120B#23ROC-AUC: 66.1
molecular-property-prediction-on-bbbp-1GAL 1.3B#24ROC-AUC: 60.4
molecular-property-prediction-on-bbbp-1GAL 30B#25ROC-AUC: 59.6
molecular-property-prediction-on-bbbp-1GAL 6.7B#26ROC-AUC: 53.5
molecular-property-prediction-on-bbbp-1GAL 125M#27ROC-AUC: 39.3
molecular-property-prediction-on-clintox-1GAL 120B#8ROC-AUC: 82.6Molecules (M): 2
molecular-property-prediction-on-clintox-1GAL 30B#9ROC-AUC: 82.2Molecules (M): 2
molecular-property-prediction-on-clintox-1GAL 6.7B#12ROC-AUC: 78.4Molecules (M): 2
molecular-property-prediction-on-clintox-1GAL 1.3B#16ROC-AUC: 58.9Molecules (M): 2
molecular-property-prediction-on-clintox-1GAL 125M#18ROC-AUC: 51.8Molecules (M): 2
molecular-property-prediction-on-hiv-datasetUni-Mol#2AUC: 0.808
molecular-property-prediction-on-hiv-datasetGAL 30B#7AUC: 0.759
molecular-property-prediction-on-hiv-datasetGAL 120B#8AUC: 0.745
molecular-property-prediction-on-hiv-datasetGAL 1.3B#9AUC: 0.724
molecular-property-prediction-on-hiv-datasetGAL 6.7B#10AUC: 0.722
molecular-property-prediction-on-hiv-datasetGAL 125M#11AUC: 0.702
molecular-property-prediction-on-moleculenetUni-Mol#1AUC: 0.77
molecular-property-prediction-on-moleculenetGAL 30B#2AUC: 0.69
molecular-property-prediction-on-moleculenetGAL 6.7B#3AUC: 0.64
molecular-property-prediction-on-moleculenetGAL 1.3B#4AUC: 0.619
molecular-property-prediction-on-moleculenetGAL 125M#5AUC: 0.581
molecular-property-prediction-on-sider-1GAL 120B#11ROC-AUC: 63.2
molecular-property-prediction-on-sider-1GAL 30B#13ROC-AUC: 61.3
molecular-property-prediction-on-sider-1GAL 125M#15ROC-AUC: 55.9
molecular-property-prediction-on-sider-1GAL 6.7B#16ROC-AUC: 55.9
molecular-property-prediction-on-sider-1GAL 1.3B#17ROC-AUC: 54.0
molecular-property-prediction-on-tox21-1Uni-Mol#5ROC-AUC: 79.6
molecular-property-prediction-on-tox21-1GAL 120B#14ROC-AUC: 68.9
molecular-property-prediction-on-tox21-1GAL 30B#15ROC-AUC: 68.5
molecular-property-prediction-on-tox21-1GAL 6.7B#16ROC-AUC: 63.9
molecular-property-prediction-on-tox21-1GAL 1.3B#17ROC-AUC: 60.6
molecular-property-prediction-on-tox21-1GAL 125M#18ROC-AUC: 54.3
multi-task-language-understanding-on-mmluGAL 120B (zero-shot)#18Average (%): 52.6
multiple-choice-question-answering-mcqa-on-10Chinchilla (few-shot, k=5)#1Accuracy: 41.5
multiple-choice-question-answering-mcqa-on-10GAL 120B (zero-shot)#2Accuracy: 38.1
multiple-choice-question-answering-mcqa-on-10Gopher (few-shot, k=5)#3Accuracy: 33.6
multiple-choice-question-answering-mcqa-on-10BLOOM (few-shot, k=5)#4Accuracy: 27.6
multiple-choice-question-answering-mcqa-on-10OPT (few-shot, k=5)#5Accuracy: 25.7
multiple-choice-question-answering-mcqa-on-11Chinchilla (few-shot, k=5)#4Accuracy: 79.9
multiple-choice-question-answering-mcqa-on-11Gopher (few-shot, k=5)#5Accuracy: 70.8
multiple-choice-question-answering-mcqa-on-11GAL 120B (zero-shot)#6Accuracy: 68.8
multiple-choice-question-answering-mcqa-on-11OPT (few-shot, k=5)#7Accuracy: 30.6
multiple-choice-question-answering-mcqa-on-11BLOOM (few-shot, k=5)#8Accuracy: 28.5
multiple-choice-question-answering-mcqa-on-12Chinchilla (few-shot, k=5)#1Accuracy: 80.3
multiple-choice-question-answering-mcqa-on-12Gopher (few-shot, k=5)#2Accuracy: 71.3
multiple-choice-question-answering-mcqa-on-12GAL 120B (zero-shot)#3Accuracy: 69.4
multiple-choice-question-answering-mcqa-on-12BLOOM (few-shot, k=5)#4Accuracy: 29.4
multiple-choice-question-answering-mcqa-on-12OPT (few-shot, k=5)#5Accuracy: 27.7
multiple-choice-question-answering-mcqa-on-13Chinchilla (few-shot, k=5)#1Accuracy: 51
multiple-choice-question-answering-mcqa-on-13GAL 120B (zero-shot)#2Accuracy: 46
multiple-choice-question-answering-mcqa-on-13Gopher (few-shot, k=5)#3Accuracy: 45
multiple-choice-question-answering-mcqa-on-13OPT (few-shot, k=5)#4Accuracy: 30
multiple-choice-question-answering-mcqa-on-13BLOOM (few-shot, k=5)#5Accuracy: 19
multiple-choice-question-answering-mcqa-on-14Chinchilla (few-shot, k=5)#1Accuracy: 58.1
multiple-choice-question-answering-mcqa-on-14GAL 120B (zero-shot)#2Accuracy: 47.8
multiple-choice-question-answering-mcqa-on-14BLOOM (few-shot, k=5)#3Accuracy: 23.2
multiple-choice-question-answering-mcqa-on-14OPT (few-shot, k=5)#4Accuracy: 21.7
multiple-choice-question-answering-mcqa-on-15Chinchilla (few-shot, k=5)#1Accuracy: 51.0
multiple-choice-question-answering-mcqa-on-15GAL 120B (zero-shot)#2Accuracy: 49
multiple-choice-question-answering-mcqa-on-15OPT (few-shot, k=5)#3Accuracy: 17.0
multiple-choice-question-answering-mcqa-on-15BLOOM (few-shot, k=5)#4Accuracy: 6.0
multiple-choice-question-answering-mcqa-on-16GAL 120B (zero-shot)#1Accuracy: 32.6
multiple-choice-question-answering-mcqa-on-16Chinchilla (few-shot, k=5)#2Accuracy: 31.9
multiple-choice-question-answering-mcqa-on-16BLOOM (few-shot, k=5)#3Accuracy: 27
multiple-choice-question-answering-mcqa-on-16OPT (few-shot, k=5)#4Accuracy: 24.4
multiple-choice-question-answering-mcqa-on-16Gopher (few-shot, k=5)#5Accuracy: 23.7
multiple-choice-question-answering-mcqa-on-17GAL 120B (zero-shot)#1Accuracy: 62.8
multiple-choice-question-answering-mcqa-on-17Chinchilla (few-shot, k=5)#2Accuracy: 62.1
multiple-choice-question-answering-mcqa-on-17Gopher (few-shot, k=5)#3Accuracy: 60
multiple-choice-question-answering-mcqa-on-17OPT (few-shot, k=5)#4Accuracy: 36.6
multiple-choice-question-answering-mcqa-on-17BLOOM (few-shot, k=5)#5Accuracy: 32.4
multiple-choice-question-answering-mcqa-on-18Chinchilla (few-shot, k=5)#1Accuracy: 46.1
multiple-choice-question-answering-mcqa-on-18GAL 120B (zero-shot)#2Accuracy: 42.2
multiple-choice-question-answering-mcqa-on-18Gopher (few-shot, k=5)#3Accuracy: 34.3
multiple-choice-question-answering-mcqa-on-18OPT (few-shot, k=5)#4Accuracy: 21.6
multiple-choice-question-answering-mcqa-on-18BLOOM (few-shot, k=5)#5Accuracy: 18.6
multiple-choice-question-answering-mcqa-on-19Chinchilla (few-shot, k=5)#1Accuracy: 36.4
multiple-choice-question-answering-mcqa-on-19GAL 120B (zero-shot)#2Accuracy: 33.8
multiple-choice-question-answering-mcqa-on-19OPT (few-shot, k=5)#3Accuracy: 29.8
multiple-choice-question-answering-mcqa-on-19BLOOM (few-shot, k=5)#4Accuracy: 25.2
multiple-choice-question-answering-mcqa-on-2Gopher (few-shot, k=5)#1Accuracy: 35.7
multiple-choice-question-answering-mcqa-on-2Chinchilla (few-shot, k=5)#2Accuracy: 33.3
multiple-choice-question-answering-mcqa-on-2GAL 120B (zero-shot)#3Accuracy: 32.5
multiple-choice-question-answering-mcqa-on-2OPT (few-shot, k=5)#4Accuracy: 29.4
multiple-choice-question-answering-mcqa-on-2BLOOM (few-shot, k=5)#5Accuracy: 26.2
multiple-choice-question-answering-mcqa-on-20Chinchilla (few-shot, k=5)#1Accuracy: 58.8
multiple-choice-question-answering-mcqa-on-20Gopher (few-shot, k=5)#2Accuracy: 50
multiple-choice-question-answering-mcqa-on-20OPT (few-shot, k=5)#3Accuracy: 43.5
multiple-choice-question-answering-mcqa-on-20GAL 120B (zero-shot)#4Accuracy: 41.2
multiple-choice-question-answering-mcqa-on-20BLOOM (few-shot, k=5)#5Accuracy: 19.4
multiple-choice-question-answering-mcqa-on-21GAL 120B (zero-shot)#16Dev Set (Acc-%): 0.529
multiple-choice-question-answering-mcqa-on-21BLOOM (few-shot, k=5)#20Dev Set (Acc-%): 0.325
multiple-choice-question-answering-mcqa-on-21OPT (few-shot, k=5)#21Dev Set (Acc-%): 0.296
multiple-choice-question-answering-mcqa-on-3GAL 30B (zero-shot)#1Accuracy: 33.3
multiple-choice-question-answering-mcqa-on-3Chinchilla (few-shot, k=5)#2Accuracy: 31
multiple-choice-question-answering-mcqa-on-3GAL 120B (zero-shot)#3Accuracy: 27
multiple-choice-question-answering-mcqa-on-3Gopher (few-shot, k=5)#4Accuracy: 25
multiple-choice-question-answering-mcqa-on-3OPT (few-shot, k=5)#5Accuracy: 21
multiple-choice-question-answering-mcqa-on-4Gopher (few-shot, k=5)#1Accuracy: 43
multiple-choice-question-answering-mcqa-on-4GAL 120B (zero-shot)#2Accuracy: 42.1
multiple-choice-question-answering-mcqa-on-4Chinchilla (few-shot, k=5)#3Accuracy: 38.6
multiple-choice-question-answering-mcqa-on-4BLOOM (few-shot, k=5)#4Accuracy: 23.7
multiple-choice-question-answering-mcqa-on-4OPT (few-shot, k=5)#5Accuracy: 21
multiple-choice-question-answering-mcqa-on-5GAL 120B (zero-shot)#1Accuracy: 70
multiple-choice-question-answering-mcqa-on-5Chinchilla (few-shot, k=5)#2Accuracy: 58
multiple-choice-question-answering-mcqa-on-5Gopher (few-shot, k=5)#3Accuracy: 54
multiple-choice-question-answering-mcqa-on-5OPT (few-shot, k=5)#4Accuracy: 30
multiple-choice-question-answering-mcqa-on-5BLOOM (few-shot, k=5)#5Accuracy: 25
multiple-choice-question-answering-mcqa-on-6Chinchilla (few-shot, k=5)#1Accuracy: 41.1
multiple-choice-question-answering-mcqa-on-6GAL 120B (zero-shot)#2Accuracy: 38.4
multiple-choice-question-answering-mcqa-on-6OPT (few-shot, k=5)#3Accuracy: 28.6
multiple-choice-question-answering-mcqa-on-6BLOOM (few-shot, k=5)#4Accuracy: 25
multiple-choice-question-answering-mcqa-on-7GAL 120B (zero-shot)#1Accuracy: 43
multiple-choice-question-answering-mcqa-on-7Gopher (few-shot, k=5)#2Accuracy: 37
multiple-choice-question-answering-mcqa-on-7OPT (few-shot, k=5)#3Accuracy: 33
multiple-choice-question-answering-mcqa-on-7Chinchilla (few-shot, k=5)#4Accuracy: 32
multiple-choice-question-answering-mcqa-on-7BLOOM (few-shot, k=5)#5Accuracy: 25
multiple-choice-question-answering-mcqa-on-8GAL 30B (zero-shot)#4Accuracy: 70
multiple-choice-question-answering-mcqa-on-8Chinchilla (few-shot, k=5)#5Accuracy: 69
multiple-choice-question-answering-mcqa-on-8GAL 120B (zero-shot)#6Accuracy: 68
multiple-choice-question-answering-mcqa-on-8BLOOM (few-shot, k=5)#7Accuracy: 36
multiple-choice-question-answering-mcqa-on-8OPT (few-shot, k=5)#8Accuracy: 35
multiple-choice-question-answering-mcqa-on-9Chinchilla (few-shot, k=5)#1Accuracy: 73.0
multiple-choice-question-answering-mcqa-on-9Gopher (few-shot, k=5)#2Accuracy: 65.8
multiple-choice-question-answering-mcqa-on-9GAL 120B (zero-shot)#3Accuracy: 65.1
multiple-choice-question-answering-mcqa-on-9BLOOM (few-shot, k=5)#4Accuracy: 25.7
multiple-choice-question-answering-mcqa-on-9OPT (few-shot, k=5)#5Accuracy: 23.0
protein-function-prediction-on-caspsimseqGAL 120B#1ROUGE-L: 0.252
protein-function-prediction-on-caspsimseqGAL 30B#2ROUGE-L: 0.137
protein-function-prediction-on-caspsimseqGAL 6.7B#3ROUGE-L: 0.109
protein-function-prediction-on-caspsimseqGAL 1.3B#4ROUGE-L: 0.069
protein-function-prediction-on-caspsimseqGAL 125M#5ROUGE-L: 0.062
protein-function-prediction-on-paenseqGAL 120B#1ROUGE-L: 0.272
protein-function-prediction-on-paenseqGAL 30B#2ROUGE-L: 0.196
protein-function-prediction-on-paenseqGAL 6.7B#3ROUGE-L: 0.137
protein-function-prediction-on-paenseqGAL 1.3B#4ROUGE-L: 0.084
protein-function-prediction-on-paenseqGAL 125M#5ROUGE-L: 0.073
protein-function-prediction-on-uniprotseqGAL 120B#1ROUGE-L: 0.252
protein-function-prediction-on-uniprotseqGAL 30B#2ROUGE-L: 0.186
protein-function-prediction-on-uniprotseqGAL 6.7B#3ROUGE-L: 0.111
protein-function-prediction-on-uniprotseqGAL 1.3B#4ROUGE-L: 0.079
protein-function-prediction-on-uniprotseqGAL 125M#5ROUGE-L: 0.061
protein-structure-prediction-on-caspseqGAL 120B#1Validation perplexity: 17.26
protein-structure-prediction-on-caspseqGAL 30B#2Validation perplexity: 17.27
protein-structure-prediction-on-caspseqGAL 6.7B#3Validation perplexity: 17.29
protein-structure-prediction-on-caspseqGAL 1.3B#4Validation perplexity: 17.58
protein-structure-prediction-on-caspseqGAL 125M#5Validation perplexity: 20.62
protein-structure-prediction-on-caspsimseqGAL 120B#1Validation perplexity: 12.77
protein-structure-prediction-on-caspsimseqGAL 30B#2Validation perplexity: 15.42
protein-structure-prediction-on-caspsimseqGAL 6.7B#3Validation perplexity: 16.35
protein-structure-prediction-on-caspsimseqGAL 1.3B#4Validation perplexity: 17.04
protein-structure-prediction-on-caspsimseqGAL 125M#5Validation perplexity: 19.18
protein-structure-prediction-on-paenseqGAL 120B#1Validation perplexity: 3.14
protein-structure-prediction-on-paenseqGAL 30B#2Validation perplexity: 4.28
protein-structure-prediction-on-paenseqGAL 6.7B#3Validation perplexity: 7.76
protein-structure-prediction-on-paenseqGAL 1.3B#4Validation perplexity: 12.53
protein-structure-prediction-on-paenseqGAL 125M#5Validation perplexity: 16.35
protein-structure-prediction-on-uniprotseqGAL 120B#1Validation perplexity: 5.54
protein-structure-prediction-on-uniprotseqGAL 30B#2Validation perplexity: 8.23
protein-structure-prediction-on-uniprotseqGAL 6.7B#3Validation perplexity: 11.58
protein-structure-prediction-on-uniprotseqGAL 1.3B#4Validation perplexity: 15.82
protein-structure-prediction-on-uniprotseqGAL 125M#5Validation perplexity: 19.05
question-answering-on-bioasqGAL 120B (zero-shot)#2Accuracy: 94.3
question-answering-on-bioasqBLOOM (zero-shot)#4Accuracy: 91.4
question-answering-on-bioasqOPT (zero-shot)#6Accuracy: 81.4
question-answering-on-medqa-usmleGAL 120B (zero-shot)#16Accuracy: 44.4
question-answering-on-medqa-usmleBLOOM (few-shot, k=5)#21Accuracy: 23.3
question-answering-on-medqa-usmleOPT (few-shot, k=5)#22Accuracy: 22.8
question-answering-on-pubmedqaGAL 120B (zero-shot)#9Accuracy: 77.6
question-answering-on-pubmedqaBLOOM (zero-shot)#15Accuracy: 73.6
question-answering-on-pubmedqaOPT (zero-shot)#19Accuracy: 70.2
question-answering-on-truthfulqaGAL 120B#7MC1: 0.26
question-answering-on-truthfulqaGAL 30B#9MC1: 0.24
question-answering-on-truthfulqaOPT 175B#15MC1: 0.21
question-answering-on-truthfulqaGAL 125M#18MC1: 0.19
question-answering-on-truthfulqaGAL 1.3B#19MC1: 0.19
question-answering-on-truthfulqaGAL 6.7B#20MC1: 0.19
stereotypical-bias-analysis-on-crows-pairsGAL 120B#1Gender: 51.9Religion: 51.9Race/Color: 59.9Sexual Orientation: 77.4…
tdc-admet-benchmarking-group-on-tdcommonsGalactica-GAL-120B#8TDC.BBB_Martins: 0.661
tdc-admet-benchmarking-group-on-tdcommonsGalactica-GAL-1.3B#9TDC.BBB_Martins: 0.604
tdc-admet-benchmarking-group-on-tdcommonsGalactica-GAL-30B#10TDC.BBB_Martins: 0.596
tdc-admet-benchmarking-group-on-tdcommonsGalactica-GAL-6.7B#11TDC.BBB_Martins: 0.535
tdc-admet-benchmarking-group-on-tdcommonsGalactica-GAL-125M#12TDC.BBB_Martins: 0.393
word-sense-disambiguation-on-big-benchOPT 175B#3Accuracy: 49.1
word-sense-disambiguation-on-big-benchGAL 120B (few-shot, k=5)#4Accuracy: 48.7
word-sense-disambiguation-on-big-benchGAL 30B (few-shot, k=5)#5Accuracy: 47.0
word-sense-disambiguation-on-big-benchBLOOM 176B#6Accuracy: 1.3

Rank counts only results with a code link.