BloombergGPT: A Large Language Model for Finance

Open paper
Benchmark Model Rank Results
common-sense-reasoning-on-arc-challengeBLOOM 176B (1-shot)#24Accuracy: 50.85
common-sense-reasoning-on-arc-challengeBloomberg GPT 50B (1-shot)#26Accuracy: 48.63
common-sense-reasoning-on-arc-challengeGPT-NeoX 20B (1-shot)#28Accuracy: 45.39
common-sense-reasoning-on-arc-challengeOPT 66B (one-shot)#30Accuracy: 44.54
common-sense-reasoning-on-arc-easyBLOOM 176B (1-shot)#19Accuracy: 75.93
common-sense-reasoning-on-arc-easyBloomberg GPT 50B (1-shot)#22Accuracy: 73.99
common-sense-reasoning-on-arc-easyOPT 66B (1-shot)#25Accuracy: 71.25
common-sense-reasoning-on-arc-easyGPT-NeoX 20B (1-shot)#28Accuracy: 70.79
common-sense-reasoning-on-big-benchPaLM 540B (few-shot, k=3)#3Accuracy: 60.8
common-sense-reasoning-on-big-benchGPT-NeoX 20B (few-shot, k=3)#6Accuracy: 40.8
common-sense-reasoning-on-big-benchOPT 66B (few-shot, k=3)#7Accuracy: 40.4
common-sense-reasoning-on-big-benchBLOOM 176B (few-shot, k=3)#8Accuracy: 40.4
common-sense-reasoning-on-big-benchBloomberg GPT 50B (few-shot, k=3)#9Accuracy: 34
common-sense-reasoning-on-big-bench-causalPaLM 540B (few-shot, k=3)#2Accuracy: 61.0
common-sense-reasoning-on-big-bench-causalGPT-NeoX 20B (few-shot, k=3)#5Accuracy: 52.41
common-sense-reasoning-on-big-bench-causalBLOOM 176B (few-shot, k=3)#6Accuracy: 51.87
common-sense-reasoning-on-big-bench-causalOPT 66B (few-shot, k=3)#7Accuracy: 51.87
common-sense-reasoning-on-big-bench-causalBloombergGPT 50B (few-shot, k=3)#9Accuracy: 49.73
common-sense-reasoning-on-big-bench-dateBloomberg GPT 50B (few-shot, k=3)#3Accuracy: 54.8
common-sense-reasoning-on-big-bench-datePaLM 540B (few-shot,k=3)#4Accuracy: 53.6
common-sense-reasoning-on-big-bench-dateBLOOM 176B (few-shot, k=3)#6Accuracy: 50.00
common-sense-reasoning-on-big-bench-dateOPT 66B (few-shot, k=3)#7Accuracy: 49.60
common-sense-reasoning-on-big-bench-dateGPT-NeoX 20B (few-shot, k=3)#8Accuracy: 45.60
common-sense-reasoning-on-big-bench-sportsPaLM 540B (few-shot, k=3)#3Accuracy: 80.4
common-sense-reasoning-on-big-bench-sportsBloomberg GPT (few-shot, k=3)#5Accuracy: 62.8
common-sense-reasoning-on-big-bench-sportsOPT 66B (few-shot, k=3)#7Accuracy: 54.4
common-sense-reasoning-on-big-bench-sportsGPT-NeoX (few-shot, k=3)#8Accuracy: 53.2
common-sense-reasoning-on-commonsenseqaOPT 66B (1-shot)#19Accuracy: 66.4
common-sense-reasoning-on-commonsenseqaBloomberg GPT 50B (1-shot)#20Accuracy: 65.5
common-sense-reasoning-on-commonsenseqaBLOOM 176B (1-shot)#22Accuracy: 64.2
common-sense-reasoning-on-commonsenseqaGPT-NeoX 20B (1-shot)#25Accuracy: 60.4
common-sense-reasoning-on-recordBloomberg GPT 50B (1-shot)#20F1: 82.8
common-sense-reasoning-on-recordOPT 66B (1-shot)#21F1: 82.5
common-sense-reasoning-on-recordBLOOM 176B (1-shot)#22F1: 78
common-sense-reasoning-on-recordGPT-NeoX 20B (1-shot)#23F1: 67.9
common-sense-reasoning-on-winograndeBLOOM 176B (1-shot)#38Accuracy: 67
common-sense-reasoning-on-winograndeOPT 66B (1-shot)#40Accuracy: 66.1
common-sense-reasoning-on-winograndeBloomberg GPT (one-shot)#42Accuracy: 64.1
common-sense-reasoning-on-winograndeGPT-NeoX (one-shot)#46Accuracy: 60.6
logical-reasoning-on-big-bench-formalOPT 66B (few-shot, k=3)#3Accuracy: 54
logical-reasoning-on-big-bench-formalPaLM 540B (few-shot, k=3)#4Accuracy: 53.6
logical-reasoning-on-big-bench-formalBLOOM 176B (few-shot, k=3)#5Accuracy: 52.8
logical-reasoning-on-big-bench-formalGPT-NeoX 20B (few-shot, k=3)#6Accuracy: 52.8
logical-reasoning-on-big-bench-formalBloomberg GPT 50B (few-shot, k=3)#8Accuracy: 50.8
logical-reasoning-on-big-bench-penguins-in-aPaLM 540B (few-shot, k=3)#4Accuracy: 44.5
logical-reasoning-on-big-bench-penguins-in-aBLOOM 176B (few-shot, k=3)#6Accuracy: 40.41
logical-reasoning-on-big-bench-penguins-in-aBloomberg GPT (few-shot, k=3)#7Accuracy: 37.67
logical-reasoning-on-big-bench-penguins-in-aGPT-NeoX (few-shot, k=3)#8Accuracy: 33.56
logical-reasoning-on-big-bench-penguins-in-aOPT 66B (few-shot, k=3)#9Accuracy: 28.08
logical-reasoning-on-big-bench-reasoningPaLM 540B (few-shot, k=3)#5Accuracy: 38
logical-reasoning-on-big-bench-reasoningBLOOM 176B (few-shot, k=3)#6Accuracy: 36.8
logical-reasoning-on-big-bench-reasoningBloomberg GPT (few-shot, k=3)#7Accuracy: 34.8
logical-reasoning-on-big-bench-reasoningOPT 66B (few-shot, k=3)#8Accuracy: 31.2
logical-reasoning-on-big-bench-reasoningGPT-NeoX (few-shot, k=3)#9Accuracy: 26
logical-reasoning-on-big-bench-temporalPaLM 540B (few-shot, k=3)#3Accuracy: 39.6
logical-reasoning-on-big-bench-temporalBLOOM 176B (few-shot, k=3)#4Accuracy: 36.8
logical-reasoning-on-big-bench-temporalBloomberg GPT (few-shot, k=3)#6Accuracy: 29.2
logical-reasoning-on-big-bench-temporalOPT 66B (few-shot, k=3)#7Accuracy: 23.6
logical-reasoning-on-big-bench-temporalGPT-NeoX (few-shot, k=3)#8Accuracy: 21.2
multi-task-language-understanding-on-mmluBloomberg GPT 50B (5-shot)#27Average (%): 39.2
multi-task-language-understanding-on-mmluBLOOM 176B (5-shot)#28Average (%): 39.1
multi-task-language-understanding-on-mmluOPT 66B (5-shot)#30Average (%): 36
multiple-choice-question-answering-mcqa-on-27Bloomberg GPT (few-shot, k=3)#1Accuracy: 92
multiple-choice-question-answering-mcqa-on-27GPT-NeoX (few-shot, k=3)#2Accuracy: 92
multiple-choice-question-answering-mcqa-on-27BLOOM 176B (few-shot, k=3)#3Accuracy: 92
multiple-choice-question-answering-mcqa-on-27OPT 66B (few-shot, k=3)#4Accuracy: 91.6
multiple-choice-question-answering-mcqa-on-27PaLM 540B (few-shot, k=3)#7Accuracy: 70.8
multiple-choice-question-answering-mcqa-on-28BLOOM 176B (few-shot, k=3)#3Accuracy: 91.2
multiple-choice-question-answering-mcqa-on-28OPT 66B (few-shot, k=3)#4Accuracy: 91.2
multiple-choice-question-answering-mcqa-on-28Bloomberg GPT (few-shot, k=3)#5Accuracy: 90.4
multiple-choice-question-answering-mcqa-on-28PaLM 540B (few-shot, k=3)#6Accuracy: 87.2
multiple-choice-question-answering-mcqa-on-28GPT-NeoX (few-shot, k=3)#7Accuracy: 86.4
multiple-choice-question-answering-mcqa-on-29PaLM 540B (few-shot, k=3)#3Accuracy: 62.4
multiple-choice-question-answering-mcqa-on-29BLOOM 176B (few-shot, k=3)#6Accuracy: 50
multiple-choice-question-answering-mcqa-on-29GPT-NeoX (few-shot, k=3)#7Accuracy: 45.2
multiple-choice-question-answering-mcqa-on-29Bloomberg GPT (few-shot, k=3)#8Accuracy: 42
multiple-choice-question-answering-mcqa-on-29OPT 66B (few-shot, k=3)#9Accuracy: 42
multiple-choice-question-answering-mcqa-on-30PaLM 540B (few-shot, k=3)#3Accuracy: 76
multiple-choice-question-answering-mcqa-on-30Bloomberg GPT (few-shot, k=3)#4Accuracy: 56
multiple-choice-question-answering-mcqa-on-30BLOOM 176B (few-shot, k=3)#5Accuracy: 54.8
multiple-choice-question-answering-mcqa-on-30GPT-NeoX (few-shot, k=3)#6Accuracy: 54
multiple-choice-question-answering-mcqa-on-30OPT 66B (few-shot, k=3)#7Accuracy: 52.8
natural-language-inference-on-anli-testBLOOM 176B (one-shot)#13A1: 33.6A2: 33.8A3: 35.17
natural-language-inference-on-anli-testOPT 66B (one-shot)#14A1: 33.1A2: 34.2A3: 34.92
natural-language-inference-on-anli-testBloomberg GPT (one-shot)#15A1: 32.9A2: 34.4A3: 37.33
natural-language-inference-on-anli-testGPT-NeoX (one-shot)#16A1: 32.6A2: 33.8A3: 36.17
natural-language-inference-on-commitmentbankBloomberg GPT (one-shot)#14Accuracy: 53.57
natural-language-inference-on-commitmentbankGPT-NeoX (one-shot)#15Accuracy: 48.21
natural-language-inference-on-commitmentbankBLOOM 176B (one-shot)#16Accuracy: 48.21
natural-language-inference-on-commitmentbankOPT 66B (one-shot)#17Accuracy: 44.64
natural-language-inference-on-rteBloomberg GPT 50B (1-shot)#49Accuracy: 69.3%
natural-language-inference-on-rteBLOOM 176B (1-shot)#71Accuracy: 57.4%
natural-language-inference-on-rteOPT 66B (1-shot)#73Accuracy: 54.9%
natural-language-inference-on-rteGPT-NeoX 20B (1-shot)#75Accuracy: 53.8%
question-answering-on-boolqBloomberg GPT 50B (1-shot)#35Accuracy: 74.6
question-answering-on-boolqOPT 66B (1-shot)#57Accuracy: 57.5
question-answering-on-boolqBLOOM 176B (1-shot)#60Accuracy: 52.9
question-answering-on-boolqGPT-NeoX 20B (1-shot)#62Accuracy: 46.4
question-answering-on-copaGPT-NeoX (one-shot)#19Accuracy: 88
question-answering-on-copaBloomberg GPT (one-shot)#24Accuracy: 86
question-answering-on-copaOPT 66B (one-shot)#25Accuracy: 86
question-answering-on-copaBLOOM 176B (one-shot)#29Accuracy: 84
question-answering-on-multircBloomberg GPT 50B (1-shot)#16F1: 62.3
question-answering-on-multircBLOOM 176B (1-shot)#21F1: 26.7
question-answering-on-multircGPT-NeoX 20B (1-shot)#22F1: 22.9
question-answering-on-multircOPT 66B (1-shot)#23F1: 18.8
question-answering-on-openbookqaOPT 66B (one-shot)#15Accuracy: 58.0
question-answering-on-openbookqaBloomberg GPT 50B (1-shot)#20Accuracy: 51.6
question-answering-on-openbookqaBLOOM 176B (2-shot)#21Accuracy: 47.2
question-answering-on-openbookqaGPT-NeoX 50B (2-shot)#22Accuracy: 44.2
question-answering-on-piqaBloomberg GPT 50B (1-shot)#38Accuracy: 77.9
question-answering-on-piqaOPT 66B (1-shot)#39Accuracy: 77.6
question-answering-on-piqaBLOOM 176B (1-shot)#41Accuracy: 77
question-answering-on-piqaGPT-NeoX 20B (1-shot)#46Accuracy: 75.8
reading-comprehension-on-raceBloomberg GPT (one-shot)#16Accuracy (Middle): 54.32Accuracy (High): 41.74
reading-comprehension-on-raceBLOOM 176B (one-shot)#17Accuracy (Middle): 52.3Accuracy (High): 39.14
reading-comprehension-on-raceOPT 66B (one-shot)#18Accuracy (Middle): 47.42Accuracy (High): 37.02
reading-comprehension-on-raceGPT-NeoX (one-shot)#19Accuracy (Middle): 41.23Accuracy (High): 34.33
sarcasm-detection-on-big-bench-snarksPaLM 540B (few-shot, k=3)#3Accuracy: 78.1
sarcasm-detection-on-big-bench-snarksBLOOM 176B (few-shot, k=3)#4Accuracy: 72.47
sarcasm-detection-on-big-bench-snarksBloomberg GPT (few-shot, k=3)#5Accuracy: 69.66
sarcasm-detection-on-big-bench-snarksGPT-NeoX (few-shot, k=3)#6Accuracy: 62.36
sentence-completion-on-hellaswagBlooombergGPT 50B (1-shot)#42Accuracy: 73.9
sentence-completion-on-hellaswagOPT 66B (1-shot)#43Accuracy: 73.5
sentence-completion-on-hellaswagBLOOM 176B (1-shot)#44Accuracy: 73.2
sentence-completion-on-hellaswagGPT-NeoX 20B (1-shot)#46Accuracy: 68.4

Rank counts only results with a code link.