BloombergGPT: A Large Language Model for Finance

Open paper

Benchmark	Model	Rank	Results
multiple-choice-question-answering-mcqa-on-big-bench-hyperbaton	Bloomberg GPT (few-shot, k=3)	#1	Accuracy: 92
multiple-choice-question-answering-mcqa-on-big-bench-hyperbaton	GPT-NeoX (few-shot, k=3)	#2	Accuracy: 92
multiple-choice-question-answering-mcqa-on-big-bench-hyperbaton	BLOOM 176B (few-shot, k=3)	#3	Accuracy: 92
multiple-choice-question-answering-mcqa-on-big-bench-hyperbaton	OPT 66B (few-shot, k=3)	#4	Accuracy: 91.6
multiple-choice-question-answering-mcqa-on-big-bench-hyperbaton	PaLM 540B (few-shot, k=3)	#7	Accuracy: 70.8
common-sense-reasoning-on-big-bench-causal-judgment	PaLM 540B (few-shot, k=3)	#2	Accuracy: 61.0
common-sense-reasoning-on-big-bench-causal-judgment	GPT-NeoX 20B (few-shot, k=3)	#5	Accuracy: 52.41
common-sense-reasoning-on-big-bench-causal-judgment	BLOOM 176B (few-shot, k=3)	#6	Accuracy: 51.87
common-sense-reasoning-on-big-bench-causal-judgment	OPT 66B (few-shot, k=3)	#7	Accuracy: 51.87
common-sense-reasoning-on-big-bench-causal-judgment	BloombergGPT 50B (few-shot, k=3)	#9	Accuracy: 49.73
common-sense-reasoning-on-big-bench-date-understanding	Bloomberg GPT 50B (few-shot, k=3)	#3	Accuracy: 54.8
common-sense-reasoning-on-big-bench-date-understanding	PaLM 540B (few-shot,k=3)	#4	Accuracy: 53.6
common-sense-reasoning-on-big-bench-date-understanding	BLOOM 176B (few-shot, k=3)	#6	Accuracy: 50.00
common-sense-reasoning-on-big-bench-date-understanding	OPT 66B (few-shot, k=3)	#7	Accuracy: 49.60
common-sense-reasoning-on-big-bench-date-understanding	GPT-NeoX 20B (few-shot, k=3)	#8	Accuracy: 45.60
common-sense-reasoning-on-big-bench-disambiguation-qa	PaLM 540B (few-shot, k=3)	#3	Accuracy: 60.8
common-sense-reasoning-on-big-bench-disambiguation-qa	GPT-NeoX 20B (few-shot, k=3)	#6	Accuracy: 40.8
common-sense-reasoning-on-big-bench-disambiguation-qa	OPT 66B (few-shot, k=3)	#7	Accuracy: 40.4
common-sense-reasoning-on-big-bench-disambiguation-qa	BLOOM 176B (few-shot, k=3)	#8	Accuracy: 40.4
common-sense-reasoning-on-big-bench-disambiguation-qa	Bloomberg GPT 50B (few-shot, k=3)	#9	Accuracy: 34
common-sense-reasoning-on-big-bench-sports-understanding	PaLM 540B (few-shot, k=3)	#3	Accuracy: 80.4
common-sense-reasoning-on-big-bench-sports-understanding	Bloomberg GPT (few-shot, k=3)	#5	Accuracy: 62.8
common-sense-reasoning-on-big-bench-sports-understanding	OPT 66B (few-shot, k=3)	#7	Accuracy: 54.4
common-sense-reasoning-on-big-bench-sports-understanding	GPT-NeoX (few-shot, k=3)	#8	Accuracy: 53.2
logical-reasoning-on-big-bench-formal-fallacies-syllogisms-negation	OPT 66B (few-shot, k=3)	#3	Accuracy: 54
logical-reasoning-on-big-bench-formal-fallacies-syllogisms-negation	PaLM 540B (few-shot, k=3)	#4	Accuracy: 53.6
logical-reasoning-on-big-bench-formal-fallacies-syllogisms-negation	BLOOM 176B (few-shot, k=3)	#5	Accuracy: 52.8
logical-reasoning-on-big-bench-formal-fallacies-syllogisms-negation	GPT-NeoX 20B (few-shot, k=3)	#6	Accuracy: 52.8
logical-reasoning-on-big-bench-formal-fallacies-syllogisms-negation	Bloomberg GPT 50B (few-shot, k=3)	#8	Accuracy: 50.8
logical-reasoning-on-big-bench-temporal-sequences	PaLM 540B (few-shot, k=3)	#3	Accuracy: 39.6
logical-reasoning-on-big-bench-temporal-sequences	BLOOM 176B (few-shot, k=3)	#4	Accuracy: 36.8
logical-reasoning-on-big-bench-temporal-sequences	Bloomberg GPT (few-shot, k=3)	#6	Accuracy: 29.2
logical-reasoning-on-big-bench-temporal-sequences	OPT 66B (few-shot, k=3)	#7	Accuracy: 23.6
logical-reasoning-on-big-bench-temporal-sequences	GPT-NeoX (few-shot, k=3)	#8	Accuracy: 21.2
multiple-choice-question-answering-mcqa-on-big-bench-movie-recommendation	BLOOM 176B (few-shot, k=3)	#3	Accuracy: 91.2
multiple-choice-question-answering-mcqa-on-big-bench-movie-recommendation	OPT 66B (few-shot, k=3)	#4	Accuracy: 91.2
multiple-choice-question-answering-mcqa-on-big-bench-movie-recommendation	Bloomberg GPT (few-shot, k=3)	#5	Accuracy: 90.4
multiple-choice-question-answering-mcqa-on-big-bench-movie-recommendation	PaLM 540B (few-shot, k=3)	#6	Accuracy: 87.2
multiple-choice-question-answering-mcqa-on-big-bench-movie-recommendation	GPT-NeoX (few-shot, k=3)	#7	Accuracy: 86.4
multiple-choice-question-answering-mcqa-on-big-bench-navigate	PaLM 540B (few-shot, k=3)	#3	Accuracy: 62.4
multiple-choice-question-answering-mcqa-on-big-bench-navigate	BLOOM 176B (few-shot, k=3)	#6	Accuracy: 50
multiple-choice-question-answering-mcqa-on-big-bench-navigate	GPT-NeoX (few-shot, k=3)	#7	Accuracy: 45.2
multiple-choice-question-answering-mcqa-on-big-bench-navigate	Bloomberg GPT (few-shot, k=3)	#8	Accuracy: 42
multiple-choice-question-answering-mcqa-on-big-bench-navigate	OPT 66B (few-shot, k=3)	#9	Accuracy: 42
multiple-choice-question-answering-mcqa-on-big-bench-ruin-names	PaLM 540B (few-shot, k=3)	#3	Accuracy: 76
multiple-choice-question-answering-mcqa-on-big-bench-ruin-names	Bloomberg GPT (few-shot, k=3)	#4	Accuracy: 56
multiple-choice-question-answering-mcqa-on-big-bench-ruin-names	BLOOM 176B (few-shot, k=3)	#5	Accuracy: 54.8
multiple-choice-question-answering-mcqa-on-big-bench-ruin-names	GPT-NeoX (few-shot, k=3)	#6	Accuracy: 54
multiple-choice-question-answering-mcqa-on-big-bench-ruin-names	OPT 66B (few-shot, k=3)	#7	Accuracy: 52.8
sarcasm-detection-on-big-bench-snarks	PaLM 540B (few-shot, k=3)	#3	Accuracy: 78.1
sarcasm-detection-on-big-bench-snarks	BLOOM 176B (few-shot, k=3)	#4	Accuracy: 72.47
sarcasm-detection-on-big-bench-snarks	Bloomberg GPT (few-shot, k=3)	#5	Accuracy: 69.66
sarcasm-detection-on-big-bench-snarks	GPT-NeoX (few-shot, k=3)	#6	Accuracy: 62.36
logical-reasoning-on-big-bench-penguins-in-a-table	PaLM 540B (few-shot, k=3)	#4	Accuracy: 44.5
logical-reasoning-on-big-bench-penguins-in-a-table	BLOOM 176B (few-shot, k=3)	#6	Accuracy: 40.41
logical-reasoning-on-big-bench-penguins-in-a-table	Bloomberg GPT (few-shot, k=3)	#7	Accuracy: 37.67
logical-reasoning-on-big-bench-penguins-in-a-table	GPT-NeoX (few-shot, k=3)	#8	Accuracy: 33.56
logical-reasoning-on-big-bench-penguins-in-a-table	OPT 66B (few-shot, k=3)	#9	Accuracy: 28.08
logical-reasoning-on-big-bench-reasoning-about-colored-objects	PaLM 540B (few-shot, k=3)	#5	Accuracy: 38
logical-reasoning-on-big-bench-reasoning-about-colored-objects	BLOOM 176B (few-shot, k=3)	#6	Accuracy: 36.8
logical-reasoning-on-big-bench-reasoning-about-colored-objects	Bloomberg GPT (few-shot, k=3)	#7	Accuracy: 34.8
logical-reasoning-on-big-bench-reasoning-about-colored-objects	OPT 66B (few-shot, k=3)	#8	Accuracy: 31.2
logical-reasoning-on-big-bench-reasoning-about-colored-objects	GPT-NeoX (few-shot, k=3)	#9	Accuracy: 26
natural-language-inference-on-anli-test	BLOOM 176B (one-shot)	#13	A1: 33.6A2: 33.8A3: 35.17
natural-language-inference-on-anli-test	OPT 66B (one-shot)	#14	A1: 33.1A2: 34.2A3: 34.92
natural-language-inference-on-anli-test	Bloomberg GPT (one-shot)	#15	A1: 32.9A2: 34.4A3: 37.33
natural-language-inference-on-anli-test	GPT-NeoX (one-shot)	#16	A1: 32.6A2: 33.8A3: 36.17
natural-language-inference-on-commitmentbank	Bloomberg GPT (one-shot)	#14	Accuracy: 53.57
natural-language-inference-on-commitmentbank	GPT-NeoX (one-shot)	#15	Accuracy: 48.21
natural-language-inference-on-commitmentbank	BLOOM 176B (one-shot)	#16	Accuracy: 48.21
natural-language-inference-on-commitmentbank	OPT 66B (one-shot)	#17	Accuracy: 44.64
question-answering-on-openbookqa	OPT 66B (one-shot)	#15	Accuracy: 58.0
question-answering-on-openbookqa	Bloomberg GPT 50B (1-shot)	#20	Accuracy: 51.6
question-answering-on-openbookqa	BLOOM 176B (2-shot)	#21	Accuracy: 47.2
question-answering-on-openbookqa	GPT-NeoX 50B (2-shot)	#22	Accuracy: 44.2
question-answering-on-multirc	Bloomberg GPT 50B (1-shot)	#16	F1: 62.3
question-answering-on-multirc	BLOOM 176B (1-shot)	#21	F1: 26.7
question-answering-on-multirc	GPT-NeoX 20B (1-shot)	#22	F1: 22.9
question-answering-on-multirc	OPT 66B (1-shot)	#23	F1: 18.8
common-sense-reasoning-on-commonsenseqa	OPT 66B (1-shot)	#19	Accuracy: 66.4
common-sense-reasoning-on-commonsenseqa	Bloomberg GPT 50B (1-shot)	#20	Accuracy: 65.5
common-sense-reasoning-on-commonsenseqa	BLOOM 176B (1-shot)	#22	Accuracy: 64.2
common-sense-reasoning-on-commonsenseqa	GPT-NeoX 20B (1-shot)	#25	Accuracy: 60.4
question-answering-on-copa	GPT-NeoX (one-shot)	#19	Accuracy: 88
question-answering-on-copa	Bloomberg GPT (one-shot)	#24	Accuracy: 86
question-answering-on-copa	OPT 66B (one-shot)	#25	Accuracy: 86
question-answering-on-copa	BLOOM 176B (one-shot)	#29	Accuracy: 84
common-sense-reasoning-on-record	Bloomberg GPT 50B (1-shot)	#20	F1: 82.8
common-sense-reasoning-on-record	OPT 66B (1-shot)	#21	F1: 82.5
common-sense-reasoning-on-record	BLOOM 176B (1-shot)	#22	F1: 78
common-sense-reasoning-on-record	GPT-NeoX 20B (1-shot)	#23	F1: 67.9
reading-comprehension-on-race	Bloomberg GPT (one-shot)	#21	Accuracy (Middle): 54.32Accuracy (High): 41.74
reading-comprehension-on-race	BLOOM 176B (one-shot)	#22	Accuracy (Middle): 52.3Accuracy (High): 39.14
reading-comprehension-on-race	OPT 66B (one-shot)	#23	Accuracy (Middle): 47.42Accuracy (High): 37.02
reading-comprehension-on-race	GPT-NeoX (one-shot)	#24	Accuracy (Middle): 41.23Accuracy (High): 34.33
common-sense-reasoning-on-arc-challenge	BLOOM 176B (1-shot)	#25	Accuracy: 50.85
common-sense-reasoning-on-arc-challenge	Bloomberg GPT 50B (1-shot)	#27	Accuracy: 48.63
common-sense-reasoning-on-arc-challenge	GPT-NeoX 20B (1-shot)	#29	Accuracy: 45.39
common-sense-reasoning-on-arc-challenge	OPT 66B (one-shot)	#31	Accuracy: 44.54
common-sense-reasoning-on-arc-easy	BLOOM 176B (1-shot)	#33	Accuracy: 75.93
common-sense-reasoning-on-arc-easy	Bloomberg GPT 50B (1-shot)	#40	Accuracy: 73.99
common-sense-reasoning-on-arc-easy	OPT 66B (1-shot)	#53	Accuracy: 71.25
common-sense-reasoning-on-arc-easy	GPT-NeoX 20B (1-shot)	#56	Accuracy: 70.79
natural-language-inference-on-rte	Bloomberg GPT 50B (1-shot)	#53	Accuracy: 69.3%
natural-language-inference-on-rte	BLOOM 176B (1-shot)	#75	Accuracy: 57.4%
natural-language-inference-on-rte	OPT 66B (1-shot)	#78	Accuracy: 54.9%
natural-language-inference-on-rte	GPT-NeoX 20B (1-shot)	#80	Accuracy: 53.8%
question-answering-on-boolq	Bloomberg GPT 50B (1-shot)	#59	Accuracy: 74.6
question-answering-on-boolq	OPT 66B (1-shot)	#110	Accuracy: 57.5
question-answering-on-boolq	BLOOM 176B (1-shot)	#114	Accuracy: 52.9
question-answering-on-boolq	GPT-NeoX 20B (1-shot)	#117	Accuracy: 46.4
multi-task-language-understanding-on-mmlu	Bloomberg GPT 50B (5-shot)	#74	Average (%): 39.2
multi-task-language-understanding-on-mmlu	BLOOM 176B (5-shot)	#75	Average (%): 39.1
multi-task-language-understanding-on-mmlu	OPT 66B (5-shot)	#78	Average (%): 36
question-answering-on-piqa	Bloomberg GPT 50B (1-shot)	#77	Accuracy: 77.9
question-answering-on-piqa	OPT 66B (1-shot)	#80	Accuracy: 77.6
question-answering-on-piqa	BLOOM 176B (1-shot)	#94	Accuracy: 77
question-answering-on-piqa	GPT-NeoX 20B (1-shot)	#108	Accuracy: 75.8
common-sense-reasoning-on-winogrande	BLOOM 176B (1-shot)	#88	Accuracy: 67
common-sense-reasoning-on-winogrande	OPT 66B (1-shot)	#92	Accuracy: 66.1
common-sense-reasoning-on-winogrande	Bloomberg GPT (one-shot)	#97	Accuracy: 64.1
common-sense-reasoning-on-winogrande	GPT-NeoX (one-shot)	#114	Accuracy: 60.6
sentence-completion-on-hellaswag	BlooombergGPT 50B (1-shot)	#98	Accuracy: 73.9
sentence-completion-on-hellaswag	OPT 66B (1-shot)	#99	Accuracy: 73.5
sentence-completion-on-hellaswag	BLOOM 176B (1-shot)	#100	Accuracy: 73.2
sentence-completion-on-hellaswag	GPT-NeoX 20B (1-shot)	#117	Accuracy: 68.4