Towards Expert-Level Medical Question Answering with Large Language Models

Benchmark Model Rank Results
multiple-choice-question-answering-mcqa-on-11Med-PaLM 2 (ER)#1Accuracy: 95.8
multiple-choice-question-answering-mcqa-on-11Med-PaLM 2 (CoT + SC)#2Accuracy: 95.1
multiple-choice-question-answering-mcqa-on-11Med-PaLM 2 (5-shot)#3Accuracy: 94.4
multiple-choice-question-answering-mcqa-on-21Med-PaLM 2 (ER)#1Test Set (Acc-%): 0.723
multiple-choice-question-answering-mcqa-on-21Med-PaLM 2 (CoT+SC)#2Test Set (Acc-%): 0.715
multiple-choice-question-answering-mcqa-on-21Med-PaLM 2 (5-shot)#3Test Set (Acc-%): 0.713
multiple-choice-question-answering-mcqa-on-23Med-PaLM 2 (ER)#1Accuracy: 88.7
multiple-choice-question-answering-mcqa-on-23Med-PaLM 2 (5-shot)#2Accuracy: 88.3
multiple-choice-question-answering-mcqa-on-23Med-PaLM 2 (CoT + SC)#3Accuracy: 88.3
multiple-choice-question-answering-mcqa-on-24Med-PaLM 2 (ER)#1Accuracy: 84.4
multiple-choice-question-answering-mcqa-on-24Med-PaLM 2 (CoT + SC)#2Accuracy: 80.0
multiple-choice-question-answering-mcqa-on-24Med-PaLM 2 (5-shot)#3Accuracy: 77.8
multiple-choice-question-answering-mcqa-on-25Med-PaLM 2 (5-shot)#1Accuracy: 95.2
multiple-choice-question-answering-mcqa-on-25Med-PaLM 2 (CoT + SC)#2Accuracy: 93.4
multiple-choice-question-answering-mcqa-on-25Med-PaLM 2 (ER)#3Accuracy: 92.3
multiple-choice-question-answering-mcqa-on-26Med-PaLM (ER)#1Accuracy: 83.2
multiple-choice-question-answering-mcqa-on-26Med-PaLM (CoT + SC)#2Accuracy: 81.5
multiple-choice-question-answering-mcqa-on-26Med-PaLM 2 (5-shot)#3Accuracy: 80.9
multiple-choice-question-answering-mcqa-on-8Med-PaLM 2 (ER)#1Accuracy: 92
multiple-choice-question-answering-mcqa-on-8Med-PaLM 2 (5-shot)#2Accuracy: 90
multiple-choice-question-answering-mcqa-on-8Med-PaLM 2 (CoT + SC)#3Accuracy: 89
question-answering-on-medqa-usmleMed-PaLM 2#2Accuracy: 85.4
question-answering-on-medqa-usmleMed-PaLM 2 (CoT + SC)#3Accuracy: 83.7
question-answering-on-medqa-usmleMed-PaLM 2 (5-shot)#4Accuracy: 79.7
question-answering-on-pubmedqaMed-PaLM 2 (5-shot)#3Accuracy: 79.2
question-answering-on-pubmedqaMed-PaLM 2 (ER)#13Accuracy: 75.0
question-answering-on-pubmedqaMed-PaLM 2 (CoT + SC)#14Accuracy: 74.0