Towards Expert-Level Medical Question Answering with Large Language Models

Benchmark Model Rank Results
multiple-choice-question-answering-mcqa-on-mmlu-anatomyMed-PaLM 2 (ER)#1Accuracy: 84.4
multiple-choice-question-answering-mcqa-on-mmlu-anatomyMed-PaLM 2 (CoT + SC)#2Accuracy: 80.0
multiple-choice-question-answering-mcqa-on-mmlu-anatomyMed-PaLM 2 (5-shot)#3Accuracy: 77.8
multiple-choice-question-answering-mcqa-on-mmlu-clinical-knowledgeMed-PaLM 2 (ER)#1Accuracy: 88.7
multiple-choice-question-answering-mcqa-on-mmlu-clinical-knowledgeMed-PaLM 2 (5-shot)#2Accuracy: 88.3
multiple-choice-question-answering-mcqa-on-mmlu-clinical-knowledgeMed-PaLM 2 (CoT + SC)#3Accuracy: 88.3
multiple-choice-question-answering-mcqa-on-mmlu-college-biologyMed-PaLM 2 (ER)#1Accuracy: 95.8
multiple-choice-question-answering-mcqa-on-mmlu-college-biologyMed-PaLM 2 (CoT + SC)#2Accuracy: 95.1
multiple-choice-question-answering-mcqa-on-mmlu-college-biologyMed-PaLM 2 (5-shot)#3Accuracy: 94.4
multiple-choice-question-answering-mcqa-on-mmlu-college-medicineMed-PaLM (ER)#1Accuracy: 83.2
multiple-choice-question-answering-mcqa-on-mmlu-college-medicineMed-PaLM (CoT + SC)#2Accuracy: 81.5
multiple-choice-question-answering-mcqa-on-mmlu-college-medicineMed-PaLM 2 (5-shot)#3Accuracy: 80.9
multiple-choice-question-answering-mcqa-on-mmlu-medical-geneticsMed-PaLM 2 (ER)#1Accuracy: 92
multiple-choice-question-answering-mcqa-on-mmlu-medical-geneticsMed-PaLM 2 (5-shot)#2Accuracy: 90
multiple-choice-question-answering-mcqa-on-mmlu-medical-geneticsMed-PaLM 2 (CoT + SC)#3Accuracy: 89
multiple-choice-question-answering-mcqa-on-mmlu-professional-medicineMed-PaLM 2 (5-shot)#1Accuracy: 95.2
multiple-choice-question-answering-mcqa-on-mmlu-professional-medicineMed-PaLM 2 (CoT + SC)#2Accuracy: 93.4
multiple-choice-question-answering-mcqa-on-mmlu-professional-medicineMed-PaLM 2 (ER)#3Accuracy: 92.3
multiple-choice-question-answering-mcqa-on-medmcqaMed-PaLM 2 (ER)#3Test Set (Acc-%): 0.723
multiple-choice-question-answering-mcqa-on-medmcqaMed-PaLM 2 (CoT+SC)#4Test Set (Acc-%): 0.715
multiple-choice-question-answering-mcqa-on-medmcqaMed-PaLM 2 (5-shot)#5Test Set (Acc-%): 0.713
question-answering-on-medqa-usmleMed-PaLM 2#4Accuracy: 85.4
question-answering-on-medqa-usmleMed-PaLM 2 (CoT + SC)#5Accuracy: 83.7
question-answering-on-medqa-usmleMed-PaLM 2 (5-shot)#6Accuracy: 79.7
question-answering-on-pubmedqaMed-PaLM 2 (5-shot)#5Accuracy: 79.2
question-answering-on-pubmedqaMed-PaLM 2 (ER)#18Accuracy: 75.0
question-answering-on-pubmedqaMed-PaLM 2 (CoT + SC)#19Accuracy: 74.0