Large Language Models Encode Clinical Knowledge

Open paper
Benchmark Model Rank Results
multiple-choice-question-answering-mcqa-on-21Flan-PaLM (540B, SC)#12Dev Set (Acc-%): 0.576
multiple-choice-question-answering-mcqa-on-21Flan-PaLM (540B, Few-shot)#13Dev Set (Acc-%): 0.565
multiple-choice-question-answering-mcqa-on-21PaLM (540B, Few-shot)#14Dev Set (Acc-%): 0.545
multiple-choice-question-answering-mcqa-on-21Flan-PaLM (540B, CoT)#15Dev Set (Acc-%): 0.536
multiple-choice-question-answering-mcqa-on-21Flan-PaLM (62B, Few-shot)#17Dev Set (Acc-%): 0.462
multiple-choice-question-answering-mcqa-on-21PaLM (62B, Few-shot)#18Dev Set (Acc-%): 0.434
multiple-choice-question-answering-mcqa-on-21Flan-PaLM (8B, Few-shot)#19Dev Set (Acc-%): 0.345
multiple-choice-question-answering-mcqa-on-21PaLM (8B, Few-shot)#22Dev Set (Acc-%): 0.267
question-answering-on-medqa-usmleFlan-PaLM (540 B)#7Accuracy: 67.6
question-answering-on-medqa-usmlePubMedGPT (2.7 B)#13Accuracy: 50.3
question-answering-on-medqa-usmleBioLinkBERT (340 M)#15Accuracy: 45.1
question-answering-on-medqa-usmleGPT-Neo (2.7 B)#20Accuracy: 33.3
question-answering-on-pubmedqaFlan-PaLM (540B, Few-shot)#4Accuracy: 79
question-answering-on-pubmedqaFlan-PaLM (62B, Few-shot)#10Accuracy: 77.2
question-answering-on-pubmedqaFlan-PaLM (540B, SC)#12Accuracy: 75.2
question-answering-on-pubmedqaFlan-PaLM (8B, Few-shot)#20Accuracy: 67.6
question-answering-on-pubmedqaPaLM (62B, Few-shot)#22Accuracy: 57.8
question-answering-on-pubmedqaPaLM (540B, Few-shot)#24Accuracy: 55
question-answering-on-pubmedqaPaLM (8B, Few-shot)#25Accuracy: 34

Rank counts only results with a code link.