Large Language Models Encode Clinical Knowledge

Open paper
Benchmark Model Rank Results
question-answering-on-pubmedqaFlan-PaLM (540B, Few-shot)#6Accuracy: 79
question-answering-on-pubmedqaFlan-PaLM (62B, Few-shot)#14Accuracy: 77.2
question-answering-on-pubmedqaFlan-PaLM (540B, SC)#17Accuracy: 75.2
question-answering-on-pubmedqaFlan-PaLM (8B, Few-shot)#25Accuracy: 67.6
question-answering-on-pubmedqaPaLM (62B, Few-shot)#30Accuracy: 57.8
question-answering-on-pubmedqaPaLM (540B, Few-shot)#33Accuracy: 55
question-answering-on-pubmedqaPaLM (8B, Few-shot)#34Accuracy: 34
question-answering-on-medqa-usmleFlan-PaLM (540 B)#9Accuracy: 67.6
question-answering-on-medqa-usmlePubMedGPT (2.7 B)#15Accuracy: 50.3
question-answering-on-medqa-usmleBioLinkBERT (340 M)#17Accuracy: 45.1
question-answering-on-medqa-usmleGPT-Neo (2.7 B)#22Accuracy: 33.3
multiple-choice-question-answering-mcqa-on-medmcqaFlan-PaLM (540B, SC)#17Dev Set (Acc-%): 0.576
multiple-choice-question-answering-mcqa-on-medmcqaFlan-PaLM (540B, Few-shot)#18Dev Set (Acc-%): 0.565
multiple-choice-question-answering-mcqa-on-medmcqaPaLM (540B, Few-shot)#19Dev Set (Acc-%): 0.545
multiple-choice-question-answering-mcqa-on-medmcqaFlan-PaLM (540B, CoT)#20Dev Set (Acc-%): 0.536
multiple-choice-question-answering-mcqa-on-medmcqaFlan-PaLM (62B, Few-shot)#22Dev Set (Acc-%): 0.462
multiple-choice-question-answering-mcqa-on-medmcqaPaLM (62B, Few-shot)#23Dev Set (Acc-%): 0.434
multiple-choice-question-answering-mcqa-on-medmcqaFlan-PaLM (8B, Few-shot)#24Dev Set (Acc-%): 0.345
multiple-choice-question-answering-mcqa-on-medmcqaPaLM (8B, Few-shot)#27Dev Set (Acc-%): 0.267

Rank counts only results with a code link.