Can large language models reason about medical questions?

Benchmark Model Rank Results
multiple-choice-question-answering-mcqa-on-21Codex 5-shot CoT#5Test Set (Acc-%): 0.627Dev Set (Acc-%): 0.597
question-answering-on-medqa-usmleCodex 5-shot CoT#9Accuracy: 60.2
question-answering-on-pubmedqaCodex 5-shot CoT#5Accuracy: 78.2