Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

Benchmark Model Rank Results
question-answering-on-piqaPhi-3.5-MoE (5-shot)#4Accuracy: 88.6
instruction-following-on-mtbenchPhi-3-Medium#5Score (1-10): 8.91
math-word-problem-solving-on-mgsm-0-shotPhi-3.5-MoE#6Accuracy: 58.7
science-question-answering-on-scienceqaPhi-3.5-Vision#6Avg. Accuracy: 91.3
multi-task-language-understanding-on-mmluPhi-3.5-MoE (5-shot)#12Average (%): 78.9
question-answering-on-triviaqaPhi-3-Medium (5-shot)#15EM: 73.9
question-answering-on-boolqPhi-3-Medium (2-shot)#16Accuracy: 86.5
common-sense-reasoning-on-winograndePhi-3-Medium (5-shot)#24Accuracy: 81.5
object-hallucination-evaluation-on-popePhi-3.5-Vision#24F1: 86.1
code-generation-on-mbppPhi-3.5-MoE (3-shot)#30Accuracy: 80.8
visual-question-answering-vqa-on-blinkPhi-3.5-Vision#33Accuracy: 57.0
sentence-completion-on-hellaswagPhi-3.5-MoE (5-shot)#44Accuracy: 83.8
math-word-problem-solving-on-mathPhi-3.5-MoE#54Accuracy: 59.5
arithmetic-reasoning-on-gsm8kPhi-3-Medium (8-shot)#61Accuracy: 91.0
chart-question-answering-on-chartqaPhi-3.5-Vision#611:1 Accuracy: 81.8
code-generation-on-humanevalPhi-3.5-MoE#85Pass@1: 70.7
multi-task-language-understanding-on-mmlu-proPhi-3.5-MoE#106Accuracy: 54.3
multimodal-reasoning-on-mathvistaPhi-3.5-Vision#109Accuracy: 43.9
multimodal-reasoning-on-mmmuPhi-3.5-Vision#117Accuracy: 43.0
mmr-total-on-mrr-benchmarkPhi-3-Vision–Total Column Score: 397