The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation

Benchmark Model Rank Results
math-word-problem-solving-on-mgsm-0-shotLlama 4 Maverick#1Accuracy: 92.3
math-word-problem-solving-on-mgsm-0-shotLlama 4 Scout#2Accuracy: 90.6
chart-question-answering-on-chartqaLlama 4 Maverick#91:1 Accuracy: 90.0
chart-question-answering-on-chartqaLlama 4 Scout#161:1 Accuracy: 88.8
code-generation-on-livecodebench-v5Llama 4 Maverick#14Pass@1: 43.4
code-generation-on-livecodebench-v5Llama 4 Scout#17Pass@1: 32.8
visual-question-answering-on-docvqa-testLlama 4 Scout#16ANLS: 0.944
visual-question-answering-on-docvqa-testLlama 4 Maverick#17ANLS: 0.944
multimodal-reasoning-on-mmmuLlama 4 Maverick#19Accuracy: 73.4
multimodal-reasoning-on-mmmuLlama 4 Scout#37Accuracy: 69.4
multimodal-reasoning-on-mmmu-proLlama 4 Maverick#30Accuracy: 59.6
multimodal-reasoning-on-mmmu-proLlama 4 Scout#40Accuracy: 52.2
multi-task-language-understanding-on-mmlu-proLlama 4 Maverick#42Accuracy: 80.5
multi-task-language-understanding-on-mmlu-proLlama 4 Scout#58Accuracy: 74.3
multimodal-reasoning-on-mathvistaLlama 4 Maverick#53Accuracy: 73.7
multimodal-reasoning-on-mathvistaLlama 4 Scout#62Accuracy: 70.7
multiple-choice-question-answering-mcqa-on-gpqa-diamondLlama 4 Maverick#85Accuracy: 69.8
multiple-choice-question-answering-mcqa-on-gpqa-diamondLlama 4 Scout#102Accuracy: 57.2