Aria: An Open Multimodal Native Mixture-of-Experts Model

Benchmark Model Rank Results
instruction-following-on-mtbenchAria#8Score (1-10): 8.53
video-question-answering-on-tvbenchAria#8Average Accuracy: 51.0
visual-question-answering-vqa-on-textvqa-valAria#13Accuracy: 81.1
zero-shot-video-question-answer-on-egoschema-fullsetAria#13Accuracy: 66.8
multi-task-language-understanding-on-mmluAria (5-shot)#22Average (%): 73.3
video-question-answering-on-video-mme-with-subtitlesAria (w/ subtitles)#22Accuracy: 72.1
vision-language-understanding-on-mmbench-v1-1Aria#22Accuracy: 80.3
visual-question-answering-on-docvqa-testAria#28ANLS: 0.926
chart-question-answering-on-chartqaAria#321:1 Accuracy: 86.4
multimodal-reasoning-on-mmmuAria#74Accuracy: 54.9
multimodal-reasoning-on-mathvistaAria#77Accuracy: 66.1
code-generation-on-humanevalAria#80Pass@1: 73.2
math-word-problem-solving-on-mathAria#82Accuracy: 50.8