Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders

Benchmark Model Rank Results
visual-question-answering-on-v-benchPenguin-VL 8B#3Accuracy: 90.1
visual-question-answering-on-v-benchPenguin-VL 2B#17Accuracy: 74.9
visual-question-answering-on-docvqa-testPenguin-VL 8B#6ANLS: 0.961
visual-question-answering-on-docvqa-testPenguin-VL 2B#20ANLS: 0.933
zero-shot-video-question-answer-on-egoschema-fullsetPenguin-VL 8B#7Accuracy: 70.2
zero-shot-video-question-answer-on-egoschema-fullsetPenguin-VL 2B#35Accuracy: 55.7
optical-character-recognition-ocr-on-ocrbenchPenguin-VL 8B#8Score: 896
visual-question-answering-vqa-on-blinkPenguin-VL 8B#8Accuracy: 69.1
visual-question-answering-vqa-on-blinkPenguin-VL 2B#40Accuracy: 53.8
chart-question-answering-on-charxiv-dqPenguin-VL 8B#11Accuracy: 83.0
chart-question-answering-on-charxiv-dqPenguin-VL 2B#16Accuracy: 62.3
mathematical-reasoning-on-mathversePenguin-VL 8B#11Accuracy: 62.1
mathematical-reasoning-on-mathversePenguin-VL 2B#17Accuracy: 52.1
chart-question-answering-on-chartqaPenguin-VL 8B#121:1 Accuracy: 89.6
chart-question-answering-on-chartqaPenguin-VL 2B#741:1 Accuracy: 76.9
zero-shot-video-question-answer-on-video-mme-w-o-subsPenguin-VL (8B)#23Accuracy (%): 71.4
chart-question-answering-on-charxiv-rqPenguin-VL 8B#24Accuracy: 46.4
chart-question-answering-on-charxiv-rqPenguin-VL 2B#29Accuracy: 26.8
video-question-answering-on-mmvuPenguin-VL 8B#25Accuracy: 58.7
video-question-answering-on-mmvuPenguin-VL 2B#32Accuracy: 41.7
multimodal-reasoning-on-mmmu-proPenguin-VL 8B#35Accuracy: 55.9
multimodal-reasoning-on-mmmu-proPenguin-VL 2B#56Accuracy: 36.5
multimodal-reasoning-on-mathvistaPenguin-VL 8B#38Accuracy: 77.2
multimodal-reasoning-on-mathvistaPenguin-VL 2B#92Accuracy: 61.3
video-question-answering-on-mvbenchPenguin-VL 8B#39Avg.: 68.7
video-question-answering-on-mvbenchPenguin-VL 2B#63Avg.: 61.7
visual-question-answering-vqa-on-realworldqaPenguin-VL 8B#43Accuracy: 71.5
visual-question-answering-vqa-on-realworldqaPenguin-VL 2B#73Accuracy: 63.9
moment-retrieval-on-charades-staPenguin-VL 8B#49R@1 IoU=0.5: 56.0
moment-retrieval-on-charades-staPenguin-VL 2B#53R@1 IoU=0.5: 54.5