StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding

Benchmark Model Rank Results
video-question-answering-on-streamingbenchClaude 3.5 Sonnet–Real-Time Visual Understanding: 72.44Overall: 57.68
video-question-answering-on-streamingbenchFlash-VStream-7B–Real-Time Visual Understanding: 23.23Overall: 24.04
video-question-answering-on-streamingbenchGPT-4o–Real-Time Visual Understanding: 73.28Overall: 60.15
video-question-answering-on-streamingbenchGemini 1.5 Pro–Real-Time Visual Understanding: 75.69Overall: 67.07
video-question-answering-on-streamingbenchInternVL2-8B–Real-Time Visual Understanding: 63.72Overall: 51.40
video-question-answering-on-streamingbenchKangaroo-7B–Real-Time Visual Understanding: 64.60Overall: 51.10
video-question-answering-on-streamingbenchLLaVA-NeXT-Video-32B–Real-Time Visual Understanding: 66.96Overall: 52.77
video-question-answering-on-streamingbenchLLaVA-OneVision-7B–Real-Time Visual Understanding: 71.12Overall: 56.36
video-question-answering-on-streamingbenchLongVA-7B–Real-Time Visual Understanding: 59.96Overall: 48.66
video-question-answering-on-streamingbenchMiniCPM-V 2.6–Real-Time Visual Understanding: 67.44Overall: 53.85
video-question-answering-on-streamingbenchQwen2-VL-7B–Real-Time Visual Understanding: 69.04Overall: 54.14
video-question-answering-on-streamingbenchVILA-1.5-8B–Real-Time Visual Understanding: 52.32Overall: 43.20
video-question-answering-on-streamingbenchVideo-CCAM-14B–Real-Time Visual Understanding: 53.96Overall: 42.53
video-question-answering-on-streamingbenchVideo-LLaMA2-7B–Real-Time Visual Understanding: 49.52Overall: 40.40
video-question-answering-on-streamingbenchVideoLLM-online-8B–Real-Time Visual Understanding: 35.99Overall: 32.48