mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video

Benchmark Model Rank Results
action-classification-on-kinetics-400mPLUG-2#36Acc@1: 87.1Acc@5: 97.7
action-classification-on-kinetics-600mPLUG-2#12Top-1 Accuracy: 89.8Top-5 Accuracy: 98.3
action-classification-on-kinetics-700mPLUG-2#13Top-1 Accuracy: 80.4Top-5 Accuracy: 94.9
video-captioning-on-msr-vtt-1mPLUG-2#1CIDEr: 80.0METEOR: 34.9ROUGE-L: 70.1BLEU-4: 57.8
video-captioning-on-msvd-1mPLUG-2#4CIDEr: 165.8BLEU-4: 70.5METEOR: 48.4ROUGE-L: 85.3
video-question-answering-on-msrvtt-qamPLUG-2#5Accuracy: 48.0
video-retrieval-on-didemomPLUG-2#12text-to-video R@1: 56.4text-to-video R@5: 79.1
video-retrieval-on-lsmdcmPLUG-2#5text-to-video R@1: 34.4text-to-video R@5: 55.2
video-retrieval-on-msr-vtt-1kamPLUG-2#8text-to-video R@1: 53.1text-to-video R@5: 77.6
visual-grounding-on-refcoco-test-bmPLUG-2#2Accuracy (%): 86.05
visual-grounding-on-refcoco-testamPLUG-2#2Accuracy (%): 92.8
visual-grounding-on-refcoco-valmPLUG-2#2Accuracy (%): 90.33
visual-question-answering-on-msrvtt-qa-1mPLUG-2#2Accuracy: 0.480
visual-question-answering-on-msvd-qa-1mPLUG-2#6Accuracy: 0.581
visual-question-answering-on-vqa-v2-test-dev-1mPLUG-2#6Accuracy: 81.11
zero-shot-video-retrieval-on-didemomPLUG-2#7text-to-video R@1: 45.7text-to-video R@5: 71.1
zero-shot-video-retrieval-on-lsmdcmPLUG-2#5text-to-video R@1: 24.1text-to-video R@5: 43.8
zero-shot-video-retrieval-on-msr-vttmPLUG-2#7text-to-video R@1: 47.1text-to-video R@5: 69.7