VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners

Benchmark Model Rank Results
video-captioning-on-activitynet-captionsVideoCoCa–BLEU4: 14.7CIDEr: 39.3ROUGE-L: 35.0
video-captioning-on-msr-vttVideoCoCa–CIDEr: 73.2ROUGE-L: 68.0BLEU-4: 53.8
video-captioning-on-vatexVideoCoCa–BLEU-4: 39.7CIDEr: 77.8ROUGE-L: 54.5
video-captioning-on-youcook2VideoCoCa–BLEU-4: 14.2CIDEr: 1.28ROUGE-L: 37.7
video-question-answering-on-activitynet-qaVideoCoCa–Accuracy: 56.1
video-question-answering-on-ivqaVideoCoCa–Accuracy: 39.0
video-retrieval-on-msr-vttVideoCoCa (zero-shot)–text-to-video R@1: 34.3text-to-video R@5: 57.8…
video-retrieval-on-youcook2VideoCoCa (zero-shot)–text-to-video R@1: 21.7text-to-video R@5: 43.9…
visual-question-answering-on-msrvtt-qaVideoCoCa–Accuracy: 0.463
visual-question-answering-on-msvd-qaVideoCoCa–Accuracy: 0.569
zero-shot-action-recognition-on-charadesVideoCoCa–mAP: 25.8
zero-shot-action-recognition-on-hmdb51VideoCoCa–Top-1 Accuracy: 58.7Top-5 Accuracy: 84.5
zero-shot-action-recognition-on-kineticsVideoCoCa–Top-1 Accuracy: 70.1Top-5 Accuracy: 88.9
zero-shot-action-recognition-on-ucf101VideoCoCa–Top-1 Accuracy: 86.6Top-5 accuracy: 98.4
zero-shot-video-retrieval-on-activitynetVideoCoCa–text-to-video R@1: 34.5text-to-video R@5: 63.2…
zero-shot-video-retrieval-on-msr-vtt-fullVideoCoCa–text-to-video R@1: 34.3text-to-video R@5: 57.8…
zero-shot-video-retrieval-on-vatexVideoCoCa–text-to-video R@1: 53.2text-to-video R@5: 83.3…
zero-shot-video-retrieval-on-youcook2VideoCOca–text-to-video R@1: 20.3text-to-video R@5: 43.0…