VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset

Benchmark Model Rank Results
audio-captioning-on-audiocapsVAST#11CIDEr: 0.781BLEU-4: 0.295METEOR: 0.247ROUGE-L: 0.509
audio-captioning-on-clothoVAST#6CIDEr: 0.519BLEU-4: 19METEOR: 19.3ROUGE-L: 40.8
audio-visual-question-answering-on-music-avqaVAST#1Acc: 80.7
cross-modal-retrieval-on-coco-2014VAST#1Text-to-image R@1: 68.0Text-to-image R@5: 87.7
cross-modal-retrieval-on-flickr30kVAST#22Text-to-image R@1: 91.0Text-to-image R@5: 98.5
image-captioning-on-coco-captionsVAST#33CIDER: 149.0SPICE: 27.0
text-to-audio-retrieval-on-audiocapsVAST#2R@1: 52.0R@5: 76.8R@10: 82.9
text-to-audio-retrieval-on-clothoVAST#3R@1: 26.9R@5: 53.2R@10: 66.1
video-captioning-on-msr-vtt-1VAST#2CIDEr: 78.0BLEU-4: 56.7
video-captioning-on-vatex-1VAST#2BLEU-4: 45.0CIDEr: 99.5
video-captioning-on-youcook2VAST#1BLEU-4: 18.2CIDEr: 1.99
video-question-answering-on-activitynet-qaVAST#1Accuracy: 50.4
video-question-answering-on-msrvtt-qaVAST#1Accuracy: 50.1
video-retrieval-on-activitynetVAST#2text-to-video R@1: 70.5text-to-video R@5: 90.9
video-retrieval-on-didemoVAST#3text-to-video R@1: 72.0text-to-video R@5: 89.0
video-retrieval-on-msr-vttVAST#2text-to-video R@1: 63.9text-to-video R@5: 84.3
video-retrieval-on-vatexVAST#2text-to-video R@1: 83.0text-to-video R@5: 98.2
video-retrieval-on-youcook2VAST#1text-to-video R@1: 50.4text-to-video R@5: 74.3
visual-question-answering-on-msvd-qa-1VAST#4Accuracy: 0.60
zero-shot-cross-modal-retrieval-on-flickr30kVAST#21Text-to-image R@1: 90.4
zero-shot-video-retrieval-on-didemoVAST#3text-to-video R@1: 55.5text-to-video R@5: 74.3
zero-shot-video-retrieval-on-msr-vttVAST#6text-to-video R@1: 49.3text-to-video R@5: 68.3