VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text

Benchmark Model Rank Results
action-classification-on-kinetics-400VATT-Large#71Acc@1: 82.1Acc@5: 95.5
action-classification-on-kinetics-600VATT-Large#31Top-1 Accuracy: 83.6Top-5 Accuracy: 96.6
action-classification-on-moments-in-timeVATT-Large#5Top 1 Accuracy: 41.1Top 5 Accuracy: 67.7
audio-classification-on-audiosetVATT-Base#37Test mAP: 0.394AUC: 0.971d-prime: 2.895
zero-shot-video-retrieval-on-msr-vttVATT-MBS#35text-to-video R@10: 29.7text-to-video Median Rank: 49
zero-shot-video-retrieval-on-youcook2VATT-MBS#6text-to-video R@10: 45.5text-to-video Mean Rank: 13