OmniVec: Learning robust representations with cross modal sharing

Benchmark Model Rank Results
3d-point-cloud-classification-on-modelnet40-cOmniVec–Error Rate: 0.156
3d-point-cloud-classification-on-scanobjectnnOmniVec–Overall Accuracy: 96.1
action-classification-on-kinetics-400OmniVec–Acc@1: 91.1
action-classification-on-moments-in-time-2OmniVec–Top 1 Accuracy: 49.8
action-recognition-in-videos-on-ucf101OmniVec–3-fold Accuracy: 99.6
audio-classification-on-audiosetOmniVec–Test mAP: 0.548
audio-classification-on-esc-50OmniVec–Top-1 Accuracy: 98.4PRE-TRAINING DATASET: Multiple…
audio-tagging-on-audiosetOmniVec (audio+visual)–mean average precision: 0.552
audio-tagging-on-audiosetOmniVec (audio-only)–mean average precision: 0.548
fine-grained-image-classification-on-oxford-iiit-petOmniVec–Accuracy: 99.2
image-classification-on-inaturalist-2018OmniVec–Top-1 Accuracy: 93.8
image-classification-on-places365OmniVec(ViT)–Top 1 Accuracy: 63.5
semantic-segmentation-on-nyu-depth-v2OmniVec–Mean IoU: 60.8
semantic-segmentation-on-s3dis-area5OmniVec–mIoU: 75.9
text-summarization-on-dialogsumOmniVec–Rouge1: 46.91Rouge2: 21.22RougeL: 40.19BertScore: 71.91
video-retrieval-on-msr-vtt-1kaOmniVec–text-to-video R@10: 89.4
video-retrieval-on-msr-vtt-1kaOmniVec (pretrained)–text-to-video R@10: 78.6
video-retrieval-on-youcook2OmniVec–text-to-video R@10: 70.8
video-retrieval-on-youcook2OmniVec (pretrained)–text-to-video R@10: 64.2