OmniVec: Learning robust representations with cross modal sharing

Benchmark Model Rank Results
3d-point-cloud-classification-on-modelnet40-cOmniVecError Rate: 0.156
3d-point-cloud-classification-on-scanobjectnnOmniVecOverall Accuracy: 96.1
action-classification-on-kinetics-400OmniVecAcc@1: 91.1
action-classification-on-moments-in-time-2OmniVecTop 1 Accuracy: 49.8
action-recognition-in-videos-on-ucf101OmniVec3-fold Accuracy: 99.6
audio-classification-on-audiosetOmniVecTest mAP: 0.548
audio-classification-on-esc-50OmniVecTop-1 Accuracy: 98.4PRE-TRAINING DATASET: Multiple
audio-tagging-on-audiosetOmniVec (audio+visual)mean average precision: 0.552
audio-tagging-on-audiosetOmniVec (audio-only)mean average precision: 0.548
fine-grained-image-classification-on-oxford-1OmniVecAccuracy: 99.2
image-classification-on-inaturalist-2018OmniVecTop-1 Accuracy: 93.8
image-classification-on-places365OmniVec(ViT)Top 1 Accuracy: 63.5
semantic-segmentation-on-nyu-depth-v2OmniVecMean IoU: 60.8
semantic-segmentation-on-s3dis-area5OmniVecmIoU: 75.9
text-summarization-on-dialogsumOmniVecRouge1: 46.91Rouge2: 21.22RougeL: 40.19BertScore: 71.91
video-retrieval-on-msr-vtt-1kaOmniVectext-to-video R@10: 89.4
video-retrieval-on-msr-vtt-1kaOmniVec (pretrained)text-to-video R@10: 78.6
video-retrieval-on-youcook2OmniVectext-to-video R@10: 70.8
video-retrieval-on-youcook2OmniVec (pretrained)text-to-video R@10: 64.2