CoCa: Contrastive Captioners are Image-Text Foundation Models

Benchmark Model Rank Results
action-classification-on-moments-in-time-2CoCa (finetuned)#1Top 1 Accuracy: 49.0
action-classification-on-moments-in-time-2CoCa (frozen)#2Top 1 Accuracy: 47.4
image-classification-on-imagenetCoCa (finetuned)#1Top 1 Accuracy: 91.0%Number of params: 2100M
visual-question-answering-on-vqa-v2-test-dev-1CoCa#1Accuracy: 82.3
zero-shot-transfer-image-classification-on-imagenet-rCoCa#1Accuracy: 96.5
zero-shot-transfer-image-classification-on-imagenet-sketchCoCa#1Accuracy (Private): 77.6
zero-shot-video-retrieval-on-msr-vtt-video-to-textCoCa#1R@1: 49.9
zero-shot-transfer-image-classification-on-imagenetCoCa#2Accuracy (Private): 86.3
zero-shot-transfer-image-classification-on-imagenet-v2CoCa#2Accuracy (Private): 80.7
visual-entailment-on-snli-ve-testCoCa#3Accuracy: 87.1
visual-entailment-on-snli-ve-valCoCa#3Accuracy: 87.0
zero-shot-transfer-image-classification-on-imagenet-aCoCa#3Accuracy (Private): 90.2
zero-shot-transfer-image-classification-on-objectnetCoCa#3Accuracy (Private): 82.7
image-captioning-on-nocaps-val-overallCoCa#4CIDEr: 122.4
image-classification-on-objectnetCoCa#5Top-1 Accuracy: 82.7
visual-reasoning-on-nlvr2-devCoCa#5Accuracy: 86.1
visual-reasoning-on-nlvr2-testCoCa#5Accuracy: 87.0
visual-question-answering-on-vqa-v2-test-devCoCa#6Accuracy: 82.3
zero-shot-cross-modal-retrieval-on-flickr30kCoCa#6Image-to-text R@1: 92.5Image-to-text R@5: 99.5…
action-classification-on-kinetics-700CoCa (finetuned)#8Top-1 Accuracy: 82.7
action-classification-on-kinetics-700CoCa (frozen)#10Top-1 Accuracy: 81.1
action-classification-on-kinetics-600CoCa (finetuned)#13Top-1 Accuracy: 89.4
action-classification-on-kinetics-600CoCa (frozen)#16Top-1 Accuracy: 88.5
image-classification-on-imagenet-v2CoCa#13Top 1 Accuracy: 80.7
image-captioning-on-coco-captionsCoCa#15BLEU-4: 40.9CIDER: 143.6METEOR: 33.9SPICE: 24.7
action-classification-on-kinetics-400CoCa (finetuned)#16Acc@1: 88.9
action-classification-on-kinetics-400CoCa (frozen)#24Acc@1: 88.0
zero-shot-cross-modal-retrieval-on-coco-2014CoCa#16Image-to-text R@1: 66.3Image-to-text R@5: 86.2…
video-retrieval-on-msr-vttCoCa (zero-shot)#19text-to-video R@1: 30.0text-to-video R@5: 52.4…