CoCa: Contrastive Captioners are Image-Text Foundation Models

Benchmark Model Rank Results
action-classification-on-kinetics-400CoCa (finetuned)#16Acc@1: 88.9
action-classification-on-kinetics-400CoCa (frozen)#23Acc@1: 88.0
action-classification-on-kinetics-600CoCa (finetuned)#13Top-1 Accuracy: 89.4
action-classification-on-kinetics-600CoCa (frozen)#16Top-1 Accuracy: 88.5
action-classification-on-kinetics-700CoCa (finetuned)#8Top-1 Accuracy: 82.7
action-classification-on-kinetics-700CoCa (frozen)#10Top-1 Accuracy: 81.1
action-classification-on-moments-in-time-2CoCa (finetuned)#1Top 1 Accuracy: 49.0
action-classification-on-moments-in-time-2CoCa (frozen)#2Top 1 Accuracy: 47.4
image-captioning-on-coco-captionsCoCa#15BLEU-4: 40.9CIDER: 143.6METEOR: 33.9SPICE: 24.7
image-classification-on-imagenetCoCa (finetuned)#1Top 1 Accuracy: 91.0%Number of params: 2100M
image-classification-on-objectnetCoCa#1Top-1 Accuracy: 82.7
video-retrieval-on-msr-vttCoCa (zero-shot)#17text-to-video R@1: 30.0text-to-video R@5: 52.4
visual-entailment-on-snli-ve-testCoCa#3Accuracy: 87.1
visual-entailment-on-snli-ve-valCoCa#3Accuracy: 87.0
visual-question-answering-on-vqa-v2-test-dev-1CoCa#1Accuracy: 82.3
visual-reasoning-on-nlvr2-devCoCa#5Accuracy: 86.1
visual-reasoning-on-nlvr2-testCoCa#4Accuracy: 87.0
zero-shot-cross-modal-retrieval-on-coco-2014CoCa#9Image-to-text R@1: 66.3Image-to-text R@5: 86.2
zero-shot-cross-modal-retrieval-on-flickr30kCoCa#5Image-to-text R@1: 92.5Image-to-text R@5: 99.5
zero-shot-transfer-image-classification-on-1CoCa#2Accuracy (Private): 86.3
zero-shot-transfer-image-classification-on-3CoCa#2Accuracy (Private): 80.7
zero-shot-transfer-image-classification-on-4CoCa#1Accuracy: 96.5
zero-shot-transfer-image-classification-on-5CoCa#1Accuracy (Private): 90.2
zero-shot-transfer-image-classification-on-6CoCa#3Accuracy (Private): 82.7
zero-shot-transfer-image-classification-on-8CoCa#1Accuracy (Private): 77.6