VinVL: Revisiting Visual Representations in Vision-Language Models

Benchmark Model Rank Results
image-captioning-on-coco-captionsVinVL#14BLEU-4: 41.0CIDER: 140.9METEOR: 31.1SPICE: 25.2
image-captioning-on-nocaps-entireVinVL (Microsoft Cognitive Services + MSR)#4CIDEr: 92.46B1: 81.59B2: 65.15B3: 45.04B4: 26.15
image-captioning-on-nocaps-in-domainVinVL (Microsoft Cognitive Services + MSR)#7CIDEr: 97.99B1: 83.24B2: 68.04B3: 49.68B4: 30.62
image-captioning-on-nocaps-near-domainVinVL (Microsoft Cognitive Services + MSR)#5CIDEr: 95.16B1: 82.77B2: 66.94B3: 47.02B4: 27.97
image-captioning-on-nocaps-out-of-domainVinVL (Microsoft Cognitive Services + MSR)#5CIDEr: 78.01B1: 75.78B2: 56.1B3: 34.02B4: 15.86
image-captioning-on-nocaps-val-in-domainVinVL#7CIDEr: 103.1SPICE: 14.2Pre-train (#images): 5.7M
image-captioning-on-nocaps-val-near-domainVinVL#7CIDEr: 96.1SPICE: 13.8Pre-train (#images): 5.7M
image-captioning-on-nocaps-val-out-domainVinVL#8CIDEr: 88.3SPICE: 12.1Pretrain (#images): 5.7M
image-captioning-on-nocaps-val-overallVinVL#7CIDEr: 95.5SPICE: 13.5Pretrain (#images): 5.7M
image-text-matching-on-commercialadsdatasetVinVL#2ADD(S) AUC: 88.56
visual-question-answering-on-gqa-test2019Single Model#2Accuracy: 64.65Binary: 82.63Open: 48.77Consistency: 94.35
visual-question-answering-on-vqa-v2-test-stdMSR + MS Cog. Svcs., X10 models#11overall: 77.45yes/no: 92.38number: 62.55other: 67.87
visual-question-answering-on-vqa-v2-test-stdMSR + MS Cog. Svcs.#12overall: 76.63yes/no: 92.04number: 61.5other: 66.68