BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

Open paper
Benchmark Model Rank Results
generative-visual-question-answering-on-pmcBLIP-2#2BLEU-1: 7.6
image-captioning-on-coco-captionsBLIP-2 ViT-G OPT 2.7B (zero-shot)#4BLEU-4: 43.7CIDER: 145.8
image-captioning-on-coco-captionsBLIP-2 ViT-G OPT 6.7B (zero-shot)#5BLEU-4: 43.5CIDER: 145.2
image-captioning-on-coco-captionsBLIP-2 ViT-G FlanT5 XL (zero-shot)#8BLEU-4: 42.4CIDER: 144.5
image-captioning-on-nocaps-val-in-domainBLIP-2 ViT-G FlanT5 XL (zero-shot)#1CIDEr: 123.7SPICE: 16.3Pre-train (#images): 1.1B
image-captioning-on-nocaps-val-in-domainBLIP-2 ViT-G OPT 6.7B (zero-shot)#2CIDEr: 123.7SPICE: 15.8Pre-train (#images): 1.1B
image-captioning-on-nocaps-val-in-domainBLIP-2 ViT-G OPT 2.7B (zero-shot)#3CIDEr: 123SPICE: 15.8Pre-train (#images): 1.1B
image-captioning-on-nocaps-val-near-domainBLIP-2 ViT-G FlanT5 XL (zero-shot)#1CIDEr: 120.2SPICE: 15.9Pre-train (#images): 1.1B
image-captioning-on-nocaps-val-near-domainBLIP-2 ViT-G OPT 6.7B (zero-shot)#2CIDEr: 119.2SPICE: 15.3Pre-train (#images): 1.1B
image-captioning-on-nocaps-val-near-domainBLIP-2 ViT-G OPT 2.7B (zero-shot)#3CIDEr: 117.8SPICE: 15.4Pre-train (#images): 1.1B
image-captioning-on-nocaps-val-out-domainBLIP-2 ViT-G FlanT5 XL (zero-shot)#1CIDEr: 124.8SPICE: 15.1Pretrain (#images): 1.1B
image-captioning-on-nocaps-val-out-domainBLIP-2 ViT-G OPT 6.7B (zero-shot)#2CIDEr: 124.4SPICE: 14.8Pretrain (#images): 1.1B
image-captioning-on-nocaps-val-out-domainBLIP-2 ViT-G OPT 2.7B (zero-shot)#3CIDEr: 123.4SPICE: 15.1Pretrain (#images): 1.1B
image-captioning-on-nocaps-val-overallBLIP-2 ViT-G FlanT5 XL (zero-shot)#1CIDEr: 121.6SPICE: 15.8Pretrain (#images): 1.1B
image-captioning-on-nocaps-val-overallBLIP-2 ViT-G OPT 6.7B (zero-shot)#2CIDEr: 121.0SPICE: 15.3Pretrain (#images): 1.1B
image-captioning-on-nocaps-val-overallBLIP-2 ViT-G OPT 2.7B (zero-shot)#3CIDEr: 119.7SPICE: 15.4Pretrain (#images): 1.1B
image-retrieval-on-cocoBLIP-2 ViT-G (fine-tuned)#1recall@1: 68.3recall@5: 87.7Recall@10: 92.6
image-retrieval-on-cocoBLIP-2 ViT-L (fine-tuned)#3recall@1: 66.3recall@5: 86.5Recall@10: 91.8
image-retrieval-on-flickr30kBLIP-2 ViT-G (zero-shot, 1K test set)#1Recall@10: 98.9Recall@5: 98.1Recall@1: 89.7
image-retrieval-on-flickr30kBLIP-2 ViT-L (zero-shot, 1K test set)#2Recall@10: 98.9Recall@5: 97.6Recall@1: 88.6
image-to-text-retrieval-on-cocoBLIP-2 (ViT-G, fine-tuned)#1Recall@1: 85.4Recall@5: 97.0Recall@10: 98.5
image-to-text-retrieval-on-cocoBLIP-2 (ViT-L, fine-tuned)#3Recall@1: 83.5Recall@5: 96.0Recall@10: 98.0
image-to-text-retrieval-on-flickr30kBLIP-2 ViT-G (zero-shot, 1K test set)#2Recall@1: 97.6Recall@5: 100Recall@10: 100
image-to-text-retrieval-on-flickr30kBLIP-2 ViT-L (zero-shot, 1K test set)#5Recall@1: 96.9Recall@5: 100Recall@10: 100
open-vocabulary-attribute-detection-on-ovad-1BLIP 2 (pretrained)#2mean average precision: 25.5
visual-instruction-following-on-llava-benchBLIP-2#8avg score: 38.1
visual-question-answering-on-gqa-test-devBLIP-2 ViT-G FlanT5 XXL (zero-shot)#8Accuracy: 44.7
visual-question-answering-on-gqa-test-devBLIP-2 ViT-L FlanT5 XL (zero-shot)#9Accuracy: 44.4
visual-question-answering-on-gqa-test-devBLIP-2 ViT-G FlanT5 XL (zero-shot)#10Accuracy: 44.2
visual-question-answering-on-gqa-test-devBLIP-2 ViT-G OPT 6.7B (zero-shot)#12Accuracy: 36.4
visual-question-answering-on-gqa-test-devBLIP-2 ViT-G OPT 2.7B (zero-shot)#13Accuracy: 34.6
visual-question-answering-on-gqa-test-devBLIP-2 ViT-L OPT 2.7B (zero-shot)#14Accuracy: 33.9
visual-question-answering-on-mm-vetBLIP-2-12B#163GPT-4 score: 22.4±0.2Params: 12B
visual-question-answering-on-ok-vqaBLIP-2 ViT-G FlanT5 XXL (zero-shot)#19Accuracy: 45.9
visual-question-answering-on-ok-vqaBLIP-2 ViT-G FlanT5 XL (zero-shot)#24Accuracy: 40.7
visual-question-answering-on-ok-vqaBLIP-2 ViT-L FlanT5 XL (zero-shot)#25Accuracy: 39.4
visual-question-answering-on-ok-vqaBLIP-2 ViT-G OPT 6.7B (zero-shot)#26Accuracy: 36.4
visual-question-answering-on-ok-vqaBLIP-2 ViT-G OPT 2.7B (zero-shot)#28Accuracy: 31.7
visual-question-answering-on-ok-vqaBLIP-2 ViT-L OPT 2.7B (zero-shot)#29Accuracy: 30.2
visual-question-answering-on-vqa-v2-test-devBLIP-2 ViT-G FlanT5 XXL (zero-shot)#39Accuracy: 65
visual-question-answering-on-vqa-v2-test-devBLIP-2 ViT-G FlanT5 XL (zero-shot)#44Accuracy: 63
visual-question-answering-on-vqa-v2-test-devBLIP-2 ViT-L FlanT5 XL (zero-shot)#45Accuracy: 62.3
visual-question-answering-on-vqa-v2-test-devBLIP-2 ViT-G OPT 6.7B (zero-shot)#47Accuracy: 52.6
visual-question-answering-on-vqa-v2-test-devBLIP-2 ViT-G OPT 2.7B (zero-shot)#48Accuracy: 52.3
visual-question-answering-on-vqa-v2-test-devBLIP-2 ViT-L OPT 2.7B (zero-shot)#50Accuracy: 49.7
visual-question-answering-on-vqa-v2-test-dev-1BLIP-2 ViT-G OPT 6.7B (fine-tuned)#2Accuracy: 82.30
visual-question-answering-on-vqa-v2-test-dev-1BLIP-2 ViT-G OPT 2.7B (fine-tuned)#4Accuracy: 81.74
visual-question-answering-on-vqa-v2-test-dev-1BLIP-2 ViT-G FlanT5 XL (fine-tuned)#5Accuracy: 81.66
visual-question-answering-on-vqa-v2-valBLIP-2 ViT-G FlanT5 XXL (zero-shot)#1Accuracy: 65.2
visual-question-answering-on-vqa-v2-valBLIP-2 ViT-G FlanT5 XL (zero-shot)#3Accuracy: 63.1
visual-question-answering-on-vqa-v2-valBLIP-2 ViT-L FlanT5 XL (zero-shot)#4Accuracy: 62.6
visual-question-answering-on-vqa-v2-valBLIP-2 ViT-G OPT 6.7B (zero-shot)#5Accuracy: 54.3
visual-question-answering-on-vqa-v2-valBLIP-2 ViT-G OPT 2.7B (zero-shot)#6Accuracy: 53.5
visual-question-answering-on-vqa-v2-valBLIP-2 ViT-L OPT 2.7B (zero-shot)#7Accuracy: 50.1
visual-question-answering-on-vqa-v2-val-1BLIP-2 ViT-G OPT 6.7B (fine-tuned)#1Accuracy: 82.19
visual-question-answering-on-vqa-v2-val-1BLIP-2 ViT-G OPT 2.7B (fine-tuned)#2Accuracy: 81.59
visual-question-answering-on-vqa-v2-val-1BLIP-2 ViT-G FlanT5 XL (fine-tuned)#3Accuracy: 81.55
visual-question-answering-vqa-on-core-mmBLIP-2-OPT2.7B#12Overall score: 19.31Deductive: 2.76Abductive: 18.96Analogical: 7.5…
visual-question-answering-vqa-on-infoseekBLIP2#6Accuracy: 14.6
visual-question-answering-vqa-on-pmc-vqaBLIP-2#4Accuracy: 24.3

Rank counts only results with a code link.