CREPE: Can Vision-Language Foundation Models Reason Compositionally?

Benchmark Model Rank Results
image-retrieval-on-crepe-vision-languageViT-L-14 (LAION400M)#1Recall@1 (HN-Atom + HN-Comp, SC): 39.44…
image-retrieval-on-crepe-vision-languageViT-B-16+240 (LAION400M)#2Recall@1 (HN-Atom + HN-Comp, SC): 37.32…
image-retrieval-on-crepe-vision-languageViT-B-16 (LAION400M)#3Recall@1 (HN-Atom + HN-Comp, SC): 37.01…
image-retrieval-on-crepe-vision-languageViT-B-32 (LAION400M)#4Recall@1 (HN-Atom + HN-Comp, SC): 34.28…
image-retrieval-on-crepe-vision-languageRN50 (YFCC15M)#5Recall@1 (HN-Atom + HN-Comp, SC): 23.38…
image-retrieval-on-crepe-vision-languageRN50 (CC12M)#6Recall@1 (HN-Atom + HN-Comp, SC): 23.26…
image-retrieval-on-crepe-vision-languageRN101 (YFCC15M)#7Recall@1 (HN-Atom + HN-Comp, SC): 22.74…
image-retrieval-on-crepe-vision-languageRandom#8Recall@1 (HN-Atom + HN-Comp, SC): 9.09…