Measuring Progress in Fine-grained Vision-and-Language Understanding

Benchmark Model Rank Results
visual-reasoning-on-winogroundX-VLM 16M#11Text Score: 46.7Image Score: 24.5Group Score: 21.2
visual-reasoning-on-winogroundX-VLM 4M#18Text Score: 44.0Image Score: 26.7Group Score: 21.5
visual-reasoning-on-winogroundBLIP 14M#31Text Score: 36.5Image Score: 18.5Group Score: 14.5
visual-reasoning-on-winogroundBLIP 129M#35Text Score: 35.5Image Score: 15.0Group Score: 11.7
visual-reasoning-on-winogroundBLIP 129M (CapFilt/L)#37Text Score: 34.7Image Score: 15.2Group Score: 12.2
visual-reasoning-on-winogroundBLIP-ViT/L 129M#38Text Score: 34.7Image Score: 14.5Group Score: 12.2
visual-reasoning-on-winogroundPEVL 14M#40Text Score: 33.2Image Score: 15.7Group Score: 12.2
visual-reasoning-on-winogroundALBEF 14M#41Text Score: 32.5Image Score: 16.2Group Score: 12.7
visual-reasoning-on-winogroundALBEF 4M#50Text Score: 29.2Image Score: 15.5Group Score: 11.0