Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality

Benchmark Model Rank Results
visual-reasoning-on-winogroundUNITER large#28Text Score: 38.00Image Score: 14.00Group Score: 10.50
visual-reasoning-on-winogroundVinVL#29Text Score: 37.75Image Score: 17.75Group Score: 14.50
visual-reasoning-on-winogroundViLLA large#30Text Score: 37.00Image Score: 13.25Group Score: 11.00
visual-reasoning-on-winogroundViLT (ViT-B/32)#36Text Score: 34.75Image Score: 14.00Group Score: 9.25
visual-reasoning-on-winogroundFLAVA (ITM)#42Text Score: 32.25Image Score: 20.50Group Score: 14.25
visual-reasoning-on-winogroundUNITER base#43Text Score: 32.25Image Score: 13.25Group Score: 10.00
visual-reasoning-on-winogroundCLIP (ViT-B/32)#44Text Score: 30.75Image Score: 10.50Group Score: 8.00
visual-reasoning-on-winogroundViLLA base#47Text Score: 30.00Image Score: 12.00Group Score: 8.00
visual-reasoning-on-winogroundFLAVA (contrastive)#58Text Score: 25.25Image Score: 13.50Group Score: 9.00
visual-reasoning-on-winogroundRandom chance#59Text Score: 25.00Image Score: 25.00Group Score: 16.67
visual-reasoning-on-winogroundViLBERT base#61Text Score: 23.75Image Score: 7.25Group Score: 4.75
visual-reasoning-on-winogroundVSE++ (COCO, ResNet)#63Text Score: 22.75Image Score: 8.00Group Score: 4.00
visual-reasoning-on-winogroundVSRN (Flickr30k)#66Text Score: 20.00Image Score: 5.00Group Score: 3.50
visual-reasoning-on-winogroundVSE++ (Flickr30k, ResNet)#67Text Score: 20.00Image Score: 5.00Group Score: 2.75
visual-reasoning-on-winogroundVSE++ (Flickr30k, VGG)#68Text Score: 19.75Image Score: 6.25Group Score: 4.50
visual-reasoning-on-winogroundUniT (ITM finetuned)#69Text Score: 19.50Image Score: 6.25Group Score: 4.00
visual-reasoning-on-winogroundLXMERT#70Text Score: 19.25Image Score: 7.00Group Score: 4.00
visual-reasoning-on-winogroundVSE++ (COCO, VGG)#72Text Score: 18.75Image Score: 5.50Group Score: 3.50
visual-reasoning-on-winogroundVSRN (COCO)#73Text Score: 17.50Image Score: 7.00Group Score: 3.75
visual-reasoning-on-winogroundVisualBERT base#74Text Score: 15.50Image Score: 2.50Group Score: 1.50