WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Models

Open paper
Benchmark Model Rank Results
visual-reasoning-on-winogavilHumans#1Jaccard Index: 90
visual-reasoning-on-winogavilViLT (Zero-Shot)#2Jaccard Index: 52
visual-reasoning-on-winogavilX-VLM (Zero-Shot)#3Jaccard Index: 46
visual-reasoning-on-winogavilCLIP-ViT-B/32 (Zero-Shot)#4Jaccard Index: 41
visual-reasoning-on-winogavilCLIP-ViT-L/14 (Zero-Shot)#5Jaccard Index: 40
visual-reasoning-on-winogavilCLIP-RN50x64/14 (Zero-Shot)#6Jaccard Index: 38
visual-reasoning-on-winogavilCLIP-RN50 (Zero-Shot)#7Jaccard Index: 35
visual-reasoning-on-winogavilCLIP-ViL (Zero-Shot)#8Jaccard Index: 15

Rank counts only results with a code link.