What You See is What You Read? Improving Text-Image Alignment Evaluation

Benchmark Model Rank Results
visual-reasoning-on-winogroundVQ2#9Text Score: 47Image Score: 42.2Group Score: 30.5
visual-reasoning-on-winogroundPaLI (ft SNLI-VE + Synthetic Data)#12Text Score: 46.5Image Score: 38Group Score: 28.75
visual-reasoning-on-winogroundPaLI (ft SNLI-VE)#16Text Score: 45.00Image Score: 41.50Group Score: 28.70
visual-reasoning-on-winogroundBLIP2 (ft COCO)#19Text Score: 44.00Image Score: 26.00Group Score: 23.50
visual-reasoning-on-winogroundCOCA ViT-L14 (f.t on COCO)#52Text Score: 28.25Image Score: 11.50Group Score: 8.25
visual-reasoning-on-winogroundOFA large (ft SNLI-VE)#55Text Score: 27.70Image Score: 14.30Group Score: 9.00
visual-reasoning-on-winogroundCLIP RN50x64#56Text Score: 26.50Image Score: 13.75Group Score: 10.25
visual-reasoning-on-winogroundTIFA#71Text Score: 19.00Image Score: 12.50Group Score: 11.30