Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images

Benchmark Model Rank Results
explanation-generation-on-whoopsBLIP2 FlanT5-XL (Fine-tuned)–Human (%): 15
explanation-generation-on-whoopsBLIP2 FlanT5-XXL (Fine-tuned)–Human (%): 27
explanation-generation-on-whoopsBLIP2 FlanT5-XXL (Zero-shot)–Human (%): 0
explanation-generation-on-whoopsGround-truth Caption -> GPT3 (Oracle)–Human (%): 68
explanation-generation-on-whoopsPredicted Caption -> GPT3–Human (%): 33