IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Models

Benchmark Model Rank Results
object-localization-on-illusionvqaGPT4-Vision 4-shot+CoT#1Accuracy: 49.7
object-localization-on-illusionvqaGPT4-Vision 4-shot#2Accuracy: 46
object-localization-on-illusionvqaGemini-Pro#3Accuracy: 43.5
object-localization-on-illusionvqaGemini-Pro 4-shot#4Accuracy: 41.8
object-localization-on-illusionvqaGPT4-Vision#5Accuracy: 40
object-localization-on-illusionvqaGemini-Pro 4-shot+CoT#6Accuracy: 33.9
object-localization-on-illusionvqaCogVLM#7Accuracy: 28
object-localization-on-illusionvqaLLaVA-1.5-13B#8Accuracy: 24.8
object-localization-on-illusionvqaInstructBLIP-13B#9Accuracy: 24.3
visual-question-answering-vqa-on-illusionvqaGPT4-Vision 4-shot#1Accuracy: 62.99
visual-question-answering-vqa-on-illusionvqaGPT4-Vision#2Accuracy: 58.85
visual-question-answering-vqa-on-illusionvqaGemini-Pro 4-shot#3Accuracy: 52.87
visual-question-answering-vqa-on-illusionvqaGemini-Pro#4Accuracy: 51.26
visual-question-answering-vqa-on-illusionvqaLLaVA-1.5-13B#5Accuracy: 40
visual-question-answering-vqa-on-illusionvqaCogVLM#6Accuracy: 38.16
visual-question-answering-vqa-on-illusionvqaInstructBLIP-13B#7Accuracy: 34.25