IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning

Benchmark Model Rank Results
visual-question-answering-on-iconqaPatch-TRM#1Sub-tasks (Img.): 82.66Sub-tasks (Txt.): 75.19
visual-question-answering-on-iconqaViLT#2Sub-tasks (Img.): 79.67Sub-tasks (Txt.): 72.69
visual-question-answering-on-iconqaViT#3Sub-tasks (Img.): 79.15Sub-tasks (Txt.): 72.34
visual-question-answering-on-iconqaUNITER#4Sub-tasks (Img.): 78.71Sub-tasks (Txt.): 72.39
visual-question-answering-on-iconqaDFAF#5Sub-tasks (Img.): 77.72Sub-tasks (Txt.): 72.17
visual-question-answering-on-iconqaMCAN#6Sub-tasks (Img.): 77.36Sub-tasks (Txt.): 71.25
visual-question-answering-on-iconqaViLBERT#7Sub-tasks (Img.): 76.66Sub-tasks (Txt.): 70.47
visual-question-answering-on-iconqaBAN#8Sub-tasks (Img.): 76.33Sub-tasks (Txt.): 70.82
visual-question-answering-on-iconqaTop-Down#9Sub-tasks (Img.): 75.92Sub-tasks (Txt.): 68.51
visual-question-answering-on-iconqaRandom#10Sub-tasks (Img.): 41.70Sub-tasks (Txt.): 36.87Sub-tasks (Blank): 0.29
visual-question-answering-on-iconqaQ-Only#11Sub-tasks (Img.): 41.64Sub-tasks (Txt.): 36.86
visual-question-answering-on-iconqaI-Only#12Sub-tasks (Img.): 41.56Sub-tasks (Txt.): 36.02