REBUS: A Robust Evaluation Benchmark of Understanding Symbols

Benchmark Model Rank Results
multimodal-reasoning-on-rebusGPT-4V#1Accuracy: 24.0
multimodal-reasoning-on-rebusGemini Pro#2Accuracy: 13.2
multimodal-reasoning-on-rebusLLaVa-1.5-13B#3Accuracy: 1.8
multimodal-reasoning-on-rebusLLaVa-1.5-7B#4Accuracy: 1.5
multimodal-reasoning-on-rebusBLIP2-FLAN-T5-XXL#5Accuracy: 0.9
multimodal-reasoning-on-rebusCogVLM#6Accuracy: 0.9
multimodal-reasoning-on-rebusQWEN#7Accuracy: 0.9
multimodal-reasoning-on-rebusInstructBLIP#8Accuracy: 0.6