Improved Baselines with Visual Instruction Tuning

Benchmark Model Rank Results
factual-inconsistency-detection-in-chart-1LLaVA-1.5-13B#4Kendall's Tau-c: 0.057
factual-inconsistency-detection-in-chart-2LLaVA-1.5-13B#2Kendall's Tau-c: 0.214
factual-inconsistency-detection-in-chart-3LLaVA-1.5-13B#3Kendall's Tau-c: 0.002
image-classification-on-coloninst-v1-seenLLaVA-v1.5 (w/ LoRA, w/ extra data)#6Accuray: 93.33
image-classification-on-coloninst-v1-seenLLaVA-v1.5 (w/ LoRA, w/o extra data)#9Accuray: 92.97
image-classification-on-coloninst-v1-unseenLLaVA-v1.5 (w/ LoRA, w/ extra data)#2Accuray: 80.89
image-classification-on-coloninst-v1-unseenLLaVA-v1.5 (w/ LoRA, w/o extra data)#6Accuray: 79.10
referring-expression-generation-on-coloninstLLaVA-v1.5 (w/ LoRA, w/ extra data)#2Accuray: 99.32
referring-expression-generation-on-coloninstLLaVA-v1.5 (w/ LoRA, w/o extra data)#5Accuray: 98.58
referring-expression-generation-on-coloninst-1LLaVA-v1.5 (w/ LoRA, w/ extra data)#9Accuray: 72.88
referring-expression-generation-on-coloninst-1LLaVA-v1.5 (w/ LoRA, w/o extra data)#11Accuray: 70.38
spatial-reasoning-on-6-dof-spatialbenchLLaVA-1.5#3Total: 27.2Position-rel: 30.9Position-abs: 24.5
visual-instruction-following-on-llava-benchLLaVA-v1.5-13B#4avg score: 70.7
visual-instruction-following-on-llava-benchLLaVA-v1.5-7B#5avg score: 63.4
visual-question-answering-on-benchlmmLLaVA-1.5-13B#3GPT-3.5 score: 55.53
visual-question-answering-on-mm-vetLLaVA-1.5-13B#99GPT-4 score: 36.3±0.2Params: 13B
visual-question-answering-on-mm-vetLLaVA-1.5-7B#143GPT-4 score: 31.1±0.2Params: 7B
visual-question-answering-on-mm-vet-v2LLaVA-v1.5-13B#14GPT-4 score: 33.2±0.1Params: 13B
visual-question-answering-on-mm-vet-v2LLaVA-v1.5-7B#15GPT-4 score: 28.3±0.2Params: 7B
visual-question-answering-on-vip-benchLLaVA-1.5-13B (Coordinates)#5GPT-4 score (bbox): 47.1
visual-question-answering-on-vip-benchLLaVA-1.5-13B (Visual Prompt)#8GPT-4 score (bbox): 41.8GPT-4 score (human): 42.9
visual-question-answering-vqa-on-5LLaVA-1.5#3Overall Accuracy: 44.5
visual-question-answering-vqa-on-core-mmLLaVA-1.5#5Overall score: 32.62Deductive: 30.94Abductive: 47.91