Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts

Benchmark Model Rank Results
chart-question-answering-on-chartqaSMoLA-PaLI-X Generalist Model1:1 Accuracy: 73.8
chart-question-answering-on-chartqaSMoLA-PaLI-X Specialist Model1:1 Accuracy: 74.6
object-counting-on-tallyqa-complexSMoLA-PaLI-X Generalist (0 shot)Accuracy: 70.7
object-counting-on-tallyqa-complexSMoLA-PaLI-X SpecialistAccuracy: 77.1
object-counting-on-tallyqa-simpleSMoLA-PaLI-X Generalist (0 shot)Accuracy: 83.3
object-counting-on-tallyqa-simpleSMoLA-PaLI-X SpecialistAccuracy: 86.3
visual-question-answering-on-a-okvqaSMoLA-PaLI-X Specialist ModelMC Accuracy: 83.75DA VQA Score: 70.55
visual-question-answering-on-docvqa-testSMoLA-PaLI-X GeneralistANLS: 0.906
visual-question-answering-on-docvqa-testSMoLA-PaLI-X SpecialistANLS: 0.908
visual-question-answering-vqa-on-ai2dSMoLA-PaLI-X Generalist ModelEM: 81.4
visual-question-answering-vqa-on-ai2dSMoLA-PaLI-X Specialist ModelEM: 82.5
visual-question-answering-vqa-on-infographicvqaSMoLA-PaLI-X GeneralistANLS: 65.6
visual-question-answering-vqa-on-infographicvqaSMoLA-PaLI-X SpecialistANLS: 66.2