Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts

Benchmark Model Rank Results
chart-question-answering-on-chartqaSMoLA-PaLI-X Generalist Model–1:1 Accuracy: 73.8
chart-question-answering-on-chartqaSMoLA-PaLI-X Specialist Model–1:1 Accuracy: 74.6
object-counting-on-tallyqa-complexSMoLA-PaLI-X Generalist (0 shot)–Accuracy: 70.7
object-counting-on-tallyqa-complexSMoLA-PaLI-X Specialist–Accuracy: 77.1
object-counting-on-tallyqa-simpleSMoLA-PaLI-X Generalist (0 shot)–Accuracy: 83.3
object-counting-on-tallyqa-simpleSMoLA-PaLI-X Specialist–Accuracy: 86.3
visual-question-answering-on-a-okvqaSMoLA-PaLI-X Specialist Model–MC Accuracy: 83.75DA VQA Score: 70.55
visual-question-answering-on-docvqa-testSMoLA-PaLI-X Generalist–ANLS: 0.906
visual-question-answering-on-docvqa-testSMoLA-PaLI-X Specialist–ANLS: 0.908
visual-question-answering-vqa-on-ai2dSMoLA-PaLI-X Generalist Model–EM: 81.4
visual-question-answering-vqa-on-ai2dSMoLA-PaLI-X Specialist Model–EM: 82.5
visual-question-answering-vqa-on-infographicvqaSMoLA-PaLI-X Generalist–ANLS: 65.6
visual-question-answering-vqa-on-infographicvqaSMoLA-PaLI-X Specialist–ANLS: 66.2