Low-Rank Approximation for Sparse Attention in Multi-Modal LLMs

Benchmark Model Rank Results
question-answering-on-piqaLoRA-Sparse (LLaMA-7B) (extra data)–Accuracy: 79.4
reading-comprehension-on-raceLoRA-Sparse (LLaMA-7B) (extra data)–Accuracy: 42.5
sentence-completion-on-hellaswagLoRA-Sparse (LLaMA-7B) (extra data)–Accuracy: 60.1
visual-question-answering-vqa-on-seed-seedbenchLoRA-Sparse (Vicuna-7B) (extra data)–Accuracy: 58.8