Mixture-of-Subspaces in Low-Rank Adaptation

Benchmark Model Rank Results
common-sense-reasoning-on-arc-challengeLLaMA 3 8B + MoSLoRA (fine-tuned)#5Accuracy: 81.5
common-sense-reasoning-on-arc-easyLLaMA 3 8B+MoSLoRA (fine-tuned)#2Accuracy: 90.5
common-sense-reasoning-on-winograndeLLaMA3 8B+MoSLoRA#9Accuracy: 85.8
question-answering-on-boolqLLaMA3+MoSLoRA#36Accuracy: 74.6
question-answering-on-openbookqaLLaMA-3 8B+MoSLoRA#3Accuracy: 86.8
question-answering-on-piqaLLaMA3 8B+MoSLoRA#2Accuracy: 89.7
question-answering-on-social-iqaLLaMA-3 8B+MoSLoRA (fine-tuned)#5Accuracy: 81.0
sentence-completion-on-hellaswagLLaMA3+MoSLoRA#5Accuracy: 95.0
visual-question-answering-on-mm-vetLLaVA-InternLM2-7B-ViT + MoSLoRA#111GPT-4 score: 35.2
visual-question-answering-on-mm-vetInternLM2+ViT (QMoSLoRA)#112GPT-4 score: 35.2
visual-question-answering-on-mmbenchLLaVA-InternLM2-ViT + MoSLoRA#1GPT-3.5 score: 73.8
visual-question-answering-on-mmbenchLLaVA-LLaMA3-8B-ViT + MoSLoRA#3GPT-3.5 score: 73.0