MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts

Benchmark Model Rank Results
question-answering-on-social-iqaLLaMA-2 13B + MixLoRA#2Accuracy: 82.5
question-answering-on-social-iqaLLaMA-3 8B + MixLoRA#10Accuracy: 78.8
question-answering-on-social-iqaLLaMA-2 7B + MixLoRA#11Accuracy: 78
question-answering-on-openbookqaLLaMA-3 8B + MixLoRA#4Accuracy: 84.8
question-answering-on-openbookqaLLaMA-2 13B + MixLoRA#6Accuracy: 83
question-answering-on-openbookqaLLaMA-2 7B + MixLoRA#10Accuracy: 81.6
common-sense-reasoning-on-arc-challengeLLaMA-3 8B + MixLoRA#6Accuracy: 79.9
common-sense-reasoning-on-arc-challengeLLaMA-2 13B + MixLoRA#7Accuracy: 69.9
common-sense-reasoning-on-arc-challengeLLaMA-2 7B + MixLoRA#16Accuracy: 58.1
common-sense-reasoning-on-arc-easyLLaMA-3 8B + MixLoRA#6Accuracy: 86.5
common-sense-reasoning-on-arc-easyLLaMA-2 13B + MixLoRA#13Accuracy: 83.5
common-sense-reasoning-on-arc-easyLLaMA-2 7B + MixLoRA#29Accuracy: 77.7
question-answering-on-piqaLLaMA-3 8B + MixLoRA#6Accuracy: 87.6
question-answering-on-piqaLLaMA-2 13B + MixLoRA#9Accuracy: 86.8
question-answering-on-piqaLLaMA-2 7B + MixLoRA#25Accuracy: 83.2
sentence-completion-on-hellaswagLLaMA-2 13B + MixLoRA#7Accuracy: 94.7
sentence-completion-on-hellaswagLLaMA-3 8B + MixLoRA#11Accuracy: 93.3
sentence-completion-on-hellaswagLLaMA-2 7B + MixLoRA#12Accuracy: 93.1
common-sense-reasoning-on-winograndeLLaMA-2 13B + MixLoRA#12Accuracy: 86.3
common-sense-reasoning-on-winograndeLLaMA-3 8B + MixLoRA#21Accuracy: 82.1
common-sense-reasoning-on-winograndeLLaMA-2 7B + MixLoRA#50Accuracy: 76.8
question-answering-on-boolqLLaMA-2 13B + MixLoRA#52Accuracy: 77.1
question-answering-on-boolqLLaMA-3 8B + MixLoRA#58Accuracy: 75
question-answering-on-boolqLLaMA-2 7B + MixLoRA#64Accuracy: 72.7