MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts

Benchmark Model Rank Results
common-sense-reasoning-on-arc-challengeLLaMA-3 8B + MixLoRA#6Accuracy: 79.9
common-sense-reasoning-on-arc-challengeLLaMA-2 13B + MixLoRA#7Accuracy: 69.9
common-sense-reasoning-on-arc-challengeLLaMA-2 7B + MixLoRA#15Accuracy: 58.1
common-sense-reasoning-on-arc-easyLLaMA-3 8B + MixLoRA#5Accuracy: 86.5
common-sense-reasoning-on-arc-easyLLaMA-2 13B + MixLoRA#9Accuracy: 83.5
common-sense-reasoning-on-arc-easyLLaMA-2 7B + MixLoRA#17Accuracy: 77.7
common-sense-reasoning-on-winograndeLLaMA-2 13B + MixLoRA#8Accuracy: 86.3
common-sense-reasoning-on-winograndeLLaMA-3 8B + MixLoRA#11Accuracy: 82.1
common-sense-reasoning-on-winograndeLLaMA-2 7B + MixLoRA#23Accuracy: 76.8
question-answering-on-boolqLLaMA-2 13B + MixLoRA#29Accuracy: 77.1
question-answering-on-boolqLLaMA-3 8B + MixLoRA#34Accuracy: 75
question-answering-on-boolqLLaMA-2 7B + MixLoRA#38Accuracy: 72.7
question-answering-on-openbookqaLLaMA-3 8B + MixLoRA#4Accuracy: 84.8
question-answering-on-openbookqaLLaMA-2 13B + MixLoRA#6Accuracy: 83
question-answering-on-openbookqaLLaMA-2 7B + MixLoRA#10Accuracy: 81.6
question-answering-on-piqaLLaMA-3 8B + MixLoRA#4Accuracy: 87.6
question-answering-on-piqaLLaMA-2 13B + MixLoRA#7Accuracy: 86.8
question-answering-on-piqaLLaMA-2 7B + MixLoRA#14Accuracy: 83.2
question-answering-on-social-iqaLLaMA-2 13B + MixLoRA#2Accuracy: 82.5
question-answering-on-social-iqaLLaMA-3 8B + MixLoRA#10Accuracy: 78.8
question-answering-on-social-iqaLLaMA-2 7B + MixLoRA#11Accuracy: 78
sentence-completion-on-hellaswagLLaMA-2 13B + MixLoRA#7Accuracy: 94.7
sentence-completion-on-hellaswagLLaMA-3 8B + MixLoRA#9Accuracy: 93.3
sentence-completion-on-hellaswagLLaMA-2 7B + MixLoRA#10Accuracy: 93.1