Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning

Benchmark Model Rank Results
question-answering-on-piqaOpen-LLaMA-3B-v2#44Accuracy: 76.2
question-answering-on-piqaSheared-LLaMA-2.7B#47Accuracy: 75.8
question-answering-on-piqaSheared-LLaMA-1.3B#49Accuracy: 73.4
sentence-completion-on-hellaswagSheared-LLaMA-2.7B (50B)#45Accuracy: 70.8
sentence-completion-on-hellaswagOpen-LLaMA-3B-v2#47Accuracy: 67.6
sentence-completion-on-hellaswagSheared-LLaMA-1.3B (50B)#49Accuracy: 60.7