LLaMA: Open and Efficient Foundation Language Models

Benchmark Model Rank Results
arithmetic-reasoning-on-gsm8kLLaMA 65B-maj1@k#73Accuracy: 69.7Parameters (Billion): 65
arithmetic-reasoning-on-gsm8kLLaMA 33B-maj1@k#92Accuracy: 53.1Parameters (Billion): 33
arithmetic-reasoning-on-gsm8kLLaMA 65B#98Accuracy: 50.9Parameters (Billion): 65
arithmetic-reasoning-on-gsm8kLLaMA 33B#103Accuracy: 35.6Parameters (Billion): 33
arithmetic-reasoning-on-gsm8kLLaMA 13B-maj1@k#106Accuracy: 29.3Parameters (Billion): 13
arithmetic-reasoning-on-gsm8kLLaMA 7B (maj1@k)#109Accuracy: 18.1Parameters (Billion): 7
arithmetic-reasoning-on-gsm8kLLaMA 13B#111Accuracy: 17.8Parameters (Billion): 13
arithmetic-reasoning-on-gsm8kLLaMA 7B#113Accuracy: 11.0Parameters (Billion): 7
code-generation-on-mbppLLaMA 65B (0-shot)#74Accuracy: 37.7
code-generation-on-mbppLLaMA 33B (0-shot)#80Accuracy: 30.2
code-generation-on-mbppLLaMA 13B (0-shot)#85Accuracy: 22
code-generation-on-mbppLLaMA 7B (0-shot)#88Accuracy: 17.7
common-sense-reasoning-on-arc-challengeLLaMA 33B (zero-shot)#16Accuracy: 57.8
common-sense-reasoning-on-arc-challengeLLaMA 65B (zero-shot)#18Accuracy: 56.0
common-sense-reasoning-on-arc-challengeLLaMA 13B (zero-shot)#21Accuracy: 52.7
common-sense-reasoning-on-arc-challengeLLaMA 7B (zero-shot)#27Accuracy: 47.6
common-sense-reasoning-on-arc-easyLLaMA 33B (0-shot)#13Accuracy: 80.0
common-sense-reasoning-on-arc-easyLLaMA 65B (0-shot)#16Accuracy: 78.9
common-sense-reasoning-on-arc-easyLLaMA 13B (0-shot)#21Accuracy: 74.8
common-sense-reasoning-on-arc-easyLLaMA 7B (0-shot)#23Accuracy: 72.8
common-sense-reasoning-on-winograndeLLaMA 65B (0-shot)#22Accuracy: 77.0
common-sense-reasoning-on-winograndeLLaMA 33B (0-shot)#24Accuracy: 76.0
common-sense-reasoning-on-winograndeLLaMA 13B (0-shot)#30Accuracy: 73.0
common-sense-reasoning-on-winograndeLLaMA 7B (0-shot)#37Accuracy: 70.1
few-shot-learning-on-medconceptsqameta-llama/Meta-Llama-3-8B-Instruct#3Accuracy: 25.653
math-word-problem-solving-on-mathLLaMA 65B (maj1@k)#88Accuracy: 20.5Parameters (Billions): 65
math-word-problem-solving-on-mathLLaMA 33B-maj1@k#94Accuracy: 15.2Parameters (Billions): 33
math-word-problem-solving-on-mathLLaMA 65B#102Accuracy: 10.6Parameters (Billions): 65
math-word-problem-solving-on-mathLLaMA 13B-maj1@k#105Accuracy: 8.8Parameters (Billions): 13
math-word-problem-solving-on-mathLLaMA 33B#106Accuracy: 7.1Parameters (Billions): 33
math-word-problem-solving-on-mathLLaMA 7B-maj1@k#108Accuracy: 6.9Parameters (Billions): 7
math-word-problem-solving-on-mathLLaMA 13B#117Accuracy: 3.9Parameters (Billions): 13
math-word-problem-solving-on-mathLLaMA 7B#120Accuracy: 2.9Parameters (Billions): 7
multi-task-language-understanding-on-mmluLLaMA 65B (fine-tuned)#8Average (%): 68.9
multi-task-language-understanding-on-mmluLLaMA 65B (5-shot)#10Average (%): 63.4
multi-task-language-understanding-on-mmluLLaMA 33B (5-shot)#15Average (%): 57.8
question-answering-on-boolqLLaMA 65B (0-shot)#15Accuracy: 85.3
question-answering-on-boolqLLaMA 33B (0-shot)#21Accuracy: 83.1
question-answering-on-boolqLLaMA 13B (zero-shot)#27Accuracy: 78.1
question-answering-on-boolqLLaMA 7B (zero-shot)#30Accuracy: 76.5
question-answering-on-natural-questionsLLaMA 65B (few-shot, k=64)#18EM: 39.9
question-answering-on-natural-questionsLLaMA 65B (few-shot, k=5)#22EM: 35.0
question-answering-on-natural-questionsLLaMA 65B (one-shot)#26EM: 31.0
question-answering-on-natural-questionsLLaMA 33B (zero-shot)#32EM: 24.9
question-answering-on-obqaLLaMA 65B (zero-shot)#3Accuracy: 60.2
question-answering-on-obqaLLaMA 33B (zero-shot)#4Accuracy: 58.6
question-answering-on-obqaLLaMA 7B (zero-shot)#6Accuracy: 57.2
question-answering-on-obqaLLaMA 13B (zero-shot)#7Accuracy: 56.4
question-answering-on-piqaLLaMA 65B (0-shot)#16Accuracy: 82.8
question-answering-on-piqaLLaMA 33B (0-shot)#19Accuracy: 82.3
question-answering-on-piqaLLaMA 13B (0-shot)#32Accuracy: 80.1
question-answering-on-piqaLLaMA 7B (0-shot)#33Accuracy: 79.8
question-answering-on-social-iqaLLaMA 65B (zero-shot)#18Accuracy: 52.3
question-answering-on-social-iqaLLaMA 13B (zero-shot)#21Accuracy: 50.4
question-answering-on-social-iqaLLaMA 33B (zero-shot)#22Accuracy: 50.4
question-answering-on-social-iqaLLaMA 7B (zero-shot)#23Accuracy: 48.9
question-answering-on-timequestionsLlama3#9P@1: 17.8
question-answering-on-triviaqaLLaMA 65B (few-shot, k=64)#12EM: 73.0
question-answering-on-triviaqaLLaMA 65B (few-shot, k=5)#13EM: 72.6
question-answering-on-triviaqaLLaMA 65B (one-shot)#15EM: 71.6
question-answering-on-triviaqaLLaMA 65B (zero-shot)#19EM: 68.2
question-answering-on-truthfulqaLLaMA 65B#22% true: 57% info: 53
question-answering-on-truthfulqaLLaMA 33B#23% true: 52% info: 48
question-answering-on-truthfulqaLLaMA 13B#24% true: 47% info: 41
question-answering-on-truthfulqaLLaMA 7B#25% true: 33% info: 29
reading-comprehension-on-raceLLaMA 65B (zero-shot)#9Accuracy (Middle): 67.9Accuracy (High): 51.6
reading-comprehension-on-raceLLaMA 33B (zero-shot)#11Accuracy (Middle): 64.1Accuracy (High): 48.3
reading-comprehension-on-raceLLaMA 13B (zero-shot)#12Accuracy (Middle): 61.6Accuracy (High): 47.2
reading-comprehension-on-raceLLaMA 7B (zero-shot)#13Accuracy (Middle): 61.1Accuracy (High): 46.9
sentence-completion-on-hellaswagLLaMA 65B (0-shot)#21Accuracy: 84.2
sentence-completion-on-hellaswagLLaMA 33B (0-shot)#28Accuracy: 82.8
sentence-completion-on-hellaswagLLaMA 13B (0-shot)#38Accuracy: 79.2
sentence-completion-on-hellaswagLLaMA 7B (0-shot)#41Accuracy: 76.1
stereotypical-bias-analysis-on-crows-pairsLLaMA 65B#4Gender: 70.6Religion: 70.6Race/Color: 57.0Sexual Orientation: 81.0
zero-shot-learning-on-medconceptsqameta-llama/Meta-Llama-3-8B-Instruct#4Accuracy: 25.840