Gemma 2: Improving Open Language Models at a Practical Size

Benchmark Model Rank Results
multiple-choice-question-answering-mcqa-on-agieval-english-3-5-shotGemma 2 27B (3-5-shot)#1Accuracy: 55.1
social-commonsense-reasoning-on-siqaGemma 2 27B#2Accuracy: 53.7
multiple-choice-question-answering-mcqa-on-arc-challenge-25-shotGemma 2 27B (25-shot)#3Accuracy: 71.4
common-sense-reasoning-on-arc-easyGemma 2 27B#4Accuracy: 88.6
question-answering-on-drop-3-shotGemma 2 27B (3-shot)#5F1: 74.2
question-answering-on-triviaqaGemma 2 27B (5-shot)#5EM: 83.7
common-sense-reasoning-on-winograndeGemma 2 27B (5-shot)#17Accuracy: 83.7
multi-task-language-understanding-on-mmluGemma 2 27B (5-shot)#18Average (%): 75.2
question-answering-on-piqaGemma 2 27B#23Accuracy: 83.2
question-answering-on-boolqGemma 2 27B#26Accuracy: 84.8
sentence-completion-on-hellaswagGemma 2 27B (10-shot)#31Accuracy: 86.4
question-answering-on-natural-questionsGemma 2 27B (5-shot)#38EM: 34.5
code-generation-on-mbppGemma 2 27B (3-shot)#78Accuracy: 62.6
code-generation-on-humanevalGemma 2 27B#99Pass@1: 51.8
math-word-problem-solving-on-mathGemma 2 27B (4-shot)#116Accuracy: 42.3
arithmetic-reasoning-on-gsm8kGemma 2 27B (5-shot)#180Accuracy: 74.0