Gemma: Open Models Based on Gemini Research and Technology

Benchmark Model Rank Results
multiple-choice-question-answering-mcqa-on-agieval-english-3-5-shotGemma 7B (3-5-shot)#4Accuracy: 41.7
social-commonsense-reasoning-on-siqaGemma 7B#5Accuracy: 51.8
multiple-choice-question-answering-mcqa-on-arc-challenge-25-shotGemma 7B (25-shot)#9Accuracy: 53.2
common-sense-reasoning-on-arc-easyGemma 7B#17Accuracy: 81.5
question-answering-on-triviaqaGemma 7B (5-shot)#29EM: 63.4
question-answering-on-boolqGemma 7B#34Accuracy: 83.2
multi-task-language-understanding-on-mmluGemma 7B (5-shot)#41Average (%): 64.3
question-answering-on-piqaGemma 7B#46Accuracy: 81.2
question-answering-on-natural-questionsGemma 7B (5-shot)#57EM: 23.0
sentence-completion-on-hellaswagGemma 7B#65Accuracy: 81.2
common-sense-reasoning-on-winograndeGemma 7B#69Accuracy: 72.3
code-generation-on-humanevalGemma 7B#115Pass@1: 32.3
code-generation-on-mbppGemma 7B (3-shot)#127Accuracy: 44.4
math-word-problem-solving-on-mathGemma 7B (4-shot)#147Accuracy: 24.3
arithmetic-reasoning-on-gsm8kGemma 7B#234Accuracy: 46.4