RecurrentGemma: Moving Past Transformers for Efficient Open Language Models

Benchmark Model Rank Results
social-commonsense-reasoning-on-siqaRecurrentGemma-9B#4Accuracy: 52.3
social-commonsense-reasoning-on-siqaRecurrentGemma-2B#6Accuracy: 51.8
multiple-choice-question-answering-mcqa-on-agieval-english-3-5-shotRecurrentGemma-9B#5Accuracy: 39.3
multiple-choice-question-answering-mcqa-on-agieval-english-3-5-shotRecurrentGemma-2B#6Accuracy: 23.8
question-answering-on-triviaqaRecurrentGemma-9B (5-shot)#23EM: 70.5
question-answering-on-triviaqaRecurrentGemma-2B (5-shot)#34EM: 52.5
common-sense-reasoning-on-arc-easyRecurrentGemma-9B#27Accuracy: 78.8
common-sense-reasoning-on-arc-easyRecurrentGemma-2B#43Accuracy: 72.9
question-answering-on-boolqRecurrentGemma-9B#44Accuracy: 80.3
question-answering-on-boolqRecurrentGemma-2B#67Accuracy: 71.3
question-answering-on-piqaRecurrentGemma-9B#45Accuracy: 81.3
question-answering-on-piqaRecurrentGemma-2B#66Accuracy: 78.5
multi-task-language-understanding-on-mmluRecurrentGemma-9B (5-shot)#50Average (%): 60.5
multi-task-language-understanding-on-mmluRecurrentGemma-2B (5-shot)#76Average (%): 38.4
question-answering-on-natural-questionsRecurrentGemma-9B (5-shot)#58EM: 21.7
question-answering-on-natural-questionsRecurrentGemma-2B (5-shot)#63EM: 11.5
common-sense-reasoning-on-winograndeRecurrentGemma-9B#62Accuracy: 73.6
common-sense-reasoning-on-winograndeRecurrentGemma-2B#86Accuracy: 67.8
sentence-completion-on-hellaswagRecurrentGemma-9B#72Accuracy: 80.4
sentence-completion-on-hellaswagRecurrentGemma-2B#105Accuracy: 71.0
code-generation-on-humanevalRecurrentGemma-9B#118Pass@1: 31.1
code-generation-on-humanevalRecurrentGemma-2B#132Pass@1: 21.3
code-generation-on-mbppRecurrentGemma-9B (3-shot)#131Accuracy: 42
code-generation-on-mbppRecurrentGemma-2B (3-shot)#151Accuracy: 28.8
math-word-problem-solving-on-mathRecurrentGemma-9B (4-shot)#148Accuracy: 23.8
math-word-problem-solving-on-mathRecurrentGemma-2B (4-shot)#174Accuracy: 11.0
arithmetic-reasoning-on-gsm8kRecurrentGemma-9B#238Accuracy: 42.6
arithmetic-reasoning-on-gsm8kRecurrentGemma-2B#259Accuracy: 13.4