Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation

Benchmark Model Rank Results
multiple-choice-question-answering-mcqa-on-agieval-english-3-5-shotT5Gemma 9B-9B PrefixLM#3Accuracy: 53.1
multi-task-language-understanding-on-mgsmT5Gemma 9B-9B PrefixLM IT#4Average (%): 80.7
question-answering-on-drop-3-shotT5Gemma 9B-9B PrefixLM#4F1: 75.7
multiple-choice-question-answering-mcqa-on-arc-challenge-25-shotT5Gemma 9B-9B PrefixLM#6Accuracy: 65.0
social-commonsense-reasoning-on-siqaT5Gemma 9B-9B PrefixLM#7Accuracy: 50.5
common-sense-reasoning-on-arc-easyT5Gemma 9B-9B PrefixLM#9Accuracy: 85.3
question-answering-on-triviaqaT5Gemma 9B-9B PrefixLM#14EM: 75.2
multi-task-language-understanding-on-mmluT5Gemma 9B-9B PrefixLM IT#15Average (%): 76.7
multi-task-language-understanding-on-mmluT5Gemma 9B-9B PrefixLM#26Average (%): 71.3
question-answering-on-boolqT5Gemma 9B-9B PrefixLM#22Accuracy: 85.6
question-answering-on-natural-questionsT5Gemma 9B-9B PrefixLM#23EM: 43.1
common-sense-reasoning-on-winograndeT5Gemma 9B-9B PrefixLM#38Accuracy: 78.7
question-answering-on-piqaT5Gemma 9B-9B PrefixLM#47Accuracy: 81.1
sentence-completion-on-hellaswagT5Gemma 9B-9B PrefixLM#66Accuracy: 81.0
code-generation-on-mbppT5Gemma 9B-9B PrefixLM IT#76Accuracy: 64.8
code-generation-on-mbppT5Gemma 9B-9B PrefixLM#96Accuracy: 55.6
code-generation-on-humanevalT5Gemma 9B-9B PrefixLM IT#83Pass@1: 72
code-generation-on-humanevalT5Gemma 9B-9B PrefixLM#111Pass@1: 40.2
arithmetic-reasoning-on-gsm8kT5Gemma 9B-9B PrefixLM IT#86Accuracy: 88.6
arithmetic-reasoning-on-gsm8kT5Gemma 9B-9B PrefixLM#186Accuracy: 72.8
multi-task-language-understanding-on-mmlu-proT5Gemma 9B-9B PrefixLM IT#104Accuracy: 55.7
multiple-choice-question-answering-mcqa-on-gpqa-diamondT5Gemma 9B-9B PrefixLM IT#124Accuracy: 40.4