The Zamba2 Suite: Technical Report

Benchmark Model Rank Results
multiple-choice-question-answering-mcqa-on-arc-challenge-25-shotZamba2-7B (25-shot)#5Accuracy: 68.9
zero-shot-question-answering-on-openbookqaZamba2-7B#10Accuracy: 48.2
instruction-following-on-mtbenchZamba2-7B-Instruct#13Score (1-10): 7.42
common-sense-reasoning-on-arc-easyZamba2-7B#15Accuracy: 82.0
question-answering-on-boolqZamba2-7B#29Accuracy: 84.1
multi-task-language-understanding-on-mmluZamba2-7B (5-shot)#36Average (%): 67.2
common-sense-reasoning-on-winograndeZamba2-7B#43Accuracy: 77.3
question-answering-on-piqaZamba2-7B#50Accuracy: 81.0
sentence-completion-on-hellaswagZamba2-7B (10-shot)#63Accuracy: 81.5
multi-task-language-understanding-on-mmlu-proZamba2-7B-Instruct#127Accuracy: 32.43