Jamba: A Hybrid Transformer-Mamba Language Model

Benchmark Model Rank Results
zero-shot-question-answering-on-truthfulqa-mc2-0-shotJamba (0-shot)#4MC2Acc: 46.4
multiple-choice-question-answering-mcqa-on-arc-challenge-25-shotJamba (25-shot)#7Accuracy: 64.4
question-answering-on-boolqJamba (10-shot)#10Accuracy: 88.2
question-answering-on-natural-questionsJamba (5-shot)#17EM: 45.9
common-sense-reasoning-on-winograndeJamba (5-shot)#20Accuracy: 82.5
question-answering-on-piqaJamba (0-shot)#21Accuracy: 83.2
sentence-completion-on-hellaswagJamba (10-shot)#26Accuracy: 87.1
multi-task-language-understanding-on-mmluJamba (5-shot)#34Average (%): 67.4
common-sense-reasoning-on-arc-easyJamba (0-shot)#40Accuracy: 73.5
code-generation-on-humanevalJamba#122Pass@1: 29.3
arithmetic-reasoning-on-gsm8kJamba (3-shot CoT)#207Accuracy: 59.9