Mamba-3: Improved Sequence Modeling using State Space Principles

Benchmark Model Rank Results
multiple-choice-question-answering-mcqa-on-arc-eMamba-3-MIMO-1.5B#3Accuracy: 76.5
multiple-choice-question-answering-mcqa-on-arc-eMamba-3-SISO-1.5B#4Accuracy: 75.9
multiple-choice-question-answering-mcqa-on-arc-eMamba-3-MIMO-880M#5Accuracy: 74.7
multiple-choice-question-answering-mcqa-on-arc-eMamba-3-SISO-880M#7Accuracy: 72.7
multiple-choice-question-answering-mcqa-on-arc-eMamba-3-SISO-440M#12Accuracy: 68.9
multiple-choice-question-answering-mcqa-on-arc-eMamba-3-MIMO-180M#17Accuracy: 60.6
multiple-choice-question-answering-mcqa-on-arc-cMamba-3-MIMO-1.5B#10Accuracy: 44.5
multiple-choice-question-answering-mcqa-on-arc-cMamba-3-SISO-1.5B#13Accuracy: 42.7
multiple-choice-question-answering-mcqa-on-arc-cMamba-3-MIMO-880M#16Accuracy: 41.2
multiple-choice-question-answering-mcqa-on-arc-cMamba-3-SISO-880M#20Accuracy: 40.2
multiple-choice-question-answering-mcqa-on-arc-cMamba-3-SISO-440M#24Accuracy: 34.4
multiple-choice-question-answering-mcqa-on-arc-cMamba-3-SISO-180M#25Accuracy: 27.9
multiple-choice-question-answering-mcqa-on-arc-cMamba-3-MIMO-180M#26Accuracy: 27.7
common-sense-reasoning-on-arc-easyMamba-3-MIMO-1.5B#29Accuracy: 76.5
language-modelling-on-lambadaMamba-3-MIMO-1.5B#42Accuracy: 51.7Perplexity: 10.2
language-modelling-on-lambadaMamba-3-MIMO-880M#45Accuracy: 49.5Perplexity: 11.8
language-modelling-on-lambadaMamba-3-SISO-1.5B#47Accuracy: 49.4Perplexity: 10.9
language-modelling-on-lambadaMamba-3-SISO-880M#50Accuracy: 47.2Perplexity: 12.9
language-modelling-on-lambadaMamba-3-MIMO-180M#59Accuracy: 34.0Perplexity: 32.1
language-modelling-on-lambadaMamba-3-SISO-180M#60Accuracy: 32.5Perplexity: 37.7
language-modelling-on-lambadaMamba-3-SISO-440M#76Perplexity: 19.6
language-modelling-on-lambadaMamba-3-MIMO-440M#77Perplexity: 17.1
question-answering-on-piqaMamba-3-MIMO-1.5B#105Accuracy: 75.3
question-answering-on-piqaMamba-3-MIMO-880M#117Accuracy: 73.7
question-answering-on-piqaMamba-3-SISO-880M#119Accuracy: 73.6
question-answering-on-piqaMamba-3-SISO-1.5B#120Accuracy: 73.6
question-answering-on-piqaMamba-3-SISO-440M#137Accuracy: 71.9
question-answering-on-piqaMamba-3-MIMO-440M#142Accuracy: 70.8
question-answering-on-piqaMamba-3-MIMO-180M#156Accuracy: 66.7
question-answering-on-piqaMamba-3-SISO-180M#158Accuracy: 66.1
common-sense-reasoning-on-winograndeMamba-3-MIMO-1.5B#115Accuracy: 60.6
common-sense-reasoning-on-winograndeMamba-3-MIMO-880M#120Accuracy: 59.9
common-sense-reasoning-on-winograndeMamba-3-SISO-1.5B#124Accuracy: 59.4
common-sense-reasoning-on-winograndeMamba-3-SISO-880M#133Accuracy: 58.4
common-sense-reasoning-on-winograndeMamba-3-MIMO-440M#141Accuracy: 56.3
common-sense-reasoning-on-winograndeMamba-3-SISO-440M#146Accuracy: 55.8
common-sense-reasoning-on-winograndeMamba-3-MIMO-180M#157Accuracy: 52.9
common-sense-reasoning-on-winograndeMamba-3-SISO-180M#162Accuracy: 52.0
sentence-completion-on-hellaswagMamba-3-MIMO-1.5B#141Accuracy: 62.3
sentence-completion-on-hellaswagMamba-3-SISO-1.5B#143Accuracy: 61.9
sentence-completion-on-hellaswagMamba-3-MIMO-880M#154Accuracy: 59.2
sentence-completion-on-hellaswagMamba-3-SISO-880M#157Accuracy: 58.8
sentence-completion-on-hellaswagMamba-3-MIMO-440M#173Accuracy: 52.8
sentence-completion-on-hellaswagMamba-3-SISO-440M#175Accuracy: 51.7
sentence-completion-on-hellaswagMamba-3-MIMO-180M#200Accuracy: 41.0
sentence-completion-on-hellaswagMamba-3-SISO-180M#201Accuracy: 40.8