Gated Delta Networks: Improving Mamba2 with Delta Rule

Benchmark Model Rank Results
social-commonsense-reasoning-on-siqaGated DeltaNet-H2#13Accuracy: 41.91
language-modelling-on-wikitext-103Gated DeltaNet-H2#14Test perplexity: 15.91
common-sense-reasoning-on-arc-easyGated DeltaNet-H1#48Accuracy: 71.75
language-modelling-on-lambadaGated DeltaNet-H2#49Accuracy: 48.76
language-modelling-on-lambadaGated DeltaNet-H1#67Perplexity: 12.12
question-answering-on-natural-questionsGated DeltaNet-H2#54EM: 24.8
question-answering-on-boolqGated DeltaNet-H1#81Accuracy: 63.21
question-answering-on-piqaGated DeltaNet-H1#131Accuracy: 72.57
common-sense-reasoning-on-winograndeGated DeltaNet-H1#132Accuracy: 58.40
sentence-completion-on-hellaswagGated DeltaNet-H2#160Accuracy: 56.88