Hungry Hungry Hippos: Towards Language Modeling with State Space Models

Benchmark Model Rank Results
coreference-resolution-on-winograd-schemaH3 125M (3-shot, rank classification)#40Accuracy: 63.5
coreference-resolution-on-winograd-schemaH3 125M (0-shot, rank classification)#46Accuracy: 61.5
coreference-resolution-on-winograd-schemaHybrid H3 125M (3-shot, logit scoring)#60Accuracy: 43.3
language-modelling-on-the-pileHybrid H3 125M#28Test perplexity: 10.2
language-modelling-on-the-pileTransformer 125M#30Test perplexity: 10.7
language-modelling-on-wikitext-103Hybrid H3 (2.7B)#2Test perplexity: 10.6Number of params: 2700M
language-modelling-on-wikitext-103Hybrid H3 (1.3B)#6Test perplexity: 12.5Number of params: 1300M
language-modelling-on-wikitext-103Hybrid H3 (355M)#18Test perplexity: 16.9Number of params: 355M
language-modelling-on-wikitext-103Hybrid H3 125M#37Test perplexity: 18.5
language-modelling-on-wikitext-103Hybrid H3 (125M)#51Test perplexity: 23.7Number of params: 125M
natural-language-inference-on-rteHybrid H3 125M (0-shot, logit scoring)#65Accuracy: 59.2%
natural-language-inference-on-rteHybrid H3 125M (3-shot, logit scoring)#68Accuracy: 58.1%
natural-language-inference-on-rteHybrid H3 125M (3-shot, rank classification)#69Accuracy: 58.1%
natural-language-inference-on-rteH3 125M (0-shot, rank classification)#76Accuracy: 53.1%
natural-language-inference-on-rteH3 125M (3-shot, rank classification)#77Accuracy: 52.3%
question-answering-on-boolqHybrid H3 1.3B (0-shot, logit scoring)#50Accuracy: 61.7
question-answering-on-boolqHybrid H3 2.7B (3-shot, logit scoring)#52Accuracy: 60.6
question-answering-on-boolqHybrid H3 125M (0-shot, logit scoring)#56Accuracy: 59.6
question-answering-on-boolqHybrid H3 125M (3-shot, logit scoring)#58Accuracy: 56.1
question-answering-on-boolqHybrid H3 125M (3-shot, rank classification)#59Accuracy: 56.1
question-answering-on-copaHybrid H3 2.7B (0-shot, logit scoring)#32Accuracy: 81
question-answering-on-copaHybrid H3 2.7B (3-shot, logit scoring)#38Accuracy: 77
question-answering-on-copaHybrid H3 125M (0-shot, logit scoring)#42Accuracy: 67
question-answering-on-copaHybrid H3 125M (0-shot, rank classification)#43Accuracy: 67
question-answering-on-copaH3 125M (0-shot, rank classification)#47Accuracy: 51
question-answering-on-multircHybrid H3 355M (3-shot, logit scoring)#25EM: 59.7
question-answering-on-multircHybrid H3 355M (0-shot, logit scoring)#26EM: 59.5
question-answering-on-multircHybrid H3 125M (0-shot, logit scoring)#27EM: 51.4
question-answering-on-multircHybrid H3 125M (3-shot, logit scoring)#28EM: 48.9
word-sense-disambiguation-on-words-in-contextHybrid H3 125M (0-shot, logit scoring)#18Accuracy: 51.4
word-sense-disambiguation-on-words-in-contextHybrid H3 125M (0-shot, rank classification)#19Accuracy: 51.4
word-sense-disambiguation-on-words-in-contextHybrid H3 125M (3-shot, logit scoring)#26Accuracy: 49.1