FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models

Benchmark Model Rank Results
common-sense-reasoning-on-winograndeFactorEngram (1B)–Accuracy: 58.48
language-modelling-on-lambadaFactorEngram (1B)–Perplexity: 11.39
question-answering-on-boolqFactorEngram (1B)–Accuracy: 60.88
question-answering-on-piqaFactorEngram (1B)–Accuracy: 72.46
sentence-completion-on-hellaswagFactorEngram (1B)–Accuracy: 57.02