MDN: Parallelizing Stepwise Momentum for Delta Linear Attention

Benchmark Model Rank Results
language-modelling-on-lambadaMDN (1.3B)#74Perplexity: 14.87
question-answering-on-boolqMDN (1.3B)#105Accuracy: 59.63
question-answering-on-piqaMDN (1.3B)#147Accuracy: 71.87
sentence-completion-on-hellaswagMDN (1.3B)#169Accuracy: 53.5