Random Feature Attention

Benchmark Model Rank Results
language-modelling-on-wikitext-103Rfa-Gate-Gaussian-Stateful (Big)Test perplexity: 23.5Validation perplexity: 22
language-modelling-on-wikitext-103Rfa-Gate-Gaussian-Stateful (Small)Test perplexity: 30.5Validation perplexity: 29.4
machine-translation-on-iwslt2014-germanRfa-Gate-arccosBLEU score: 34.4
machine-translation-on-wmt2014-english-frenchRfa-Gate-arccosBLEU score: 39.2
machine-translation-on-wmt2014-english-germanRfa-Gate-arccosBLEU score: 28.2