Random Feature Attention

Benchmark Model Rank Results
language-modelling-on-wikitext-103Rfa-Gate-Gaussian-Stateful (Big)–Test perplexity: 23.5Validation perplexity: 22
language-modelling-on-wikitext-103Rfa-Gate-Gaussian-Stateful (Small)–Test perplexity: 30.5Validation perplexity: 29.4
machine-translation-on-iwslt2014-german-englishRfa-Gate-arccos–BLEU score: 34.4
machine-translation-on-wmt2014-english-frenchRfa-Gate-arccos–BLEU score: 39.2
machine-translation-on-wmt2014-english-germanRfa-Gate-arccos–BLEU score: 28.2