Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

Open paper
Benchmark Model Rank Results
d4rl-on-d4rlLinear Transformer#6Average Reward: 64.4
language-modelling-on-wikitext-103Linear Attention 125M#58Test perplexity: 25.6

Rank counts only results with a code link.