Rethinking Attention with Performers

Benchmark Model Rank Results
d4rl-on-d4rlPerformer#8Average Reward: 63.8
image-generation-on-imagenet-64x64Performer (12 layers)#40Bits per dim: 3.636
image-generation-on-imagenet-64x64Performer (6 layers)#45Bits per dim: 3.719
language-modelling-on-wikitext-103Performer 125M#62Test perplexity: 26.8