Rethinking Attention with Performers

Benchmark Model Rank Results
d4rl-on-d4rlPerformer#8Average Reward: 63.8
image-generation-on-imagenet-64x64Performer (12 layers)#43Bits per dim: 3.636
image-generation-on-imagenet-64x64Performer (6 layers)#48Bits per dim: 3.719
language-modelling-on-wikitext-103Performer 125M#66Test perplexity: 26.8