Efficient Content-Based Sparse Attention with Routing Transformers

Benchmark Model Rank Results
image-generation-on-imagenet-64x64Routing Transformer#33Bits per dim: 3.43
language-modelling-on-enwiki8Routing Transformer (12 layers)#13Bit per Character (BPC): 0.99
language-modelling-on-wikitext-103Routing Transformer#11Test perplexity: 15.8