Combiner: Full Attention Transformer with Sparse Computation Cost

Benchmark Model Rank Results
language-modelling-on-wiki-40bCombiner-Axial-8k#2Perplexity: 16.49
language-modelling-on-wiki-40bCombiner-Fixed-8k#3Perplexity: 16.60
image-generation-on-imagenet-64x64Combiner-Axial#35Bits per dim: 3.42
image-generation-on-imagenet-64x64Combiner-Mixture#40Bits per dim: 3.504