Combiner: Full Attention Transformer with Sparse Computation Cost

Open paper
Benchmark Model Rank Results
image-generation-on-imagenet-64x64Combiner-Axial#32Bits per dim: 3.42
image-generation-on-imagenet-64x64Combiner-Mixture#37Bits per dim: 3.504
language-modelling-on-wiki-40bCombiner-Axial-8k#2Perplexity: 16.49
language-modelling-on-wiki-40bCombiner-Fixed-8k#3Perplexity: 16.60

Rank counts only results with a code link.