Improving Transformer Models by Reordering their Sublayers

Benchmark Model Rank Results
language-modelling-on-enwiki8Sandwich Transformer (adaptive span)#6Bit per Character (BPC): 0.968Number of params: 209M
language-modelling-on-wikitext-103Sandwich Transformer#32Test perplexity: 17.96Number of params: 247M