Improving Transformer Models by Reordering their Sublayers

Benchmark Model Rank Results
language-modelling-on-enwiki8Sandwich Transformer (adaptive span)#6Bit per Character (BPC): 0.968Number of params: 209M
language-modelling-on-wikitext-103Sandwich Transformer#28Test perplexity: 17.96Number of params: 247M