DeLighT: Deep and Light-weight Transformer

Benchmark Model Rank Results
machine-translation-on-wmt2016-english-romanianDeLighT#1BLEU score: 34.7
machine-translation-on-wmt2016-english-germanDeLighT#3BLEU score: 28.0
machine-translation-on-iwslt2014-german-englishDeLighT#21BLEU score: 35.3
language-modelling-on-wikitext-103DeLighT#57Test perplexity: 24.14Number of params: 99M