DeLighT: Deep and Light-weight Transformer

Benchmark Model Rank Results
language-modelling-on-wikitext-103DeLighT#53Test perplexity: 24.14Number of params: 99M
machine-translation-on-iwslt2014-germanDeLighT#21BLEU score: 35.3
machine-translation-on-wmt2016-english-1DeLighT#1BLEU score: 34.7
machine-translation-on-wmt2016-english-germanDeLighT#3BLEU score: 28.0