Accelerating Neural Transformer via an Average Attention Network

Benchmark Model Rank Results
machine-translation-on-wmt2014-english-germanAverage Attention Network#53BLEU score: 26.31
machine-translation-on-wmt2014-english-germanAverage Attention Network (w/o FFN)#56BLEU score: 26.05
machine-translation-on-wmt2014-english-germanAverage Attention Network (w/o gate)#58BLEU score: 25.91