Mega: Moving Average Equipped Gated Attention

Benchmark Model Rank Results
image-classification-on-imagenetMega#525Top 1 Accuracy: 82.4%Number of params: 90M
language-modelling-on-wikitext-103Mega#30Test perplexity: 18.07Number of params: 252M
machine-translation-on-wmt2014-english-germanMega#30BLEU score: 29.01SacreBLEU: 27.96Number of Params: 67M
machine-translation-on-wmt2014-german-englishMega#4BLEU score: 33.12