On the adequacy of untuned warmup for adaptive optimization

Benchmark Model Rank Results
machine-translation-on-wmt2016-english-germanTransformer#5BLEU score: 26.7
language-modelling-on-wikitext-103Transformer (Adaptive inputs)#82Validation perplexity: 19.5
image-classification-on-imagenetResNet-50#1051Top 1 Accuracy: 72.1%