On the adequacy of untuned warmup for adaptive optimization

Benchmark Model Rank Results
image-classification-on-imagenetResNet-50#938Top 1 Accuracy: 72.1%
language-modelling-on-wikitext-103Transformer (Adaptive inputs)#78Validation perplexity: 19.5
machine-translation-on-wmt2016-english-germanTransformer#5BLEU score: 26.7