Primer: Searching for Efficient Transformers for Language Modeling

Benchmark Model Rank Results
language-modelling-on-c4Primer#1Perplexity: 12.35TPUv3 Hours: 17.3KSteps: 1M
language-modelling-on-c4T5++#3Perplexity: 12.69TPUv3 Hours: 16.5KSteps: 1M
language-modelling-on-c4Original T5#4Perplexity: 13.25TPUv3 Hours: 15.7KSteps: 1M