| Paper | Code | GLUE (test) | ModelName | ReleaseDate |
|---|---|---|---|---|
| Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer | ✓ Link | 90.3 | T5-11B | 2019-10-23 |
| Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing | ✓ Link | 89.7 | Funnel-Transformer Large | 2020-06-05 |
| ALBERT: A Lite BERT for Self-supervised Learning of Language Representations | ✓ Link | 89.4 | ALBERT | 2019-09-26 |
| ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators | ✓ Link | 89.4 | ELECTRA-1.75M | 2020-03-23 |
| RoBERTa: A Robustly Optimized BERT Pretraining Approach | ✓ Link | 88.5 | RoBERTa | 2019-07-26 |
| MPNet: Masked and Permuted Pre-training for Language Understanding | ✓ Link | 86.5 | MPNet Base | 2020-04-20 |
| BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding | ✓ Link | 80.5 | BERT Large | 2018-10-11 |
| MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices | ✓ Link | 78.5 | MobileBERT w/o OPT | 2020-04-06 |
| Improving Language Understanding by Generative Pre-Training | ✓ Link | 72.8 | GPT-1 | 2018-06-11 |