| Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism | ✓ Link | 91.2 | | Megatron-3.9B | 2019-09-17 |
| UltraBERT: Beyond the Limit of BERT Pre-Training | | 90.9 | | UltraBERT 380M | 2026-10-10 |
| DeBERTa: Decoding-enhanced BERT with Disentangled Attention | ✓ Link | 90.7 | | DeBERTa Large | 2020-06-05 |
| XLNet: Generalized Autoregressive Pretraining for Language Understanding | ✓ Link | 90.6 | 87.9 | XLNet (single model) | 2019-06-19 |
| Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing | ✓ Link | 90.4 | | Funnel-Transformer Large | 2020-06-05 |
| Dice Loss for Data-imbalanced NLP Tasks | ✓ Link | 89.51 | 87.65 | XLNet+DSC | 2019-11-07 |
| RoBERTa: A Robustly Optimized BERT Pretraining Approach | ✓ Link | 89.4 | 86.5 | RoBERTa (no data aug) | 2019-07-26 |
| BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension | ✓ Link | 89.2 | | BART | 2019-10-29 |
| ALBERT: A Lite BERT for Self-supervised Learning of Language Representations | ✓ Link | 88.1 | 85.1 | ALBERT xxlarge | 2019-09-26 |
| SG-Net: Syntax-Guided Machine Reading Comprehension | ✓ Link | 87.9 | 85.1 | SG-Net | 2019-08-14 |
| ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding | ✓ Link | 87.1 | | ERNIE-Gram | 2020-10-23 |
| SpanBERT: Improving Pre-training by Representing and Predicting Spans | ✓ Link | 86.8 | | SpanBERT | 2019-07-24 |
| ALBERT: A Lite BERT for Self-supervised Learning of Language Representations | ✓ Link | 85.9 | 83.1 | ALBERT xlarge | 2019-09-26 |
| Semantics-aware BERT for Language Understanding | ✓ Link | 83.6 | 80.9 | SemBERT large | 2019-09-05 |
| ALBERT: A Lite BERT for Self-supervised Learning of Language Representations | ✓ Link | 82.1 | 79.0 | ALBERT large | 2019-09-26 |
| MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices | ✓ Link | 80.2 | | MobileBERT w/o OPT | 2020-04-06 |
| ALBERT: A Lite BERT for Self-supervised Learning of Language Representations | ✓ Link | 79.1 | 76.1 | ALBERT base | 2019-09-26 |
| Read + Verify: Machine Reading Comprehension with Unanswerable Questions | | 74.8 | 72.3 | RMR + ELMo (Model-III) | 2018-08-17 |
| U-Net: Machine Reading Comprehension with Unanswerable Questions | ✓ Link | 74.0 | 70.3 | U-Net | 2018-10-12 |
| TinyBERT: Distilling BERT for Natural Language Understanding | ✓ Link | 73.4 | 69.9 | TinyBERT-6 67M | 2019-09-23 |