| Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism | ✓ Link | 91.4 | | Megatron-3.9B | 2019-09-17 |
| DeBERTa: Decoding-enhanced BERT with Disentangled Attention | ✓ Link | 91.1 | | DeBERTa Large | 2020-06-05 |
| ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators | ✓ Link | 90.9 | | ELECTRA-1.75M | 2020-03-23 |
| Larger-Scale Transformers for Multilingual Masked Language Modeling | ✓ Link | 90.9 | | XLM-R XXL | 2021-05-02 |
| MPNet: Masked and Permuted Pre-training for Language Understanding | ✓ Link | 90.5 | | MPNet Large | 2020-04-20 |
| RoBERTa: A Robustly Optimized BERT Pretraining Approach | ✓ Link | 90.2 | | RoBERTa | 2019-07-26 |
| MVP: Multi-task Supervised Pre-training for Natural Language Generation | ✓ Link | 89.70 | | MVP | 2022-06-24 |
| ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding | ✓ Link | 89.1 | | ERNIE-Gram | 2020-10-23 |
| LFM2.5-Encoders: Fast at Long Context, Even on CPU | ✓ Link | 89.03 | | LFM2.5-Encoder-350M | 2026-07-28 |
| Unsupervised Cross-lingual Representation Learning at Scale | ✓ Link | 88.9 | | XLM-R | 2019-11-05 |
| Multi Resolution Analysis (MRA) for Approximate Self-Attention | ✓ Link | 87.4 | | MRA-2-s | 2022-07-21 |
| GLU Variants Improve Transformer | ✓ Link | 86.90 | | FFN Bilinear | 2020-02-12 |
| TinyBERT: Distilling BERT for Natural Language Understanding | ✓ Link | 84.5 | 84.5 | TinyBERT-6 67M | 2019-09-23 |
| Prune Once for All: Sparse Pre-Trained Language Models | ✓ Link | 83.74 | 84.2 | BERT-Large-uncased-PruneOFA (90% unstruct sparse) | 2021-11-10 |
| Prune Once for All: Sparse Pre-Trained Language Models | ✓ Link | 83.47 | 84.08 | BERT-Large-uncased-PruneOFA (90% unstruct sparse, QAT Int8) | 2021-11-10 |
| Prune Once for All: Sparse Pre-Trained Language Models | ✓ Link | 82.71 | 83.67 | BERT-Base-uncased-PruneOFA (85% unstruct sparse) | 2021-11-10 |
| DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter | ✓ Link | 82.2 | | DistilBERT | 2019-10-02 |
| You Only Sample (Almost) Once: Linear Cost Self-Attention Via Bernoulli Sampling | ✓ Link | 82.2 | | YOSO-E (MNLI-m) | 2021-11-18 |
| Prune Once for All: Sparse Pre-Trained Language Models | ✓ Link | 81.45 | 82.43 | BERT-Base-uncased-PruneOFA (90% unstruct sparse) | 2021-11-10 |
| Prune Once for All: Sparse Pre-Trained Language Models | ✓ Link | 81.4 | 82.51 | BERT-Base-uncased-PruneOFA (85% unstruct sparse, QAT Int8) | 2021-11-10 |
| Prune Once for All: Sparse Pre-Trained Language Models | ✓ Link | 81.35 | 82.03 | DistilBERT-uncased-PruneOFA (85% unstruct sparse) | 2021-11-10 |
| Prune Once for All: Sparse Pre-Trained Language Models | ✓ Link | 80.68 | 81.47 | DistilBERT-uncased-PruneOFA (90% unstruct sparse) | 2021-11-10 |
| Prune Once for All: Sparse Pre-Trained Language Models | ✓ Link | 80.66 | 81.14 | DistilBERT-uncased-PruneOFA (85% unstruct sparse, QAT Int8) | 2021-11-10 |
| Prune Once for All: Sparse Pre-Trained Language Models | ✓ Link | 78.8 | 80.4 | DistilBERT-uncased-PruneOFA (90% unstruct sparse, QAT Int8) | 2021-11-10 |