| Paper | Code | Pearson/Spearman Correlation | ModelName | ReleaseDate |
|---|---|---|---|---|
| DeBERTa: Decoding-enhanced BERT with Disentangled Attention | ✓ Link | 92.8 | DeBERTa Large | 2020-06-05 |
| ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators | ✓ Link | 92.6 | ELECTRA-1.75M | 2020-03-23 |
| RoBERTa: A Robustly Optimized BERT Pretraining Approach | ✓ Link | 92.4 | RoBERTa | 2019-07-26 |
| MPNet: Masked and Permuted Pre-training for Language Understanding | ✓ Link | 92.4 | MPNet Large | 2020-04-20 |
| I-BERT: Integer-only BERT Quantization | ✓ Link | 92.2 | I-BERT RoBERTa-Large | 2021-01-05 |
| Unsupervised Cross-lingual Representation Learning at Scale | ✓ Link | 91.2 | XLM-R | 2019-11-05 |
| MVP: Multi-task Supervised Pre-training for Natural Language Generation | ✓ Link | 91.1 | MVP | 2022-06-24 |
| GLU Variants Improve Transformer | ✓ Link | 90.23 | FFN SwiGLU | 2020-02-12 |
| DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter | ✓ Link | 86.9 | DistilBERT | 2019-10-02 |
| FLAVA: A Foundational Language And Vision Alignment Model | ✓ Link | 85.7 | FLAVA | 2021-12-08 |