| TinyBERT: Distilling BERT for Natural Language Understanding | ✓ Link | 84.5 | 84.5 | TinyBERT-6 67M | 2019-09-23 |
| Prune Once for All: Sparse Pre-Trained Language Models | ✓ Link | 83.74 | 84.2 | BERT-Large-uncased-PruneOFA (90% unstruct sparse) | 2021-11-10 |
| Prune Once for All: Sparse Pre-Trained Language Models | ✓ Link | 83.47 | 84.08 | BERT-Large-uncased-PruneOFA (90% unstruct sparse, QAT Int8) | 2021-11-10 |
| Prune Once for All: Sparse Pre-Trained Language Models | ✓ Link | 82.71 | 83.67 | BERT-Base-uncased-PruneOFA (85% unstruct sparse) | 2021-11-10 |
| Prune Once for All: Sparse Pre-Trained Language Models | ✓ Link | 81.45 | 82.43 | BERT-Base-uncased-PruneOFA (90% unstruct sparse) | 2021-11-10 |
| Prune Once for All: Sparse Pre-Trained Language Models | ✓ Link | 81.4 | 82.51 | BERT-Base-uncased-PruneOFA (85% unstruct sparse, QAT Int8) | 2021-11-10 |
| Prune Once for All: Sparse Pre-Trained Language Models | ✓ Link | 81.35 | 82.03 | DistilBERT-uncased-PruneOFA (85% unstruct sparse) | 2021-11-10 |
| Prune Once for All: Sparse Pre-Trained Language Models | ✓ Link | 80.68 | 81.47 | DistilBERT-uncased-PruneOFA (90% unstruct sparse) | 2021-11-10 |
| Prune Once for All: Sparse Pre-Trained Language Models | ✓ Link | 80.66 | 81.14 | DistilBERT-uncased-PruneOFA (85% unstruct sparse, QAT Int8) | 2021-11-10 |
| Prune Once for All: Sparse Pre-Trained Language Models | ✓ Link | 78.8 | 80.4 | DistilBERT-uncased-PruneOFA (90% unstruct sparse, QAT Int8) | 2021-11-10 |