OpenCodePapers

natural-language-understanding-on-glue-dev

Natural Language Understanding
Dataset Link
Results over time
Click legend items to toggle metrics. Hover points for model names.
Leaderboard
Click a sortable column header to sort.
PaperCodeGLUE (dev)ModelNameReleaseDate
UltraBERT: Beyond the Limit of BERT Pre-Training91.0UltraBERT 380M2026-10-10
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference✓ Link90.4ModernBERT Large2024-12-18
ByT5: Towards a token-free future with pre-trained byte-to-byte models✓ Link90.1ByT5 XXL2021-05-28
DeBERTa: Decoding-enhanced BERT with Disentangled Attention✓ Link90.00DeBERTa Large2020-06-05
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators✓ Link89.5ELECTRA-1.75M2020-03-23
I-BERT: Integer-only BERT Quantization✓ Link89.5I-BERT RoBERTa-Large2021-01-05
MPNet: Masked and Permuted Pre-training for Language Understanding✓ Link89.4MPNet Large2020-04-20
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference✓ Link88.4ModernBERT Base2024-12-18
ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding✓ Link88.1ERNIE-Gram2020-10-23
MVP: Multi-task Supervised Pre-training for Natural Language Generation✓ Link85.88MVP2022-06-24
GLU Variants Improve Transformer✓ Link84.67FFN ReGLU2020-02-12
FNet: Mixing Tokens with Fourier Transforms✓ Link83.6FNet-Hybrid-Large2021-05-09
Perceiver IO: A General Architecture for Structured Inputs & Outputs✓ Link81.8Perceiver IO++2021-07-30
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter✓ Link77.0DistilBERT2019-10-02
Check out how to contribute new results. Then edit this file on GitLab or GitHub.