| Improving Machine Reading Comprehension with Single-choice Decision and Transfer Learning | | 91.4 | | | ALBERT (Ensemble) | 2020-11-06 |
| Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism | ✓ Link | 90.9 | 93.1 | 90.0 | Megatron-BERT (ensemble) | 2019-09-17 |
| UL2: Unifying Language Learning Paradigms | ✓ Link | 90.9 | | | UL2 20B | 2022-05-10 |
| DUMA: Reading Comprehension with Transposition Thinking | ✓ Link | 89.8 | 88.7 | 92.6 | ALBERTxxlarge+DUMA(ensemble) | 2020-01-26 |
| Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism | ✓ Link | 89.5 | 91.8 | 88.6 | Megatron-BERT | 2019-09-17 |
| ALBERT: A Lite BERT for Self-supervised Learning of Language Representations | ✓ Link | 89.4 | | | ALBERT ensemble | 2019-09-26 |
| Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning | ✓ Link | 88.04 | | | Nemotron 3 Nano 30B-A3B Base | 2025-12-23 |
| DeBERTa: Decoding-enhanced BERT with Disentangled Attention | ✓ Link | 86.8 | | | DeBERTalarge | 2020-06-05 |
| Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing | ✓ Link | 85.7 | 88.8 | 84.4 | B10-10-10 | 2020-06-05 |
| Reinforcement Learning from Meta-Evaluation: Aligning Language Models Without Ground-Truth Labels | | 84.5 | | | Qwen3-4B-Base (RLVR+RLME) | 2026-01-29 |
| RoBERTa: A Robustly Optimized BERT Pretraining Approach | ✓ Link | 83.2 | 86.5 | 81.3 | RoBERTa | 2019-07-26 |
| Darwin Family: MRI-Trust-Weighted Evolutionary Merging for Training-Free Scaling of Language-Model Reasoning | | 83.1 | | | Darwin-27B-Opus | 2026-05-14 |
| Orca 2: Teaching Small Language Models How to Reason | | 82.87 | | | Orca 2-13B | 2023-11-18 |
| Orca 2: Teaching Small Language Models How to Reason | | 80.79 | | | Orca 2-7B | 2023-11-18 |
| ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding | ✓ Link | 77.7 | | | ERNIE-Gram | 2020-10-23 |
| MPNet: Masked and Permuted Pre-training for Language Understanding | ✓ Link | 76.1 | | | MPNet Base | 2020-04-20 |
| Hierarchical Learning for Generation with Long Source Sequences | | 67.3 | | | HAT (Encoder) | 2021-04-15 |
| XLNet: Generalized Autoregressive Pretraining for Language Understanding | ✓ Link | | 88.6 | 84.0 | XLNet | 2019-06-19 |
| PaLM: Scaling Language Modeling with Pathways | ✓ Link | | 68.1 | 49.1 | PaLM 540B (zero-shot) | 2022-04-05 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | | 67.9 | 51.6 | LLaMA 65B (zero-shot) | 2023-02-27 |
| PaLM: Scaling Language Modeling with Pathways | ✓ Link | | 64.3 | 47.5 | PaLM 62B (zero-shot) | 2022-04-05 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | | 64.1 | 48.3 | LLaMA 33B (zero-shot) | 2023-02-27 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | | 61.6 | 47.2 | LLaMA 13B (zero-shot) | 2023-02-27 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | | 61.1 | 46.9 | LLaMA 7B (zero-shot) | 2023-02-27 |
| Language Models are Few-Shot Learners | ✓ Link | | 58.4 | | GPT-3 175B (0-shot) | 2020-05-28 |
| PaLM: Scaling Language Modeling with Pathways | ✓ Link | | 57.9 | 42.3 | PaLM 8B (zero-shot) | 2022-04-05 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | | 54.32 | 41.74 | Bloomberg GPT (one-shot) | 2023-03-30 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | | 52.3 | 39.14 | BLOOM 176B (one-shot) | 2023-03-30 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | | 47.42 | 37.02 | OPT 66B (one-shot) | 2023-03-30 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | | 41.23 | 34.33 | GPT-NeoX (one-shot) | 2023-03-30 |
| Language Models are Few-Shot Learners | ✓ Link | | | 45.5 | GPT-3 175B (zero-shot) | 2020-05-28 |
| Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms | | 59.9 | | | Instruction-Queryable LoRA | 2026-05-08 |
| Improving Language Understanding by Generative Pre-Training | ✓ Link | 59.0 | | | GPT-1 | 2018-06-11 |
| The Curse and Blessing of Mean Bias in FP4-Quantized LLM Training | | 50.3600 | | | Qwen3-0.6B (Averis FP4) | 2026-03-11 |
| Triplet-Block Diffusion RWKV | ✓ Link | 49.7 | | | B3D-RWKV-7.2B | 2026-05-25 |
| Low-Rank Approximation for Sparse Attention in Multi-Modal LLMs | | 42.5 | | | LoRA-Sparse (LLaMA-7B) (extra data) | 2024-06-17 |
| Sumi: Open Uniform Diffusion Language Model from Scratch | ✓ Link | 41.4 | | | Sumi-7B | 2026-06-17 |
| GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression | | 40.77 | | | GeoPair (Llama3 8B, CR=0.2) | 2026-09-22 |
| Circuit Hypernetworks for Quantum-Augmented Diffusion Language Models | | 40.3 | | | HyperQ, 64 q | 2026-09-21 |
| daVinci-LLM:Towards the Science of Pretraining | ✓ Link | 38.56 | | | daVinci-LLM (3B) | 2026-03-28 |
| Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior | ✓ Link | 37.2 | | | PPT (k-Shuffle Dyck), 3B, C4, 100B PT tokens | 2026-09-30 |
| Variable-Width Transformers | ✓ Link | 34.3 | | | 3B/1B MoE > <former | 2026-06-16 |
| SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization | ✓ Link | 34.2 | | | SpectralShift (1.5B-A0.6B GDN-MoE, 32K extension, 10B CPT) | 2026-09-13 |
| What Makes Recurrence Effective in Looped Language Models? | | 30.62 | | | BaseLoop (Llama3.1-1B, 3x4, Muon) | 2026-09-29 |
| UniPool: A Globally Shared Expert Pool for Mixture-of-Experts | ✓ Link | 29.38 | | | UniPool (978M) | 2026-05-07 |