| PaLM: Scaling Language Modeling with Pathways | ✓ Link | 89.7 | | PaLM-540B (Few-Shot) | 2022-04-05 |
| PaLM 2 Technical Report | ✓ Link | 86.9 | | PaLM 2-L (one-shot) | 2023-05-17 |
| Language Models are Few-Shot Learners | ✓ Link | 86.4 | 1.92 | GPT-3 175B (Few-Shot) | 2020-05-28 |
| The Falcon Series of Open Language Models | ✓ Link | 84.4 | | Falcon-180B (1-shot) | 2023-11-28 |
| Stay on topic with Classifier-Free Guidance | | 84.0 | | LLaMA-65B+CFG (Zero-Shot) | 2023-06-30 |
| Stay on topic with Classifier-Free Guidance | | 83.9 | | LLaMA-30B+CFG (zero-shot) | 2023-06-30 |
| PaLM 2 Technical Report | ✓ Link | 83.7 | | PaLM 2-M (one-shot) | 2023-05-17 |
| | 82.33 | | Cohere Large | |
| Stay on topic with Classifier-Free Guidance | | 82.2 | | LLaMA-13B+CFG (zero-shot) | 2023-06-30 |
| PaLM: Scaling Language Modeling with Pathways | ✓ Link | 81.8 | | PaLM-540B (One-Shot) | 2022-04-05 |
| GLaM: Efficient Scaling of Language Models with Mixture-of-Experts | | 80.9 | | GLaM 62B/64E (One-Shot) | 2021-12-13 |
| PaLM 2 Technical Report | ✓ Link | 80.7 | | PaLM 2-S (one-shot) | 2023-05-17 |
| GLM-130B: An Open Bilingual Pre-trained Model | ✓ Link | 80.2 | | GLM-130B (bidirectional attention) | 2022-10-05 |
| SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot | ✓ Link | 79.47 | | SparseGPT (175B, 2:4 Sparsity) | 2023-01-02 |
| SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot | ✓ Link | 78.77 | | SparseGPT (175B, 4:8 Sparsity) | 2023-01-02 |
| PaLM: Scaling Language Modeling with Pathways | ✓ Link | 77.9 | | PaLM-540B (Zero-Shot) | 2022-04-05 |
| Training Compute-Optimal Large Language Models | ✓ Link | 77.7 | | Chinchilla (Zero-Shot) | 2022-03-29 |
| SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot | ✓ Link | 76.51 | | SparseGPT (175B, 50% Sparsity) | 2023-01-02 |
| Language Models are Few-Shot Learners | ✓ Link | 76.2 | 3.00 | GPT-3 175B (Zero-Shot) | 2020-05-28 |
| SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot | ✓ Link | 75.59 | | OPT-175B | 2023-01-02 |
| Scaling Language Models: Methods, Analysis & Insights from Training Gopher | ✓ Link | 74.5 | | Gopher 280B (zero-shot) | 2021-12-08 |
| Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior | ✓ Link | 74.3 | | PPT (k-Shuffle Dyck), 3B, Marin, 100B PT tokens | 2026-09-30 |
| Language Models are Few-Shot Learners | ✓ Link | 72.5 | 3.56 | GPT-3 13B (Zero-Shot) | 2020-05-28 |
| GLM: General Language Model Pretraining with Autoregressive Blank Infilling | ✓ Link | 72.35 | | GLM-XXLarge (bidirectional) | 2021-03-18 |
| RWKV: Reinventing RNNs for the Transformer Era | ✓ Link | 71.05 | 3.81 | RWKV-4 14B | 2023-05-22 |
| Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling | ✓ Link | 70.46 | | Pythia 12B (0-shot) | 2023-04-03 |
| Language Models are Few-Shot Learners | ✓ Link | 70.3 | 4.00 | GPT-3 6.7B (Zero-Shot) | 2020-05-28 |
| Introducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs | ✓ Link | 70.3 | | MPT-7B | 2023-05-05 |
| TIDE: Every Layer Knows the Token Beneath the Context | | 70.2 | | TIDE-8E (3B) | 2026-05-07 |
| | 69.7 | 3.99 | GPT-J-6B | |
| GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model | ✓ Link | 69.7 | 3.99 | GPT-J 6B | 2021-06-04 |
| Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality | ✓ Link | 69.7 | 4.10 | Mamba-2-2.7B | 2024-05-31 |
| Mamba: Linear-Time Sequence Modeling with Selective State Spaces | ✓ Link | 69.2 | 4.23 | Mamba-2.8B | 2023-12-01 |
| Scaling Language Models: Methods, Analysis & Insights from Training Gopher | ✓ Link | 68.5 | | Gopher 7.1B (zero-shot) | 2021-12-08 |
| Initialisation Determines the Basin: Efficient Codebook Optimisation for Extreme LLM Quantization | ✓ Link | 68.5 | 4.64 | OA-EM (Llama 3.2 3B) | 2026-04-09 |
| Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling | ✓ Link | 67.28 | | Pythia 6.9B (0-shot) | 2023-04-03 |
| GLM: General Language Model Pretraining with Autoregressive Blank Infilling | ✓ Link | 67.18 | | GLM-XXLarge (unidirectional) | 2021-03-18 |
| Language Models are Few-Shot Learners | ✓ Link | 67.1 | 4.60 | GPT-3 2.7B (Zero-Shot) | 2020-05-28 |
| Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism | ✓ Link | 66.51 | | Megatron-LM GPT-2 8.3B | 2019-09-17 |
| Language Models are Unsupervised Multitask Learners | ✓ Link | 63.24 | 8.63 | GPT-2 1.5B (Zero Shot) | 2019-02-14 |
| Cola2: Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation | | 63.0 | | Cola2 | 2026-10-22 |
| GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow | ✓ Link | 62.22 | 5.626 | GPT-Neo 2.7B | 2021-03-21 |
| Scaling Language Models: Methods, Analysis & Insights from Training Gopher | ✓ Link | 62.1 | | Gopher 1.4B (zero-shot) | 2021-12-08 |
| xLSTM: Extended Long Short-Term Memory | ✓ Link | 57.83 | 6.86 | xLSTM[1:0] 1.3B | 2024-05-07 |
| Universal Transformers | ✓ Link | 56.25 | | Universal Transformer (w/ dynamic halting) | 2018-07-10 |
| Towards Looped Models Done Right, Part II: Rethinking at Fixed Points | ✓ Link | 55.17 | | LoopedLM-P2 learned-depth lambda_H=0.01 (1.6B) | 2026-10-05 |
| Residual Shuffle-Exchange Networks for Fast Processing of Long Sequences | ✓ Link | 54.34 | | Residual Shuffle-Exchange network | 2020-04-06 |
| Safin-1: Safety from Within through Memory-Native State Evolution | ✓ Link | 52.57 | | Kimi Delta Attention with MARCH | 2026-08-31 |
| Kalman Delta Networks: Uncertainty-aware Associative Memory | ✓ Link | 51.99 | | Diagonal KDN + SWA (1.3B) | 2026-09-07 |
| Scaling Language Models: Methods, Analysis & Insights from Training Gopher | ✓ Link | 51.9 | | Gopher 417M (zero-shot) | 2021-12-08 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 51.7 | 10.2 | Mamba-3-MIMO-1.5B | 2026-03-16 |
| Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention | ✓ Link | 50.9 | 10.43 | Gated DeltaNet-2 (1.3B) | 2026-05-21 |
| Parallax: Parameterized Local Linear Attention for Language Modeling | ✓ Link | 50.26 | 10.8 | Parallax (1.7B) | 2026-05-27 |
| MeKi: Memory-based Expert Knowledge Injection for Efficient LLM Scaling | ✓ Link | 50.1 | | MeKi (4B) | 2026-02-03 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 49.5 | 11.8 | Mamba-3-MIMO-880M | 2026-03-16 |
| The State-Prediction Separation Hypothesis | ✓ Link | 49.5 | | SPS XL | 2026-07-01 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 49.4 | 10.9 | Mamba-3-SISO-1.5B | 2026-03-16 |
| dLLM: Simple Diffusion Language Modeling | ✓ Link | 49.3 | | ModernBERT-base-chat-v0.1 | 2026-02-26 |
| Broad Context Language Modeling as Reading Comprehension | | 49.0 | | Gated-Attention Reader (+ features) | 2016-10-26 |
| Gated Delta Networks: Improving Mamba2 with Delta Rule | ✓ Link | 48.76 | | Gated DeltaNet-H2 | 2024-12-09 |
| Maglev: Sliding Recurrent Memory | | 48.7 | 7.73 | Maglev (separate parameters, lambda=0.1) | 2026-08-05 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 47.2 | 12.9 | Mamba-3-SISO-880M | 2026-03-16 |
| Copy-Paste Language Models | ✓ Link | 46.92 | | Copy-Paste Language Model (1.7B, Full) | 2026-10-13 |
| Multi-Token Prediction Done Right | ✓ Link | 46.48 | | Soft-MTP (1.2B, zero-shot) | 2026-10-28 |
| LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation | ✓ Link | 45.92 | | LatentLoop (7B / 1B active) | 2026-10-14 |
| Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation | ✓ Link | 45.27 | | GRT Large (isoParams, zero-shot) | 2026-08-15 |
| UniPool: A Globally Shared Expert Pool for Mixture-of-Experts | ✓ Link | 44.94 | | UniPool (978M) | 2026-05-07 |
| Parcae: Scaling Laws For Stable Looped Language Models | ✓ Link | 43.22401 | | Parcae-770M (T=8, 0-shot) | 2026-04-14 |
| Error-Free Linear Attention is a Free Lunch: Exact Solution from Continuous-Time Dynamics | ✓ Link | 43.2 | 16.54 | EFLA | 2025-12-14 |
| Do Language Models Need a Trainable Input Embedding Table? Fixed Minimal Token Codes at 1.7B-Class Scale | | 42.75 | | ab_ext_binary16 | 2026-10-02 |
| Scaling Language Models: Methods, Analysis & Insights from Training Gopher | ✓ Link | 41.2 | | Gopher 117M (zero-shot) | 2021-12-08 |
| The Curse and Blessing of Mean Bias in FP4-Quantized LLM Training | | 38.62 | | Qwen3-0.6B (Averis FP4) | 2026-03-11 |
| LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation | ✓ Link | 38.27 | | LoopFormer (3x8) | 2026-02-11 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 34.0 | 32.1 | Mamba-3-MIMO-180M | 2026-03-16 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 32.5 | 37.7 | Mamba-3-SISO-180M | 2026-03-16 |
| MultiHashFormer: Hash-based Generative Language Models | ✓ Link | 30.27 | | MHF (H4B4K) | 2026-06-26 |
| Scaling Language Models: Methods, Analysis & Insights from Training Gopher | ✓ Link | 30.1 | | Gopher 44M (zero-shot) | 2021-12-08 |
| Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models | | 23.2 | | B frozen table, float | 2026-10-15 |
| SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot | ✓ Link | 0.02 | | OPT-175B (50% Sparsity) | 2023-01-02 |
| Test-Time Training with Self-Supervision for Generalization under Distribution Shifts | ✓ Link | 0.01 | | test | 2019-09-29 |
| Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling | ✓ Link | | 3.92 | Pythia 12B(Zero-Shot) | 2023-04-03 |
| Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling | ✓ Link | | 4.45 | Pythia 6.9B(Zero-Shot) | 2023-04-03 |
| Gated Delta Networks: Improving Mamba2 with Delta Rule | ✓ Link | | 12.12 | Gated DeltaNet-H1 | 2024-12-09 |
| Rethinking Selective Knowledge Distillation | ✓ Link | | 6.9 | SE-KD | 2026-02-01 |
| Memory Caching: RNNs with Growing Memory | | | 11.29 | Titans + GRM | 2026-02-27 |
| LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling | ✓ Link | | 46.93 | LangFlow (130M) | 2026-04-15 |
| Parcae: Scaling Laws For Stable Looped Language Models | ✓ Link | | 14.71 | Parcae (1.3B) | 2026-04-14 |
| MDN: Parallelizing Stepwise Momentum for Delta Linear Attention | ✓ Link | | 14.87 | MDN (1.3B) | 2026-05-07 |
| LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Models | ✓ Link | | 3.49 | Phi-4 (14B) with LoopUS | 2026-05-10 |
| Solve the Loop: Attractor Models for Language and Reasoning | ✓ Link | | 15.21 | Attractor Model (770M) | 2026-05-12 |
| Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation | ✓ Link | | 45.87 | Absorbing Uniform Diffusion (ELBO, Denoiser) | 2026-05-21 |
| SQ-format: A Unified Sparse-Quantized Hardware-friendly Data Format for LLMs | | | 2.64 | Llama-3-70B with SQ-format (W4A(SQ6), static, bb=16) | 2025-12-05 |
| Kalman Delta Networks: Uncertainty-aware Associative Memory | ✓ Link | | 9.75 | Diagonal KDN (1.3B) | 2026-09-07 |
| FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models | | | 11.39 | FactorEngram (1B) | 2026-09-28 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | | 19.6 | Mamba-3-SISO-440M | 2026-03-16 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | | 17.1 | Mamba-3-MIMO-440M | 2026-03-16 |
| Modular TTT: Rethinking Test-Time Training as Composable Modules | ✓ Link | | 12.19 | Modular TTT 1.45B (inner-product M) | 2026-08-07 |
| Fast Weight Attention for Continual Learning | ✓ Link | | 47.93 | Falcon-1A.1130M | 2026-08-27 |
| WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixing | ✓ Link | | 60.73 | WhiteMatter k=16 | 2026-09-27 |