| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | 60.2 | LLaMA-65B | 2023-02-27 |
| OPT: Open Pre-trained Transformer Language Models | ✓ Link | 58.6 | OPT-175B | 2022-05-02 |
| Language Models are Few-Shot Learners | ✓ Link | 57.6 | GPT-3 175B (zero-shot) | 2020-05-28 |
| Language Models are Few-Shot Learners | ✓ Link | 55.6 | GPT-3 13B (zero-shot) | 2020-05-28 |
| Chameleon: Mixed-Modal Early-Fusion Foundation Models | ✓ Link | 54.0 | Chameleon 34B | 2024-05-16 |
| Moshi: a speech-text foundation model for real-time dialogue | ✓ Link | 53.6 | Helium | 2024-09-17 |
| Language Models are Few-Shot Learners | ✓ Link | 50.4 | GPT-3 6.7B (zero-shot) | 2020-05-28 |
| OLMo: Accelerating the Science of Language Models | ✓ Link | 50.4 | OLMo 7B | 2024-02-01 |
| Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models | ✓ Link | 48.8 | Nemotron-H-47B-Base | 2025-04-04 |
| The Zamba2 Suite: Technical Report | ✓ Link | 48.2 | Zamba2-7B | 2024-11-22 |
| Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models | ✓ Link | 47.2 | Nemotron-H-8B-Base | 2025-04-04 |
| Language Models are Few-Shot Learners | ✓ Link | 46.8 | GPT-3 XL (zero-shot) | 2020-05-28 |
| OLMo: Accelerating the Science of Language Models | ✓ Link | 46.4 | OLMo 1B | 2024-02-01 |
| Zamba: A Compact 7B SSM Hybrid Model | ✓ Link | 44.6 | Zamba 7B | 2024-05-26 |
| Parcae: Scaling Laws For Stable Looped Language Models | ✓ Link | 40.666669 | Parcae-770M (T=8, 0-shot) | 2026-04-14 |
| SmolLM3: smol, multilingual, long-context reasoner | ✓ Link | 40.6 | SmolLM3 3B Base | 2025-07-08 |
| Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality | ✓ Link | 38.8 | Mamba-2-2.7B | 2024-05-31 |
| Disentangling Representation Evolution in Transformers through Directional Decomposition | ✓ Link | 38.2 | Transformer Geometry 2.7B V-Para (zero-shot) | 2026-09-14 |
| Language Models are Few-Shot Learners | ✓ Link | 35.6 | GPT-3 Small (zero-shot) | 2020-05-28 |
| Full-bandwidth transformer | | 35.0 | Full-bandwidth transformer 1B (200B, 1 feedback pass) | 2026-08-09 |
| Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts | ✓ Link | 34.8 | LOOM (1.7B, 12 loops) | 2026-10-01 |
| Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction | | 33.6 | GVA + DRoPE d_r=16 | 2026-09-08 |
| Fast Weight Attention for Continual Learning | ✓ Link | 33.2 | Falcon-1A.2130M | 2026-08-27 |
| Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation | ✓ Link | 30.2 | GRT Large (isoParams, zero-shot) | 2026-08-15 |
| Differential Transformer | ✓ Link | 29.0 | Diff-3B | 2024-10-07 |
| GPT-SW3: An Autoregressive Language Model for the Nordic Languages | | 27.4 | GPT-SW3 40B | 2023-05-22 |
| LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation | ✓ Link | 26.8 | LoopFormer (3x8) | 2026-02-11 |
| ModuleFormer: Modularity Emerges from Mixture-of-Experts | ✓ Link | 23.8 | MoLM-8B-K2 (0-shot) | 2023-06-07 |
| What Makes Recurrence Effective in Looped Language Models? | | 20.8 | BaseLoop (Llama3.1-1B, 3x4, Muon) | 2026-09-29 |