| ST-MoE: Designing Stable and Transferable Sparse Expert Models | ✓ Link | 96.1 | ST-MoE-32B 269B (fine-tuned) | 2022-02-17 |
| UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark | ✓ Link | 91.3 | Unicorn 11B (fine-tuned) | 2021-03-24 |
| Task Compass: Scaling Multi-task Pre-training with Task Prefix | ✓ Link | 90.5 | CompassMTL 567M with Tailor | 2022-10-12 |
| UL2: Unifying Language Learning Paradigms | ✓ Link | 90.1 | UL2 20B | 2022-05-10 |
| Task Compass: Scaling Multi-task Pre-training with Task Prefix | ✓ Link | 89.6 | CompassMTL 567M | 2022-10-12 |
| Nemotron-4 340B Technical Report | ✓ Link | 89.50 | Nemotron-4-340B-Base | 2024-06-17 |
| UnifiedQA: Crossing Format Boundaries With a Single QA System | ✓ Link | 89.4 | UnifiedQA 11B (fine-tuned) | 2020-05-02 |
| The Claude 3 Model Family: Opus, Sonnet, Haiku | | 88.5 | Claude 3 Opus (5-shot) | 2024-03-04 |
| InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models | ✓ Link | 87.8 | InternVL3-78B | 2025-04-14 |
| GPT-4 Technical Report | ✓ Link | 87.5 | GPT-4 (5-shot) | 2023-03-15 |
| The Falcon Series of Open Language Models | ✓ Link | 87.1 | Falcon-180B (5-shot) | 2023-11-28 |
| Task Compass: Scaling Multi-task Pre-training with Task Prefix | ✓ Link | 87 | ExDeBERTa 567M | 2022-10-12 |
| MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts | ✓ Link | 86.3 | LLaMA-2 13B + MixLoRA | 2024-04-22 |
| Mixture-of-Subspaces in Low-Rank Adaptation | ✓ Link | 85.8 | LLaMA3 8B+MoSLoRA | 2024-06-16 |
| Qwen2 Technical Report | ✓ Link | 85.1 | Qwen2-72B | 2024-07-15 |
| Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models | ✓ Link | 84.5 | Nemotron-H-56B-Base (5-shot) | 2025-04-04 |
| MiMo-V2-Flash Technical Report | ✓ Link | 83.8 | MiMo-V2-Flash (Base) | 2026-01-06 |
| Gemma 2: Improving Open Language Models at a Practical Size | ✓ Link | 83.7 | Gemma 2 27B (5-shot) | 2024-07-31 |
| PaLM 2 Technical Report | ✓ Link | 83.0 | PaLM 2-L (1-shot) | 2023-05-17 |
| 2 OLMo 2 Furious | ✓ Link | 83.0 | OLMo 2 32B | 2024-12-31 |
| Jamba: A Hybrid Transformer-Mamba Language Model | ✓ Link | 82.5 | Jamba (5-shot) | 2024-03-28 |
| MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts | ✓ Link | 82.1 | LLaMA-3 8B + MixLoRA | 2024-04-22 |
| CoWindow Attention: Full Causal Coverage Is a Collective Property | | 81.72 | CoWA (32B) | 2026-09-26 |
| ST-MoE: Designing Stable and Transferable Sparse Expert Models | ✓ Link | 81.7 | ST-MoE-L 4.1B (fine-tuned) | 2022-02-17 |
| GPT-4 Technical Report | ✓ Link | 81.6 | GPT-3.5 (5-shot) | 2023-03-15 |
| Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone | ✓ Link | 81.5 | Phi-3-Medium (5-shot) | 2024-04-22 |
| 2 OLMo 2 Furious | ✓ Link | 81.5 | OLMo 2 13B | 2024-12-31 |
| PaLM: Scaling Language Modeling with Pathways | ✓ Link | 81.1 | PaLM 540B (0-shot) | 2022-04-05 |
| Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks | ✓ Link | 80.9 | Camelidae-8×34B | 2024-01-05 |
| Motif 3: Technical Report | ✓ Link | 80.9 | Motif 3 Base (5-shot) | 2026-08-10 |
| MassAlloc Attention: Let Attention Allocate Its Own Compute | ✓ Link | 80.87 | MALA (32B) | 2026-09-26 |
| Arcee Trinity Large Technical Report | ✓ Link | 80.82 | Trinity Large Base (5-shot) | 2026-02-19 |
| Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models | ✓ Link | 80.5 | Nemotron-H-8B-Base (5-shot) | 2025-04-04 |
| Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning | ✓ Link | 79.64 | Nemotron 3 Nano 30B-A3B Base | 2025-12-23 |
| Qwen2 Technical Report | ✓ Link | 79.5 | Qwen2-57B-A14B | 2024-07-15 |
| PaLM 2 Technical Report | ✓ Link | 79.2 | PaLM 2-M (1-shot) | 2023-05-17 |
| WinoGrande: An Adversarial Winograd Schema Challenge at Scale | ✓ Link | 79.1 | RoBERTa-Winogrande 355M (fine-tuned) | 2019-07-24 |
| Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters | ✓ Link | 79.1 | Step-3.5-Flash Base | 2026-02-11 |
| Compact Language Models via Pruning and Knowledge Distillation | ✓ Link | 79.0 | Minitron 8B | 2024-07-19 |
| Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation | ✓ Link | 78.7 | T5Gemma 9B-9B PrefixLM | 2025-04-08 |
| Chameleon: Mixed-Modal Early-Fusion Foundation Models | ✓ Link | 78.5 | Chameleon 34B | 2024-05-16 |
| Falcon2-11B Technical Report | | 78.30 | Falcon2-11B (5-shot) | 2024-07-20 |
| Nemotron-4 15B Technical Report | ✓ Link | 78.0 | Nemotron-4 15B | 2024-02-26 |
| PaLM 2 Technical Report | ✓ Link | 77.9 | PaLM 2-S (1-shot) | 2023-05-17 |
| LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Models | ✓ Link | 77.5 | Phi-4 (14B) with LoopUS | 2026-05-10 |
| The Zamba2 Suite: Technical Report | ✓ Link | 77.3 | Zamba2-7B | 2024-11-22 |
| Mixtral of Experts | ✓ Link | 77.2 | Mixtral 8x7B (0-shot) | 2024-01-08 |
| 2 OLMo 2 Furious | ✓ Link | 77.2 | OLMo 2 7B | 2024-12-31 |
| Advancing Polish Language Modeling through Tokenizer Optimization in the Bielik v3 7B and 11B Series | | 77.19 | Bielik-11B-v3.0-Instruct | 2026-04-12 |
| Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs | ✓ Link | 77.03 | SRR (DeepSeekMoE-16B-A3B, WUDI-Merge) | 2026-09-26 |
| PaLM: Scaling Language Modeling with Pathways | ✓ Link | 77.0 | PaLM 62B (0-shot) | 2022-04-05 |
| PaLM: Scaling Language Modeling with Pathways | ✓ Link | 77.0 | PaLM-cont 62B (0-shot) | 2022-04-05 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | 77.0 | LLaMA 65B (0-shot) | 2023-02-27 |
| MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts | ✓ Link | 76.8 | LLaMA-2 7B + MixLoRA | 2024-04-22 |
| LoopRPT: Reinforcement Pre-Training for Looped Language Models | | 76.47 | Ouro (2.6B) LoopRPT | 2026-03-20 |
| Zamba: A Compact 7B SSM Hybrid Model | ✓ Link | 76.4 | Zamba 7B | 2024-05-26 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | 76.0 | LLaMA 33B (0-shot) | 2023-02-27 |
| DeepSeek LLM: Scaling Open-Source Language Models with Longtermism | ✓ Link | 76.0 | DeepSeek LLM 67B Chat | 2024-01-05 |
| FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach | ✓ Link | 75.69 | FineRMoE (Qwen2.5-7B) | 2026-03-09 |
| Mistral 7B | ✓ Link | 75.3 | Mistral 7B (0-shot) | 2023-10-10 |
| The Claude 3 Model Family: Opus, Sonnet, Haiku | | 75.1 | Claude 3 Sonnet (5-shot) | 2024-03-04 |
| Training Compute-Optimal Large Language Models | ✓ Link | 74.9 | Chinchilla 70B (0-shot) | 2022-03-29 |
| Large Language Diffusion Models | ✓ Link | 74.8 | LLaDA 8B Base (5-shot) | 2025-02-14 |
| Dream 7B: Diffusion Large Language Models | | 74.5 | Dream 7B (5-shot) | 2025-08-21 |
| Mixtral of Experts | ✓ Link | 74.2 | Mistral 7B (0-shot) | 2024-01-08 |
| The Claude 3 Model Family: Opus, Sonnet, Haiku | | 74.2 | Claude 3 Haiku (5-shot) | 2024-03-04 |
| Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See | | 74.2 | Sophea-Qwen3.6-v1 | 2026-08-18 |
| Textbooks Are All You Need II: phi-1.5 technical report | ✓ Link | 74.0 | phi-1.5-web 1.3B (zero-shot) | 2023-09-11 |
| Compact Language Models via Pruning and Knowledge Distillation | ✓ Link | 74.0 | Minitron 4B | 2024-07-19 |
| Who Should Lead Decoding Now? Tracking Reliable Trajectories for Ensembling Masked Diffusion Language Models | | 73.95 | TIE (LLaDA + Dream; Top-1 probability) | 2026-06-15 |
| Falcon Mamba: The First Competitive Attention-free 7B Language Model | ✓ Link | 73.64 | Falcon Mamba 7B | 2024-10-07 |
| RecurrentGemma: Moving Past Transformers for Efficient Open Language Models | ✓ Link | 73.6 | RecurrentGemma-9B | 2024-04-11 |
| DFM Mimir v1: An Open HRM Delivering Frontier Performance at 1B Parameters Using Only Permissible Post-Training Data | ✓ Link | 73.5 | Mimir 1B | 2026-08-13 |
| UnifiedQA: Crossing Format Boundaries With a Single QA System | ✓ Link | 73.3 | Unified QA 406M (fine-tuned) | 2020-05-02 |
| Apertus: Democratizing Open and Compliant LLMs for Global Language Environments | ✓ Link | 73.3 | Apertus-70B | 2025-09-17 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | 73.0 | LLaMA 13B (0-shot) | 2023-02-27 |
| Finetuned Language Models Are Zero-Shot Learners | ✓ Link | 72.8 | FLAN 137B (few-shot, k=16) | 2021-09-03 |
| HRM-Text: Efficient Pretraining Beyond Scaling | ✓ Link | 72.4 | HRM-Text-1B | 2026-05-20 |
| Decoding Looped Transformers Better for (Almost) Free | | 72.31 | LoopCD-Ouro-2.6B (adaptive) | 2026-10-01 |
| Gemma: Open Models Based on Gemini Research and Technology | ✓ Link | 72.3 | Gemma 7B | 2024-03-13 |
| HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing | | 72.1 | HySparse (80B MoE) | 2026-02-03 |
| BitNet b1.58 2B4T Technical Report | ✓ Link | 71.90 | BitNet b1.58 2B4T | 2025-04-16 |
| T5Gemma 2: Seeing, Reading, and Understanding Longer | | 71.6 | T5Gemma 2 4B-4B | 2025-12-16 |
| Generative Data Augmentation for Commonsense Reasoning | ✓ Link | 71.4 | G-DAUG-Combo + RoBERTa-Large | 2020-04-24 |
| OPT: Open Pre-trained Transformer Language Models | ✓ Link | 71.4 | OPT-175B | 2022-05-02 |
| Finetuned Language Models Are Zero-Shot Learners | ✓ Link | 71.2 | FLAN 137B (0-shot) | 2021-09-03 |
| | 70.8 | RWKV v5 Eagle 7B | |
| Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging | ✓ Link | 70.8 | MERIT (Proposed, 2D split, 4 groups, Qwen2.5-3B) | 2026-06-01 |
| Revisiting Complete Reasoning Traces for Post-Training | ✓ Link | 70.64 | Qwen3-4B-Base + E-SFT (OpenThoughts3-100K) | 2026-09-07 |
| Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM | ✓ Link | 70.6 | Branch-Train-MiX 4x7B (sampling top-1 expert) | 2024-03-12 |
| Language Models are Few-Shot Learners | ✓ Link | 70.2 | GPT-3 175B (0-shot) | 2020-05-28 |
| OLMoE: Open Mixture-of-Experts Language Models | ✓ Link | 70.2 | OLMoE-1B-7B (5-shot) | 2024-09-03 |
| Scaling Language Models: Methods, Analysis & Insights from Training Gopher | ✓ Link | 70.1 | Gopher 280B (0-shot) | 2021-12-08 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | 70.1 | LLaMA 7B (0-shot) | 2023-02-27 |
| Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler | ✓ Link | 70.0 | PowerLM-3B | 2024-08-23 |
| Moshi: a speech-text foundation model for real-time dialogue | ✓ Link | 70.0 | Helium | 2024-09-17 |
| Universal YOCO for Efficient Depth Scaling | | 69.85 | YOCO-U (Equal Steps) | 2026-04-01 |
| GPT-NeoX-20B: An Open-Source Autoregressive Language Model | ✓ Link | 68.3 | GPT-NeoX-20B (5-shot) | 2022-04-14 |
| Introducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs | ✓ Link | 68.3 | MPT-7B | 2023-05-05 |
| OLMo: Accelerating the Science of Language Models | ✓ Link | 67.9 | OLMo 7B | 2024-02-01 |
| RecurrentGemma: Moving Past Transformers for Efficient Open Language Models | ✓ Link | 67.8 | RecurrentGemma-2B | 2024-04-11 |
| Differential Transformer | ✓ Link | 67.1 | Diff-3B | 2024-10-07 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | 67 | BLOOM 176B (1-shot) | 2023-03-30 |
| Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs | ✓ Link | 67.0 | Phi-4-Mini (5-shot) | 2025-02-24 |
| daVinci-LLM:Towards the Science of Pretraining | ✓ Link | 66.77 | daVinci-LLM-3B | 2026-03-28 |
| Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling | ✓ Link | 66.6 | Pythia 12B (5-shot) | 2023-04-03 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | 66.1 | OPT 66B (1-shot) | 2023-03-30 |
| MobileMoE: Scaling On-Device Mixture of Experts | | 66.1 | MobileMoE-L (922M) | 2026-05-26 |
| GPT-SW3: An Autoregressive Language Model for the Nordic Languages | | 65.6 | GPT-SW3 40B | 2023-05-22 |
| GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model | ✓ Link | 65.3 | GPT-J 6B | 2021-06-04 |
| WinoGrande: An Adversarial Winograd Schema Challenge at Scale | ✓ Link | 64.9 | BERT-Winogrande 345M (fine-tuned) | 2019-07-24 |
| Scaling Laws for Looped Mixture of Experts | | 64.4 | A0.3B-1.3B LoopMoE (R=4) | 2026-09-30 |
| ECHO: Efficient Causal Self-Recursion Beyond Auxiliary Multi-Token Prediction | ✓ Link | 64.4 | ECHO-1.5(3) | 2026-10-17 |
| Scaling Inherently Interpretable Language Models | ✓ Link | 64.2 | Steerling-8B | 2026-08-06 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | 64.1 | Bloomberg GPT (one-shot) | 2023-03-30 |
| Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality | ✓ Link | 64.0 | Mamba-2-2.7B | 2024-05-31 |
| LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation | ✓ Link | 63.93 | LatentLoop (7B / 1B active) | 2026-10-14 |
| Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling | ✓ Link | 63.9 | Pythia 12B (0-shot) | 2023-04-03 |
| Mamba: Linear-Time Sequence Modeling with Selective State Spaces | ✓ Link | 63.5 | Mamba-2.8B | 2023-12-01 |
| Efficient Pre-Training with Token Superposition | ✓ Link | 63.0 | MoE TST (10B A1B) | 2026-05-07 |
| RWKV: Reinventing RNNs for the Transformer Era | ✓ Link | 62.98 | RWKV-4 14B | 2023-05-22 |
| Disentangling Representation Evolution in Transformers through Directional Decomposition | ✓ Link | 62.6 | Transformer Geometry 2.7B V-Para (5-shot) | 2026-09-14 |
| Full-bandwidth transformer | | 62.59 | Full-bandwidth transformer 1B (200B, 1 feedback pass) | 2026-08-09 |
| Distributionally Robust Mixture-of-Experts Training | ✓ Link | 62.35 | DRMoET (1.7B active, 10.3B total, eta=0.01) | 2026-10-05 |
| Exploring the Benefits of Training Expert Language Models over Instruction Tuning | ✓ Link | 61.60 | RoE-3B | 2023-02-07 |
| MeKi: Memory-based Expert Knowledge Injection for Efficient LLM Scaling | ✓ Link | 61.6 | MeKi (4B) | 2026-02-03 |
| Multi-Head Low-Rank Attention | ✓ Link | 61.48 | MLRA-4 (2.9B) | 2026-03-02 |
| Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090 | ✓ Link | 61.48 | Puro-2B CMA215k average | 2026-08-27 |
| Kalman Delta Networks: Uncertainty-aware Associative Memory | ✓ Link | 61.33 | Isotropic KDN (1.3B) | 2026-09-07 |
| Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling | ✓ Link | 60.9 | Pythia 6.9B (0-shot) | 2023-04-03 |
| Parallax: Parameterized Local Linear Attention for Language Modeling | ✓ Link | 60.77 | Parallax (1.7B) | 2026-05-27 |
| xLSTM: Extended Long Short-Term Memory | ✓ Link | 60.62 | xLSTM[1:0] 1.3B | 2024-05-07 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | 60.6 | GPT-NeoX (one-shot) | 2023-03-30 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 60.6 | Mamba-3-MIMO-1.5B | 2026-03-16 |
| Variable-Width Transformers | ✓ Link | 60.2 | 2B > <former | 2026-06-16 |
| Sumi: Open Uniform Diffusion Language Model from Scratch | ✓ Link | 60.0 | Sumi-7B (5-shot) | 2026-06-17 |
| Modular TTT: Rethinking Test-Time Training as Composable Modules | ✓ Link | 59.98 | Modular TTT 1.45B (inner-product M) | 2026-08-07 |
| LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions | ✓ Link | 59.9 | FLAN-T5-Large 783M | 2023-04-27 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 59.9 | Mamba-3-MIMO-880M | 2026-03-16 |
| Copy-Paste Language Models | ✓ Link | 59.59 | Copy-Paste Language Model (1.7B, Hybrid 3:1) | 2026-10-13 |
| Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling | ✓ Link | 59.4 | Pythia 2.8B (0-shot) | 2023-04-03 |
| SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model | ✓ Link | 59.4 | SmolLM2 1.7B | 2025-02-04 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 59.4 | Mamba-3-SISO-1.5B | 2026-03-16 |
| Data Science and Technology Towards AGI Part I: Tiered Data Management | ✓ Link | 59.04 | MiniCPM-1.2B (Math-L3) | 2026-02-09 |
| WinoGrande: An Adversarial Winograd Schema Challenge at Scale | ✓ Link | 58.9 | RoBERTa-DPR 355M (0-shot) | 2019-07-24 |
| OLMo: Accelerating the Science of Language Models | ✓ Link | 58.9 | OLMo 1B | 2024-02-01 |
| SmolLM3: smol, multilingual, long-context reasoner | ✓ Link | 58.88 | SmolLM3 3B Base | 2025-07-08 |
| Parcae: Scaling Laws For Stable Looped Language Models | ✓ Link | 58.800316 | Parcae-1.3B (T=8, 0-shot) | 2026-04-14 |
| Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema | | 58.7 | ALBERT-xxlarge 235M | 2021-04-16 |
| Maglev: Sliding Recurrent Memory | | 58.7 | Maglev (lambda=0.1) | 2026-08-05 |
| Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners | ✓ Link | 58.56 | Flipped-3B | 2022-10-06 |
| Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention | ✓ Link | 58.56 | Gated DeltaNet-2 (1.3B) | 2026-05-21 |
| FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models | | 58.48 | FactorEngram (1B) | 2026-09-28 |
| Gated Delta Networks: Improving Mamba2 with Delta Rule | ✓ Link | 58.40 | Gated DeltaNet-H1 | 2024-12-09 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 58.4 | Mamba-3-SISO-880M | 2026-03-16 |
| LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions | ✓ Link | 58.3 | GPT-2-XL 1.5B | 2023-04-27 |
| MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookup | ✓ Link | 58.25 | MoME (100B tokens) | 2026-09-14 |
| olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models | ✓ Link | 58.0 | OLMo-2-7B-1124 (olmOCR-peS2o) | 2025-02-25 |
| The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning | ✓ Link | 57.5 | T0-3B (CoT fine-tuned) | 2023-05-23 |
| Language Models are Few-Shot Learners | ✓ Link | 57.4 | GPT-3 Large 760M (0-shot) | 2020-05-28 |
| Memory Caching: RNNs with Growing Memory | | 57.4 | Titans + GRM | 2026-02-27 |
| Group Representational Position Encoding | ✓ Link | 56.99 | GRAPE-AP | 2025-12-08 |
| GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow | ✓ Link | 56.50 | GPT-Neo 2.7B | 2021-03-21 |
| OpenESM: Energy-Steered Models Are Scalable Foundation Learners | ✓ Link | 56.35 | OpenESM DCLM d26 | 2026-10-07 |
| Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema | | 56.3 | RoBERTa-base 125M | 2021-04-16 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 56.3 | Mamba-3-MIMO-440M | 2026-03-16 |
| Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall | ✓ Link | 56.1 | Switch Distillation (1B; 7B teacher, after SFT) | 2026-09-01 |
| LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions | ✓ Link | 56 | LaMini-F-T5 783M | 2023-04-27 |
| LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions | ✓ Link | 56 | LaMini-GPT 1.5B | 2023-04-27 |
| SmolLM - blazingly fast and remarkably powerful | ✓ Link | 56.0 | SmolLM 1.7B | 2024-07-16 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 55.8 | Mamba-3-SISO-440M | 2026-03-16 |
| Language Is Not All You Need: Aligning Perception with Language Models | ✓ Link | 55.7 | KOSMOS-1 (4-shot) | 2023-02-27 |
| Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema | | 55.6 | BERT-large 345M | 2021-04-16 |
| Kosmos-2: Grounding Multimodal Large Language Models to the World | ✓ Link | 55.6 | KOSMOS-2 | 2023-06-26 |
| Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training | ✓ Link | 55.43 | Qwen3 (1.7B) with DeMix (224 proxies) | 2026-01-31 |
| Knowledge-in-Context: Towards Knowledgeable Semi-Parametric Language Models | | 55.30 | KiC-770M | 2022-10-28 |
| LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions | ✓ Link | 55.2 | T5-Large 738M | 2023-04-27 |
| Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema | | 54.9 | RoBERTa-large 355M | 2021-04-16 |
| LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions | ✓ Link | 54.9 | LaMini-T5 738M | 2023-04-27 |
| Releasing Persimmon-8B | ✓ Link | 54.6 | Persimmon-8B-FT | 2023-09-07 |
| Efficient Language Modeling with Sparse all-MLP | | 54.3 | sMLP – deterministic 9.4B (0-shot) | 2022-03-14 |
| Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts | ✓ Link | 53.6 | LOOM (1.7B, 9 loops) | 2026-10-01 |
| LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation | ✓ Link | 53.43 | LoopFormer (3x4) | 2026-02-11 |
| Efficient Language Modeling with Sparse all-MLP | | 53.4 | Switch Transformer 9B (0-shot) | 2022-03-14 |
| Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction | | 53.35 | GVA + DRoPE d_r=16 | 2026-09-08 |
| Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation | ✓ Link | 53.28 | GRT Large (isoParams, zero-shot) | 2026-08-15 |
| Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema | | 53.1 | BERT-base 110M | 2021-04-16 |
| Fast Weight Attention for Continual Learning | ✓ Link | 52.96 | Falcon-1A.2130M | 2026-08-27 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 52.9 | Mamba-3-MIMO-180M | 2026-03-16 |
| What Makes Recurrence Effective in Looped Language Models? | | 52.88 | BaseLoop (Llama3.1-1B, 3x4, Muon) | 2026-09-29 |
| Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema | | 52.8 | ALBERT-base 11M | 2021-04-16 |
| Normalized Low-Rank Adaptation | ✓ Link | 52.72 | MLA342M NoRA-init | 2026-08-31 |
| Safin-1: Safety from Within through Memory-Native State Evolution | ✓ Link | 52.6 | Gated DeltaNet with MARCH | 2026-08-31 |
| Dynamic Linear Attention | | 52.4 | Gated DeltaNet-1.3B w/ DLA | 2026-06-09 |
| UniPool: A Globally Shared Expert Pool for Mixture-of-Experts | ✓ Link | 52.17 | UniPool (978M) | 2026-05-07 |
| Error-Free Linear Attention is a Free Lunch: Exact Solution from Continuous-Time Dynamics | ✓ Link | 52.1 | EFLA | 2025-12-14 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 52.0 | Mamba-3-SISO-180M | 2026-03-16 |
| WinoGrande: An Adversarial Winograd Schema Challenge at Scale | ✓ Link | 51.9 | BERT-large 345M (0-shot) | 2019-07-24 |
| Efficient Language Modeling with Sparse all-MLP | | 51.7 | HASH Layers 10B (0-shot) | 2022-03-14 |
| WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixing | ✓ Link | 51.38 | WhiteMatter k=16 | 2026-09-27 |
| NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction | ✓ Link | 51.22 | NCP-ArchPreview Stage-1 | 2026-09-09 |
| Efficient Language Modeling with Sparse all-MLP | | 51.1 | Gshard 9B (0-shot) | 2022-03-14 |
| WinoGrande: An Adversarial Winograd Schema Challenge at Scale | ✓ Link | 51 | BERT-DPR 345M (0-shot) | 2019-07-24 |
| Efficient Language Modeling with Sparse all-MLP | | 51 | Base Layers 10B (0-shot) | 2022-03-14 |
| Looped Uniform Diffusion Language Models | ✓ Link | 50.83 | Looped UDLM | 2026-10-21 |
| Daedalus-150M: A Convolution-Attention Hybrid Designed for CPU Inference | ✓ Link | 50.04 | Daedalus-150M | 2026-08-20 |
| WinoGrande: An Adversarial Winograd Schema Challenge at Scale | ✓ Link | 50 | RoBERTa-large 355M (0-shot) | 2019-07-24 |
| Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema | | 50 | Random baseline | 2021-04-16 |
| PLDR-LLMs Reason At Self-Organized Criticality | ✓ Link | 49.72 | PLDRv51-SOC-110M-5 | 2026-03-12 |