| ST-MoE: Designing Stable and Transferable Sparse Expert Models | ✓ Link | 95.2 | ST-MoE-32B 269B (fine-tuned) | 2022-02-17 |
| Mixture-of-Subspaces in Low-Rank Adaptation | ✓ Link | 90.5 | LLaMA 3 8B+MoSLoRA (fine-tuned) | 2024-06-16 |
| PaLM 2 Technical Report | ✓ Link | 89.7 | PaLM 2-L (1-shot) | 2023-05-17 |
| Gemma 2: Improving Open Language Models at a Practical Size | ✓ Link | 88.6 | Gemma 2 27B | 2024-07-31 |
| PaLM 2 Technical Report | ✓ Link | 88.0 | PaLM 2-M (1-shot) | 2023-05-17 |
| MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts | ✓ Link | 86.5 | LLaMA-3 8B + MixLoRA | 2024-04-22 |
| Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks | ✓ Link | 86.2 | Camelidae-8×34B | 2024-01-05 |
| PaLM 2 Technical Report | ✓ Link | 85.6 | PaLM 2-S (1-shot) | 2023-05-17 |
| Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation | ✓ Link | 85.3 | T5Gemma 9B-9B PrefixLM | 2025-04-08 |
| Stay on topic with Classifier-Free Guidance | | 84.2 | LLaMA 65B + CFG (0-shot) | 2023-06-30 |
| OLMoE: Open Mixture-of-Experts Language Models | ✓ Link | 84.2 | OLMoE-1B-7B (5-shot) | 2024-09-03 |
| Chameleon: Mixed-Modal Early-Fusion Foundation Models | ✓ Link | 84.1 | Chameleon 34B | 2024-05-16 |
| Dream 7B: Diffusion Large Language Models | | 83.9 | Dream 7B | 2025-08-21 |
| Galactica: A Large Language Model for Science | ✓ Link | 83.8 | GAL 120B (0-shot) | 2022-11-16 |
| MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts | ✓ Link | 83.5 | LLaMA-2 13B + MixLoRA | 2024-04-22 |
| LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Models | ✓ Link | 83.5 | Phi-4 (14B) with LoopUS | 2026-05-10 |
| Stay on topic with Classifier-Free Guidance | | 83.2 | LLaMA 30B + CFG (0-shot) | 2023-06-30 |
| Mixtral of Experts | ✓ Link | 83.1 | Mixtral 8x7B (0-shot) | 2024-01-08 |
| The Zamba2 Suite: Technical Report | ✓ Link | 82.0 | Zamba2-7B | 2024-11-22 |
| River-LLM: Large Language Model Seamless Exit Based on KV Share | | 82.0 | River-LLM (Llama3.1 8B, τ=0.5) | 2026-04-20 |
| MobileMoE: Scaling On-Device Mixture of Experts | | 81.9 | MobileMoE-L (922M) | 2026-05-26 |
| DeepSeek LLM: Scaling Open-Source Language Models with Longtermism | ✓ Link | 81.6 | DeepSeek LLM 67B Chat | 2024-01-05 |
| Gemma: Open Models Based on Gemini Research and Technology | ✓ Link | 81.5 | Gemma 7B | 2024-03-13 |
| Nemotron-4 15B Technical Report | ✓ Link | 80.9 | Nemotron-4 15B | 2024-02-26 |
| Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation | ✓ Link | 80.72 | Taylor-Calibrate qwen3_8b 700M_ga_s2_taylor | 2026-06-15 |
| Finetuned Language Models Are Zero-Shot Learners | ✓ Link | 80.7 | FLAN 137B (few-shot, k=14) | 2021-09-03 |
| Mixtral of Experts | ✓ Link | 80.5 | Mistral 7B (0-shot) | 2024-01-08 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | 80.0 | LLaMA 33B (0-shot) | 2023-02-27 |
| Mistral 7B | ✓ Link | 80.0 | Mistral 7B (0-shot) | 2023-10-10 |
| HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Experts | | 79.9 | Qwen 3.5-122B-A10B (HodgeCover+Wanda) | 2026-05-13 |
| Finetuned Language Models Are Zero-Shot Learners | ✓ Link | 79.6 | FLAN 137B (0-shot) | 2021-09-03 |
| Triplet-Block Diffusion RWKV | ✓ Link | 79.3 | B3D-RWKV-7.2B | 2026-05-25 |
| Stay on topic with Classifier-Free Guidance | | 79.1 | LLaMA 13B + CFG (0-shot) | 2023-06-30 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | 78.9 | LLaMA 65B (0-shot) | 2023-02-27 |
| RecurrentGemma: Moving Past Transformers for Efficient Open Language Models | ✓ Link | 78.8 | RecurrentGemma-9B | 2024-04-11 |
| MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts | ✓ Link | 77.7 | LLaMA-2 7B + MixLoRA | 2024-04-22 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 76.5 | Mamba-3-MIMO-1.5B | 2026-03-16 |
| Textbooks Are All You Need II: phi-1.5 technical report | ✓ Link | 76.1 | phi-1.5-web 1.3B (0-shot) | 2023-09-11 |
| Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts | ✓ Link | 76.0 | Upcycled 32→64 (100% CPT) | 2026-04-21 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | 75.93 | BLOOM 176B (1-shot) | 2023-03-30 |
| TIDE: Every Layer Knows the Token Beneath the Context | | 75.5 | TIDE-8E (3B) | 2026-05-07 |
| ST-MoE: Designing Stable and Transferable Sparse Expert Models | ✓ Link | 75.4 | ST-MoE-L 4.1B (fine-tuned) | 2022-02-17 |
| GLaM: Efficient Scaling of Language Models with Mixture-of-Experts | | 74.8 | GLaM (64B/64E) (5-shot) | 2021-12-13 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | 74.8 | LLaMA 13B (0-shot) | 2023-02-27 |
| Kalman Delta Networks: Uncertainty-aware Associative Memory | ✓ Link | 74.71 | Isotropic KDN (1.3B) | 2026-09-07 |
| GPT-NeoX-20B: An Open-Source Autoregressive Language Model | ✓ Link | 74.6 | GPT-NeoX-20B (5-shot) | 2022-04-14 |
| Zamba: A Compact 7B SSM Hybrid Model | ✓ Link | 74.5 | Zamba 7B | 2024-05-26 |
| T5Gemma 2: Seeing, Reading, and Understanding Longer | | 74.3 | T5Gemma 2 4B-4B | 2025-12-16 |
| Efficient Pre-Training with Token Superposition | ✓ Link | 74.2 | MoE TST (10B A1B) | 2026-05-07 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | 73.99 | Bloomberg GPT 50B (1-shot) | 2023-03-30 |
| Modular TTT: Rethinking Test-Time Training as Composable Modules | ✓ Link | 73.74 | Modular TTT 1.45B (inner-product M-SiLU) | 2026-08-07 |
| Jamba: A Hybrid Transformer-Mamba Language Model | ✓ Link | 73.5 | Jamba (0-shot) | 2024-03-28 |
| Parallax: Parameterized Local Linear Attention for Language Modeling | ✓ Link | 73.27 | Parallax (1.7B) | 2026-05-27 |
| VersatileFFN: Achieving Parameter Efficiency in LLMs via Adaptive Wide-and-Deep Reuse | ✓ Link | 73.16 | VersatileFFN | 2025-12-16 |
| RecurrentGemma: Moving Past Transformers for Efficient Open Language Models | ✓ Link | 72.9 | RecurrentGemma-2B | 2024-04-11 |
| Mixture-of-Depths Attention | ✓ Link | 72.81 | MoDA (1.5B) | 2026-03-16 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | 72.8 | LLaMA 7B (0-shot) | 2023-02-27 |
| Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in {pm 1, pm i} | ✓ Link | 72.73 | Fairy2i-W2 (2-bit) | 2025-12-02 |
| Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention | ✓ Link | 72.43 | Gated DeltaNet-2 (1.3B) | 2026-05-21 |
| Distributionally Robust Mixture-of-Experts Training | ✓ Link | 72.1 | DRMoET (1.7B active, 10.3B total, eta=0.01) | 2026-10-05 |
| Gated Delta Networks: Improving Mamba2 with Delta Rule | ✓ Link | 71.75 | Gated DeltaNet-H1 | 2024-12-09 |
| ECO: Quantized Training without Full-Precision Master Weights | | 71.59 | DeepSeek-MoE-16B (INT4 w/o MW ECO + RTN) | 2026-01-29 |
| Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling | ✓ Link | 71.5 | Pythia 12B (5-shot) | 2023-04-03 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | 71.25 | OPT 66B (1-shot) | 2023-03-30 |
| Language Models are Few-Shot Learners | ✓ Link | 71.2 | GPT-3 175B (1 shot) | 2020-05-28 |
| SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot | ✓ Link | 71.04 | OPT-175B | 2023-01-02 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | 70.79 | GPT-NeoX 20B (1-shot) | 2023-03-30 |
| Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling | ✓ Link | 70.2 | Pythia 12B (0-shot) | 2023-04-03 |
| Sumi: Open Uniform Diffusion Language Model from Scratch | ✓ Link | 70.0 | Sumi-7B | 2026-06-17 |
| UL2: Unifying Language Learning Paradigms | ✓ Link | 69.8 | UL2 20B (chain-of-thought + self-consistency) | 2022-05-10 |
| Mamba: Linear-Time Sequence Modeling with Selective State Spaces | ✓ Link | 69.7 | Mamba-2.8B (0-shot) | 2023-12-01 |
| SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot | ✓ Link | 69.65 | SparseGPT 175B (50% sparsity) | 2023-01-02 |
| Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality | ✓ Link | 69.6 | Mamba-2-2.7B | 2024-05-31 |
| Parcae: Scaling Laws For Stable Looped Language Models | ✓ Link | 68.981483 | Parcae-1.3B (T=8, 10-shot) | 2026-04-14 |
| Language Models are Few-Shot Learners | ✓ Link | 68.8 | GPT-3 175B (0-shot) | 2020-05-28 |
| Galactica: A Large Language Model for Science | ✓ Link | 68.8 | GPT-3 (zero-shot) | 2022-11-16 |
| GPT-SW3: An Autoregressive Language Model for the Nordic Languages | | 68.7 | GPT-SW3 40B | 2023-05-22 |
| SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot | ✓ Link | 68.35 | SparseGPT (175B, 4:8 Sparsity) | 2023-01-02 |
| Copy-Paste Language Models | ✓ Link | 68.1 | Copy-Paste Language Model (1.7B, Full) | 2026-10-13 |
| GLaM: Efficient Scaling of Language Models with Mixture-of-Experts | | 68.0 | GLaM 64B/64E (0-shot) | 2021-12-13 |
| Multi-Head Low-Rank Attention | ✓ Link | 67.89 | MLRA-2 (2.9B) | 2026-03-02 |
| Introducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs | ✓ Link | 67.3 | MPT-7B | 2023-05-05 |
| SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot | ✓ Link | 67.08 | SparseGPT 175B (2:4 sparsity) | 2023-01-02 |
| OPT: Open Pre-trained Transformer Language Models | ✓ Link | 67 | OPT-175B | 2022-05-02 |
| Delta Attention Residuals | | 66.5 | Delta Block (0.6B) | 2026-05-13 |
| The State-Prediction Separation Hypothesis | ✓ Link | 66.3 | SPS XL | 2026-07-01 |
| OLMo: Accelerating the Science of Language Models | ✓ Link | 65.4 | OLMo 7B | 2024-02-01 |
| xLSTM: Extended Long Short-Term Memory | ✓ Link | 65.11 | xLSTM[7:1] 1.3B | 2024-05-07 |
| Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression | ✓ Link | 65.0 | LLaMA-7B with Swift-SVD* (dynamic) | 2026-04-02 |
| Releasing Persimmon-8B | ✓ Link | 64.0 | Persimmon-8B-FT | 2023-09-07 |
| Group Representational Position Encoding | ✓ Link | 63.89 | GRAPE-AP | 2025-12-08 |
| Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction | | 63.81 | GVA + DRoPE d_r=16 | 2026-09-08 |
| SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations | ✓ Link | 63.0 | Token Rounding (1.8B MoE) | 2025-12-16 |
| ModuleFormer: Modularity Emerges from Mixture-of-Experts | ✓ Link | 62.46 | MoLM-8B-K2 (0-shot) | 2023-06-07 |
| When Does Sparsity Mitigate the Curse of Depth in LLMs | ✓ Link | 61.74 | Transformer (2.3B, L=32) | 2026-03-16 |
| Disentangling Representation Evolution in Transformers through Directional Decomposition | ✓ Link | 60.4 | Transformer Geometry 2.7B V-Para (zero-shot) | 2026-09-14 |
| Data Science and Technology Towards AGI Part I: Tiered Data Management | ✓ Link | 59.79 | MiniCPM-1.2B (Math-L3) | 2026-02-09 |
| Stay on topic with Classifier-Free Guidance | | 58.9 | LLaMA 7B + CFG (0-shot) | 2023-06-30 |
| OLMo: Accelerating the Science of Language Models | ✓ Link | 58.1 | OLMo 1B | 2024-02-01 |
| Circuit Hypernetworks for Quantum-Augmented Diffusion Language Models | | 55.5 | HyperQ, 64 q | 2026-09-21 |
| UniPool: A Globally Shared Expert Pool for Mixture-of-Experts | ✓ Link | 54.34 | UniPool (978M) | 2026-05-07 |
| The Curse and Blessing of Mean Bias in FP4-Quantized LLM Training | | 50.72 | Qwen3-0.6B (Averis FP4) | 2026-03-11 |
| OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration | | 50.62 | GPT-2 XL + OPUS | 2026-02-05 |
| Daedalus-150M: A Convolution-Attention Hybrid Designed for CPU Inference | ✓ Link | 50.42 | Daedalus-150M | 2026-08-20 |
| ParameterNet: Parameters Are All You Need for Large-scale Visual Pretraining of Mobile Networks | ✓ Link | 49.58 | ParameterNet-MoE-up-proj-8 | 2024-06-17 |
| Parcae: Scaling Laws For Stable Looped Language Models | ✓ Link | 48.78 | Parcae (350M) | 2026-04-14 |
| Dynamic Linear Attention | | 48.1 | Mamba-2-780M w/ DLA | 2026-06-09 |
| MDM-Prime-v2: Binary Encoding and Index Shuffling Enable Scaling of Diffusion Language Models | ✓ Link | 47.81 | MDM-Prime-v2 (1.1B) | 2026-03-17 |
| Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity | ✓ Link | 47.32 | Sparse-BitNet (Qwen2.5-3B, 6:8) | 2026-03-05 |
| Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation | ✓ Link | 45.54 | GRT Large (isoParams, zero-shot) | 2026-08-15 |
| MeKi: Memory-based Expert Knowledge Injection for Efficient LLM Scaling | ✓ Link | 42.2 | MeKi (4B) | 2026-02-03 |
| Galactica: A Large Language Model for Science | ✓ Link | 40.7 | BLOOM (5-shot) | 2022-11-16 |
| PLDR-LLMs Reason At Self-Organized Criticality | ✓ Link | 38.55 | PLDRv51-SOC-110M-5 | 2026-03-12 |
| UL2: Unifying Language Learning Paradigms | ✓ Link | 38.4 | UL2 20B (chain-of-thought) | 2022-05-10 |
| Galactica: A Large Language Model for Science | ✓ Link | 37.4 | OPT (5-shot) | 2022-11-16 |
| UL2: Unifying Language Learning Paradigms | ✓ Link | 32.2 | UL2 20B (0-shot) | 2022-05-10 |
| Looped Uniform Diffusion Language Models | ✓ Link | 29.17 | Looped UDLM | 2026-10-21 |
| SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot | ✓ Link | 28.03 | OPT 175B (50% Sparsity) | 2023-01-02 |