| UL2: Unifying Language Learning Paradigms | ✓ Link | 90.7 | UL2 20B | 2022-05-10 |
| UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark | ✓ Link | 90.1 | Unicorn 11B (fine-tuned) | 2021-03-24 |
| Mixture-of-Subspaces in Low-Rank Adaptation | ✓ Link | 89.7 | LLaMA3 8B+MoSLoRA | 2024-06-16 |
| Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone | ✓ Link | 88.6 | Phi-3.5-MoE (5-shot) | 2024-04-22 |
| Task Compass: Scaling Multi-task Pre-training with Task Prefix | ✓ Link | 88.3 | CompassMTL 567M with Tailor | 2022-10-12 |
| MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts | ✓ Link | 87.6 | LLaMA-3 8B + MixLoRA | 2024-04-22 |
| Two is Better than Many? Binary Classification as an Effective Approach to Multi-Choice Question Answering | ✓ Link | 87.4 | DeBERTa-Large 304M | 2022-10-29 |
| Task Compass: Scaling Multi-task Pre-training with Task Prefix | ✓ Link | 87.3 | CompassMTL 567M | 2022-10-12 |
| MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts | ✓ Link | 86.8 | LLaMA-2 13B + MixLoRA | 2024-04-22 |
| SHAKTI: A 2.5 Billion Parameter Small Language Model Optimized for Edge AI and Low-Resource Environments | | 86.2 | Shakti-LLM (2.5B) | 2024-10-15 |
| MassAlloc Attention: Let Attention Allocate Its Own Compute | ✓ Link | 85.94 | MALA (32B) | 2026-09-26 |
| Two is Better than Many? Binary Classification as an Effective Approach to Multi-Choice Question Answering | ✓ Link | 85.9 | DeBERTa-Large 304M (classification-based) | 2022-10-29 |
| CoWindow Attention: Full Causal Coverage Is a Collective Property | | 85.75 | CoWA (32B) | 2026-09-26 |
| A Sovereign, Open-Source Foundation Model for German and English | ✓ Link | 85.7 | Soofi S 30B-A3B Base | 2026-07-10 |
| Task Compass: Scaling Multi-task Pre-training with Task Prefix | ✓ Link | 85.5 | ExDeBERTa 567M | 2022-10-12 |
| UnifiedQA: Crossing Format Boundaries With a Single QA System | ✓ Link | 85.3 | UnifiedQA 3B | 2020-05-02 |
| PaLM: Scaling Language Modeling with Pathways | ✓ Link | 85.2 | PaLM 540B | 2022-04-05 |
| Motif 3: Technical Report | ✓ Link | 85.14 | Motif 3 Base (zero-shot) | 2026-08-10 |
| PaLM 2 Technical Report | ✓ Link | 85.0 | PaLM 2-L (1-shot) | 2023-05-17 |
| Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models | ✓ Link | 85.0 | Nemotron-H-56B-Base | 2025-04-04 |
| DeepSeek-V3 Technical Report | ✓ Link | 84.7 | DeepSeek-V3 Base | 2024-12-27 |
| SQ-format: A Unified Sparse-Quantized Hardware-friendly Data Format for LLMs | | 84.06 | Llama-3-70B with SQ-format (W4A(SQ6)) | 2025-12-05 |
| DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model | ✓ Link | 83.7 | DeepSeek-V2 Base | 2024-05-07 |
| Mixtral of Experts | ✓ Link | 83.6 | Mixtral 8x7B (0-shot) | 2024-01-08 |
| HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Experts | | 83.4 | Qwen 3.5-122B-A10B (HodgeCover+Wanda) | 2026-05-13 |
| Chameleon: Mixed-Modal Early-Fusion Foundation Models | ✓ Link | 83.3 | Chameleon 34B | 2024-05-16 |
| PaLM 2 Technical Report | ✓ Link | 83.2 | PaLM 2-M (1-shot) | 2023-05-17 |
| Jamba: A Hybrid Transformer-Mamba Language Model | ✓ Link | 83.2 | Jamba (0-shot) | 2024-03-28 |
| MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts | ✓ Link | 83.2 | LLaMA-2 7B + MixLoRA | 2024-04-22 |
| Gemma 2: Improving Open Language Models at a Practical Size | ✓ Link | 83.2 | Gemma 2 27B | 2024-07-31 |
| Mistral 7B | ✓ Link | 83.0 | Mistral 7B (0-shot) | 2023-10-10 |
| Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs | ✓ Link | 82.86 | SRR (Qwen3-30B-A3B, Task Arithmetic) | 2026-09-26 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | 82.8 | LLaMA 65B (0-shot) | 2023-02-27 |
| Llama 2: Open Foundation and Fine-Tuned Chat Models | ✓ Link | 82.8 | LLaMA 2 70B (0-shot) | 2023-07-18 |
| Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks | ✓ Link | 82.7 | Camelidae-8×34B | 2024-01-05 |
| DeepSeek LLM: Scaling Open-Source Language Models with Longtermism | ✓ Link | 82.6 | DeepSeek LLM 67B Chat | 2024-01-05 |
| Nemotron-4 15B Technical Report | ✓ Link | 82.4 | Nemotron-4 15B | 2024-02-26 |
| PaLM: Scaling Language Modeling with Pathways | ✓ Link | 82.3 | PaLM 62B | 2022-04-05 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | 82.3 | LLaMA 33B (0-shot) | 2023-02-27 |
| PaLM 2 Technical Report | ✓ Link | 82.2 | PaLM 2-S (1-shot) | 2023-05-17 |
| Mixtral of Experts | ✓ Link | 82.2 | Mistral 7B (0-shot) | 2024-01-08 |
| Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models | ✓ Link | 82.2 | Nemotron-H-8B-Base | 2025-04-04 |
| Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism | ✓ Link | 82.0 | MT-NLG 530B (0-shot) | 2019-09-17 |
| Llama 2: Open Foundation and Fine-Tuned Chat Models | ✓ Link | 81.9 | LLaMA 2 34B (0-shot) | 2023-07-18 |
| Scaling Language Models: Methods, Analysis & Insights from Training Gopher | ✓ Link | 81.8 | Gopher 280B (0-shot) | 2021-12-08 |
| Training Compute-Optimal Large Language Models | ✓ Link | 81.8 | Chinchilla 70B (0-shot) | 2022-03-29 |
| LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Models | ✓ Link | 81.8 | Phi-4 (14B) with LoopUS | 2026-05-10 |
| Finetuned Language Models Are Zero-Shot Learners | ✓ Link | 81.7 | FLAN 137B (few-shot, k=10) | 2021-09-03 |
| ECO: Quantized Training without Full-Precision Master Weights | | 81.5 | DeepSeek-MoE-16B (INT4 w/o MW ECO + SR) | 2026-01-29 |
| NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction | ✓ Link | 81.45 | NCP-ArchPreview Stage-2 | 2026-09-09 |
| Zamba: A Compact 7B SSM Hybrid Model | ✓ Link | 81.37 | Zamba 7B | 2024-05-26 |
| G^2PTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensation | ✓ Link | 81.34 | G^2PTQ* (Qwen3-32B, W4A16) | 2026-09-25 |
| RecurrentGemma: Moving Past Transformers for Efficient Open Language Models | ✓ Link | 81.3 | RecurrentGemma-9B | 2024-04-11 |
| Gemma: Open Models Based on Gemini Research and Technology | ✓ Link | 81.2 | Gemma 7B | 2024-03-13 |
| Byte Latent Transformer: Patches Scale Better Than Tokens | ✓ Link | 81.1 | BLT-Space 8B | 2024-12-13 |
| Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation | ✓ Link | 81.1 | T5Gemma 9B-9B PrefixLM | 2025-04-08 |
| SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot | ✓ Link | 81.07 | OPT-175B | 2023-01-02 |
| Language Models are Few-Shot Learners | ✓ Link | 81.0 | GPT-3 175B (0-shot) | 2020-05-28 |
| The Zamba2 Suite: Technical Report | ✓ Link | 81.0 | Zamba2-7B | 2024-11-22 |
| Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models | ✓ Link | 80.69 | HSA-UltraLong-MoE | 2025-11-28 |
| Decoding Looped Transformers Better for (Almost) Free | | 80.69 | LoopCD-Ouro-2.6B (adaptive) | 2026-10-01 |
| SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot | ✓ Link | 80.63 | SparseGPT 175B (50% Sparsity) | 2023-01-02 |
| Finetuned Language Models Are Zero-Shot Learners | ✓ Link | 80.5 | FLAN 137B (0-shot) | 2021-09-03 |
| Llama 2: Open Foundation and Fine-Tuned Chat Models | ✓ Link | 80.5 | LLaMA 2 13B (0-shot) | 2023-07-18 |
| mHC: Manifold-Constrained Hyper-Connections | | 80.5 | mHC (27B) | 2025-12-31 |
| OPT: Open Pre-trained Transformer Language Models | ✓ Link | 80.1 | OPT-175B | 2022-05-02 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | 80.1 | LLaMA 13B (0-shot) | 2023-02-27 |
| Introducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs | ✓ Link | 79.9 | MPT-7B | 2023-05-05 |
| Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler | ✓ Link | 79.9 | PowerLM-3B | 2024-08-23 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | 79.8 | LLaMA 7B (0-shot) | 2023-02-27 |
| OLMoE: Open Mixture-of-Experts Language Models | ✓ Link | 79.8 | OLMoE-1B-7B (5-shot) | 2024-09-03 |
| SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot | ✓ Link | 79.54 | SparseGPT 175B (4:8 Sparsity) | 2023-01-02 |
| SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot | ✓ Link | 79.54 | SparseGPT 175B (2:4 Sparsity) | 2023-01-02 |
| RoBERTa: A Robustly Optimized BERT Pretraining Approach | ✓ Link | 79.4 | RoBERTa-Large 355M | 2019-07-26 |
| Low-Rank Approximation for Sparse Attention in Multi-Modal LLMs | | 79.4 | LoRA-Sparse (LLaMA-7B) (extra data) | 2024-06-17 |
| Moshi: a speech-text foundation model for real-time dialogue | ✓ Link | 79.4 | Helium | 2024-09-17 |
| The Falcon 3 Family of Open Models | | 79.4 | Falcon3-10B-Base | 2024-12-17 |
| CoRe3D: Collaborative Reasoning as a Foundation for 3D Intelligence | | 79.4 | CoRe3D | 2025-12-14 |
| T5Gemma 2: Seeing, Reading, and Understanding Longer | | 79.2 | T5Gemma 2 4B-4B | 2025-12-16 |
| MobileMoE: Scaling On-Device Mixture of Experts | | 78.9 | MobileMoE-L (922M) | 2026-05-26 |
| SmolLM3: smol, multilingual, long-context reasoner | ✓ Link | 78.89 | SmolLM3 3B Base | 2025-07-08 |
| Llama 2: Open Foundation and Fine-Tuned Chat Models | ✓ Link | 78.8 | LLaMA 2 7B (0-shot) | 2023-07-18 |
| GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression | | 78.73 | GeoPair (Llama3 8B, CR=0.2) | 2026-09-22 |
| RecurrentGemma: Moving Past Transformers for Efficient Open Language Models | ✓ Link | 78.5 | RecurrentGemma-2B | 2024-04-11 |
| Improved Large Language Diffusion Models | ✓ Link | 78.5 | Confidence-based scoring | 2026-06-24 |
| Improved Large Language Diffusion Models | ✓ Link | 78.5 | iLLaDA 8B Base | 2026-06-24 |
| OLMo: Accelerating the Science of Language Models | ✓ Link | 78.4 | OLMo 7B | 2024-02-01 |
| Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation | ✓ Link | 78.4 | Taylor-Calibrate qwen2_3b 100M_ga_s2_taylor | 2026-06-15 |
| ECHO: Efficient Causal Self-Recursion Beyond Auxiliary Multi-Token Prediction | ✓ Link | 78.38 | ECHO-1.5(3) | 2026-10-17 |
| TIDE: Every Layer Knows the Token Beneath the Context | | 78.3 | TIDE-8E (3B) | 2026-05-07 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | 77.9 | Bloomberg GPT 50B (1-shot) | 2023-03-30 |
| Post-Training Leaves Behavioral Shadows on Unrelated Decisions | ✓ Link | 77.71 | ATD student (Qwen2.5-1.5B-Instruct) | 2026-09-24 |
| Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090 | ✓ Link | 77.69 | Puro-2B UD decay average | 2026-08-27 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | 77.6 | OPT 66B (1-shot) | 2023-03-30 |
| SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model | ✓ Link | 77.6 | SmolLM2 1.7B | 2025-02-04 |
| Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs | ✓ Link | 77.6 | Phi-4-Mini (5-shot) | 2025-02-24 |
| Disentangling Representation Evolution in Transformers through Directional Decomposition | ✓ Link | 77.6 | Transformer Geometry 2.7B V-Para (zero-shot) | 2026-09-14 |
| PaLM: Scaling Language Modeling with Pathways | ✓ Link | 77.5 | PaLM 8B | 2022-04-05 |
| RWKV: Reinventing RNNs for the Transformer Era | ✓ Link | 77.42 | RWKV-4 14B | 2023-05-22 |
| GPT-NeoX-20B: An Open-Source Autoregressive Language Model | ✓ Link | 77.4 | GPT-NeoX-20B (5-shot) | 2022-04-14 |
| Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts | ✓ Link | 77.4 | Upcycled 32→64 (100% CPT) | 2026-04-21 |
| Efficient Pre-Training with Token Superposition | ✓ Link | 77.4 | MoE TST (10B A1B) | 2026-05-07 |
| SmolLM - blazingly fast and remarkably powerful | ✓ Link | 77.3 | SmolLM 1.7B | 2024-07-16 |
| daVinci-LLM:Towards the Science of Pretraining | ✓ Link | 77.26 | daVinci-LLM-3B | 2026-03-28 |
| GPT-SW3: An Autoregressive Language Model for the Nordic Languages | | 77.2 | GPT-SW3 40B | 2023-05-22 |
| PIQA: Reasoning about Physical Commonsense in Natural Language | ✓ Link | 77.1 | RoBERTa-large 355M (fine-tuned) | 2019-11-26 |
| Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior | ✓ Link | 77.1 | PPT (k-Shuffle Dyck), 3B, C4, 100B PT tokens | 2026-09-30 |
| BitNet b1.58 2B4T Technical Report | ✓ Link | 77.09 | BitNet b1.58 2B4T | 2025-04-16 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | 77 | BLOOM 176B (1-shot) | 2023-03-30 |
| Textbooks Are All You Need II: phi-1.5 technical report | ✓ Link | 77 | phi-1.5-web (1.3B) | 2023-09-11 |
| Mixture-of-Depths Attention | ✓ Link | 76.82 | MoDA (1.5B) | 2026-03-16 |
| Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling | ✓ Link | 76.7 | Pythia 12B (5-shot) | 2023-04-03 |
| Distributionally Robust Mixture-of-Experts Training | ✓ Link | 76.66 | DRMoET (1.7B active, 10.3B total, eta=0.01) | 2026-10-05 |
| GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model | ✓ Link | 76.5 | GPT-J 6B | 2021-06-04 |
| Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality | ✓ Link | 76.4 | Mamba-2-2.7B | 2024-05-31 |
| Parallax: Parameterized Local Linear Attention for Language Modeling | ✓ Link | 76.39 | Parallax (1.7B) | 2026-05-27 |
| LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation | ✓ Link | 76.33 | LatentLoop (7B / 1B active) | 2026-10-14 |
| Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning | ✓ Link | 76.2 | Open-LLaMA-3B-v2 | 2023-10-10 |
| Initialisation Determines the Basin: Efficient Codebook Optimisation for Extreme LLM Quantization | ✓ Link | 76.2 | OA-EM (Llama 3.2 3B) | 2026-04-09 |
| Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in {pm 1, pm i} | ✓ Link | 76.17 | Fairy2i-W2 (2-bit) | 2025-12-02 |
| Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling | ✓ Link | 76 | Pythia 12B (0-shot) | 2023-04-03 |
| Scaling Inherently Interpretable Language Models | ✓ Link | 75.9 | Steerling-8B | 2026-08-06 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | 75.8 | GPT-NeoX 20B (1-shot) | 2023-03-30 |
| Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning | ✓ Link | 75.8 | Sheared-LLaMA-2.7B | 2023-10-10 |
| Dream 7B: Diffusion Large Language Models | | 75.8 | Dream 7B | 2025-08-21 |
| Self-Improving Pretraining: using post-trained models to pretrain better models | | 75.8 | Self-Improving Pretraining (1.4B) | 2026-01-29 |
| Cross-Tokenizer LLM Distillation through a Byte-Level Interface | | 75.68 | Llama3.2 (3B) with BLCTD (transferred to Qwen2 tokenizer) | 2026-04-13 |
| Cross-Tokenizer LLM Distillation through a Byte-Level Interface | | 75.68 | BLD | 2026-04-13 |
| Multi-Head Low-Rank Attention | ✓ Link | 75.52 | MLRA-2 (2.9B) | 2026-03-02 |
| MeKi: Memory-based Expert Knowledge Injection for Efficient LLM Scaling | ✓ Link | 75.4 | MeKi (4B) | 2026-02-03 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 75.3 | Mamba-3-MIMO-1.5B | 2026-03-16 |
| Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling | ✓ Link | 75.2 | Pythia 6.9B (0-shot) | 2023-04-03 |
| Mamba: Linear-Time Sequence Modeling with Selective State Spaces | ✓ Link | 75.2 | Mamba-2.8B | 2023-12-01 |
| xLSTM: Extended Long Short-Term Memory | ✓ Link | 74.92 | xLSTM[7:1] 1.3B | 2024-05-07 |
| Copy-Paste Language Models | ✓ Link | 74.86 | Copy-Paste Language Model (1.7B, Full) | 2026-10-13 |
| SoloQ: Calibration-Free Quantization for Diffusion Language Models | | 74.76 | SoloQ-N (LLaDA-1.5-8B) | 2026-10-05 |
| VersatileFFN: Achieving Parameter Efficiency in LLMs via Adaptive Wide-and-Deep Reuse | ✓ Link | 74.65 | VersatileFFN | 2025-12-16 |
| Modular TTT: Rethinking Test-Time Training as Composable Modules | ✓ Link | 74.43 | Modular TTT 1.45B (MSE M) | 2026-08-07 |
| SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model | ✓ Link | 74.4 | SmolLM2 1.7B Instruct | 2025-02-04 |
| Kalman Delta Networks: Uncertainty-aware Associative Memory | ✓ Link | 74.21 | Diagonal KDN (1.3B) | 2026-09-07 |
| Refuse without Refusal: A Structural Analysis of Safety-Tuning Responses for Reducing False Refusals in Language Models | ✓ Link | 74.16 | Llama-3.1-8B + Rationale-Only | 2026-09-04 |
| Scaling Laws for Looped Mixture of Experts | | 74.0 | A0.3B-1.3B LoopMoE (R=5) | 2026-09-30 |
| Towards Looped Models Done Right, Part II: Rethinking at Fixed Points | ✓ Link | 73.72 | LoopedLM-P2 learned-depth lambda_H=0.01 (1.6B) | 2026-10-05 |
| OLMo: Accelerating the Science of Language Models | ✓ Link | 73.7 | OLMo 1B | 2024-02-01 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 73.7 | Mamba-3-MIMO-880M | 2026-03-16 |
| Large Language Diffusion Models | ✓ Link | 73.6 | LLaDA 8B Base | 2025-02-14 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 73.6 | Mamba-3-SISO-880M | 2026-03-16 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 73.6 | Mamba-3-SISO-1.5B | 2026-03-16 |
| Triplet-Block Diffusion RWKV | ✓ Link | 73.5 | B3D-RWKV-7.2B | 2026-05-25 |
| Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning | ✓ Link | 73.4 | Sheared-LLaMA-1.3B | 2023-10-10 |
| In-Place Test-Time Training | ✓ Link | 73.29 | In-Place TTT (4B, Full Attention) | 2026-04-07 |
| Efficient Language Modeling with Sparse all-MLP | | 73 | sMLP - deterministic 9.4B (0-shot) | 2022-03-14 |
| Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression | ✓ Link | 73.0 | LLaMA-7B with Swift-SVD* (dynamic) | 2026-04-02 |
| Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training | ✓ Link | 72.94 | Qwen3 (1.7B) with DeMix (224 proxies) | 2026-01-31 |
| ModuleFormer: Modularity Emerges from Mixture-of-Experts | ✓ Link | 72.91 | MoLM-8B-K2 (0-shot) | 2023-06-07 |
| Language Models are Few-Shot Learners | ✓ Link | 72.9 | GPT-3 Large 760M (0-shot) | 2020-05-28 |
| Language Is Not All You Need: Aligning Perception with Language Models | ✓ Link | 72.9 | KOSMOS-1 (0-shot) | 2023-02-27 |
| Kosmos-2: Grounding Multimodal Large Language Models to the World | ✓ Link | 72.9 | KOSMOS-2 | 2023-06-26 |
| Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention | ✓ Link | 72.8 | Gated DeltaNet-2 (1.3B) | 2026-05-21 |
| Maglev: Sliding Recurrent Memory | | 72.6 | Maglev (separate parameters, lambda=1) | 2026-08-05 |
| Gated Delta Networks: Improving Mamba2 with Delta Rule | ✓ Link | 72.57 | Gated DeltaNet-H1 | 2024-12-09 |
| FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models | | 72.46 | FactorEngram (1B) | 2026-09-28 |
| Parcae: Scaling Laws For Stable Looped Language Models | ✓ Link | 72.343125 | Parcae-1.3B (T=8, 10-shot) | 2026-04-14 |
| LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions | ✓ Link | 72.2 | FLAN-T5-Large 783M | 2023-04-27 |
| OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration | | 72.2 | GPT-2 XL + OPUS | 2026-02-05 |
| GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow | ✓ Link | 72.14 | GPT-Neo 2.7B | 2021-03-21 |
| Multi-Token Prediction Done Right | ✓ Link | 72.03 | Soft-MTP (1.2B, zero-shot) | 2026-10-28 |
| Group Representational Position Encoding | ✓ Link | 71.98 | GRAPE-AP | 2025-12-08 |
| Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity | ✓ Link | 71.93 | Sparse-BitNet (Qwen2.5-3B, 6:8) | 2026-03-05 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 71.9 | Mamba-3-SISO-440M | 2026-03-16 |
| The State-Prediction Separation Hypothesis | ✓ Link | 71.9 | SPS XL | 2026-07-01 |
| MDN: Parallelizing Stepwise Momentum for Delta Linear Attention | ✓ Link | 71.87 | MDN (1.3B) | 2026-05-07 |
| Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus | ✓ Link | 71.55 | GDN 1.3B (3:1 hybrid, 50B tokens) | 2026-08-12 |
| Full-bandwidth transformer | | 71.49 | Full-bandwidth transformer 1B (200B, 1 feedback pass) | 2026-08-09 |
| LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions | ✓ Link | 71.3 | LaMini-GPT 1.5B | 2023-04-27 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 70.8 | Mamba-3-MIMO-440M | 2026-03-16 |
| Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts | ✓ Link | 70.7 | LOOM (1.7B, 12 loops) | 2026-10-01 |
| LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions | ✓ Link | 70.6 | LaMini-F-T5 783M | 2023-04-27 |
| LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions | ✓ Link | 70.5 | GPT-2-XL 1.5B | 2023-04-27 |
| Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling | ✓ Link | 70.4 | Pythia 1B (5-shot) | 2023-04-03 |
| OpenESM: Energy-Steered Models Are Scalable Foundation Learners | ✓ Link | 70.02 | OpenESM FineWeb d26 | 2026-10-07 |
| SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations | ✓ Link | 69.6 | Token Rounding (1.8B MoE) | 2025-12-16 |
| Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs | | 69.53 | FLAN-T5 (large, 783M) with Similarity-based Router | 2026-02-01 |
| UniPool: A Globally Shared Expert Pool for Mixture-of-Experts | ✓ Link | 69.21 | UniPool (978M) | 2026-05-07 |
| PIQA: Reasoning about Physical Commonsense in Natural Language | ✓ Link | 69.2 | GPT-2-small 124M (fine-tuned) | 2019-11-26 |
| Error-Free Linear Attention is a Free Lunch: Exact Solution from Continuous-Time Dynamics | ✓ Link | 68.9 | EFLA | 2025-12-14 |
| Revisiting the Shape Convention of Transformer Language Models | | 68.2 | Hourglass (1074M) | 2026-02-06 |
| Efficient Language Modeling with Sparse all-MLP | | 68.1 | Gshard 9B | 2022-03-14 |
| LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions | ✓ Link | 67.2 | LaMini-T5 738M | 2023-04-27 |
| Fast Weight Attention for Continual Learning | ✓ Link | 67.03 | Falcon-1A.2130M | 2026-08-27 |
| Safin-1: Safety from Within through Memory-Native State Evolution | ✓ Link | 66.9 | Gated DeltaNet with MARCH | 2026-08-31 |
| PIQA: Reasoning about Physical Commonsense in Natural Language | ✓ Link | 66.8 | BERT-large 340M (fine-tuned) | 2019-11-26 |
| BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding | ✓ Link | 66.7 | BERT-Large 340M | 2018-10-11 |
| The Curse and Blessing of Mean Bias in FP4-Quantized LLM Training | | 66.7 | Qwen3-0.6B (Averis FP4) | 2026-03-11 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 66.7 | Mamba-3-MIMO-180M | 2026-03-16 |
| Sumi: Open Uniform Diffusion Language Model from Scratch | ✓ Link | 66.4 | Sumi-7B | 2026-06-17 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 66.1 | Mamba-3-SISO-180M | 2026-03-16 |
| Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation | ✓ Link | 66.05 | GRT Large (isoParams, zero-shot) | 2026-08-15 |
| Daedalus-150M: A Convolution-Attention Hybrid Designed for CPU Inference | ✓ Link | 65.78 | Daedalus-150M | 2026-08-20 |
| What Makes Recurrence Effective in Looped Language Models? | | 65.18 | BaseLoop (Llama3.1-1B, 3x4, Muon) | 2026-09-29 |
| Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention | | 64.69 | GQE (renorm. scoring + shared head) | 2026-06-18 |
| Low-Rank Ternary Adaptation for Fine-Tuning Transformers | ✓ Link | 64.6 | Low-Rank Ternary Adaptation (Llama-3.2-3B) | 2026-08-25 |
| PLDR-LLMs Reason At Self-Organized Criticality | ✓ Link | 63.98 | PLDRv51-SOC-110M-5 | 2026-03-12 |
| Efficient Language Modeling with Sparse all-MLP | | 63.8 | Base Layers 10B (0-shot) | 2022-03-14 |
| Efficient Language Modeling with Sparse all-MLP | | 63.8 | HASH Layers 10B (0-shot) | 2022-03-14 |
| Normalized Low-Rank Adaptation | ✓ Link | 63.76 | MLA342M NoRA-init | 2026-08-31 |
| Dynamic Linear Attention | | 63.7 | Mamba-2-780M w/ DLA | 2026-06-09 |
| WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixing | ✓ Link | 63.55 | WhiteMatter k=16 | 2026-09-27 |
| LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation | ✓ Link | 63.33 | LoopFormer (3x8) | 2026-02-11 |
| Parcae: Scaling Laws For Stable Looped Language Models | ✓ Link | 62.08 | Parcae (350M) | 2026-04-14 |
| Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models | | 61.5 | B frozen table, float | 2026-10-15 |
| Circuit Hypernetworks for Quantum-Augmented Diffusion Language Models | | 60.86 | HyperQ, 64 q | 2026-09-21 |
| LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions | ✓ Link | 55.9 | T5-Large 738M | 2023-04-27 |
| Recover-LoRA: Data-Free Accuracy Recovery of Degraded Language Models via Low-Rank Adaptation | ✓ Link | 54.79 | DeepSeek-R1-Distill-Qwen-1.5B + Recover-LoRA | 2025-10-06 |
| SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot | ✓ Link | 54.73 | OPT-175B (50% Sparsity) | 2023-01-02 |
| Looped Uniform Diffusion Language Models | ✓ Link | 53.26 | Looped UDLM | 2026-10-21 |
| PIQA: Reasoning about Physical Commonsense in Natural Language | ✓ Link | 50 | Random chance baseline | 2019-11-26 |