| Task Compass: Scaling Multi-task Pre-training with Task Prefix | ✓ Link | 96.1 | CompassMTL 567M with Tailor | 2022-10-12 |
| Task Compass: Scaling Multi-task Pre-training with Task Prefix | ✓ Link | 95.6 | CompassMTL 567M | 2022-10-12 |
| Two is Better than Many? Binary Classification as an Effective Approach to Multi-Choice Question Answering | ✓ Link | 95.6 | DeBERTa-Large 304M (classification-based) | 2022-10-29 |
| GPT-4 Technical Report | ✓ Link | 95.3 | GPT-4 (10-shot) | 2023-03-15 |
| Mixture-of-Subspaces in Low-Rank Adaptation | ✓ Link | 95.0 | LLaMA3+MoSLoRA | 2024-06-16 |
| Two is Better than Many? Binary Classification as an Effective Approach to Multi-Choice Question Answering | ✓ Link | 94.7 | DeBERTa-Large 304M | 2022-10-29 |
| MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts | ✓ Link | 94.7 | LLaMA-2 13B + MixLoRA | 2024-04-22 |
| UL2: Unifying Language Learning Paradigms | ✓ Link | 94.1 | UL2 20B | 2022-05-10 |
| UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark | ✓ Link | 93.9 | Unicorn 11B (fine-tuned) | 2021-03-24 |
| Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context | ✓ Link | 93.3 | Gemini 1.5 Pro (10-shot) | 2024-03-08 |
| MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts | ✓ Link | 93.3 | LLaMA-3 8B + MixLoRA | 2024-04-22 |
| MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts | ✓ Link | 93.1 | LLaMA-2 7B + MixLoRA | 2024-04-22 |
| DeBERTa: Decoding-enhanced BERT with Disentangled Attention | ✓ Link | 93 | DeBERTa++ | 2020-06-05 |
| DiscoSense: Commonsense Reasoning with Discourse Connectives | ✓ Link | 91.5 | ELECTRA-Large 335M (fine-tuned on DiscoSense and HellaSwag) | 2022-10-22 |
| Nemotron-4 340B Technical Report | ✓ Link | 90.53 | Nemotron-4-340B-Base | 2024-06-17 |
| Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters | ✓ Link | 90.2 | Step-3.5-Flash Base | 2026-02-11 |
| Arcee Trinity Large Technical Report | ✓ Link | 90.11 | Trinity Large Base (5-shot) | 2026-02-19 |
| 2 OLMo 2 Furious | ✓ Link | 89.7 | OLMo 2 32B | 2024-12-31 |
| | 89 | DBRX Instruct 132B (10-shot) | |
| Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models | ✓ Link | 89.0 | Nemotron-H-56B-Base (10-shot) | 2025-04-04 |
| MiMo-V2-Flash Technical Report | ✓ Link | 88.5 | MiMo-V2-Flash (Base) | 2026-01-06 |
| | 88.3 | TheBloke/llama-2-70b-Guanaco-QLoRA-fp16 (10-shot) | |
| Motif 3: Technical Report | ✓ Link | 88.3 | Motif 3 Base (10-shot) | 2026-08-10 |
| | 88 | ALBERT-XXL 235M | |
| Gemini: A Family of Highly Capable Multimodal Models | ✓ Link | 87.8 | Gemini Ultra (10-shot) | 2023-12-19 |
| Qwen2 Technical Report | ✓ Link | 87.6 | Qwen2-72B | 2024-07-15 |
| PaLM 2 Technical Report | ✓ Link | 87.4 | PaLM 2-L (1-shot) | 2023-05-17 |
| DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression | ✓ Link | 87.2 | DeepSeek-V4.1-Flash Base | 2026-09-17 |
| Jamba: A Hybrid Transformer-Mamba Language Model | ✓ Link | 87.1 | Jamba (10-shot) | 2024-03-28 |
| GLM-5.3-Flash: Frontier Intelligence, Flash Cost | ✓ Link | 87.1 | GLM-5.3-Flash-Base | 2026-08-26 |
| Darwin Family: MRI-Trust-Weighted Evolutionary Merging for Training-Free Scaling of Language-Model Reasoning | | 87.0 | Darwin-27B-Opus | 2026-05-14 |
| DiscoSense: Commonsense Reasoning with Discourse Connectives | ✓ Link | 86.9 | ELECTRA-Large 335M (fine-tuned on HellaSwag) | 2022-10-22 |
| PaLM 2 Technical Report | ✓ Link | 86.7 | PaLM 2-M (1-shot) | 2023-05-17 |
| Muppet: Massive Multi-task Representations with Pre-Finetuning | ✓ Link | 86.4 | MUPPET Roberta Large | 2021-01-26 |
| Gemma 2: Improving Open Language Models at a Practical Size | ✓ Link | 86.4 | Gemma 2 27B (10-shot) | 2024-07-31 |
| 2 OLMo 2 Furious | ✓ Link | 86.4 | OLMo 2 13B | 2024-12-31 |
| Stay on topic with Classifier-Free Guidance | | 86.3 | LLaMA 65B + CFG (0-shot) | 2023-06-30 |
| The Falcon Series of Open Language Models | | 85.9 | Falcon-180B (0-shot) | 2023-11-28 |
| PaLM 2 Technical Report | ✓ Link | 85.6 | PaLM 2-S (1-shot) | 2023-05-17 |
| LLaDA2.1: Speeding Up Text Diffusion via Token Editing | ✓ Link | 85.6 | LLaDA2.1-flash (100B) | 2026-02-09 |
| HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Experts | | 85.6 | Qwen 3.5-122B-A10B (HodgeCover+Wanda) | 2026-05-13 |
| RoBERTa: A Robustly Optimized BERT Pretraining Approach | ✓ Link | 85.5 | RoBERTa-Large Ensemble | 2019-07-26 |
| GPT-4 Technical Report | ✓ Link | 85.5 | GPT-3.5 (10-shot) | 2023-03-15 |
| Stay on topic with Classifier-Free Guidance | | 85.3 | LLaMA 30B + CFG (0-shot) | 2023-06-30 |
| Llama 2: Open Foundation and Fine-Tuned Chat Models | ✓ Link | 85.3 | LLaMA 2 70B (0-shot) | 2023-07-18 |
| Qwen2 Technical Report | ✓ Link | 85.2 | Qwen2-57B-A14B | 2024-07-15 |
| Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering | | 85.0 | HyKAS+CSKG | 2019-10-30 |
| LLaDA2.0: Scaling Up Diffusion Language Models to 100B | ✓ Link | 84.97 | LLaDA2.0-flash | 2025-12-10 |
| MassAlloc Attention: Let Attention Allocate Its Own Compute | ✓ Link | 84.69 | MALA (32B) | 2026-09-26 |
| SQ-format: A Unified Sparse-Quantized Hardware-friendly Data Format for LLMs | | 84.44 | Llama-3-70B with SQ-format (W4A(SQ6)) | 2025-12-05 |
| Mixtral of Experts | ✓ Link | 84.4 | Mixtral 8x7B | 2024-01-08 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | 84.2 | LLaMA 65B (0-shot) | 2023-02-27 |
| PaLM: Scaling Language Modeling with Pathways | ✓ Link | 83.8 | PaLM-540B (Few-Shot) | 2022-04-05 |
| Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone | ✓ Link | 83.8 | Phi-3.5-MoE (5-shot) | 2024-04-22 |
| 2 OLMo 2 Furious | ✓ Link | 83.8 | OLMo 2 7B | 2024-12-31 |
| PaLM: Scaling Language Modeling with Pathways | ✓ Link | 83.6 | PaLM-540B (1-shot) | 2022-04-05 |
| Task Compass: Scaling Multi-task Pre-training with Task Prefix | ✓ Link | 83.6 | ExDeBERTa 567M | 2022-10-12 |
| Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs | ✓ Link | 83.52 | SRR (Qwen3-30B-A3B, TIES) | 2026-09-26 |
| PaLM: Scaling Language Modeling with Pathways | ✓ Link | 83.4 | PaLM-540B (0-shot) | 2022-04-05 |
| Attention Residuals | ✓ Link | 83.4 | Kimi Linear (48B) with Block AttnRes | 2026-03-16 |
| Llama 2: Open Foundation and Fine-Tuned Chat Models | ✓ Link | 83.3 | LLaMA 2 34B (0-shot) | 2023-07-18 |
| Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks | ✓ Link | 83.2 | Camelidae-8×34B (10-shot) | 2024-01-05 |
| Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models | ✓ Link | 83.2 | Nemotron-H-8B-Base (10-shot) | 2025-04-04 |
| CoWindow Attention: Full Causal Coverage Is a Collective Property | | 83.1 | CoWA (32B) | 2026-09-26 |
| Introducing LFM-7B: Setting New Standards for Efficient Language Models | | 83.07 | LFM-7B (10-shot) | 2025-01-20 |
| OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training | ✓ Link | 82.98 | OptiMer (Gemma 3 27B) | 2026-03-30 |
| Falcon2-11B Technical Report | | 82.91 | Falcon2-11B (10-shot) | 2024-07-20 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | 82.8 | LLaMA 33B (0-shot) | 2023-02-27 |
| The Falcon Series of Open Language Models | | 82.7 | Falcon-40B (0-shot) | 2023-11-28 |
| Chameleon: Mixed-Modal Early-Fusion Foundation Models | ✓ Link | 82.7 | Chameleon 34B | 2024-05-16 |
| Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model | ✓ Link | 82.4 | Megatron-Turing NLG 530B (Few-Shot) | 2022-01-28 |
| Nemotron-4 15B Technical Report | ✓ Link | 82.4 | Nemotron-4 15B | 2024-02-26 |
| Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks | ✓ Link | 82.3 | Qwen2idae-16x14B (10-shot) | 2024-01-05 |
| Stay on topic with Classifier-Free Guidance | | 82.1 | LLaMA 13B + CFG (0-shot) | 2023-06-30 |
| Liquid Foundation Models: Our First Series of Generative AI Models | | 82.07 | LFM-40B Preview (10-shot) | 2024-09-30 |
| Training language models to follow instructions with human feedback | ✓ Link | 82.0 | InstructGPT 175B (few-shot) | 2022-03-04 |
| Advancing Polish Language Modeling through Tokenizer Optimization in the Bielik v3 7B and 11B Series | | 81.96 | Bielik-11B-v3.0-Instruct | 2026-04-12 |
| Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance | ✓ Link | 81.94 | Falcon-H1-34B-Instruct | 2025-07-30 |
| RoBERTa: A Robustly Optimized BERT Pretraining Approach | ✓ Link | 81.7 | RoBERTa-Large 355M | 2019-07-26 |
| Decoding Looped Transformers Better for (Almost) Free | | 81.66 | LoopCD-Ouro-2.6B (adaptive) | 2026-10-01 |
| The Zamba2 Suite: Technical Report | ✓ Link | 81.5 | Zamba2-7B (10-shot) | 2024-11-22 |
| Mistral 7B | ✓ Link | 81.3 | Mistral 7B (0-shot) | 2023-10-10 |
| Gemma: Open Models Based on Gemini Research and Technology | ✓ Link | 81.2 | Gemma 7B | 2024-03-13 |
| Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation | ✓ Link | 81.0 | T5Gemma 9B-9B PrefixLM | 2025-04-08 |
| Falcon Mamba: The First Competitive Attention-free 7B Language Model | ✓ Link | 80.82 | Falcon Mamba 7B | 2024-10-07 |
| Training Compute-Optimal Large Language Models | ✓ Link | 80.8 | Chinchilla 70B (0-shot) | 2022-03-29 |
| Llama 2: Open Foundation and Fine-Tuned Chat Models | ✓ Link | 80.7 | LLaMA 2 13B (0-shot) | 2023-07-18 |
| Compact Language Models via Pruning and Knowledge Distillation | ✓ Link | 80.7 | Minitron 8B | 2024-07-19 |
| Byte Latent Transformer: Patches Scale Better Than Tokens | ✓ Link | 80.6 | BLT-Entropy 8B | 2024-12-13 |
| RecurrentGemma: Moving Past Transformers for Efficient Open Language Models | ✓ Link | 80.4 | RecurrentGemma-9B | 2024-04-11 |
| Zamba: A Compact 7B SSM Hybrid Model | ✓ Link | 80.24 | Zamba 7B | 2024-05-26 |
| Fast-Decoding Diffusion Language Models via Progress-Aware Confidence Schedules | ✓ Link | 80.21 | Dream (Instruct) with SchED (Exp k=2, τhigh=7.5, τlow=2.5) | 2025-12-02 |
| Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model | ✓ Link | 80.2 | Megatron-Turing NLG 530B (1-shot) | 2022-01-28 |
| LoopRPT: Reinforcement Pre-Training for Looped Language Models | | 80.03 | Ouro (2.6B) LoopRPT | 2026-03-20 |
| OLMoE: Open Mixture-of-Experts Language Models | ✓ Link | 80.0 | OLMoE-1B-7B (5-shot) | 2024-09-03 |
| FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach | ✓ Link | 79.51 | FineRMoE (Qwen2.5-7B) | 2026-03-09 |
| Language Models are Few-Shot Learners | ✓ Link | 79.3 | GPT-3 175B (few-shot, k=32) | 2020-05-28 |
| Scaling Language Models: Methods, Analysis & Insights from Training Gopher | ✓ Link | 79.2 | Gopher 280B (0-shot) | 2021-12-08 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | 79.2 | LLaMA 13B (0-shot) | 2023-02-27 |
| HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing | | 79.0 | HySparse (80B MoE) | 2026-02-03 |
| Language Models are Few-Shot Learners | ✓ Link | 78.9 | GPT-3 (0-shot) | 2020-05-28 |
| ECO: Quantized Training without Full-Precision Master Weights | | 78.88 | DeepSeek-MoE-16B (INT4 w/o MW ECO + RTN) | 2026-01-29 |
| WUSH: Near-Optimal Adaptive Transforms for LLM Quantization | ✓ Link | 78.82 | Qwen3-14B with WUSH (NVFP4) | 2025-11-30 |
| OPT: Open Pre-trained Transformer Language Models | ✓ Link | 78.5 | OPT-175B | 2022-05-02 |
| T5Gemma 2: Seeing, Reading, and Understanding Longer | | 77.4 | T5Gemma 2 4B-4B | 2025-12-16 |
| Llama 2: Open Foundation and Fine-Tuned Chat Models | ✓ Link | 77.2 | LLaMA 2 7B (0-shot) | 2023-07-18 |
| LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models | | 77.19 | LLaDA MoE v2 Base | 2026-08-04 |
| Scheduling Recursive Reasoning in Looped Transformers | | 76.74 | Ouro-2.6B + TAPS (P/S-Sign) | 2026-09-29 |
| Improved Large Language Diffusion Models | ✓ Link | 76.6 | iLLaDA 8B Base | 2026-06-24 |
| OLMo: Accelerating the Science of Language Models | ✓ Link | 76.4 | OLMo 7B | 2024-02-01 |
| The Falcon Series of Open Language Models | | 76.3 | Falcon-7B (0-shot) | 2023-11-28 |
| Moshi: a speech-text foundation model for real-time dialogue | ✓ Link | 76.3 | Helium | 2024-09-17 |
| SmolLM3: smol, multilingual, long-context reasoner | ✓ Link | 76.15 | SmolLM3 3B Base | 2025-07-08 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | 76.1 | LLaMA 7B (0-shot) | 2023-02-27 |
| Introducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs | ✓ Link | 76.1 | MPT-7B | 2023-05-05 |
| The More Popular, The Harder to Forget: Adaptive Popularity for LLM Unlearning | ✓ Link | 76.0 | AdaPop (Llama-3.1-8B-Instruct) | 2026-08-14 |
| Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See | | 75.9 | Sophea-Nemo-3.5-Lightning-v1 | 2026-08-18 |
| DeepSeek LLM: Scaling Open-Source Language Models with Longtermism | ✓ Link | 75.7 | DeepSeek LLM 67B Chat | 2024-01-05 |
| GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression | | 75.38 | GeoPair (Llama3 8B, CR=0.2) | 2026-09-22 |
| Compact Language Models via Pruning and Knowledge Distillation | ✓ Link | 75.0 | Minitron 4B | 2024-07-19 |
| mHC: Manifold-Constrained Hyper-Connections | | 74.7 | mHC (27B) | 2025-12-31 |
| Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging | ✓ Link | 74.7 | MERIT (Proposed, 2D split, 4 groups, Qwen2.5-3B) | 2026-06-01 |
| Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler | ✓ Link | 74.6 | PowerLM-3B | 2024-08-23 |
| Large Language Diffusion Models | ✓ Link | 74.6 | LLaDA 8B Instruct | 2025-02-14 |
| Memory Caching: RNNs with Growing Memory | | 74.5 | Titans + GRM | 2026-02-27 |
| Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation | ✓ Link | 74.37 | Taylor-Calibrate qwen3_8b 700M_uniform_taylor | 2026-06-15 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | 73.9 | BlooombergGPT 50B (1-shot) | 2023-03-30 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | 73.5 | OPT 66B (1-shot) | 2023-03-30 |
| Dream 7B: Diffusion Large Language Models | | 73.3 | Dream 7B | 2025-08-21 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | 73.2 | BLOOM 176B (1-shot) | 2023-03-30 |
| MobileMoE: Scaling On-Device Mixture of Experts | | 73.0 | MobileMoE-L (922M) | 2026-05-26 |
| TIDE: Every Layer Knows the Token Beneath the Context | | 72.2 | TIDE-8E (3B) | 2026-05-07 |
| Differential Transformer | ✓ Link | 71.4 | Diff-3B | 2024-10-07 |
| Removing the NEEDLE in the Haystack: Backdoor Removal in LLMs via Weight Orthogonalisation | ✓ Link | 71.27 | NEEDLE (Qwen3-4B-Instruct-2507, Targeted refusal, BadNet) | 2026-09-29 |
| Efficient Pre-Training with Token Superposition | ✓ Link | 71.2 | MoE TST (10B A1B) | 2026-05-07 |
| daVinci-LLM:Towards the Science of Pretraining | ✓ Link | 71.17 | daVinci-LLM-3B | 2026-03-28 |
| RecurrentGemma: Moving Past Transformers for Efficient Open Language Models | ✓ Link | 71.0 | RecurrentGemma-2B | 2024-04-11 |
| Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning | ✓ Link | 70.8 | Sheared-LLaMA-2.7B (50B) | 2023-10-10 |
| Large Language Diffusion Models | ✓ Link | 70.5 | LLaDA 8B Base | 2025-02-14 |
| Initialisation Determines the Basin: Efficient Codebook Optimisation for Extreme LLM Quantization | ✓ Link | 70.4 | OA-EM (Llama 3.2 3B) | 2026-04-09 |
| Scaling Inherently Interpretable Language Models | ✓ Link | 70.3 | Steerling-8B | 2026-08-06 |
| RWKV: Reinventing RNNs for the Transformer Era | ✓ Link | 70.24 | RWKV-4 14B | 2023-05-22 |
| Llama 3.2: Revolutionizing edge AI and vision with open, customizable models | ✓ Link | 69.8 | Llama 3.2 3B Instruct | 2024-09-25 |
| Self-Improving Pretraining: using post-trained models to pretrain better models | | 69.4 | Self-Improving Pretraining (1.4B) | 2026-01-29 |
| Prefill-Free Cross-Family KV Cache Transfer for Heterogeneous Multi-Agent LLMs | ✓ Link | 69.16 | HeteroFold (Ministral-3-14B to Llama-3.1-8B) | 2026-09-29 |
| Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs | ✓ Link | 69.1 | Phi-4-Mini (5-shot) | 2025-02-24 |
| SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model | ✓ Link | 68.7 | SmolLM2 1.7B | 2025-02-04 |
| BitNet b1.58 2B4T Technical Report | ✓ Link | 68.44 | BitNet b1.58 2B4T | 2025-04-16 |
| LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation | ✓ Link | 68.44 | LatentLoop (7B / 1B active) | 2026-10-14 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | 68.4 | GPT-NeoX 20B (1-shot) | 2023-03-30 |
| DeepSeek-VL: Towards Real-World Vision-Language Understanding | ✓ Link | 68.4 | DeepSeek-VL 7B Chat | 2024-03-08 |
| Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning | ✓ Link | 67.6 | Open-LLaMA-3B-v2 | 2023-10-10 |
| Octrees as an Explicit 3D Language | ✓ Link | 67.44 | OctLLM | 2026-10-01 |
| Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models | ✓ Link | 67.43 | HSA-UltraLong-MoE | 2025-11-28 |
| Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts | ✓ Link | 67.3 | Upcycled 32→64 (100% CPT) | 2026-04-21 |
| DFM Mimir v1: An Open HRM Delivering Frontier Performance at 1B Parameters Using Only Permissible Post-Training Data | ✓ Link | 67.3 | Mimir 1B | 2026-08-13 |
| Universal YOCO for Efficient Depth Scaling | | 67.12 | YOCO-U (Equal Steps) | 2026-04-01 |
| Disentangling Representation Evolution in Transformers through Directional Decomposition | ✓ Link | 67.1 | Transformer Geometry 2.7B V-Para (10-shot) | 2026-09-14 |
| Post-Training Leaves Behavioral Shadows on Unrelated Decisions | ✓ Link | 66.82 | ATD student (Qwen2.5-1.5B-Instruct) | 2026-09-24 |
| Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality | ✓ Link | 66.6 | Mamba-2-2.7B | 2024-05-31 |
| NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction | ✓ Link | 66.4 | NCP-ArchPreview Stage-2 | 2026-09-09 |
| Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior | ✓ Link | 66.3 | PPT (k-Shuffle Dyck), 3B, C4, 100B PT tokens | 2026-09-30 |
| Mixture-of-Depths Attention | ✓ Link | 66.24 | MoDA (1.5B) | 2026-03-16 |
| GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model | ✓ Link | 66.1 | GPT-J 6B | 2021-06-04 |
| Mamba: Linear-Time Sequence Modeling with Selective State Spaces | ✓ Link | 66.1 | Mamba-2.8B | 2023-12-01 |
| SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model | ✓ Link | 66.1 | SmolLM2 1.7B Instruct | 2025-02-04 |
| Parallax: Parameterized Local Linear Attention for Language Modeling | ✓ Link | 64.54 | Parallax (1.7B) | 2026-05-27 |
| SmolLM - blazingly fast and remarkably powerful | ✓ Link | 64.1 | SmolLM 1.7B | 2024-07-16 |
| A Sovereign, Open-Source Foundation Model for German and English | ✓ Link | 64.1 | Soofi S 30B-A3B Base | 2026-07-10 |
| Apertus: Democratizing Open and Compliant LLMs for Global Language Environments | ✓ Link | 64.0 | Apertus-70B | 2025-09-17 |
| Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090 | ✓ Link | 63.62 | Puro-2B UD1/2 | 2026-08-27 |
| Scaling Laws for Looped Mixture of Experts | | 63.1 | A0.3B-1.3B LoopMoE (R=5) | 2026-09-30 |
| From Reweighting to Rewriting: Unlocking the Intervention Effects of Influential Samples in Training Data Attribution | | 62.93 | OLMo2-1B + influence rewriting (Opposed Harmful) | 2026-09-02 |
| MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning | ✓ Link | 62.8 | MatryoshkaLora (Llama-3.1-8B-Instruct) | 2026-05-08 |
| OLMo: Accelerating the Science of Language Models | ✓ Link | 62.5 | OLMo 1B | 2024-02-01 |
| MeKi: Memory-based Expert Knowledge Injection for Efficient LLM Scaling | ✓ Link | 62.3 | MeKi (4B) | 2026-02-03 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 62.3 | Mamba-3-MIMO-1.5B | 2026-03-16 |
| Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms | | 62.3 | Instruction-Queryable LoRA | 2026-05-08 |
| Multi-Head Low-Rank Attention | ✓ Link | 62.16 | MLRA-4 (2.9B) | 2026-03-02 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 61.9 | Mamba-3-SISO-1.5B | 2026-03-16 |
| Copy-Paste Language Models | ✓ Link | 61.78 | Copy-Paste Language Model (1.7B, SW-256) | 2026-10-13 |
| xLSTM: Extended Long Short-Term Memory | ✓ Link | 60.91 | xLSTM[1:0] 1.3B | 2024-05-07 |
| Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning | ✓ Link | 60.7 | Sheared-LLaMA-1.3B (50B) | 2023-10-10 |
| Kalman Delta Networks: Uncertainty-aware Associative Memory | ✓ Link | 60.68 | Diagonal KDN (1.3B) | 2026-09-07 |
| Target-Oriented Pretraining Data Selection via Neuron-Activated Graph | ✓ Link | 60.6 | NAG (Qwen3-1.7B) | 2026-04-17 |
| LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Models | ✓ Link | 60.58 | Phi-4 (14B) with LoopUS | 2026-05-10 |
| VersatileFFN: Achieving Parameter Efficiency in LLMs via Adaptive Wide-and-Deep Reuse | ✓ Link | 60.17 | VersatileFFN | 2025-12-16 |
| Low-Rank Approximation for Sparse Attention in Multi-Modal LLMs | | 60.1 | LoRA-Sparse (LLaMA-7B) (extra data) | 2024-06-17 |
| Sumi: Open Uniform Diffusion Language Model from Scratch | ✓ Link | 60.0 | Sumi-7B | 2026-06-17 |
| Debias-SparseGPT: Bias-Aware Pruning for Large Language Models | ✓ Link | 59.72 | Debias-SparseGPT (Mistral7B-v0.3-IT,1:4 sparse) | 2026-09-02 |
| Finetuned Language Models Are Zero-Shot Learners | ✓ Link | 59.2 | FLAN 137B (3-shot) | 2021-09-03 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 59.2 | Mamba-3-MIMO-880M | 2026-03-16 |
| Mamba: Linear-Time Sequence Modeling with Selective State Spaces | ✓ Link | 59.1 | Mamba-1.4B | 2023-12-01 |
| Modular TTT: Rethinking Test-Time Training as Composable Modules | ✓ Link | 58.96 | Modular TTT 1.45B (MSE M) | 2026-08-07 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 58.8 | Mamba-3-SISO-880M | 2026-03-16 |
| River-LLM: Large Language Model Seamless Exit Based on KV Share | | 58.6 | River-LLM (Llama3.1 8B, τ=0.7) | 2026-04-20 |
| Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention | ✓ Link | 58.46 | Gated DeltaNet-2 (1.3B) | 2026-05-21 |
| Data Science and Technology Towards AGI Part I: Tiered Data Management | ✓ Link | 58.27 | MiniCPM-1.2B (Math-L3) | 2026-02-09 |
| Circuit Hypernetworks for Quantum-Augmented Diffusion Language Models | | 57.75 | HyperQ, 64 q | 2026-09-21 |
| FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models | | 57.02 | FactorEngram (1B) | 2026-09-28 |
| Gated Delta Networks: Improving Mamba2 with Delta Rule | ✓ Link | 56.88 | Gated DeltaNet-H2 | 2024-12-09 |
| Finetuned Language Models Are Zero-Shot Learners | ✓ Link | 56.7 | FLAN 137B (0-shot) | 2021-09-03 |
| Parcae: Scaling Laws For Stable Looped Language Models | ✓ Link | 56.598949 | Parcae-1.3B (T=8, 10-shot) | 2026-04-14 |
| The State-Prediction Separation Hypothesis | ✓ Link | 56.3 | SPS XL | 2026-07-01 |
| In-Place Test-Time Training | ✓ Link | 55.85 | In-Place TTT (4B, Full Attention) | 2026-04-07 |
| Towards Looped Models Done Right, Part I: Topology, Input Injection, Recurrent-State Design | ✓ Link | 55.1 | Huginn MoE (8.0B resident, 0.8B active) | 2026-10-16 |
| Revisiting Complete Reasoning Traces for Post-Training | ✓ Link | 54.86 | Qwen3-4B-Base + E-SFT (OpenThoughts3-100K) | 2026-09-07 |
| Efficient Language Modeling with Sparse all-MLP | | 54.5 | sMLP – deterministic 9.4B (0-shot) | 2022-03-14 |
| Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action | ✓ Link | 54.3 | Unified-IO 2 XXL (zero-shot) | 2023-12-28 |
| GPT-NeoX-20B: An Open-Source Autoregressive Language Model | ✓ Link | 53.8 | GPT-NeoX-20B (5-shot) | 2022-04-14 |
| MDN: Parallelizing Stepwise Momentum for Delta Linear Attention | ✓ Link | 53.5 | MDN (1.3B) | 2026-05-07 |
| Refuse without Refusal: A Structural Analysis of Safety-Tuning Responses for Reducing False Refusals in Language Models | ✓ Link | 53.39 | Llama-3.1-8B + Rationale-Only | 2026-09-04 |
| Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in {pm 1, pm i} | ✓ Link | 53.33 | Fairy2i-W2 (2-bit) | 2025-12-02 |
| GPT-SW3: An Autoregressive Language Model for the Nordic Languages | | 53.2 | GPT-SW3 40B | 2023-05-22 |
| ICP: Immediate Compensation Pruning for Mid-to-high Sparsity | ✓ Link | 53.03 | ICP (Llama-2-7B, 50% sparsity) | 2025-06-11 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 52.8 | Mamba-3-MIMO-440M | 2026-03-16 |
| Efficient Language Modeling with Sparse all-MLP | | 52.5 | Switch Transformer 9B | 2022-03-14 |
| Group Representational Position Encoding | ✓ Link | 52.29 | GRAPE-AP (w/ KV-shift) | 2025-12-08 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 51.7 | Mamba-3-SISO-440M | 2026-03-16 |
| Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity | ✓ Link | 51.2 | Sparse-BitNet (Qwen2.5-3B, 6:8) | 2026-03-05 |
| Language Models are Few-Shot Learners | ✓ Link | 51.0 | GPT-3 Large 760M (0-shot) | 2020-05-28 |
| LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions | ✓ Link | 50.9 | GPT-2-XL 1.5B | 2023-04-27 |
| Learnable Multipliers: Freeing the Scale of Language Model Matrix Layers | ✓ Link | 50.56 | Falcon-H1-0.5B with Learnable Multipliers (Muon) | 2026-01-08 |
| OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration | | 50.54 | GPT-2 XL + OPUS | 2026-02-05 |
| Language Is Not All You Need: Aligning Perception with Language Models | ✓ Link | 50.3 | KOSMOS-1 (4-shot) | 2023-02-27 |
| LLM in a flash: Efficient Large Language Model Inference with Limited Memory | | 50.3 | OPT-6.7B | 2023-12-12 |
| Distributionally Robust Mixture-of-Experts Training | ✓ Link | 49.96 | DRMoET (1.7B active, 10.3B total, eta=0.001) | 2026-10-05 |
| LLM in a flash: Efficient Large Language Model Inference with Limited Memory | | 49.8 | LLM in a Flash (OPT-6.7B with Predictor) | 2023-12-12 |
| Kosmos-2: Grounding Multimodal Large Language Models to the World | ✓ Link | 49.4 | KOSMOS-2 | 2023-06-26 |
| Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts | ✓ Link | 48.8 | LOOM (1.7B, 9 loops) | 2026-10-01 |
| LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions | ✓ Link | 48.7 | FLAN-T5-Large 783M | 2023-04-27 |
| When Does Sparsity Mitigate the Curse of Depth in LLMs | ✓ Link | 48.63 | Transformer (2.3B, L=32) | 2026-03-16 |
| LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions | ✓ Link | 48.3 | LaMini-GPT 1.5B | 2023-04-27 |
| Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression | ✓ Link | 48.0 | LLaMA-7B with Swift-SVD* (dynamic) | 2026-04-02 |
| HellaSwag: Can a Machine Really Finish Your Sentence? | ✓ Link | 47.3 | BERT-Large 340M | 2019-05-19 |
| dLLM: Simple Diffusion Language Modeling | ✓ Link | 45.0 | ModernBERT-large-chat-v0.1 | 2026-02-26 |
| Error-Free Linear Attention is a Free Lunch: Exact Solution from Continuous-Time Dynamics | ✓ Link | 44.5 | EFLA | 2025-12-14 |
| ParameterNet: Parameters Are All You Need for Large-scale Visual Pretraining of Mobile Networks | ✓ Link | 44.21 | ParameterNet-MoE-up-proj-8 | 2024-06-17 |
| LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions | ✓ Link | 43.7 | LaMini-F-T5 783M | 2023-04-27 |
| ModuleFormer: Modularity Emerges from Mixture-of-Experts | ✓ Link | 43.33 | MoLM-8B-K2 (0-shot) | 2023-06-07 |
| GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow | ✓ Link | 42.73 | GPT-Neo 2.7B | 2021-03-21 |
| Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction | | 42.69 | GVA + DRoPE d_r=16 | 2026-09-08 |
| dLLM: Simple Diffusion Language Modeling | ✓ Link | 42.1 | Qwen3-0.6B-diffusion-mdlm-v0.1 | 2026-02-26 |
| HellaSwag: Can a Machine Really Finish Your Sentence? | ✓ Link | 41.7 | GPT-1 117M | 2019-05-19 |
| Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners | ✓ Link | 41.6 | Flipped-3B | 2022-10-06 |
| Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation | ✓ Link | 41.52 | GRT Large (isoParams, zero-shot) | 2026-08-15 |
| The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning | ✓ Link | 41.1 | T0-3B (CoT fine-tuned) | 2023-05-23 |
| Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention | | 41.01 | GQE (renorm. scoring + shared head) | 2026-06-18 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 41.0 | Mamba-3-MIMO-180M | 2026-03-16 |
| Mamba-3: Improved Sequence Modeling using State Space Principles | ✓ Link | 40.8 | Mamba-3-SISO-180M | 2026-03-16 |
| LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions | ✓ Link | 40.6 | LaMini-T5 738M | 2023-04-27 |
| HellaSwag: Can a Machine Really Finish Your Sentence? | ✓ Link | 40.5 | BERT-Base 110M | 2019-05-19 |
| Revisiting the Shape Convention of Transformer Language Models | | 40.3 | Hourglass (1074M) | 2026-02-06 |
| LinguDistill: Recovering Linguistic Ability in Vision- Language Models via Selective Cross-Modal Distillation | | 39.4 | LinguDistill | 2026-04-01 |
| LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions | ✓ Link | 38.9 | T5-Large 738M | 2023-04-27 |
| Fast Weight Attention for Continual Learning | ✓ Link | 38.38 | Falcon-1.3130M | 2026-08-27 |
| Efficient Language Modeling with Sparse all-MLP | | 38 | Gshard 9B | 2022-03-14 |
| Daedalus-150M: A Convolution-Attention Hybrid Designed for CPU Inference | ✓ Link | 37.93 | Daedalus-150M | 2026-08-20 |
| The Curse and Blessing of Mean Bias in FP4-Quantized LLM Training | | 37.51 | Qwen3-0.6B (Averis FP4) | 2026-03-11 |
| HellaSwag: Can a Machine Really Finish Your Sentence? | ✓ Link | 36.2 | LSTM + BERT-Base | 2019-05-19 |
| UniPool: A Globally Shared Expert Pool for Mixture-of-Experts | ✓ Link | 36.19 | UniPool (978M) | 2026-05-07 |
| Normalized Low-Rank Adaptation | ✓ Link | 36.11 | MLA342M NoRA-init | 2026-08-31 |
| Exploring the Benefits of Training Expert Language Models over Instruction Tuning | ✓ Link | 34.6 | RoE-3B | 2023-02-07 |
| Low-Rank Ternary Adaptation for Fine-Tuning Transformers | ✓ Link | 34.0 | Low-Rank Ternary Adaptation (Llama-3.2-3B) | 2026-08-25 |
| WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixing | ✓ Link | 33.8 | WhiteMatter k=16 | 2026-09-27 |
| HellaSwag: Can a Machine Really Finish Your Sentence? | ✓ Link | 33.3 | ESIM + ElMo | 2019-05-19 |
| Efficient Language Modeling with Sparse all-MLP | | 33 | HASH Layers 10B (0-shot) | 2022-03-14 |
| LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation | ✓ Link | 32.3 | LoopFormer (3x8) | 2026-02-11 |
| HellaSwag: Can a Machine Really Finish Your Sentence? | ✓ Link | 31.7 | LSTM + GloVe | 2019-05-19 |
| HellaSwag: Can a Machine Really Finish Your Sentence? | ✓ Link | 31.6 | fastText | 2019-05-19 |
| What Makes Recurrence Effective in Looped Language Models? | | 31.48 | BaseLoop (Llama3.1-1B, 3x4, Muon) | 2026-09-29 |
| HellaSwag: Can a Machine Really Finish Your Sentence? | ✓ Link | 31.4 | LSTM + ElMo | 2019-05-19 |
| Dynamic Linear Attention | | 30.8 | Mamba-2-780M w/ DLA | 2026-06-09 |
| PLDR-LLMs Reason At Self-Organized Criticality | ✓ Link | 30.55 | PLDRv51-SOC-110M-5 | 2026-03-12 |
| Efficient Language Modeling with Sparse all-MLP | | 30.2 | Base Layers 10B (0-shot) | 2022-03-14 |
| Knowledge-in-Context: Towards Knowledgeable Semi-Parametric Language Models | | 29.6 | KiC-770M | 2022-10-28 |
| Parcae: Scaling Laws For Stable Looped Language Models | ✓ Link | 29.23 | Parcae (350M) | 2026-04-14 |
| Recover-LoRA: Data-Free Accuracy Recovery of Degraded Language Models via Low-Rank Adaptation | ✓ Link | 26.52 | DeepSeek-R1-Distill-Qwen-1.5B + Recover-LoRA | 2025-10-06 |
| HellaSwag: Can a Machine Really Finish Your Sentence? | ✓ Link | 25 | Random chance baseline | 2019-05-19 |