| Hierarchical Prompting Taxonomy: A Universal Evaluation Framework for Large Language Models Aligned with Human Cognitive Principles | ✓ Link | 99.87 | Mistral-Nemo 12B (HPT) | 2024-06-18 |
| ST-MoE: Designing Stable and Transferable Sparse Expert Models | ✓ Link | 92.4 | ST-MoE-32B 269B (fine-tuned) | 2022-02-17 |
| PaLM: Scaling Language Modeling with Pathways | ✓ Link | 92.2 | PaLM 540B (fine-tuned) | 2022-04-05 |
| Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUE | | 92 | Turing NLR v5 XXL 5.4B (fine-tuned) | 2022-12-04 |
| HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Experts | | 91.3 | Qwen 3.5-35B-A3B (HodgeCover) | 2026-05-13 |
| Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer | ✓ Link | 91.2 | T5-XXL 11B (fine-tuned) | 2019-10-23 |
| PaLM 2 Technical Report | ✓ Link | 90.9 | PaLM 2-L (1-shot) | 2023-05-17 |
| UL2: Unifying Language Learning Paradigms | ✓ Link | 90.8 | UL2 20B (fine-tuned) | 2022-05-10 |
| Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUE | | 90.5 | Vega v2 6B (fine-tuned) | 2022-12-04 |
| DeBERTa: Decoding-enhanced BERT with Disentangled Attention | ✓ Link | 90.4 | DeBERTa-1.5B | 2020-06-05 |
| ST-MoE: Designing Stable and Transferable Sparse Expert Models | ✓ Link | 88.6 | ST-MoE-L 4.1B (fine-tuned) | 2022-02-17 |
| PaLM 2 Technical Report | ✓ Link | 88.6 | PaLM 2-M (1-shot) | 2023-05-17 |
| Jamba: A Hybrid Transformer-Mamba Language Model | ✓ Link | 88.2 | Jamba (10-shot) | 2024-03-28 |
| Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs | ✓ Link | 88.13 | SRR (Qwen3-30B-A3B, Task Arithmetic) | 2026-09-26 |
| PaLM 2 Technical Report | ✓ Link | 88.1 | PaLM 2-S (1-shot) | 2023-05-17 |
| BEAM: Binary Expert Activation Masking for Dynamic Routing in MoE | | 88.07 | Qwen3-30B-A3B (BEAM, beta=0.01) | 2026-05-14 |
| The Falcon Series of Open Language Models | ✓ Link | 87.8 | Falcon-180B | 2023-11-28 |
| DFM Mimir v1: An Open HRM Delivering Frontier Performance at 1B Parameters Using Only Permissible Post-Training Data | ✓ Link | 87.8 | Mimir 1B | 2026-08-13 |
| Muppet: Massive Multi-task Representations with Pre-Finetuning | ✓ Link | 87.5 | MUPPET Roberta Large | 2021-01-26 |
| Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone | ✓ Link | 86.5 | Phi-3-Medium (2-shot) | 2024-04-22 |
| Finetuned Language Models Are Zero-Shot Learners | ✓ Link | 86.3 | FLAN 137B (prompt-tuned) | 2021-09-03 |
| Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging | ✓ Link | 86.3 | MERIT (Proposed, 1D split, 2 groups, Qwen2.5-3B) | 2026-06-01 |
| HRM-Text: Efficient Pretraining Beyond Scaling | ✓ Link | 86.2 | HRM-Text-1B | 2026-05-20 |
| Entailment as Few-Shot Learner | ✓ Link | 86.0 | RoBERTa-large 355M + Entailment as Few-shot Learner | 2021-04-29 |
| Chameleon: Mixed-Modal Early-Fusion Foundation Models | ✓ Link | 86.0 | Chameleon 34B | 2024-05-16 |
| Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation | ✓ Link | 85.6 | T5Gemma 9B-9B PrefixLM | 2025-04-08 |
| Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer | ✓ Link | 85.4 | T5-Large 770M (fine-tuned) | 2019-10-23 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | 85.3 | LLaMA 65B (0-shot) | 2023-02-27 |
| Llama 2: Open Foundation and Fine-Tuned Chat Models | ✓ Link | 85 | LLaMA 2 70B (0-shot) | 2023-07-18 |
| Gemma 2: Improving Open Language Models at a Practical Size | ✓ Link | 84.8 | Gemma 2 27B | 2024-07-31 |
| Finetuned Language Models Are Zero-Shot Learners | ✓ Link | 84.6 | FLAN 137B (4-shot) | 2021-09-03 |
| Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation | ✓ Link | 84.56 | Taylor-Calibrate qwen3_8b 700M_uniform_taylor | 2026-06-15 |
| The Zamba2 Suite: Technical Report | ✓ Link | 84.1 | Zamba2-7B | 2024-11-22 |
| Muppet: Massive Multi-task Representations with Pre-Finetuning | ✓ Link | 83.8 | MUPPET Roberta Base | 2021-01-26 |
| Training Compute-Optimal Large Language Models | ✓ Link | 83.7 | Chinchilla 70B (0-shot) | 2022-03-29 |
| Llama 2: Open Foundation and Fine-Tuned Chat Models | ✓ Link | 83.7 | LLaMA 2 34B (0-shot) | 2023-07-18 |
| Zamba: A Compact 7B SSM Hybrid Model | ✓ Link | 83.6 | Zamba 7B | 2024-05-26 |
| River-LLM: Large Language Model Seamless Exit Based on KV Share | | 83.4 | River-LLM (Llama3.1 8B, τ=0.5) | 2026-04-20 |
| Gemma: Open Models Based on Gemini Research and Technology | ✓ Link | 83.2 | Gemma 7B | 2024-03-13 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | 83.1 | LLaMA 33B (0-shot) | 2023-02-27 |
| Finetuned Language Models Are Zero-Shot Learners | ✓ Link | 82.9 | FLAN 137B (0-shot) | 2021-09-03 |
| Llama 2: Open Foundation and Fine-Tuned Chat Models | ✓ Link | 81.7 | LLaMA 2 13B (0-shot) | 2023-07-18 |
| GLU Variants Improve Transformer | ✓ Link | 81.53 | FFN Bilinear | 2020-02-12 |
| LFM2.5-Encoders: Fast at Long Context, Even on CPU | ✓ Link | 81.52 | LFM2.5-Encoder-350M | 2026-07-28 |
| Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer | ✓ Link | 81.4 | T5-Base 220M (fine-tuned) | 2019-10-23 |
| Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs | ✓ Link | 81.2 | Phi-4-Mini (2-shot) | 2025-02-24 |
| MobileMoE: Scaling On-Device Mixture of Experts | | 81.1 | MobileMoE-L (922M) | 2026-05-26 |
| ReRound: Reconstructive Rounding to Resolve Midpoint Ambiguity in Calibration-Free LLM Quantization | ✓ Link | 80.7 | ReRound (Phi-2 2.7B, W4A16, G=128) | 2026-08-11 |
| BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions | ✓ Link | 80.4 | BERT-MultiNLI 340M (fine-tuned) | 2019-05-24 |
| RecurrentGemma: Moving Past Transformers for Efficient Open Language Models | ✓ Link | 80.3 | RecurrentGemma-9B | 2024-04-11 |
| BitNet b1.58 2B4T Technical Report | ✓ Link | 80.18 | BitNet b1.58 2B4T | 2025-04-16 |
| Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090 | ✓ Link | 79.48 | Puro-2B UD full | 2026-08-27 |
| Scaling Language Models: Methods, Analysis & Insights from Training Gopher | ✓ Link | 79.3 | Gopher (zero-shot) | 2021-12-08 |
| T5Gemma 2: Seeing, Reading, and Understanding Longer | | 79.3 | T5Gemma 2 4B-4B | 2025-12-16 |
| SmolLM3: smol, multilingual, long-context reasoner | ✓ Link | 78.99 | SmolLM3 3B Base | 2025-07-08 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | 78.1 | LLaMA 13B (zero-shot) | 2023-02-27 |
| Llama 2: Open Foundation and Fine-Tuned Chat Models | ✓ Link | 77.4 | LLaMA 2 7B (zero-shot) | 2023-07-18 |
| Cross-Tokenizer LLM Distillation through a Byte-Level Interface | | 77.34 | Llama3.2 (3B) with BLCTD (transferred to Qwen2 tokenizer) | 2026-04-13 |
| Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action | ✓ Link | 77.3 | Unified-IO 2 XXL | 2023-12-28 |
| MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts | ✓ Link | 77.1 | LLaMA-2 13B + MixLoRA | 2024-04-22 |
| LLaMA: Open and Efficient Foundation Language Models | ✓ Link | 76.5 | LLaMA 7B (zero-shot) | 2023-02-27 |
| Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer | ✓ Link | 76.4 | T5-Small 60M (fine-tuned) | 2019-10-23 |
| Language Models are Few-Shot Learners | ✓ Link | 76.4 | GPT-3 175B (few-shot, k=32) | 2020-05-28 |
| BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions | ✓ Link | 75.57 | BiDAF-MultiNLI (fine-tuned) | 2019-05-24 |
| Introducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs | ✓ Link | 75.0 | MPT-7B | 2023-05-05 |
| MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts | ✓ Link | 75 | LLaMA-3 8B + MixLoRA | 2024-04-22 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | 74.6 | Bloomberg GPT 50B (1-shot) | 2023-03-30 |
| Mixture-of-Subspaces in Low-Rank Adaptation | ✓ Link | 74.6 | LLaMA3+MoSLoRA | 2024-06-16 |
| MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookup | ✓ Link | 74.25 | MoME (100B tokens) | 2026-09-14 |
| OLMo: Accelerating the Science of Language Models | ✓ Link | 73.4 | OLMo 7B | 2024-02-01 |
| BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions | ✓ Link | 72.87 | GPT-1 117M (fine-tuned) | 2019-05-24 |
| MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts | ✓ Link | 72.7 | LLaMA-2 7B + MixLoRA | 2024-04-22 |
| TIDE: Every Layer Knows the Token Beneath the Context | | 72.3 | TIDE-8E (3B) | 2026-05-07 |
| Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler | ✓ Link | 72.0 | PowerLM-3B | 2024-08-23 |
| BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions | ✓ Link | 71.41 | BiDAF + ELMo (fine-tuned) | 2019-05-24 |
| OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization | ✓ Link | 71.4 | OPT-IML 175B | 2022-12-22 |
| RecurrentGemma: Moving Past Transformers for Efficient Open Language Models | ✓ Link | 71.3 | RecurrentGemma-2B | 2024-04-11 |
| Self-Improving Pretraining: using post-trained models to pretrain better models | | 70.3 | Self-Improving Pretraining (1.4B) | 2026-01-29 |
| AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model | ✓ Link | 69.4 | AlexaTM 20B | 2022-08-02 |
| Efficient Pre-Training with Token Superposition | ✓ Link | 69.4 | MoE TST (10B A1B) | 2026-05-07 |
| Differential Transformer | ✓ Link | 69.0 | Diff-3B | 2024-10-07 |
| Ask Me Anything: A simple strategy for prompting language models | ✓ Link | 67.2 | Neo-6B (QA + WS) | 2022-10-05 |
| OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization | ✓ Link | 66.9 | OPT-IML 30B | 2022-12-22 |
| Ask Me Anything: A simple strategy for prompting language models | ✓ Link | 66.5 | Neo-6B (few-shot) | 2022-10-05 |
| OPT: Open Pre-trained Transformer Language Models | ✓ Link | 65 | OPT-175B | 2022-05-02 |
| N-Grammer: Augmenting Transformers with latent n-grams | ✓ Link | 65 | N-Grammer 343M | 2022-07-13 |
| Ask Me Anything: A simple strategy for prompting language models | ✓ Link | 64.9 | Neo-6B (QA) | 2022-10-05 |
| Parallax: Parameterized Local Linear Attention for Language Modeling | ✓ Link | 64.59 | Parallax (1.7B) | 2026-05-27 |
| MeKi: Memory-based Expert Knowledge Injection for Efficient LLM Scaling | ✓ Link | 64.4 | MeKi (4B) | 2026-02-03 |
| Disentangling Representation Evolution in Transformers through Directional Decomposition | ✓ Link | 64.4 | Transformer Geometry 2.7B V-Para (zero-shot) | 2026-09-14 |
| OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization | ✓ Link | 64 | OPT 30B (0-shot) | 2022-12-22 |
| Maglev: Sliding Recurrent Memory | | 64.0 | Maglev (separate parameters, lambda=1) | 2026-08-05 |
| Gated Delta Networks: Improving Mamba2 with Delta Rule | ✓ Link | 63.21 | Gated DeltaNet-H1 | 2024-12-09 |
| UL2: Unifying Language Learning Paradigms | ✓ Link | 63.1 | UL2 20B (0-shot) | 2022-05-10 |
| Parcae: Scaling Laws For Stable Looped Language Models | ✓ Link | 63.088683 | Parcae-1.3B (T=8, 10-shot) | 2026-04-14 |
| Circuit Hypernetworks for Quantum-Augmented Diffusion Language Models | | 62.74 | HyperQ, 64 q | 2026-09-21 |
| Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention | ✓ Link | 62.57 | Gated DeltaNet-2 (1.3B) | 2026-05-21 |
| Modular TTT: Rethinking Test-Time Training as Composable Modules | ✓ Link | 62.35 | Modular TTT 1.45B (inner-product M) | 2026-08-07 |
| BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions | ✓ Link | 62.17 | Majority baseline | 2019-05-24 |
| Recover-LoRA: Data-Free Accuracy Recovery of Degraded Language Models via Low-Rank Adaptation | ✓ Link | 62.17 | AMD OLMo SFT 1B + Recover-LoRA | 2025-10-06 |
| MDM-Prime-v2: Binary Encoding and Index Shuffling Enable Scaling of Diffusion Language Models | ✓ Link | 62.05 | MDM-Prime-v2 (1.1B) | 2026-03-17 |
| Kosmos-2: Grounding Multimodal Large Language Models to the World | ✓ Link | 62.0 | KOSMOS-2 | 2023-06-26 |
| Multi-Head Low-Rank Attention | ✓ Link | 61.74 | MLRA-4 (2.9B) | 2026-03-02 |
| Hungry Hungry Hippos: Towards Language Modeling with State Space Models | ✓ Link | 61.7 | Hybrid H3 1.3B (0-shot, logit scoring) | 2022-12-28 |
| OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization | ✓ Link | 61.5 | OPT-IML 1.3B (0-shot) | 2022-12-22 |
| SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations | ✓ Link | 61.4 | Token Rounding (1.8B MoE) | 2025-12-16 |
| SHAKTI: A 2.5 Billion Parameter Small Language Model Optimized for Edge AI and Low-Resource Environments | | 61.1 | Shakti-LLM (2.5B) | 2024-10-15 |
| WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixing | ✓ Link | 61.04 | WhiteMatter k=16 | 2026-09-27 |
| Variable-Width Transformers | ✓ Link | 60.9 | 2B > <former | 2026-06-16 |
| FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models | | 60.88 | FactorEngram (1B) | 2026-09-28 |
| Parcae: Scaling Laws For Stable Looped Language Models | ✓ Link | 60.73 | Parcae (350M) | 2026-04-14 |
| OLMo: Accelerating the Science of Language Models | ✓ Link | 60.7 | OLMo 1B | 2024-02-01 |
| Hungry Hungry Hippos: Towards Language Modeling with State Space Models | ✓ Link | 60.6 | Hybrid H3 2.7B (3-shot, logit scoring) | 2022-12-28 |
| Language Models are Few-Shot Learners | ✓ Link | 60.5 | GPT-3 75B (0-shot) | 2020-05-28 |
| OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization | ✓ Link | 60.5 | OPT 1.3B (zero-shot) | 2022-12-22 |
| Error-Free Linear Attention is a Free Lunch: Exact Solution from Continuous-Time Dynamics | ✓ Link | 60.4 | EFLA | 2025-12-14 |
| Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation | ✓ Link | 60.18 | GRT Large (isoFLOPs, zero-shot) | 2026-08-15 |
| OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization | ✓ Link | 60.1 | OPT 175B | 2022-12-22 |
| MDN: Parallelizing Stepwise Momentum for Delta Linear Attention | ✓ Link | 59.63 | MDN (1.3B) | 2026-05-07 |
| Hungry Hungry Hippos: Towards Language Modeling with State Space Models | ✓ Link | 59.6 | Hybrid H3 125M (0-shot, logit scoring) | 2022-12-28 |
| Language Is Not All You Need: Aligning Perception with Language Models | ✓ Link | 59.2 | KOSMOS-1 (4-shot) | 2023-02-27 |
| ParameterNet: Parameters Are All You Need for Large-scale Visual Pretraining of Mobile Networks | ✓ Link | 58.58 | ParameterNet-MoE-down-proj-8 | 2024-06-17 |
| OpenESM: Energy-Steered Models Are Scalable Foundation Learners | ✓ Link | 57.83 | OpenESM FineWeb d26 | 2026-10-07 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | 57.5 | OPT 66B (1-shot) | 2023-03-30 |
| The Curse and Blessing of Mean Bias in FP4-Quantized LLM Training | | 57.46 | Qwen3-0.6B (Averis FP4) | 2026-03-11 |
| Hungry Hungry Hippos: Towards Language Modeling with State Space Models | ✓ Link | 56.1 | Hybrid H3 125M (3-shot, logit scoring) | 2022-12-28 |
| Hungry Hungry Hippos: Towards Language Modeling with State Space Models | ✓ Link | 56.1 | Hybrid H3 125M (3-shot, rank classification) | 2022-12-28 |
| Looped Uniform Diffusion Language Models | ✓ Link | 53.31 | Looped UDLM | 2026-10-21 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | 52.9 | BLOOM 176B (1-shot) | 2023-03-30 |
| Hyena Hierarchy: Towards Larger Convolutional Language Models | ✓ Link | 51.8 | Hyena | 2023-02-21 |
| Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity | ✓ Link | 51.35 | Sparse-BitNet (Qwen2.5-3B, 6:8) | 2026-03-05 |
| BloombergGPT: A Large Language Model for Finance | ✓ Link | 46.4 | GPT-NeoX 20B (1-shot) | 2023-03-30 |
| Memory Caching: RNNs with Growing Memory | | 43.9 | Titans + GRM | 2026-02-27 |