OpenCodePapers

multi-task-language-understanding-on-mmlu-pro

Natural Language UnderstandingMulti-task Language Understanding
Dataset Link
Results over time
Click legend items to toggle metrics. Hover points for model names.
Leaderboard
Click a sortable column header to sort.
PaperCodeAccuracyModelNameReleaseDate
Intern-S2-Preview: Scientific Agentic Foundation Model✓ Link89.75Intern-S2-Preview-397B2026-08-13
Intern-S2-Mobius: Foundation Model with Decoupled Knowledge and Reasoning✓ Link89.05Intern-S2-Mobius-35B2026-08-14
Qwen3.5: Towards Native Multimodal Agents✓ Link87.8Qwen3.5-397B-A17B2026-02-16
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence✓ Link87.5DeepSeek-V4-Pro Max2026-04-26
Kimi K2.5: Visual Agentic Intelligence✓ Link87.1Kimi K2.52026-02-02
Seed2.0 Model Card: Towards Intelligence Frontier for Real-World Complexity87.0Seed2.0 Pro2026-06-30
Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning✓ Link86.8Nemotron 3 Ultra2026-06-03
Qwen3.5: Towards Native Multimodal Agents✓ Link86.7Qwen3.5-122B-A10B2026-02-16
SenseNova-U1.5: Towards Native Unified Visual Intelligence✓ Link86.67SenseNova-U1.5 8B-Think2026-09-10
Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion Scale86.6Intern-S1-Pro2026-03-26
Qwen3.6✓ Link86.2Qwen3.6-27B2026-04-21
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence✓ Link86.2DeepSeek-V4-Flash Max2026-04-26
Qwen3.5: Towards Native Multimodal Agents✓ Link86.1Qwen3.5-27B2026-02-16
Qwen3.5-Omni Technical Report85.9Qwen3.5-Omni-Plus2026-04-21
Introducing efficiency focused "Thinking CAP" model series✓ Link85.4ThinkingCap-Qwen3.6-27B2026-07-06
Qwen3.5: Towards Native Multimodal Agents✓ Link85.3Qwen3.5-35B-A3B2026-02-16
Qwen3.6✓ Link85.2Qwen3.6-35B-A3B2026-04-21
Gemma 4 Technical Report✓ Link85.2Gemma 4 31B2026-07-02
DeepSeek-V3.2 Technical Report✓ Link85.0DeepSeek-V3.22025-12-02
MAI-Thinking-1: Building a Hill-Climbing Machine85MAI-Thinking-12026-06-02
MiMo-V2-Flash Technical Report✓ Link84.9MiMo-V2-Flash (Thinking)2026-01-06
Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters✓ Link84.8Step-3.5-Flash PaCoRe2026-02-11
Capek 0.5: An Execution-Centric Vision-Language Model for Embodied Intelligence84.77Capek0.5-35B-A3B2026-08-07
GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models✓ Link84.6GLM-4.5 (355B)2025-08-08
Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters✓ Link84.4Step-3.5-Flash2026-02-11
GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models✓ Link84.3GLM-4.72025-08-08
SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillation✓ Link84.1SecOPD (Qwen3.6-27B)2026-08-21
SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecture✓ Link84.04SenseNova-U1-A3B-MoT2026-05-12
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning✓ Link84.0DeepSeek-R12025-01-22
Qwen3-VL Technical Report✓ Link83.8Qwen3-VL-235B-A22B-Thinking2025-11-26
K-EXAONE Technical Report✓ Link83.8K-EXAONE (236B, reasoning)2026-01-05
ERNIE 5.0 Technical Report83.8ERNIE 5.02026-02-04
NVIDIA Nemotron 3: Efficient and Open Intelligence✓ Link83.73Nemotron 3 Super2025-12-24
Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning✓ Link83.73Nemotron 3 Super (post-trained)2026-04-14
EXAONE 4.5 Technical Report✓ Link83.3EXAONE 4.5 (Reasoning)2026-04-09
LongCat-Flash Technical Report✓ Link82.68LongCat-Flash2025-09-01
Gemma 4 Technical Report✓ Link82.6Gemma 4 26B-A4B2026-07-02
Qwen3.5: Towards Native Multimodal Agents✓ Link82.5Qwen3.5-9B2026-02-16
NVIDIA Nemotron 3.5 Lightning✓ Link81.94NVIDIA Nemotron 3.5 Lightning 30B-A3B2026-08-11
INTELLECT-3: Technical Report✓ Link81.9INTELLECT-32025-12-18
EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modes✓ Link81.8EXAONE 4.0 32B (Reasoning)2025-07-15
The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence✓ Link81.8MiniMax-M2.72026-05-26
GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models✓ Link81.4GLM-4.5-Air2025-08-08
QwenLong-L1.5: Post-Training Recipe for Long-Context Reasoning and Memory Management✓ Link81.33QwenLong-L1.5-30B-A3B2025-12-15
Kimi K2: Open Agentic Intelligence✓ Link81.1Kimi K22025-07-28
The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction✓ Link81.0Edge0-35B-A3B-preview2026-09-16
Qwen3-Next80.6Qwen3-Next-80B-A3B-Instruct2025-09-09
Qwen3-Coder-Next Technical Report✓ Link80.52Qwen3-Coder-Next2026-02-28
The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation✓ Link80.5Llama 4 Maverick2025-04-05
Solar Open Technical Report✓ Link80.4Solar Open (102B)2026-01-11
Kolibri: A Sovereign European Model on the Pareto Frontier✓ Link80.0Kolibri2026-10-03
Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation79.8Nemotron-Cascade-2-30B-A3B2026-03-19
Introspective Diffusion Language Models✓ Link79.7I-DLM-32B2026-04-13
Unified Audio Intelligence Without Regressing on Text Intelligence78.9Audex 30B-A3B2026-07-06
ERNIE 4.5 Technical Report✓ Link78.4ERNIE-4.5-300B-A47B2025-06-30
Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning✓ Link78.3NVIDIA-Nemotron-3-Nano-30B-A3B-BF162025-12-23
Scaling Embeddings Outperforms Scaling Experts in Language Models78.29LongCat-Flash-Lite (68.5B)2026-01-29
DiffusionGemma Technical Report✓ Link77.6DiffusionGemma 26B A4B2026-07-31
Granite 4.2 LLMs: How They're Built✓ Link77.6Granite-4.2-30B2026-08-25
Apriel-Reasoner: RL Post-Training for General-Purpose and Efficient Reasoning✓ Link77.3Apriel-1.5-OpenReasoner2026-04-02
Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence✓ Link77.3Nemotron 3 Nano Omni2026-04-27
Gemma 4 Technical Report✓ Link77.2Gemma 4 12B Unified2026-07-02
STEP3-VL-10B Technical Report✓ Link77.09Step3-VL-10B (PaCoRe)2026-01-14
Qwen3.8-27B-Uncensored76.8Qwen3.8-27B-Uncensored2026-08-18
Introducing Gemini 2.0: our new AI model for the agentic era76.4Gemini 2.0 Flash Experimental2024-12-11
ACC: Compiling Agent Trajectories for Long-Context Training76.0Qwen3-30B-A3B-Thinking + ACC2026-05-21
DeepSeek-V3 Technical Report✓ Link75.9DeepSeek-V32024-12-27
Is Next-Chunk Reasoning RL Really Better than SFT? Revisiting Training Strategies under no-CoT Data75.84Mixed SFT + RLVR2026-08-24
MiniMax-01: Scaling Foundation Models with Lightning Attention✓ Link75.7MiniMax-Text-012025-01-14
Arcee Trinity Large Technical Report✓ Link75.25Trinity Large Preview2026-02-19
Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability✓ Link74.4Qwen3-14B (Math-CoT SFT)2026-04-08
The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation✓ Link74.3Llama 4 Scout2025-04-05
RISE: Recursive Improvement via Self-Extrapolating Policy Distillation74.3RISE (logit,Qwen3-8B)2026-09-04
ZAYA1-8B Technical Report✓ Link74.2ZAYA1-8B2026-05-06
DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression✓ Link74.1DeepSeek-V4.1-Flash Base2026-09-17
LLaDA2.0: Scaling Up Diffusion Language Models to 100B✓ Link73.36LLaDA2.0-flash2025-12-10
Introducing Llama 3.1: Our most capable models to date✓ Link73.3Llama 3.1 (405B, Instruct)2024-07-23
On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability✓ Link73.23Qwen3.8-Flash-Next-Base2026-08-31
daVinci-Env: Open SWE Environment Synthesis at Scale✓ Link72.7OpenSWE-72B (Qwen2.5-72B-Base)2026-03-13
Qwen2.5 Technical Report✓ Link72.5Qwen2.5-Plus2024-12-19
Self-Hinting Language Models Enhance Reinforcement Learning✓ Link72.5SAGE (Qwen3-4B-Instruct)2026-02-03
DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards✓ Link72.27DelTA (Qwen3-14B-Base)2026-05-20
From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs✓ Link71.9NBDiff-7B (Instruct)2025-12-07
SEIF: Self-Evolving Reinforcement Learning for Instruction Following✓ Link71.4SEIF (Distill-Qwen-14B)2026-05-08
Qwen2.5-VL Technical Report✓ Link71.2Qwen2.5-VL-72B2025-02-19
Data Science and Technology Towards AGI Part I: Tiered Data Management✓ Link70.8MiniCPM5-2B2026-02-09
MiniCPM5-2B✓ Link70.8MiniCPM5-2B2026-09-07
dots.llm1 Technical Report✓ Link70.4dots.llm1.inst2025-06-06
SCALER:Synthetic Scalable Adaptive Learning Environment for Reasoning✓ Link70.0Qwen3-4B-base + SCALER2026-01-08
Gemma 4 Technical Report✓ Link69.4Gemma 4 E4B2026-07-02
Motif 3: Technical Report✓ Link68.56Motif 3 Base (5-shot CoT)2026-08-10
CoWindow Attention: Full Causal Coverage Is a Collective Property68.51CoWA (32B)2026-09-26
MiMo-V2.5-Pro✓ Link68.5MiMo-V2.5-Pro2026-04-27
MassAlloc Attention: Let Attention Allocate Its Own Compute✓ Link68.33MALA (32B)2026-09-26
Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients68.2ZPPO (Qwen3.5-2B)2026-06-16
Qwen3 Technical Report✓ Link68.18Qwen3-235B-A22B (Base)2025-05-14
Gemma 3 Technical Report✓ Link67.5Gemma 3 (27B, IT) (5-shot)2025-03-25
MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling67.04MiniCPM-SALA (9B)2026-02-12
Introducing Llama 3.1: Our most capable models to date✓ Link66.4Llama 3.1 (70B, Instruct)2024-07-23
Recursive Think-Answer Process for LLMs and VLMs65.9MiMo-RL-7B-R-TAP2026-03-02
PretrainZero: Reinforcement Active Pretraining64.59PretrainZero (Qwen3-30B-A3B-MoE-Base)2025-12-03
Qwen2 Technical Report✓ Link64.4Qwen2-72B-Instruct2024-07-15
Sci-CoE: Co-evolving Scientific Reasoning LLMs via Geometric Consensus with Sparse Supervision✓ Link64.34Sci-CoE-Stage 2-30k (Qwen3-8B)2026-02-12
Raon-Speech Technical Report✓ Link64.05Raon-Speech2026-04-08
J-Zero: Unified Challenger--Solver--Judge Co-Evolution from Zero Data✓ Link63.8J-Zero-8B2026-08-27
From P(y|x) to P(y): Investigating Reinforcement Learning in Pre-train Space✓ Link63.79DSRL (Qwen3-8B)2026-04-15
Guided Self-Evolving LLMs with Minimal Human Supervision63.2Qwen3-8B-Base (R-Few 5%)2025-12-02
How2Everything: Mining the Web for How-To Procedures to Evaluate and Improve LLMs✓ Link63.11Qwen3-8B (Instruct) + RL2026-02-09
Coupled Variational Reinforcement Learning for Language Model General Reasoning✓ Link63.1Qwen3-14B with CoVRL2025-12-14
DiagEvo: Diagnosis-Guided Self-Evolution via Hierarchical Error Memory63.1DiagEvo (Qwen3-8B, 235B-A22B diagnostician)2026-09-01
DARC: Decoupled Asymmetric Reasoning Curriculum for LLM Evolution✓ Link62.3DARC (Qwen3-8B-Base)2026-01-20
Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters✓ Link62.3Step-3.5-Flash Base2026-02-11
Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models✓ Link61.8Nemotron-H-47B-Base (5-shot)2025-04-04
Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Models✓ Link61.6Youtu-LLM 2B (5-shot)2025-12-31
Learning to Hint for Reinforcement Learning✓ Link61.6HiLL (Qwen2.5-7B-Instruct)2026-04-01
Co-Evolving LLM Decision and Skill Bank Agents for Long-Horizon Tasks✓ Link61.15COS-PLAY2026-04-22
Can LLMs Learn to Reason Robustly under Noisy Supervision?✓ Link60.6Qwen3-4B-Base (with OLR)2026-04-05
Gemma 4 Technical Report✓ Link60.0Gemma 4 E2B2026-07-02
Mellum2 Technical Report✓ Link59.3Mellum 2 (base)2026-05-29
RelayLLM: Efficient Reasoning via Collaborative Decoding✓ Link59.03Qwen3-1.7B (RelayLLM Difficulty-Aware-Reward)2026-01-08
Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance✓ Link58.73Falcon-H1-34B-Instruct (5-shot)2025-07-30
DarkForest: Less Talk, Higher Accuracy for Multi-Agent LLMs✓ Link58.38DarkForest2026-05-24
Stratagem: Learning Transferable Reasoning via Trajectory-Modulated Game Self-Play✓ Link57.83Stratagem (All Games)2026-04-20
Difficulty-Adaptive Tree-Structured Policy Optimization for Expanding Reasoning Coverage in RLVR✓ Link57.7DATPO (Qwen3-4B-Base)2026-09-08
Decoding Looped Transformers Better for (Almost) Free57.5LoopCD-Ouro-2.6B (fixed)2026-10-01
LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models57.28LLaDA MoE v2 Base2026-08-04
Qwen2.5 Technical Report✓ Link56.3Qwen2.5-7B-Instruct2024-12-19
Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation✓ Link55.7T5Gemma 9B-9B PrefixLM IT2025-04-08
Liquid Foundation Models: Our First Series of Generative AI Models55.63LFM-40B Preview (5-shot)2024-09-30
Qwen2 Technical Report✓ Link55.6Qwen2-72B2024-07-15
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone✓ Link54.3Phi-3.5-MoE2024-04-22
LoopRPT: Reinforcement Pre-Training for Looped Language Models54.19Ouro (2.6B) LoopRPT2026-03-20
Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation✓ Link53.7OPDLM-8B2026-06-04
Laguna M.1/XS.2 Technical Report✓ Link53.0Laguna XS.22026-05-26
Qwen2 Technical Report✓ Link52.8Qwen2-57B-A14B-Instruct2024-07-15
Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs✓ Link52.8Phi-4-Mini (0-shot, CoT)2025-02-24
Improved Large Language Diffusion Models✓ Link52.3iLLaDA 8B Instruct2026-06-24
Attention Residuals✓ Link52.2Kimi Linear (48B) with Block AttnRes2026-03-16
SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training51.44SlimQwen (23A2B, NTP KD + LM)2026-05-09
A Sovereign, Open-Source Foundation Model for German and English✓ Link51.4Soofi S 30B-A3B Base2026-07-10
TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning✓ Link50.7Qwen2.5-Math-7B with TraPO (4K labels, 12K unlabeled)2025-12-15
LFM2.5-1.2B-Thinking: On-Device Reasoning Under 1GB49.65LFM2.5-1.2B-Thinking2026-01-20
WildReward: Learning Reward Models from In-the-Wild Human Interactions✓ Link48.9Llama3.1-8B-Instruct (online DPO with WildReward)2026-02-09
MiniCPM5-1B✓ Link48.85MiniCPM5-1B (Thinking)2026-05-25
Introducing Llama 3.1: Our most capable models to date✓ Link48.3Llama 3.1 (8B, Instruct)2024-07-23
The Llama 3 Herd of Models✓ Link48.3Llama 3.1 (8B, Instruct)2024-07-31
Qwen2.5-Omni Technical Report✓ Link47.0Qwen2.5-Omni-7B2025-03-26
2 OLMo 2 Furious✓ Link46.9OLMo 2 32B2024-12-31
ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decoding✓ Link45.94ReFusion2025-12-15
T5Gemma 2: Seeing, Reading, and Understanding Longer44.4T5Gemma 2 4B-4B (post-trained)2025-12-16
Introducing LFM2.5: The Next Generation of On-Device AI44.35LFM2.5-1.2B-Instruct2026-01-05
Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models✓ Link44.0Nemotron-H-8B-Base (5-shot)2025-04-04
daVinci-LLM:Towards the Science of Pretraining✓ Link43.5daVinci-LLM-3B2026-03-28
Introducing LFM-7B: Setting New Standards for Efficient Language Models42.42LFM-7B (5-shot)2025-01-20
The Falcon 3 Family of Open Models38.10Falcon3-10B-Instruct2024-12-17
LFM2 Technical Report✓ Link37.42LFM2-8B-A1B (5-shot CoT)2025-11-28
2 OLMo 2 Furious✓ Link35.1OLMo 2 13B2024-12-31
SmolLM3: smol, multilingual, long-context reasoner✓ Link32.7SmolLM3 3B Base (MCF)2025-07-08
The Zamba2 Suite: Technical Report✓ Link32.43Zamba2-7B-Instruct2024-11-22
xHC: Expanded Hyper-Connections✓ Link32.0xHC (18B, pre-training only) with Muon2026-07-16
Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss31.9MoE + ERC2025-12-29
2 OLMo 2 Furious✓ Link31.0OLMo 2 7B2024-12-31
FlexOlmo: Open Language Models for Flexible Data Use✓ Link30.9FlexOlmo2025-07-09
Introducing Command R7B: Fast and efficient generative AI28.5Command R7B2024-12-13
FlexMoRE: A Flexible Mixture of Rank-heterogeneous Experts for Efficient Federatedly-trained Large Language Models25.64FlexMoRE-a4 (heterogen. MC9)2026-02-09
NITP: Next Implicit Token Prediction for LLM Pre-training✓ Link21.09B MoE (NITP)2026-05-24
LFM2.5-230M: Built to Run Anywhere20.25LFM2.5-230M2026-06-25
Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall✓ Link20.1Switch Distillation (1B; 7B teacher, after DPO)2026-09-01
LFM2.5-350M: No Size Left Behind20.01LFM2.5-350M2026-03-31
Learnable Multipliers: Freeing the Scale of Language Model Matrix Layers✓ Link19.92Falcon-H1-0.5B with Learnable Multipliers (Adam)2026-01-08
SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model✓ Link19.4SmolLM2 1.7B2025-02-04
LFM2.5-VL-450M: Structured Visual Intelligence, Edge to Cloud19.32LFM2.5-VL-450M2026-04-08
SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model✓ Link19.3SmolLM2 1.7B Instruct2025-02-04
Falcon Mamba: The First Competitive Attention-free 7B Language Model✓ Link14.47Falcon Mamba 7B (5-shot)2024-10-07
MARS: Enabling Autoregressive Models Multi-Token Generation✓ Link12.4MARS-0.5B (B=4)2026-04-08
Check out how to contribute new results. Then edit this file on GitLab or GitHub.