OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset

Open paper
Benchmark Model Rank Results
math-word-problem-solving-on-mawpsOpenMath-CodeLlama-70B (w/ code)#1Accuracy (%): 95.7
math-word-problem-solving-on-svampOpenMath-CodeLlama-70B (w/ code)#4Execution Accuracy: 87.8
math-word-problem-solving-on-asdiv-aOpenMath-CodeLlama-70B (w/ code)#5Execution Accuracy: 84.7
math-word-problem-solving-on-mathOpenMath-CodeLlama-70B (w/ code, SC, k=50)#52Accuracy: 60.4Parameters (Billions): 70
math-word-problem-solving-on-mathOpenMath-CodeLlama-34B (w/ code, SC, k=50)#53Accuracy: 60.2Parameters (Billions): 34
math-word-problem-solving-on-mathOpenMath-Llama2-70B (w/ code, SC, k=50)#59Accuracy: 58.3Parameters (Billions): 70
math-word-problem-solving-on-mathOpenMath-CodeLlama-13B (w/ code, SC, k=50)#61Accuracy: 57.6Parameters (Billions): 13
math-word-problem-solving-on-mathOpenMath-Mistral-7B (w/ code, SC, k=50)#62Accuracy: 57.2Parameters (Billions): 7
math-word-problem-solving-on-mathOpenMath-CodeLlama-7B (w/ code, SC, k=50)#67Accuracy: 55.6Parameters (Billions): 7
math-word-problem-solving-on-mathOpenMath-CodeLlama-70B (w/ code)#84Accuracy: 50.7Parameters (Billions): 70
math-word-problem-solving-on-mathOpenMath-CodeLlama-34B (w/ code)#92Accuracy: 48.3Parameters (Billions): 34
math-word-problem-solving-on-mathOpenMath-Llama2-70B (w/ code)#100Accuracy: 46.3Parameters (Billions): 70
math-word-problem-solving-on-mathOpenMath-CodeLlama-13B (w/ code)#102Accuracy: 45.5Parameters (Billions): 13
math-word-problem-solving-on-mathOpenMath-Mistral-7B (w/ code)#109Accuracy: 44.5Parameters (Billions): 7
math-word-problem-solving-on-mathOpenMath-CodeLlama-7B (w/ code)#112Accuracy: 43.6Parameters (Billions): 7
arithmetic-reasoning-on-gsm8kOpenMath-CodeLlama-70B (w/ code, SC, k=50)#67Accuracy: 90.8Parameters (Billion): 70
arithmetic-reasoning-on-gsm8kOpenMath-Llama2-70B (w/ code, SC, k=50)#72Accuracy: 90.1Parameters (Billion): 70
arithmetic-reasoning-on-gsm8kOpenMath-CodeLlama-34B (w/ code, SC, k=50)#96Accuracy: 88.0Parameters (Billion): 34
arithmetic-reasoning-on-gsm8kOpenMath-Mistral-7B (w/ code, SC, k=50)#106Accuracy: 86.9Parameters (Billion): 7
arithmetic-reasoning-on-gsm8kOpenMath-CodeLlama-13B (w/ code, SC, k=50)#107Accuracy: 86.8Parameters (Billion): 13
arithmetic-reasoning-on-gsm8kOpenMath-CodeLlama-7B (w/ code, SC, k=50)#118Accuracy: 84.8Parameters (Billion): 7
arithmetic-reasoning-on-gsm8kOpenMath-Llama2-70B (w/ code)#120Accuracy: 84.7Parameters (Billion): 70
arithmetic-reasoning-on-gsm8kOpenMath-CodeLlama-70B (w/ code)#122Accuracy: 84.6Parameters (Billion): 70
arithmetic-reasoning-on-gsm8kOpenMath-CodeLlama-34B (w/ code)#157Accuracy: 80.7Parameters (Billion): 34
arithmetic-reasoning-on-gsm8kOpenMath-Mistral-7B (w/ code)#162Accuracy: 80.2Parameters (Billion): 7
arithmetic-reasoning-on-gsm8kOpenMath-CodeLlama-13B (w/ code)#165Accuracy: 78.8Parameters (Billion): 13
arithmetic-reasoning-on-gsm8kOpenMath-CodeLlama-7B (w/ code)#179Accuracy: 75.9Parameters (Billion): 7

Rank counts only results with a code link.