Orca 2: Teaching Small Language Models How to Reason

Benchmark Model Rank Results
arithmetic-reasoning-on-gsm8kOrca 2 13B–Accuracy: 59.14Parameters (Billion): 13
arithmetic-reasoning-on-gsm8kOrca 2 7B–Accuracy: 47.23Parameters (Billion): 7
crass-ai-on-big-benchOrca 2-13B–Accuracy: 86.86
crass-ai-on-big-benchOrca 2-7B–Accuracy: 84.31
multi-task-language-understanding-on-bbh-nlpOrca 2-13B–Average (%): 50.18
multi-task-language-understanding-on-bbh-nlpOrca 2-7B–Average (%): 45.93
question-answering-on-drop-testOrca 2-13B–F1: 57.97
question-answering-on-drop-testOrca 2-7B–F1: 60.26
reading-comprehension-on-raceOrca 2-13B–Accuracy: 82.87
reading-comprehension-on-raceOrca 2-7B–Accuracy: 80.79