Orca 2: Teaching Small Language Models How to Reason

Benchmark Model Rank Results
arithmetic-reasoning-on-gsm8kOrca 2 13BAccuracy: 59.14Parameters (Billion): 13
arithmetic-reasoning-on-gsm8kOrca 2 7BAccuracy: 47.23Parameters (Billion): 7
crass-ai-on-big-benchOrca 2-13BAccuracy: 86.86
crass-ai-on-big-benchOrca 2-7BAccuracy: 84.31
multi-task-language-understanding-on-bbh-nlpOrca 2-13BAverage (%): 50.18
multi-task-language-understanding-on-bbh-nlpOrca 2-7BAverage (%): 45.93
question-answering-on-drop-testOrca 2-13BF1: 57.97
question-answering-on-drop-testOrca 2-7BF1: 60.26
reading-comprehension-on-raceOrca 2-13BAccuracy: 82.87
reading-comprehension-on-raceOrca 2-7BAccuracy: 80.79