Model Card and Evaluations for Claude Models

Benchmark Model Rank Results
arithmetic-reasoning-on-gsm8kClaude 1.3 (0-shot chain-of-thought)Accuracy: 85.2
arithmetic-reasoning-on-gsm8kClaude 2 (0-shot chain-of-thought)Accuracy: 88
arithmetic-reasoning-on-gsm8kClaude Instant 1.1 (0-shot chain-of-thought)Accuracy: 80.9
common-sense-reasoning-on-arc-challengeClaude 1.3 (few-shot, k=5)Accuracy: 90
common-sense-reasoning-on-arc-challengeClaude 2 (few-shot, k=5)Accuracy: 91
common-sense-reasoning-on-arc-challengeClaude Instant 1.1 (few-shot, k=5)Accuracy: 85.7
question-answering-on-qualityClaude 1.3 (5-shot)Accuracy: 84.1
question-answering-on-qualityClaude 2 (5-shot)Accuracy: 83.2
question-answering-on-qualityClaude Instant 1.1 (5-shot)Accuracy: 80.5
question-answering-on-triviaqaClaude 1.3 (few-shot, k=5)EM: 86.7
question-answering-on-triviaqaClaude 2 (few-shot, k=5)EM: 87.5
question-answering-on-triviaqaClaude Instant 1.1 (few-shot, k=5)EM: 78.9