US AISI and UK AISI Joint Pre-Deployment Test: Anthropic’s Claude 3.5 Sonnet (October 2024 Release)

Benchmark Model Rank Results
cybersecurity-on-cybenchClaude 3.5 Sonnet (old, US AISI scaffold, pass@10)–Unguided Performance: 35%
cybersecurity-on-cybencho1-preview (US AISI scaffold, pass@10)–Unguided Performance: 35%