US AISI and UK AISI Joint Pre-Deployment Test: Anthropic’s Claude 3.5 Sonnet (October 2024 Release)
Open paper
Benchmark
Model
Rank
Results
on-cybench
Claude 3.5 Sonnet (old, US AISI scaffold, pass@10)
–
Unguided Performance: 35%
on-cybench
o1-preview (US AISI scaffold, pass@10)
–
Unguided Performance: 35%
Rank counts only results with a code link.