| code-generation-on-programbench | Claude Opus 5 (xhigh) (mini-SWE-agent) | #2 | Average Test Pass Rate: 74.7Resolved: 4.5Almost Resolved: 37.0 |
| code-generation-on-programbench | GPT-5.6 Sol (xhigh) (mini-SWE-agent) | #3 | Average Test Pass Rate: 69.9Resolved: 1.0Almost Resolved: 15.5 |
| code-generation-on-programbench | Gemini 3.7 Flash (mini-SWE-agent) | #5 | Average Test Pass Rate: 61.2Resolved: 0.0Almost Resolved: 5.5 |
| code-generation-on-programbench | GPT-5.6 Sol (mini-SWE-agent) | #7 | Average Test Pass Rate: 57.8Resolved: 0.5Almost Resolved: 2.5 |
| code-generation-on-programbench | Muse Spark 1.1 (xhigh) (mini-SWE-agent) | #9 | Average Test Pass Rate: 47.0Resolved: 0.0Almost Resolved: 4.0 |
| code-generation-on-programbench | Claude Haiku 4.5 (mini-SWE-agent) | – | Average Test Pass Rate: 30.0Resolved: 0.0Almost Resolved: 0.0 |
| code-generation-on-programbench | Claude Opus 4.6 (mini-SWE-agent) | – | Average Test Pass Rate: 52.1Resolved: 0.0Almost Resolved: 2.5 |
| code-generation-on-programbench | Claude Opus 4.7 (mini-SWE-agent) | – | Average Test Pass Rate: 50.9Resolved: 0.0Almost Resolved: 3.0 |
| code-generation-on-programbench | Claude Opus 4.7 (xhigh) (mini-SWE-agent) | – | Average Test Pass Rate: 55.1Resolved: 0.0Almost Resolved: 4.5 |
| code-generation-on-programbench | Claude Opus 4.8 (xhigh) (mini-SWE-agent) | – | Average Test Pass Rate: 70.9Resolved: 0.0Almost Resolved: 16.5 |
| code-generation-on-programbench | Claude Sonnet 4.6 (mini-SWE-agent) | – | Average Test Pass Rate: 47.5Resolved: 0.0Almost Resolved: 1.0 |
| code-generation-on-programbench | GPT 5 mini (mini-SWE-agent) | – | Average Test Pass Rate: 16.0Resolved: 0.0Almost Resolved: 0.0 |
| code-generation-on-programbench | GPT 5.4 (mini-SWE-agent) | – | Average Test Pass Rate: 37.7Resolved: 0.0Almost Resolved: 0.0 |
| code-generation-on-programbench | GPT 5.4 mini (mini-SWE-agent) | – | Average Test Pass Rate: 16.4Resolved: 0.0Almost Resolved: 0.0 |
| code-generation-on-programbench | Gemini 3 Flash (mini-SWE-agent) | – | Average Test Pass Rate: 31.8Resolved: 0.0Almost Resolved: 0.0 |
| code-generation-on-programbench | Gemini 3.5 Flash (mini-SWE-agent) | – | Average Test Pass Rate: 53.6Resolved: 0.0Almost Resolved: 3.0 |
| code-generation-on-programbench | Gemini 3.6 Flash (mini-SWE-agent) | – | Average Test Pass Rate: 55.7Resolved: 0.5Almost Resolved: 4.0 |
| code-generation-on-programbench | Muse Spark 1.2 (xhigh) (mini-SWE-agent) | – | Average Test Pass Rate: 57.2Resolved: 0.0Almost Resolved: 4.5 |
| code-generation-on-programbench | Muse Spark 1.3 (max) (mini-SWE-agent) | – | Average Test Pass Rate: 70.8Resolved: 2.5Almost Resolved: 25.0 |
| code-generation-on-programbench | Muse Spark 1.3 (xhigh) (mini-SWE-agent) | – | Average Test Pass Rate: 68.6Resolved: 1.0Almost Resolved: 16.5 |