| Benchmark | Model | Rank | Results |
|---|---|---|---|
| code-generation-on-apps | MapCoder APPS-150-cherrypicked (GPT-4) | #16 | Introductory Pass@1: 1.30%Interview Pass@1: 0.70%… |
| code-generation-on-codecontests | MapCoder (GPT-4) | #3 | Test Set pass@1: 28.5Test Set pass@5: 35.2Val Set pass@1: 28.5 |
| code-generation-on-humaneval | Mistral 7B | #9 | Pass@1: 93.9 |
| code-generation-on-mbpp | o1-mini + MapCoder (Hamming.ai) | #2 | Accuracy: 93.2 |
| code-generation-on-mbpp | MapCoder (GPT-4o) | #7 | Accuracy: 89.7 |
| code-generation-on-mbpp | MapCoder (GPT-4) | #11 | Accuracy: 83.1 |