CAPO: Cost-Aware Prompt Optimization

Benchmark Model Rank Results
arithmetic-reasoning-on-gsm8kLlama-3.3-70B + CAPO#67Accuracy: 73.73
arithmetic-reasoning-on-gsm8kMistral-Small-24B + CAPO#78Accuracy: 65.07
arithmetic-reasoning-on-gsm8kQwen2.5-32B + CAPO#83Accuracy: 60.2
sentiment-analysis-on-sst-5-fine-grainedLlama-3.3-70B + CAPO#1Accuracy: 62.27
sentiment-analysis-on-sst-5-fine-grainedMistral-Small-24B + CAPO#25Accuracy: 60.2
sentiment-analysis-on-sst-5-fine-grainedQwen2.5-32B + CAPO#26Accuracy: 59.07
subjectivity-analysis-on-subjLlama-3.3-70B + CAPO#15Accuracy: 91.6
subjectivity-analysis-on-subjQwen2.5-32B + CAPO#16Accuracy: 91
subjectivity-analysis-on-subjMistral-Small-24B + CAPO#18Accuracy: 81.67
text-classification-on-ag-newsLlama-3.3-70B + CAPO#19Error: 11.2
text-classification-on-ag-newsQwen2.5-32B + CAPO#20Error: 12.93
text-classification-on-ag-newsMistral-Small-24B + CAPO#22Error: 15.7
text-classification-on-bala-copaQwen2.5-32B + CAPO#1Accuracy: 98.47
text-classification-on-bala-copaLlama-3.3-70B + CAPO#2Accuracy: 98.27
text-classification-on-bala-copaMistral-Small-24B + CAPO#3Accuracy: 95.13