TheoremQA: A Theorem-driven Question Answering dataset

Benchmark Model Rank Results
natural-questions-on-theoremqaGPT-4 (PoT)#1Accuracy: 52.4
natural-questions-on-theoremqaGPT-4 (CoT)#2Accuracy: 43.8
natural-questions-on-theoremqaGPT-3.5-turbo (PoT)#3Accuracy: 35.6
natural-questions-on-theoremqaPaLM-2-unicorn (CoT)#6Accuracy: 31.8
natural-questions-on-theoremqaGPT-3.5-turbo (CoT)#7Accuracy: 30.2
natural-questions-on-theoremqaClaude-v1 (PoT)#10Accuracy: 25.9
natural-questions-on-theoremqaClaude-v1 (CoT)#11Accuracy: 24.9
natural-questions-on-theoremqacode-davinci-002#12Accuracy: 23.9
natural-questions-on-theoremqaClaude-instant (CoT)#13Accuracy: 23.6
natural-questions-on-theoremqatext-davinci-003#14Accuracy: 22.8
natural-questions-on-theoremqaPaLM-2-bison (CoT)#15Accuracy: 21.0