ProTrix: Building Models for Planning and Reasoning over Tables with Sentence Context

Benchmark Model Rank Results
code-generation-on-wikisqlGPT-4#4Execution Accuracy: 75.8
code-generation-on-wikisqlProTrix-Coder#6Execution Accuracy: 72.3
code-generation-on-wikisqlProTrix#8Execution Accuracy: 67.4
code-generation-on-wikisqlGPT-3.5-turbo#11Execution Accuracy: 55.0
code-generation-on-wikisqlLlama-2-7B#13Execution Accuracy: 17.4
code-generation-on-wikisqlCodeLlama-7B#14Execution Accuracy: 17.3
fact-verification-on-feverousProTrix#1Label Accuracy: 75.6
fact-verification-on-feverousProTrix-Coder#2Label Accuracy: 71.4
fact-verification-on-feverousGPT-4#3Label Accuracy: 71.0
fact-verification-on-feverousPlan-then-Reason (Ours)#4Label Accuracy: 65.8
fact-verification-on-feverousGPT-3.5-turbo#7Label Accuracy: 61.0
fact-verification-on-feverousLlama-2-7B#8Label Accuracy: 47.1
fact-verification-on-feverousCodeLlama-7B#9Label Accuracy: 43.0
question-answering-on-hybridqaGPT-4#2ANS-EM: 64.1
question-answering-on-hybridqaGPT-3.5-turbo#4ANS-EM: 55.1
question-answering-on-hybridqaProTrix-Coder#5ANS-EM: 45.1
question-answering-on-hybridqaProTrix#7ANS-EM: 42.9
question-answering-on-hybridqaCodeLlama-7B#8ANS-EM: 28.5
question-answering-on-hybridqaLlama-2-7B#9ANS-EM: 27.6
question-answering-on-tat-qaGPT-4#8Exact Match: 80.8
question-answering-on-tat-qaGPT-3.5-turbo#9Exact Match: 59.1
question-answering-on-tat-qaProTrix-Coder#10Exact Match: 52.2
question-answering-on-tat-qaProTrix#11Exact Match: 50.1
question-answering-on-tat-qaLlama-2-7B#12Exact Match: 28.7
question-answering-on-tat-qaCodeLlama-7B#13Exact Match: 28.4
semantic-parsing-on-wikitablequestionsGPT-4#4Accuracy (Test): 72.9
semantic-parsing-on-wikitablequestionsPlan-then-Reason (Ours)#11Accuracy (Test): 65.2
semantic-parsing-on-wikitablequestionsProTrix-Coder#17Accuracy (Test): 57.8
semantic-parsing-on-wikitablequestionsProTrix#19Accuracy (Test): 56.2
semantic-parsing-on-wikitablequestionsGPT-3.5-turbo#21Accuracy (Test): 51.8
semantic-parsing-on-wikitablequestionsLlama-2-7B#25Accuracy (Test): 21.4
semantic-parsing-on-wikitablequestionsCodeLlama-7B#26Accuracy (Test): 13.1
table-based-fact-verification-on-tabfactPlan-then-Reason (Ours)#8Test: 83.5
table-based-fact-verification-on-tabfactProTrix#12Test: 71.6
table-based-fact-verification-on-tabfactGPT-4#13Test: 71.5
table-based-fact-verification-on-tabfactProTrix-Coder#14Test: 70.6
table-based-fact-verification-on-tabfactGPT-3.5-turbo#16Test: 68.8
table-based-fact-verification-on-tabfactCodeLlama-7B#20Test: 49.5
table-based-fact-verification-on-tabfactLlama-2-7B#21Test: 48.6
table-claim-verification-on-scitabProTrix#2Macro-F1 (3-class): 0.4500
table-claim-verification-on-scitabProTrix-Coder#3Macro-F1 (3-class): 0.4120