SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables

Benchmark Model Rank Results
table-claim-verification-on-scitabAlpaca-7B (3-shot in-context)–Macro-F1 (3-class): 0.2895
table-claim-verification-on-scitabAlpaca-7B (zero-shot)–Macro-F1 (3-class): 0.2759
table-claim-verification-on-scitabFlan-T5-XL (3-shot in-context)–Macro-F1 (3-class): 0.2921
table-claim-verification-on-scitabFlan-T5-XL (zero-shot)–Macro-F1 (3-class): 0.3805
table-claim-verification-on-scitabFlan-T5-XXL (3-shot in-context)–Macro-F1 (3-class): 0.3404
table-claim-verification-on-scitabFlan-T5-XXL (zero-shot)–Macro-F1 (3-class): 0.3491
table-claim-verification-on-scitabFlan-T5-base (3-shot in-context)–Macro-F1 (3-class): 0.2409
table-claim-verification-on-scitabFlan-T5-base (zero-shot)–Macro-F1 (3-class): 0.2656
table-claim-verification-on-scitabFlan-T5-large (3-shot in-context)–Macro-F1 (3-class): 0.2730
table-claim-verification-on-scitabFlan-T5-large (zero-shot)–Macro-F1 (3-class): 0.3255
table-claim-verification-on-scitabGPT-4 (3-shot in-context)–Macro-F1 (3-class): 0.6321
table-claim-verification-on-scitabGPT-4 (zero-shot)–Macro-F1 (3-class): 0.6480
table-claim-verification-on-scitabGPT-4 + CoT (3-shot in-context)–Macro-F1 (3-class): 0.6277
table-claim-verification-on-scitabInstructGPT (3-shot in-context)–Macro-F1 (3-class): 0.4158
table-claim-verification-on-scitabInstructGPT (zero-shot)–Macro-F1 (3-class): 0.4141
table-claim-verification-on-scitabInstructGPT + CoT (3-shot in-context)–Macro-F1 (3-class): 0.4260
table-claim-verification-on-scitabLLaMA-13B (3-shot in-context)–Macro-F1 (3-class): 0.3266
table-claim-verification-on-scitabLLaMA-13B (zero-shot)–Macro-F1 (3-class): 0.3718
table-claim-verification-on-scitabLLaMA-7B (3-shot in-context)–Macro-F1 (3-class): 0.2717
table-claim-verification-on-scitabLLaMA-7B (zero-shot)–Macro-F1 (3-class): 0.3226
table-claim-verification-on-scitabTAPEX-Zero-XL (3-shot in-context)–Macro-F1 (3-class): 0.2562
table-claim-verification-on-scitabTAPEX-Zero-XL (zero-shot)–Macro-F1 (3-class): 0.3430
table-claim-verification-on-scitabTAPEX-Zero-large (3-shot in-context)–Macro-F1 (3-class): 0.2347
table-claim-verification-on-scitabTAPEX-Zero-large (zero-shot)–Macro-F1 (3-class): 0.2972
table-claim-verification-on-scitabVicuna-13B (3-shot in-context)–Macro-F1 (3-class): 0.3516
table-claim-verification-on-scitabVicuna-13B (zero-shot)–Macro-F1 (3-class): 0.2963
table-claim-verification-on-scitabVicuna-7B (3-shot in-context)–Macro-F1 (3-class): 0.3426
table-claim-verification-on-scitabVicuna-7B (zero-shot)–Macro-F1 (3-class): 0.3247