Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs

Open paper
Benchmark Model Rank Results
text-to-sql-on-birdCoT + ChatGPT#8Execution Accuracy % (Test): 40.08Execution Accuracy % (Dev): 36.64
text-to-sql-on-birdChatGPT (Baseline)#9Execution Accuracy % (Test): 39.30Execution Accuracy % (Dev): 37.22
text-to-sql-on-birdCodex (Baseline)#10Execution Accuracy % (Test): 36.47Execution Accuracy % (Dev): 34.35
text-to-sql-on-birdPalm-2 (Baseline)#11Execution Accuracy % (Test): 33.04Execution Accuracy % (Dev): 27.38
text-to-sql-on-birdHuman Performance#14Execution Accurarcy (Human): 92.96

Rank counts only results with a code link.