PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating the Text-to-SQL Capabilities of Large Language Models

Nitarshan Rajkumar, Raymond Li, Dzmitry Bahdanau

arXiv:2204.00498v1cs.CLcs.DBcs.LG

TL;DR

This paper asks whether large language models can perform Text-to-SQL without task-specific finetuning. It evaluates Codex across zero-shot Spider and few-shot GeoQuery and Scholar settings, varying prompts and in-domain examples. Codex provides a strong Spider baseline and can outperform smaller finetuned models when prompted with a small number of domain examples.

  • Problem

    The paper addresses whether pretrained generative models can provide competitive Text-to-SQL solutions without task-specific finetuning.

  • Method

    The evaluation compares GPT-3 and Codex models across Spider, GeoQuery, and Scholar using varied prompt structures and few-shot examples.

  • Results

    Codex reaches up to 67% execution accuracy on the Spider development set without task-specific finetuning.

  • Takeaways & Limitations

    Prompt-based few-shot learning can outperform finetuning-based few-shot learning of smaller models on GeoQuery and Scholar.

Abstract

from arXiv · show

We perform an empirical evaluation of Text-to-SQL capabilities of the Codex language model. We find that, without any finetuning, Codex is a strong baseline on the Spider benchmark; we also analyze the failure modes of Codex in this setting. Furthermore, we demonstrate on the GeoQuery and Scholar benchmarks that a small number of in-domain examples provided in the prompt enables Codex to perform better than state-of-the-art models finetuned on such few-shot examples.

1 Introduction

This paper evaluates whether large language models can perform Text-to-SQL competitively without task-specific finetuning. Codex reaches strong Spider performance, while prompting with a few in-domain examples improves adaptation to GeoQuery and Scholar.

  • Motivation: Text-to-SQL translates natural-language questions into SQL using questions, database schemas, and possibly database content.The paper studies pretrained generative models as alternatives to task-specific finetuning.
  • Main findings: Many automatically rejected Codex queries would be judged correct by humans, while others could likely be fixed without finetuning.The paper analyzes predicted queries that automatic evaluation marked wrong.
  • Main findings: Few in-domain examples let Codex adapt to GeoQuery and Scholar more effectively than finetuning a smaller language model on the same examples.This compares prompt-based adaptation with few-shot finetuning across the two benchmarks.

2 Experimental Setup

The experiments compare pretrained GPT-3 and code-finetuned Codex models across zero-shot and few-shot Text-to-SQL settings. They vary benchmark, prompt structure, database information, and the number of in-domain examples.

  • Models: The evaluation compares GPT-3 and Codex models of multiple sizes, including ada through davinci and cushman-codex through davinci-codex.GPT-3 is pretrained on diverse internet sources, whereas Codex is additionally finetuned on GitHub code.
  • Baselines: The comparison includes finetuned T5 and BRIDGE v2 baselines alongside API-accessible GPT-3 and Codex models.T5 is finetuned on Spider to predict SQL from the question and schema, while BRIDGE v2 is BERT-based.
  • Benchmarks and metrics: Spider zero-shot experiments measure valid SQL, execution accuracy, and test-suite execution accuracy.Test-suite accuracy captures semantic equivalence among differently styled SQL queries.
  • Few-shot design: Few-shot GeoQuery and Scholar experiments select one random example from each of the n most frequent query templates.Examples sharing a template have the same SQL structure but may differ in English questions and SQL literals.
  • Prompt design: Six prompt structures vary the supplied information, from the question alone to schemas, database contents, and question-query examples.The structures include Question, API Docs, Select X, Create Table, Create Table + Select X, and Fewshot.

3 Zero-Shot Results

Zero-shot Codex is a strong Spider baseline, with performance shaped substantially by prompt design and database content. Error analysis also finds that many evaluation failures are human-acceptable, while model scaling yields limited gains beyond context effects.

  • Overall performance: 67% execution accuracy and 56.5% test-suite execution accuracy make davinci-codex a strong Spider baseline, comparable to BRIDGE v2's 68% execution accuracy.The reported result uses the Create Table + Select 3 prompt.
  • Prompt design: Execution accuracy rises from 8.3% with only the question to 67.0% with Create Table + Select 3.Intermediate results are 56.8% with API Docs and 59.9% with Create Table.
  • Database content: Create Table + Select X accuracy peaks at three rows before significantly decreasing as more database rows are added.Select Limit X prompts show negligible change when additional rows are provided.
  • Model size: Cushman-codex is generally within 1 percentage point of davinci-codex and performs 3 percentage points better on the Create Table prompt.The authors suggest that davinci-codex's longer context window may contribute more to peak performance than its larger parameter count.
  • Error analysis: The error analysis annotated 100 valid SQL predictions judged incorrect by test-suite evaluation and categorized semantic, ambiguous, shortcut, grouping, selection, and argmax behaviors.The categories were chosen to surface Codex-specific behaviors among the observed errors.
  • Error analysis: 31% of valid yet erroneous predictions were judged acceptable by a human annotator despite being penalized by Spider evaluation.The paper identifies prompt design or few examples as possible ways to control such ambiguous errors.

4 Few-Shot

The study tests few-shot Text-to-SQL adaptation by comparing prompt-based Codex with a T5-3B model fine-tuned on the same examples. Codex beats the fine-tuned baseline in several settings, with task-dependent differences between GeoQuery and Scholar.

  • Setup: The few-shot task repurposes GeoQuery and Scholar question-splits, grouping examples by query template to evaluate adaptation to new domains.Templates share SQL structure while allowing different English questions and SQL literals.
  • Setup: The comparison uses a T5-3B model fine-tuned on Spider and then further fine-tuned on the new domain, while Codex receives the same examples in its prompt.The baseline reaches 71% exact-match accuracy on Spider validation; Codex uses greedy decoding at temperature 0.
  • Results: The baseline reaches 85.7% test-suite accuracy on GeoQuery and 87.2% on Scholar after training on the complete datasets.These results use 549 GeoQuery examples and 499 Scholar examples.
  • Results: In zero-shot GeoQuery, both Codex models beat the baseline, while on Scholar the T5 model initially outperforms Codex.The authors speculate that Codex performs well on GeoQuery because it uses the dataset’s argmax convention.
  • Results: With 5 and 10 examples on Scholar, Codex beats the T5 baseline, and with up to 40 GeoQuery examples davinci-codex outperforms the fine-tuned T5-3B model by a large margin.GPT-3 davinci performs poorly on GeoQuery in this comparison.

5 Conclusion

The paper concludes that code-trained generative language models are strong Text-to-SQL baselines and that prompt-based few-shot learning can rival fine-tuning smaller models. It also identifies failure-mode analysis and Codex fine-tuning as directions for improving the approach.

  • Conclusion: Code-trained generative language models provide a strong baseline for Text-to-SQL.The conclusion frames this as an empirical result of the evaluation.
  • Conclusion: Failure-mode analysis is intended to guide future prompt design using few-shot examples or natural-language instructions.The paper presents this analysis as a way to improve prompting in this setting.
  • Conclusion: Prompt-based few-shot learning with Codex performs competitively with fine-tuning-based few-shot learning of smaller models.The comparison concerns few-shot adaptation rather than full-dataset training.
  • Conclusion: Evaluating the benefits of fine-tuning Codex models is identified as a clear direction for future work.

A API Details

The API details identify the OpenAI API as the access point for the models and link to the example underlying the API Docs prompt.

  • API Details: The experiments use the OpenAI API, whose documented access point was openai.com/api/ at the time of writing.The passage also links the source example for the API Docs prompt.
  • API Details: The API Docs prompt draws from the default-sql-translate example.

A.1 Hyperparameters

The appendix describes generation settings and cautions that API model sizes and versions are not fully observable or stable. Parameter-count comparisons therefore rely partly on guesses.

  • Generation: GPT-3 and Codex outputs are sampled for 200 tokens at temperature 0, using “--”, “\n\n”, “;”, and “#” as stop tokens.
  • Parameter Counts: OpenAI API parameter counts are not openly available, so the paper estimates model sizes by comparing API behavior with published GPT-3 results.
  • Parameter Counts: The paper presumes davinci-codex matches GPT-3 davinci in size and guesses cushman-codex is similar in size to GPT-3 curie, while warning these estimates should not be relied on.
  • Model Versions: The exact API models may vary over time, although one fixed version was used for each model type in the reported results.

A.4 Finetuning

The paper reports preliminary GPT-3 fine-tuning results on Spider using the full training set and default settings, without a hyperparameter sweep.

  • The preliminary experiments fine-tuned GPT-3 models on the full Spider training set.The authors used four epochs, batch size 8, and a learning-rate multiplier of 0.1.
  • No hyperparameter sweep was performed because of its substantial cost.
  • The reported settings were 4 epochs, batch size 8, and learning-rate multiplier 0.1.

A.5 Memorization

The authors examine whether Codex’s Spider performance could reflect memorization, arguing that prompt and query-style mismatches make this unlikely and that Spider-Realistic supports the same trend.

  • Spider’s development set may have appeared in Codex’s training data because it is available on GitHub.
  • The evaluation repository format differs from the prompts, and verbatim fragments failed to reproduce file contents.
  • Most Codex performance comes from schemas included in prompts, whereas the repository’s dev.sql file does not contain those schemas.
  • Codex predictions use a different style from gold queries, especially because Codex does not use the gold queries’ consistent T1, T2 aliasing strategy.
  • Spider-Realistic shows a similar cross-model performance trend despite removing column-name references and not being publicly available on GitHub.
  • The authors also note that Spider data is rare on GitHub and their long prompts differ substantially from repository files.

A.6 Choice of Spider Evaluation Set

The authors did not evaluate Spider’s held-out test set offline because doing so through the API could expose the examples to Codex retraining.

  • The held-out Spider test set was not evaluated because offline evaluation was unavailable.
  • The evaluation therefore used a set other than Spider’s held-out test set.
  • Sending held-out examples through OpenAI’s API risked inadvertently leaking them for Codex retraining.

B Additional Tables and Figures

The appendix provides Spider performance definitions and supplementary prompts, schema-and-content examples, few-shot demonstrations, and error-analysis figures for Codex evaluation.

  • Additional tables: Table 4 reports Spider performance using valid/executable SQL, execution accuracy, and test-suite accuracy.Results are shown for Spider and, in parentheses, Spider-Realistic.
  • Error analysis: Figure 3 presents davinci-codex error examples from the Create Table + Select 3 prompt and reports each error type’s percentage.NL denotes the natural-language question.
  • Few-shot examples: The few-shot examples pair natural-language questions with SQL queries, including questions about rivers, populations, cities, states, and student relationships.
  • Few-shot prompting: Figure 4 compares cherry-picked GeoQuery validation examples from 0-shot and 10-shot prompting.The examples illustrate changes toward gold-SQL style and adaptation to dataset conventions such as argmax and selected columns.
  • Prompt examples: The appendix includes prompts containing questions, API documentation, SELECT results, CREATE TABLE statements, schemas, and three example rows per table.
Loading 2204.00498v1…