Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback

• The proposed interaction generates and explains an SQL parse, displays its execution result, and uses human feedback to produce a correction.Source: Figure 1 · p. 1, §1 · p. 2
• One-shot Text-to-SQL systems often make minor parsing mistakes that humans can detect and correct through precise feedback.Source: §1 · p. 1
• The system must explain generated SQL, support natural-language feedback, and combine that feedback with the original question to improve interpretation.Source: §1 · p. 2
• The paper defines SQL parse correction, constructs SPLASH with human feedback paired with inaccurate parses, and establishes baseline correction models.Source: §1 · p. 2
• State-of-the-art correction models remain challenged despite feedback being useful, leaving substantial room for improvement.Source: §1 · p. 2

• Given a question, schema, mispredicted parse, and feedback, the task is to generate a corrected parse matching the gold parse.Source: §2 · p. 2, Figure 2 · p. 2
• Models consume q, s, p′, and f, then hypothesize a corrected parse ˆp evaluated against gold p using exact set-match accuracy.Source: §2 · p. 2

3 Dataset Construction

• SPLASH construction combines erroneous parser outputs, natural-language SQL explanations, crowd feedback, and paraphrased feedback.Source: §3 · p. 2

3.1 Generating Questions and Incorrect SQL Pairs

• Seq2Struct outputs are used to model contemporary parser errors rather than injecting artificial errors into gold SQL.Source: §3.1 · p. 3
• The dataset construction does not explicitly depend on Seq2Struct, so other models could generate erroneous parses.Source: §3.1 · p. 3

3.2 Explaining SQL

• SQL explanations replace literals, schema names, aggregations, and comparisons with placeholders while handling joins, set operations, and limits through structured templates.Source: §3.2 · p. 3

3.3 Crowdsourcing Feedback

• Annotators judge whether generated SQL satisfies the question and provide feedback when it does not, using database tables, columns, and sample rows as context.Source: §3.3 · p. 4

3.4 Dataset Summary

• Annotators identified correct parses with 96.4% accuracy, and manual review found 95.5% of their equivalence judgments correct when predicted and gold SQL differed.Source: §3.4 · p. 4
• After paraphrasing, SPLASH contains 9,314 question-feedback pairs, including 8,352 from Spider training questions and 962 from development questions.Source: §3.4 · p. 4

4.1 Error Characteristics

• 78%+ of inaccurate SQL predictions lie within an edit distance of 3 or less, supporting human-assisted correction of mostly minor errors.The analysis measures edit distance using grouped token-level insert, delete, and replace segments.Source: §4.1 · p. 5
• 58% of edits are replacements, 31% insertions, and 11% deletions; 68.1% modify a single token, most often a schema item.Among single-token edits, 89.2% concern table or column names, followed by SQL keywords at 7.4%.Source: §4.1 · p. 5
• Joins account for many edits, followed by where clauses, aggregation, counting, and unique-value selection, indicating recurring difficulty with relational query structure.A single edit can involve multiple keywords, such as multiple joins or a join combined with a where clause.Source: §4.1 · p. 5

4.2 Feedback Characteristics

• In a 200-example sample, feedback was 81.5% complete, 13.5% partial, and 5.0% paraphrastic; partial and paraphrase feedback mostly accompanied large SQL distances.Annotators used partial feedback to correct some errors or rewrote the question when the parsing mistake was major.Source: §4.2 · p. 5
• Feedback covers diverse correction types, especially retrieved-information and condition errors, with replacements more common than insertions or removals.Missing retrieved information often reflects an absent inner one-to-one join; aggregation and ordering errors are also represented.Source: §4.2 · pp. 5–6
• The dataset’s top 60 feedback patterns are heuristically extracted from predicates and arguments and center on operations, column names, and values.Column mentions are normalized to the token “item” to reduce vocabulary before visualizing pattern frequencies.Source: §4.2 · p. 6, Figure 6 · p. 6

Text-to-SQL Parsing

• Unlike WikiSQL, Spider, SParC, and CoSQL lines focused on mapping questions or conversational dependencies, SPLASH evaluates applying feedback to generated SQL parses.The distinction is empirically examined in Section 6.3.Source: §5 · p. 7

Learning from Feedback

• Prior feedback-learning work uses binary correctness, execution validation, or token-level signals, whereas this work immediately incorporates natural-language feedback without retraining the parser.This positions parse correction as an interaction-time task rather than general parser improvement.Source: §5 · p. 7

Interactive Semantic Parsing

• Interactive semantic parsing studies have used soft constraints, graphical editing, multiple-choice questions, or yes/no component queries; SPLASH instead permits richer free-form feedback.Its closest comparison is Labutov et al. (2018), but SPLASH targets complex multi-domain text-to-SQL parsing and unseen domains.Source: §5 · p. 7
• SPLASH pairs mispredicted parses and feedback with gold parses and reports that incorporating the mispredicted parse improves correction accuracy, contrary to Labutov et al.’s finding.The gold parse represents the final parse approved after one or more correction rounds.Source: §5 · pp. 7–8

Asking Clarifying Questions

• Clarifying-question research resolves ambiguity or missing information, whereas SPLASH corrects an initial parse of a fully specified question through human feedback.The distinction concerns correcting an existing interpretation rather than eliciting clarification before parsing.Source: §5 · p. 8

6 Experiments

• Correction accuracy measures the percentage of test examples whose predicted SQL matches the gold parse under exact set match, alongside an end-to-end two-turn accuracy.The end-to-end measure counts an example as correct if either the initial parse or corrected parse is correct.Source: §6 · p. 8, §6.1 · p. 8

6.1 Baselines

• Baseline comparisons include feedback-agnostic beam reranking, handcrafted feedback-based reranking, Seq2Struct+Feedback, and EditSQL+Feedback.The handcrafted baseline uses schema-item differences between the mispredicted parse and beam candidates to match feedback.Source: §6.1 · p. 8

6.2 Main Results

• EditSQL+Feedback achieves the best baseline performance, but state-of-the-art models still struggle with SQL parse correction, leaving a large improvement gap.The comparison is reported in Table 4.Source: §6.2 · p. 9, Table 4 · p. 9

6.3 Analysis

• Randomizing feedback reduces EditSQL+Feedback accuracy to 5.6%, indicating that the model uses the feedback rather than simply ignoring it.The test shuffles feedback across SPLASH examples.Source: §6.3 · p. 9
• Models trained for SParC and CoSQL achieve only 3.4% and 3.2% accuracy on SPLASH, showing that SPLASH targets different modeling aspects from conversational text-to-SQL.The comparison uses trained EditSQL models evaluated on the SPLASH test set.Source: §6.3 · p. 9
• The SPLASH-trained EditSQL+Feedback model reaches 14.6% correction accuracy on errors produced by EditSQL, extending evaluation beyond Seq2Struct errors.Feedback was collected for 179 mispredicted EditSQL parses.Source: §6.3 · p. 9

• The paper introduces SQL parse correction with natural-language feedback and a dataset pairing human-authored feedback with mispredicted and gold parses.Source: §7 · p. 9
• Natural-language feedback helps correct parses, but state-of-the-art models still struggle with the task.Future work includes improving correction models, using feedback logs to improve text-to-SQL parsers, and expanding the dataset to multiple correction turns.Source: §7 · p. 9

• Figure 7 shows the instructions presented to annotators.Source: §A.1 · p. 12

• Figure 8 shows the data-collection interface and notes that annotators are not shown either the gold or predicted SQL.The example predicted query selects instructor names and salaries for mathematics departments.Source: §A.2 · p. 12

A.3 Example of Explanations

• The data-collection interface includes a predicted SQL query selecting instructor names and salaries for mathematics departments.Source: Figure 8 · p. 13
• Annotators are not shown either the gold SQL or the predicted SQL during data collection.Source: Figure 8 · p. 13
• Figure 9 presents several examples of how different SQL components can be explained in natural language.Source: §A.3 · p. 12, Figure 9 · p. 13