NL-EDIT: Correcting semantic parse errors through natural language interaction

• NL-EDIT converts natural-language feedback into a sequence of edits that can be applied to an initial SQL parse to correct errors.Source: §1 · pp. 1–2
• The model interprets feedback using the interaction context, including the original question, schema, and explanation of the initial parse.Source: §1 · p. 2
• Feedback commonly describes only a few desired edits, motivating an edit-based formulation of semantic parse correction.Source: §1 · p. 2
• The edit-based framing supports synthetic-data generation that improves correction-model accuracy.Source: §1 · p. 2
• NL-EDIT outperforms prior baselines by more than 16 points on natural-language feedback correction.Source: §1 · p. 2

• Text-to-SQL parsing generates a SQL query from a natural-language question and database schema containing tables, columns, and key relations.Source: §2 · p. 2
• Natural-language feedback correction takes a question, schema, incorrect initial parse, and feedback as input, then generates a corrected parse.Source: §2 · p. 2
• SPLASH was created by asking annotators to correct incorrect parses after reading natural-language explanations of them.Source: §2 · p. 2
• SPLASH contains 9,314 question-feedback pairs and uses database-level train-dev-test splits to encourage generalization to unseen databases.Source: §2 · p. 2

• The representation balances edit granularity because overly fine edits are difficult to learn, whereas overly coarse edits are harder to generate compactly.Source: §3 · p. 2
• SQL queries are represented as sets of clauses with ordered arguments, including recursively defined SUBS clauses for subqueries.Source: §3 · p. 2
• An edit is the set of clause-level changes between source and target queries, with unmatched source arguments removed and unmatched target arguments added.Source: §3 · pp. 2–3
• Subquery edits are computed recursively and pruned when removing a reference leaves the subquery unreferenced.Source: §3 · p. 3, Figure 2 · p. 3
• Edit Size is the number of add and remove operations in an edit; the Figure 2 example has size four.Source: §3 · p. 3, Figure 2 · p. 3

4.1 Intuitions

• Context resolves feedback ambiguities because users refer to explanation steps, schema items, and question-specific meanings rather than using SQL terminology directly.For example, “find last name” is interpreted using the question’s request for a full name.Source: §4.1 · p. 3

4 Model

• NL-EDIT converts natural-language feedback into canonical SQL edits applied deterministically to the initial parse.The model generates edits rather than complete SQL queries.Source: §4.1 · pp. 3–4, §4.3 · p. 4

4.2 Encoder

• The encoder grounds feedback using the explanation, question, and schema through BERT followed by relation-aware self-attention.It combines self-learned transformer relations with hard-coded interaction relations.Source: §4.1 · p. 3, Figure 3 · p. 4, §4.2 · p. 4

4.3 Decoder

• Training linearizes clause-level add/remove operations in a fixed order, while decoding can copy schema columns and tables before applying the resulting edit.SQL operators in edit arguments are expressed with natural-language descriptions.Source: §4.3 · p. 4
• A feedback-only decoder-attention variant improved accuracy by only 0.5%, so experiments use standard decoder-encoder attention.The authors leave alternative attention patterns for future investigation.Source: §4.3 · p. 5

5 Synthetic Feedback

• Synthetic examples start from 8,099 SPIDER question–gold-parse pairs and apply SQL editing operations to create erroneous initial parses.The process ensures no overlap between seed examples and the SPLASH development set.Source: §5 · p. 5
• The synthesis framework defines 26 SQL editors, each paired with frequent feedback templates and constraints governing when the edit is feasible.Constraints include requiring a relevant clause before applying an editor such as Remove-Limit.Source: §5 · p. 5
• Each cloned example receives a randomly chosen sequence of up to four feasible edits, whose feedback templates are combined into one utterance.Edit probabilities are manually set to balance editor frequencies, and edit arguments are sampled from the parse or schema.Source: §5 · p. 5

6 Experiments

• The experiments use SPLASH with 7,481 training, 871 development, and 962 test examples, plus 50,000 synthesized training examples.Synthetic data are trained first for 20,000 steps, followed by SPLASH training through 100,000 total steps.Source: §6 · p. 5
• Correction accuracy measures exact-set agreement between the predicted corrected parse and gold parse, but cannot credit partial error reduction after one feedback turn.This motivates reporting Edit ↓, Edit ↑, and Progress.Source: §6 · p. 6
• Progress averages the relative reduction in edit-set size from the initial parse toward the gold parse, assigning partial credit and penalizing more erroneous corrections.A perfect model would achieve 100% Progress.Source: §6 · p. 6
• NL-EDIT improves correction accuracy by more than 16% over EditSQL+Feedback, beats oracle re-ranking by around 5%, and achieves 37% Progress.It strictly reduces initial-parse errors in 72.4% of test examples.Source: §6 · p. 6, Table 2 · p. 6

7.1 Ablations

• Removing feedback, explanation, question, interaction relations, or synthetic data tests the contribution of each model component under a common training and evaluation setup.Variants are selected by development correction accuracy and evaluated on the SPLASH test set.Source: §7.1 · pp. 6–7
• 19.8% of examples are corrected without feedback, showing that the question and explanation alone capture systematic parser errors.The ablated-feedback model gains less than 1% from additional SPLASH training after synthetic-data training.Source: §7.1 · p. 7

7.2 Error Analysis

• Correction accuracy drops sharply when the reference edit exceeds two operations, while increasing feedback or explanation length causes a more gradual decline.Among feedback longer than 24 tokens, 8% rewrites the whole query, whereas 92% enumerate several errors averaging edit size 5.5.Source: §7.2 · p. 7
• Even for initial parses with four or five errors, the model reduces the error count in most cases.This analysis measures errors by edit size before and after correction.Source: §7.2 · p. 7, Figure 4 · p. 7
• 15% of one-error failures involve wrong or non-editing feedback, 29% add irrelevant edits, and remaining failures mainly involve linking feedback to the schema.These cases identify both feedback quality and schema grounding as practical failure sources.Source: §7.2 · p. 7

7.3 Cross-Parser Generalization

• NL-EDIT significantly boosts newer parsers, but gains decline as baseline parser accuracy improves because errors increasingly occur in complex queries with long explanations.The authors hypothesize that localizing errors in these complex parses is the main generalization bottleneck.Source: §7.3 · p. 8

Natural language to SQL

• Natural-language interfaces to databases have been studied alongside datasets such as WikiSQL and SPIDER and increasingly capable text-to-SQL models.This work situates interactive correction within the broader natural-language-to-SQL literature.Source: §8 · p. 8

Interactive Semantic Parsing

• Prior interactive semantic parsers use feedback such as yes/no or multiple-choice answers, while SPLASH enables richer open-form natural-language corrections.NL-EDIT uses SPLASH and treats correction as a parser-editing problem.Source: §8 · pp. 8–9

Discussion

• Table 6 evaluates zero-shot generalization from SPLASH, built on Seq2Struct, to test sets based on EditSQL, TaBERT, and RAT-SQL.The comparison reports both correction accuracy and end-to-end parsing accuracy with and without interaction.Source: Table 6 · p. 9

Semantic Parsing with Synthetic Data

• The paper extends synthetic-data approaches by automatically simulating edits between erroneous and correct parses without human involvement.This synthetic-data framing supports training for semantic parse correction.Source: §8 · p. 9

Conclusions and Future Work

• The paper introduces a model, a data-augmentation method, and analysis tools for correcting semantic parse errors through natural-language feedback.These components form the paper’s overall contribution.Source: §9 · p. 9
• NL-EDIT improves correction accuracy by 16% and boosts end-to-end parsing accuracy by up to 20% with one turn of feedback.The conclusion also identifies future work on interaction modeling, synthetic-data training, user-centered progress optimization, and conversational text-to-SQL.Source: §9 · p. 9