Learning to Simulate Natural Language Feedback for Interactive Semantic Parsing

1 Introduction

• Interactive semantic parsing allows users to give feedback after a parser presents its result, whereas single-turn models produce an output and terminate the task.Semantic parsing translates natural-language sentences into logical forms; the feedback describes errors and how to correct them.Source: §1 · pp. 1–2
• Prior feedback collection used 10 crowd workers and took six minutes per annotation; because it is parser-specific, the collected feedback may not generalize to other parsers.Source: §1 · p. 2
• The paper formulates feedback simulation as generating a natural-language correction from the command, incorrect and ground-truth logical forms, and contextual information.The simulator is intended to extend feedback to different parsers at scale, starting from a small set of human annotations.Source: §1 · p. 2
• The proposed evaluator assesses whether simulated feedback matches correction intent by comparing it with template feedback, and the authors use it to select among T5-based simulators.Source: §1 · p. 2
• Simulated feedback improves error correction for both Seq2Struct and EditSQL, and low-data training reaches performance comparable to training with the full SPLASH set.The experiments use SPLASH, which contains human-annotated feedback for Seq2Struct mistakes.Source: §1 · p. 2

2.1 Overview

• The simulator is intended to generate high-quality feedback at scale from a small set of human annotations, then support error-correction-model training.The interactive parser presents an initial logical form explanation and receives NL feedback describing errors and corrections; the explanation uses NL templates in this work.Source: §2.1 · pp. 2–3

2.2 Feedback Evaluation

• The evaluator scores whether simulated feedback is logically consistent with the correction intent, using template feedback rather than human feedback as its reference.The template is described as a cleaner standard than human annotations, which may contain inaccurate or incomplete error descriptions, and can be generated automatically at scale.Source: §2.2.1 · p. 3

2.2.1 Task Formulation & Architecture

• Its architecture compares contextualized candidate and reference tokens, aggregates token similarities into sentence-level precision and recall, and trains with a contrastive hinge loss.It also uses task-derived token-alignment supervision with a mask that excludes tokens whose correct alignments cannot be derived; negative feedback is created by replacing values or schema items.Source: §2.2.1 · pp. 3–4

2.3 Feedback Simulation

• CWQES supplies correct and incorrect logical forms, DQES supplies linearized edits, and TQES supplies template descriptions of the edits as simulation inputs.TQES frames simulation as paraphrasing template feedback into language closer to how real users speak.Source: §2.3 · pp. 4–5

3.1 Experimental Setup

• The experiments use SPLASH feedback for Seq2Struct and test simulated feedback on EditSQL, with roughly 2,400 simulated examples augmenting the original SPLASH training set.In the low-data setting, only 5%, 10%, or 20% of SPLASH training examples are used to construct the simulator and evaluator; the remainder is simulated.Source: §3.1 · p. 5

3.2 Can the Feedback Simulator Generate Useful Feedback for a Specific Parser?

• Adding simulated feedback for EditSQL improves its correction accuracy by 3.5% absolute and improves partial-correction metrics by 5–8%.On the SPLASH test set, it also yields around 2% gain on correction accuracy and 2.5–3.5% on other metrics; the authors suggest the smaller gains may reflect that the simulated feedback is based on EditSQL.Source: §3.2 · p. 6, Table 1 · p. 6

3.3 Can the Feedback Evaluator Properly Assess Each Simulator?

• The proposed evaluator substantially outperforms BLEU and BERTScore on both feedback-ranking metrics, better assessing logical consistency and distinguishing feedback quality.Table 2 reports MRR for ranking positive feedback on SPLASH-dev and Spearman ranking correlations with human judgments.Source: Table 2 · p. 6, §3.3 · p. 7

3.4 How Does Each Feedback Simulator Variant Perform?

• TQES achieves the best performance according to the proposed evaluator, so the authors use it as their default feedback simulator.TQES directly includes edit information and translates it into text, which the authors say better fits T5’s textual pre-training than the alternatives.Source: §3.4 · p. 7, Table 3 · p. 7

3.5 Can the Feedback Simulator Work Well in the Low-data Setting?

• With only a small amount of training data, simulated feedback yields error-correction performance comparable to training with the full SPLASH set.Performance increases as the amount of human-annotated data grows; template feedback also helps in low-data settings, but its gains are smaller than those from simulated feedback.Source: §3.5 · pp. 7–8, Figure 4 · p. 7

Interactive Semantic Parsing

• Interactive semantic parsing has used selectable logical-form components or paraphrases, while NL feedback lets users describe and correct parsing errors.Prior work showed that an error-correction model can learn from NL feedback; this paper complements that work by simulating feedback rather than changing the correction-model architecture.Source: §4 · p. 8

General NLP Research with Human Feedback

• Other NLP work uses feedback such as chatbot responses, ratings, explanations, or expert rankings, whereas this work targets corrective NL feedback.The authors describe corrective NL feedback as still largely under-explored and leave improving parsers from such feedback as future work.Source: §4 · p. 8

User Simulation in Dialogue Systems

• Dialogue-system user simulators model utterances and often user goals or agendas, while this work targets utterance generation for correcting parsing mistakes.Other dimensions of user simulation, including the agenda of error correction, are left for future work.Source: §4 · p. 8

Text Evaluation

• The authors report that surface-form metrics such as BLEU miss semantic modifications and that general-purpose neural metrics fail to identify subtler logical differences.They therefore train a text evaluator to assess whether simulated feedback is logically consistent.Source: §4 · pp. 8–9

• The work proposes the task of simulating NL feedback for interactive semantic parsing.Source: §5 · p. 9
• It presents models for both feedback evaluation and feedback simulation.Source: §5 · p. 9
• Experiments demonstrate the effectiveness of both models and the promise of saving human-annotation effort with simulated feedback.Source: §5 · p. 9

• Both the feedback simulator and evaluator can be further improved.Suggested directions include specialized simulator architectures, evaluator-guided training, finer-grained user behavior modeling, and continually improving parsers from feedback.Source: Limitations · p. 9
• Experiments evaluate the approaches only on text-to-SQL, although they make no assumptions about logical-form type.Source: Limitations · p. 9
• Applying the simulator to other NLP tasks may require meaning representations for both gold and model predictions.For retrieval-based QA, the authors say representations of the retrieval processes would need defining; for open-ended story generation, doing so would take non-trivial effort.Source: Limitations · p. 9

Ethics Statement

• Generated feedback is used to train the error-correction model and is not exposed to real users, though it may not perfectly match the intended correction.The authors therefore do not anticipate ethical issues and note the potential positive impact of feedback-driven human-AI interaction.Source: Ethics Statement · p. 9

A.1 Template Feedback

• Template feedback represents SQL corrections through replace, add, and remove operations, using placeholders for schema items.Subscripts distinguish wrong and correct items in replacements, new and old items in additions, and multiple items in a template.Source: §A.1 · p. 13

A.2 Post-processing of Feedback Evaluation

• The evaluator weights primary feedback spans more heavily than secondary spans because they contribute differently to describing correction intent.The method uses one-to-one bipartite matching before calculating similarity and normalizes for span weighting.Source: §A.2 · p. 13

A.3 Error Correction Model

• Fine-tuned pre-trained T5 generates the gold logical form from feedback, explanation, the initial question, and contextual information.The model corrects the initial logical form; investigating more advanced error-correction architectures is left for future work.Source: §A.3 · pp. 13–14

B.1 Implementation Details

• The evaluator’s best configuration used learning rate 1e-8, batch size 64, m = 0.1, and λ = γ = 1e-3; its primary-span weight was 0.9.It was trained for at most 200 epochs, with model selection based on dev-set MRR.Source: §B.1 · p. 14
• The T5-large simulator used learning rate 1e-4, batch size 5, and up to 10,500 training steps.Training took roughly 10 hours for the simulator and 48 hours for the evaluator on one NVIDIA A100 80GB GPU; experiments used one random seed.Source: §B.1 · p. 14

B.2 Dataset and Prepossessing

• SPLASH is an English dataset whose feedback was collected from anonymized Amazon Mechanical Turk crowd workers.The dataset is distributed under CC BY-SA 4.0.Source: §B.2 · p. 14
• 652 train, 61 dev, and 92 test examples were filtered as structural errors involving an entire missed or incorrect subquery.The authors report that feedback for errors requiring an entire subquery to be added or removed is noisy and inaccurate.Source: §B.2 · pp. 14–15, Table 6 · p. 15

B.3 Error Correction Model Implementation

• The error correction model is based on T5-base; comparisons include reported results for EditSQL+Feedback and NL-Edit, which are not publicly available.EditSQL+Feedback is distinct from the EditSQL model used in the main experiments.Source: §B.3 · p. 15
• A 10% gap separates the authors’ model from NL-Edit, while their performances are very comparable in all other metrics.The authors attribute the gap partly to strict full-correction scoring and noisy or incomplete human feedback, which can leave only partial corrections.Source: §B.3 · p. 15

C.1 Example of Feedback Simulation

• For an uncommon SPLASH error pattern also present in EditSQL, training correction with both datasets fixes an error that SPLASH-only training cannot fix.The authors describe the simulated feedback for this example as high quality, although not perfect.Source: §C.1 · p. 16

C.2 Human Evaluation

• Human evaluators rated 50 SPLASH dev examples on a five-level Likert scale, and metric quality was assessed by Spearman rank correlation with those ratings.The evaluation targeted whether metrics could rank feedback sentences similarly to humans; Table 8 reports that the evaluator correctly ranks the three simulated feedbacks in two examples.Source: §C.2 · pp. 16–17, Table 8 · p. 17, Table 9 · p. 17

C.3 Case Study of Evaluation Metrics

• In the easy and complex examples, the evaluator makes the reported distinctions among simulated feedback, while BLEU and BERTScore fail on the comparisons described.For the easy example, BLEU ranks CWQES highest; for the complex example, both baselines fail to detect the logical inconsistencies described for CWQES and TQES.Source: §C.3 · p. 17
• Figures 5 and 6 show token-level similarity matrices generated by BERTScore and the evaluator; the evaluator’s matrix is reported as sparser and more accurate.Source: §C.3 · p. 17
• The feedback simulator produces high-quality feedback in the TQES setting for both reported examples.Source: §C.3 · p. 17

C.4 Feedback Simulation in Low-data Settings

• 20% of SPLASH training data yields a simulator with comparable generation quality, according to the authors.The low-data simulators are evaluated with an evaluator trained on full SPLASH.Source: §C.4 · p. 18
• For selecting the best simulator in each low-data setting, the evaluator is trained using the same small amount of SPLASH data.Source: §C.4 · p. 18
• The authors connect the comparable generation quality at 20% data with the small gap between error correction models trained on full SPLASH and simulated feedback.Source: §C.4 · p. 18