Source-linked AI summary
Grounded Adaptation for Zero-shot Executable Semantic Parsing
Victor Zhong, Mike Lewis, Sida I. Wang, Luke Zettlemoyer
TL;DR
Semantic parsers are usually engineered for specific environments, raising the question of how to adapt them when a new environment has no training data. GAZP synthesizes data in the new environment and uses cycle-consistent examples to adapt an existing parser. It improves logical-form and execution accuracy across three zero-shot tasks, with analyses identifying cycle-consistency as central to successful adaptation.
Problem
Semantic parsers are usually engineered for individual application environments, and adapting them to new environments without training data remains a challenge.
Source: §1 · p. 1Method
GAZP combines a forward parser and backward utterance generator to synthesize new-environment examples, then adapts the parser using pairs whose parsed and sampled logical forms are equivalent.
Source: §1 · p. 1Results
Takeaways & Limitations
The analyses identify cycle-consistency as central to successful adaptation and report that performance improves with more GAZP-synthesized data.
Source: §1 · p. 1, §5 · p. 8The stated setup assumes logical forms include syntax candidates from a fixed syntax vocabulary and environment candidates from the environment description.
Source: §2 · p. 2
Abstract
from arXiv · showhide
We propose Grounded Adaptation for Zero-shot Executable Semantic Parsing (GAZP) to adapt an existing semantic parser to new environments (e.g. new database schemas). GAZP combines a forward semantic parser with a backward utterance generator to synthesize data (e.g. utterances and SQL queries) in the new environment, then selects cycle-consistent examples to adapt the parser. Unlike data-augmentation, which typically synthesizes unverified examples in the training environment, GAZP synthesizes examples in the new environment whose input-output consistency are verified. On the Spider, Sparc, and CoSQL zero-shot semantic parsing tasks, GAZP improves logical form and execution accuracy of the baseline parser. Our analyses show that GAZP outperforms data-augmentation in the training environment, performance increases with the amount of GAZP-synthesized data, and cycle-consistency is central to successful adaptation.
1 Introduction
Semantic parsers are often engineered for individual environments, leaving adaptation to new environments without training data as a challenge. GAZP synthesizes cycle-consistent data in the new environment and improves baseline accuracy across three zero-shot tasks.
1 Introduction
- Semantic parsers are commonly engineered for each application environment, and a parser trained on one database may struggle on another.The introduction frames the challenge as adapting parsers where no training data exists for the new environment.Source: §1 · p. 1
- GAZP adapts existing parsers by synthesizing data in the new environment and using cycle-consistent examples for retraining.The synthesized examples pair generated utterances with sampled logical forms that the parser maps back to equivalent forms.Source: §1 · p. 1
- Unlike typical data augmentation in the training environment, GAZP synthesizes consistency-verified examples in the new environment.The approach is complementary to prior modeling work and applies where logical-form equivalence can be checked.Source: §1 · p. 1
- GAZP improves the baseline parser’s logical-form and execution accuracy on Spider, Sparc, and CoSQL zero-shot tasks.The tasks test on unseen databases.Source: §1 · p. 1
2 Grounded Adaptation for Zero-shot Executable Semantic Parsing
GAZP adapts a semantic parser to unseen environments by generating examples there and retaining examples whose generated utterance and parsed logical form form a consistent cycle. Its method samples environment-specific queries, uses forward and backward models to produce and check examples, and retrains the parser with the retained examples.
2 Grounded Adaptation for Zero-shot Executable Semantic Parsing
- GAZP frames semantic parsing as producing an executable logical form for an utterance in an environment, where execution yields the desired denotation.For language-to-SQL parsing, the logical form and environment are a SQL query and database.Source: §2 · p. 2
2.4 Synthesizing cycle-consistent examples
- GAZP samples logical forms in a new environment, generates utterances, and keeps examples whose reparsed forms are equivalent before adapting the parser.For SQL, equivalence is assessed by whether the queries execute to the same denotation in the new database.Source: §2 · p. 2, §2.4 · p. 5
2.1 Query sampling
- Query sampling builds coarse templates from training queries, fills their slots with columns and values from a sampled database, and discards invalid or empty-result queries.Columns and values are replaced with typed slots when constructing the coarse templates.Source: §2.1 · pp. 2–3
2.2 Forward semantic parser
- The forward parser cross-encodes utterances with environment documents, predicts a logical-form template from syntax and environment candidates, then fills its slots with utterance values.The separate value decoder produces fully specified executable logical forms.Source: §2.2 · pp. 3–4
2.3 Backward utterance generator
- The backward generator augments logical-form environment candidates with corresponding spans from the environment description, then selects tokens to produce an utterance.Its pointer-decoder chooses from the augmented logical form and a fixed vocabulary.Source: §2.3 · pp. 4–5
3 Experiments
GAZP is evaluated on three zero-shot language-to-SQL tasks, with results showing gains across benchmarks and metrics. Analyses compare adaptation with training-environment augmentation and test the roles of cycle-consistency and synthesized-data quantity.
3 Experiments
- The evaluation covers Spider, Sparc, and CoSQL, measuring template exact match, execution accuracy, and fuzz-test accuracy.For multi-turn tasks, Sparc uses the previous system-produced query at evaluation, while training and synthesis use the ground-truth previous query; CoSQL focuses on state-tracking.Source: §3 · pp. 5–6
3.1 Results
- GAZP improves EM, EX, and FX across Spider, Sparc, and CoSQL compared with the unadapted baseline.Table 2 reports development-set evaluation; EM measures template exact match, EX execution accuracy, and FX fuzz-test execution accuracy.Source: §3.1 · p. 6, Table 2 · p. 7
- 57.2 EM for the dual objective trails GAZP adaptation at 59.1 EM on Spider.The dual objective jointly supervises the model in the reverse direction; the paper also reports that training-environment synthesis leads to overfitting in those environments.Source: §3.1 · p. 7
- Without cycle-consistency, synthesized examples significantly degrade performance; cycle filtering prunes about 60% of them.The ablation compares adaptation without cycle-consistency against the cycle-consistent approach.Source: §3.1 · p. 7, Table 3 · p. 7
- String-match EM consistency also improves performance and performs similarly to execution consistency, though typically with lower execution accuracy.The variant retains examples when the original and reparsed logical forms match as strings.Source: §3.1 · p. 7, Table 3 · p. 7
- Adaptation performance generally rises with more cycle-consistent examples, with diminishing returns after 30–40k examples.Figure 5 plots development-set EM and EX against the number of cycle-consistent examples synthesized in inference environments.Source: §3.1 · p. 8, Figure 5 · p. 8
4 Related work
Related work addresses zero-shot parsing, data augmentation, and cycle-consistent generation. GAZP differs by synthesizing forms in new environments, filtering examples for consistency, and checking query–utterance alignment through execution.
4 Related work
- Prior language-to-SQL methods generalize to new databases by selecting from their schemas (Zhang et al., 2019; Guo et al., 2019), while GAZP’s synthesis, consistency, and adaptation steps can complement any parser with a backward generator and logical-form equivalence checks.The paper situates GAZP within zero-shot semantic parsing, where models must handle environments unseen during training (Pasupat and Liang, 2015; Zhong et al., 2017).Source: §4 · p. 8
- Unlike training-domain data augmentation, GAZP samples logical forms from the new environment and uses cycle-consistency to filter low-quality examples.The paper reports that these differences are central to improving parsing performance.Source: §4 · p. 8
- Unlike cycle-GANs, GAZP uses no adversarial objective and verifies consistency between synthesized utterances and outputs by executing queries.The paper contrasts this with cycle-GANs’ discriminator-based check that generated inputs resemble natural inputs.Source: §4 · p. 8
5 Conclusion and Future work
GAZP adapts existing semantic parsers to new environments by synthesizing cycle-consistent data, and improves performance on three zero-shot parsing tasks. The paper identifies robotics as a possible area for future application.
5 Conclusion and Future work
- GAZP improved parsing performance on three zero-shot parsing tasks, and its analyses found gains over data augmentation, scaling with synthesized-data amount, and cycle-consistency central to adaptation.Source: §5 · p. 8
- The authors identify robotics as a possible application because annotated real-world trajectories are prohibitively expensive to collect.Future work would interpret environment specifications to support grounded adaptation in other areas.Source: §5 · p. 8
A.1 Coverage and multi-turn sampling
The Spider template-filling sampler covers many development-set examples, while Sparc and CoSQL use conditional sampling to produce paired previous and current queries.
A.1 Coverage and multi-turn sampling
- 85% of Spider development-set examples have a query template found in the training set, though template filling does not cover the full development set.Coverage is measured by the share of development examples whose query template occurs in training.Source: §A.1 · p. 11
- For Sparc and CoSQL, sampling first draws a previous query from the empirical template distribution, then draws the current query conditioned on that previous query.The template distributions are estimated by counting templates in the training set.Source: §A.1 · p. 11
A.2 Hyperparameters
The implementation uses LSTMs with DistilBERT and Adam, tunes dropout by three-fold grid search, and reports substantial training compute and model sizes.
A.2 Hyperparameters
- The models use 300-dimensional LSTMs and DistilBERT, optimized with Adam at an initial learning rate of 5e-5 for 50 epochs.Training uses batch size 10 and gradient clipping with norm 20.Source: §A.2 · p. 11
- Dropout follows BERT and encoder LSTMs and precedes the pointer scorer; rates are tuned using three-fold cross-validation over 0.1 and 0.3.Tables 4 and 5 report dropout rates for the forward parser and backward generator, respectively.Source: §A.2 · p. 11, Table 4 · p. 11, Table 5 · p. 11
- A forward-parser run took approximately 16 hours on one Titan X GPU, and the final parser and generator contain 142 million and 73 million parameters, respectively.The reported 27 runs include three folds per task and the final official train/dev run.Source: §A.2 · p. 11
A.3 Synthesized examples
Synthesized Spider queries resemble development-set queries on several structural measures, and a sampled set of synthesized examples was mostly correct, fluent, and sensible.
A.3 Synthesized examples
- 0.60 vs. 0.47 joins, 1.21 vs. 1.37 conditions, 0.20 vs. 0.26 group by’s, 0.23 vs. 0.25 order by’s, 0.07 vs. 0.04 intersections, and 1.25 vs. 1.32 selection columns per query compare synthesized data with Spider development examples.The authors say this suggests synthesized queries are similar to real data.Source: §A.3 · p. 13
- 51 of 60 sampled synthesized examples were correct; reported mistakes included aggregation over wrong columns and underspecified questions.Some examples also had grammatical errors, though most questions were described as fluent and sensible.Source: §A.3 · p. 13
A.4 Performance breakdown
GAZP’s gains are generally larger for harder queries and later interaction turns across the evaluated tasks, with possible explanations in new-environment query coverage and multi-turn synthesis.
A.4 Performance breakdown
- GAZP’s gains are generally more pronounced on harder queries across Spider, Sparc, and CoSQL, with some variance across tasks and evaluation metrics.Difficulty classes reflect the number of SQL components, selections, and conditions; the tables break results down by task.Source: §A.4 · p. 13, Table 7 · p. 13, Table 8 · p. 13
- The “extra” difficulty category is a catch-all for queries exceeding the hard-category criteria and may include confounding artifacts such as set exclusion operations.The authors caution that these artifacts may introduce other confounding factors.Source: §A.4 · p. 13
- GAZP’s larger gains on harder queries may reflect its synthesis of difficult queries in the new database, where training data is lacking.The authors suggest simpler queries may be easier to learn, leaving less room for adaptation to help.Source: §A.4 · p. 14
- For Sparc and CoSQL, GAZP’s gains are more pronounced in later interaction turns across tasks and evaluation metrics.The turn breakdowns are reported for Sparc and CoSQL.Source: Table 10 · p. 13, §A.4 · p. 14, Table 11 · p. 14
- Conditional sampling may support later-turn parsing by synthesizing multi-turn examples and expanding query variety.The authors suggest this variety may improve later-turn queries and reduce cascading errors.Source: §A.4 · p. 14