Source-linked AI summary
Generate-and-Retrieve: use your predictions to improve retrieval for semantic parsing
Yury Zemlyanskiy, Michiel de Jong, Joshua Ainslie, Panupong Pasupat, Peter Shaw, Linlu Qiu, Sumit Sanghai, Fei Sha
TL;DR
Low-resource semantic parsing needs informative retrieved exemplars, but existing retrieval mainly matches inputs. GandR generates a preliminary prediction, retrieves exemplars balancing input and output similarity, and achieves state-of-the-art results across several low-resource semantic parsing benchmarks.
Problem
Retrieval based mainly on input similarity may fail to identify informative exemplars for low-resource semantic parsing.
Method
GandR generates a preliminary prediction with input-based retrieval, then retrieves exemplars using a relevance score combining input similarity with output similarity.
Results
GandR achieves state-of-the-art results on several low-resource semantic parsing benchmarks.
Takeaways & Limitations
Balancing input similarity with template-relevant output similarity retrieves exemplars that are more helpful for the model.
Takeaways & Limitations
Output similarity can distract the model from a lexically similar and informative exemplar, causing an incorrect prediction.
Abstract
from arXiv · showhide
A common recent approach to semantic parsing augments sequence-to-sequence models by retrieving and appending a set of training samples, called exemplars. The effectiveness of this recipe is limited by the ability to retrieve informative exemplars that help produce the correct parse, which is especially challenging in low-resource settings. Existing retrieval is commonly based on similarity of query and exemplar inputs. We propose GandR, a retrieval procedure that retrieves exemplars for which outputs are also similar. GandRfirst generates a preliminary prediction with input-based retrieval. Then, it retrieves exemplars with outputs similar to the preliminary prediction which are used to generate a final prediction. GandR sets the state of the art on multiple low-resource semantic parsing tasks.
1 Introduction
Low-resource semantic parsing benefits from retrieved training exemplars, but input similarity alone may miss examples with useful output structures. GandR addresses this by generating a preliminary prediction, retrieving exemplars similar in both input and output, and achieving state-of-the-art results on low-resource benchmarks.
- 1 Introduction: Low-resource performance suffers when limited training data cannot match task complexity, motivating retrieval of salient input-output patterns at test time.Prior work shows that appending retrieved exemplars can improve low-resource performance.
- 1 Introduction: Existing retrieval asks which outputs correspond to similar inputs, whereas GandR also asks which inputs correspond to similar outputs.This output-space signal is intended to provide complementary information for exemplar selection.
- 1 Introduction: GandR first generates a preliminary prediction with input-similar exemplars, then retrieves exemplars balancing input similarity with similarity between the prediction and exemplar outputs.The final prediction is generated from the second retrieval set.
- 1 Introduction: GandR evaluates task-oriented semantic parsing and reports that output similarity provides a complementary retrieval signal, including cases where less similarly phrased examples have more similar outputs.The model can also verify whether preliminary predictions are valid target-language outputs.
- 1 Introduction: GandR strongly improves low-resource semantic parsing and achieves state-of-the-art results on MTOP and TopV2 low-resource and transfer benchmarks.The benchmark result is reported for low-resource and transfer settings.
2 Method
GandR uses a sequence-to-sequence semantic parser whose retrieval function combines input similarity with similarity between a preliminary predicted structure and candidate outputs. Training and evaluation use retrieved input-output exemplars to augment the query before final parsing.
- 2 Method: The parser retrieves K = 4 training exemplars, appends their input-output pairs to the sample input, and applies a T5 sequence-to-sequence model to predict the parse.The augmented input concatenates the retrieved pairs with the original query.
- 2 Method: During evaluation GandR selects the top K relevant exemplars, while training samples retrievals according to a geometric distribution over relevance rank.The rank-based sampling probability is p(1 −p)^r, with p serving as a temperature hyperparameter.
- 2 Method: The method follows the input-retrieval approach of Pasupat et al. (2021) but replaces its primary relevance choice with a hybrid input-output similarity measure.A simple TF-IDF input-similarity baseline is reported as comparable to or better than the Universal Sentence Encoder baseline.
- 2 Method: GandR builds a hybrid relevance measure from sample-input similarity and similarity between the model’s predicted structure and candidate output.Output similarity uses TF-IDF over predicted and true structures, represented here by intents and slots.
- 2 Method: For convenience, GandR trains in two stages: M1 produces preliminary predictions and M2 produces final predictions using retrievals based on M1.A single-weight alternative is possible if training mixes input-only and GandR retrieval augmentations.
3 Related Work
Sequence-to-sequence semantic parsers commonly augment inputs with retrieved exemplars, using input similarity or learned retrieval methods to select examples. Related work also examines exemplar selection for in-context prompting.
- 3 Related Work: Sequence-to-sequence models are pretrained with language modeling and fine-tuned for specific domains, achieving strong results in semantic parsing and structured prediction.
- 3 Related Work: Several semantic parsing methods retrieve exemplars whose input encodings resemble the query, while BM25 similarity has also been used for exemplar retrieval.
- 3 Related Work: Dense retrievers incorporate label information but require a separately trained retrieval model, unlike approaches using sparse similarity over model predictions.
- 3 Related Work: In-context prompting likewise selects training exemplars with pretrained or fine-tuned sentence encoders.
- 3 Related Work: Table 2 reports the number of examples for each dataset and split.
4 Experiments
Experiments show that GandR improves low-resource semantic parsing by retrieving exemplars using both input and output similarity, while retrieval is less useful in high-resource settings.
- 4.2 Main results: GandR retrieval outperforms all baselines, setting the state of the art on evaluated low-resource and transfer settings.The primary experiments cover MTOP and TOPv2 benchmarks, including low-resource and transfer conditions.
- 4.3 Ablations and discussion: High-resource results show that exemplar augmentation is less effective and retrieval methods produce nearly identical results on the highest-resource TOPv2 dataset.
- 4.3 Ablations and discussion: Input and output similarity are strongly complementary, with hybrid similarity improving MTOPboot performance across output-similarity weights.
- 4.3 Ablations and discussion: Considering output similarity increases template recall, although output-only retrieval can match or exceed GandR on this proxy while achieving lower performance.GandR’s results indicate that balancing input similarity with template recall yields more informative exemplars for the model.
- 4.3 Ablations and discussion: GandR outperforms input TF-IDF on MTOP1k when retrieving 1, 2, or 4 exemplars, showing robustness to the number of retrieved examples.
- 4.4 Error analysis: An error-analysis example shows input TF-IDF selecting lexical overlap without relevance, whereas GandR balances lexical and label similarity to produce a correct prediction.Output similarity can also distract the model from a lexically similar informative exemplar, causing an incorrect result.
5 Conclusion
GandR generates a preliminary prediction, retrieves exemplars with similar outputs and inputs, and uses them to produce a final prediction. Output similarity improves low-resource semantic parsing and achieves state-of-the-art results on several benchmarks.
- GandR generates a preliminary prediction, retrieves training exemplars with similar outputs and inputs, and augments the input to generate a final prediction.
- Using output similarity improves semantic parsing in low-resource settings and achieves state-of-the-art results on several semantic parsing benchmarks.
Output TF-IDF
Output similarity is measured by applying TF-IDF to structured parse representations. Each intent and slot becomes a token, while slot values are omitted because input similarity is intended to capture them.
- Output similarity measures TF-IDF similarity between parses represented as sentences of intent and slot tokens.
- Slot values are discarded because their similarity should already be captured by input TF-IDF similarity.
- The method therefore uses a simple, non-learned similarity measure over structured parse elements.
B Training
The experiments use T5.1.1-base with fixed optimization settings and vary sampling temperature, batch size, and output-similarity weight. Input TF-IDF and GandR receive matched training opportunities across tasks.
- Models are initialized from the public T5.1.1-base checkpoint and fine-tuned with ADAM, dropout 0.1, and weight decay 0.01.
- Experiments vary sampling temperature across {0.5, 0.1, 0.05}, batch size across {128, 256}, and output similarity weight α across six values.
- The α = 0 setting is the input TF-IDF baseline.
- Input TF-IDF and GandR are trained for the same number of steps, with task-specific schedules and early stopping on MTOP25%.
- Hyperparameters are selected using development-set performance, and reported results average three random seeds.
D Error analysis
The error analysis examines how retrieval choices and output-similarity weighting affect predictions across MTOP settings. Examples show that GandR can correct invalid or lexically misleading slot and intent assignments, though retrieval can also be beneficial or harmful.
- Retrieval examples: The paper provides additional retrieval examples to illustrate cases where GandR retrieval can be beneficial or harmful.
- Sensitivity analysis: Figures 4 and 5 vary output similarity weight α on MTOP1k and MTOP25 development sets, while Figure 6 varies retrieved exemplars K on MTOP25%.
- Error analysis: Table 7 shows GandR removes an invalid SL:ORDINAL slot because Weather training exemplars do not contain that slot.
- Error analysis: Table 8 shows GandR avoids assigning SL:CONTACT myself when matching the correct template without that slot outweighs lexical similarity.
- Error analysis: Table 9 shows balancing lexical input similarity with label similarity can avoid an irrelevant exemplar and produce a correct prediction.