Constrained Language Models Yield Few-Shot Semantic Parsers

Richard Shin; Christopher H. Lin; Sam Thomson; Charles Chen; Subhro Roy; Emmanouil Antonios Platanios; Adam Pauls; Dan Klein; Jason Eisner; Benjamin Van Durme

Source-linked AI summary

Constrained Language Models Yield Few-Shot Semantic Parsers

Richard Shin, Christopher H. Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, Benjamin Van Durme

arXiv:2104.08768v2Apr 18, 2021cs.CL

TL;DR

Semantic parsing requires structured meaning representations, while pretrained language models naturally generate natural language and lack abundant task-specific representation data. The paper bridges this gap by prompting and constraining language models to produce controlled canonical paraphrases that map to meaning representations, achieving effective few-shot performance across multiple tasks. The approach also supports rapid parser bootstrapping with limited data, though dialogue-history omission hurts performance relative to prior work.

Problem
Semantic parsing must generate structured meaning representations, but language models are trained for natural-language generation and task-specific representations are scarce in web data.
Method
The method primes pretrained language models with few-shot examples and constrains decoding so user utterances become canonical natural-language paraphrases that can be mapped to task-specific representations.
Results
Across multiple datasets, constrained decoding outperformed unconstrained decoding, and controlled natural-language targets were better than meaning representations for pretrained language models.
Takeaways & Limitations
With small hundreds of examples, the approach quickly bootstraps semantic parsers and supports future human-in-the-loop iterative model refinement.
Takeaways & Limitations
Ignoring dialogue history hurts performance relative to prior work, though incorporating history into prompts remains future work.

Abstract

from arXiv · show

We explore the use of large pretrained language models as few-shot semantic parsers. The goal in semantic parsing is to generate a structured meaning representation given a natural language input. However, language models are trained to generate natural language. To bridge the gap, we use language models to paraphrase inputs into a controlled sublanguage resembling English that can be automatically mapped to a target meaning representation. Our results demonstrate that with only a small amount of data and very little code to convert into English-like representations, our blueprint for rapidly bootstrapping semantic parsers leads to surprisingly effective performance on multiple community tasks, greatly exceeding baseline methods also trained on the same limited data.

1 Introduction

The paper uses pretrained language models for few-shot semantic parsing by generating controlled, canonical paraphrases that a grammar maps to task-specific meaning representations. With limited examples, this approach supports rapid parser prototyping and performs effectively across community datasets.

Semantic parsing is difficult to fit into few-shot text-to-text prompting because task-specific meaning representations rarely appear in web data at sufficient scale.
A synchronous context-free grammar defines valid meaning representations and canonical language constructions, while the language model maps an unconstrained utterance to a canonical form.
Given a grammar, language model, and handful of examples, the method immediately produces a working parser for a new domain.
Experiments cover Overnight, Break, and SMCalFlow with GPT-2, GPT-3, and BART, reporting better natural-language-like targets, gains from explicit decoder constraints, and strong few-shot performance.
The workflow generates paraphrases, constrains them to canonical utterances, and mechanically converts the highest-scoring candidate into a task-specific meaning representation.

2 Background

The paper builds on few-shot generation with autoregressive language models and reframes semantic parsing as paraphrasing through canonical utterances. Its approach uses a grammar to constrain generation rather than merely score previously generated candidates.

Autoregressive language models generate token sequences incrementally, supporting prompt-based few-shot task completion without task-specific fine-tuning.
Few-shot prompting places several input-output examples before a test input and asks the language model to generate the subsequent task output.
Semantic parsing can be represented as a triple where the parser maps a natural utterance u to a canonical utterance c and then to a meaning representation m.
The paper’s innovation is prompting and constraining a generative language model to perform the u 7→c mapping.
Unlike prior approaches that generate or enumerate candidate canonical forms and score them, this procedure uses the utterance to generate candidates while the grammar constrains them.
The grammar-based setup is motivated by human-in-the-loop bootstrapping, with accuracy at K measuring whether an annotator finds the correct parse among K options.

3 Approach

The approach combines few-shot language-model priming with constrained decoding over valid target outputs. Prompts use at most 1,000 training examples, and generation can target either canonical linguistic representations or original meaning representations.

Dynamic Prompt Creation: GPT-3 is used for prompt-based few-shot generation because it was the largest available language model with an accessible API at the time.
The method has two components: language-model priming through dynamic prompts or fine-tuning, and constrained decoding that ensures well-formed outputs.
Dynamic Prompt Creation: Dynamic prompts contain a small representative set of utterance-output examples, with each task using 1,000 or fewer training examples.
Dynamic Prompt Creation: Examples target either the original meaning representation m_i or a canonical linguistic representation c_i that can be translated into m_i.
Dynamic Prompt Creation: GPT-3 selects prompt examples by ranking training pairs according to p(u | u_i), while the prompt is shortened when necessary to satisfy the 2,048-token API limit.
Fine-tuning: Fine-tuning uses the same candidate examples to train models to map utterances directly to target outputs, including sequence-to-sequence semantic parsing.
Constrained Decoding: Constrained decoding defines valid next tokens for each partial sequence and uses length-normalized beam search to generate the output.

4 Case Studies

Across Overnight, Break, and SMCalFlow, the case studies evaluate few-shot semantic parsing with natural-language-like canonical representations and constrained decoding. The approach performs competitively in low-data settings, with especially strong results on Overnight and consistent gains from constraints and more natural representations.

Experimental scope: The case studies evaluate whether language models can predict original meaning representations directly or through canonical natural-language forms, using datasets including Overnight, Break, and SMCalFlow.Results compare mappings u → m with two-step mappings u → c → m.
4.1 Overnight: Overnight experiments simulate low-data prototyping with 200 randomly selected training examples, while GPT-3 is also tested with 20 examples.GPT-3 prompts select up to P = 20 training examples, and constrained decoding uses beam search with beam size 10.
4.1 Overnight: On Overnight, GPT-3 performs best among models trained on 200 examples and approaches models trained on all available training examples.On Calendar, Housing, and Restaurants, the method obtains similar numbers to the state-of-the-art approach using 7 to 13 times less training data.
4.1 Overnight: 0.98 accuracy@5 is achieved by Constrained Canonical on Calendar with GPT-3 using only 200 training examples.The top-K score counts an input as correct if any of the top K hypotheses is correct.
4.2 Break: On Break, canonical representations are easier to predict than meaning representations, and constrained decoding adds 7–11% absolute accuracy gains.All methods outperform the BART Unconstrained Meaning baseline trained on the same number of examples, while constrained canonical accuracy improves as training examples increase from 25 to 1,000.
4.3 SMCalFlow: On SMCalFlow, targeting a more natural representation and constraining decoding improves results, and the methods significantly outperform the BART Unconstrained Meaning baseline.The original meaning representation is used for evaluation; mapping back can require guessing because canonical utterances may be ambiguous and intermediate representations may omit information.

5 Discussion

The experiments show that constrained decoding improves semantic parsing, while controlled natural-language representations outperform meaning representations with pretrained LMs. The latter advantage is illustrated by better preservation of negation and interpretation of novel language.

Constrained decoding is better than unconstrained decoding because unconstrained generation can produce non-canonical answers.For example, GPT-3 produced a non-canonical form that omitted the required canonical construction.
Controlled natural languages outperform meaning representations when used with pretrained LMs, although this effect is harder to isolate.The authors provide suggestive examples rather than a definitive isolation of the effect.
Preserving the input word “not” helped the method correctly represent that meetings were not attended by Alice.Constrained prediction of the meaning representation instead dropped the negation by using = rather than !=.
The method correctly interpreted the novel bigram “as many” as equality between the numbers of matte and metallic objects.Constrained QDMR prediction instead selected a higher-than relation.

6 Further Related Work

The paper situates its grammar-constrained, pretrained-LM approach among work on constrained generation, neural semantic parsing, and parser bootstrapping. It distinguishes its few-shot focus from concurrent work emphasizing unsupervised learning.

Grammar-constrained decoding restricts successful generation paths to sequences parseable under an externally provided grammar.Earlier applications included task-oriented speech recognition, where users were assumed to phrase commands precisely.
Semantic parsing has increasingly used encoder-decoder neural architectures and pretrained language models, including BART for encoder and decoder initialization.The paper places its approach within this progression from neural architectures to pretrained models.
Prior parser-bootstrapping work used grammar-generated synthetic examples or crowdworker paraphrases to create training data.These approaches include generating canonical forms from grammars and paraphrasing them into training examples.
The paper independently confirms positive Overnight results while adding studies on Break and SMCalFlow in a few-shot setting.Unlike concurrent work focused primarily on unsupervised learning, these experiments assume small hundreds of examples from a domain expert.
Grammar restrictions were historically used in speech recognition because limiting possible word sequences was considered necessary for high accuracy.The paper connects this earlier assumption to contemporary task-oriented parsing with task-specific features.

7 Conclusion

The approach uses constrained decoding and synchronous context-free grammars to paraphrase utterances into controlled English that maps to task-specific meaning representations. With small hundreds of examples, it quickly bootstraps parsers across datasets and supports future human-in-the-loop refinement.

7 Conclusion: Small hundreds of examples enable quickly bootstrapped semantic parsers for a variety of datasets.The paper connects this rapid prototyping to future human-in-the-loop model refinement.
7 Conclusion: An SCFG maps between meaning representations and canonical English by synchronizing source and target expansions through shared nonterminals.Parsing a canonical string yields a tree that can generate its corresponding meaning representation.
7 Conclusion: The SCFG framework defines context-free languages through recursive rule expansion and synchronizes paired source-target grammars for translation.A CFG uses nonterminals, terminals, rules, and a start symbol; an SCFG shares nonterminals across two CFGs.
7 Conclusion: Domain functions receive typed arguments and return typed values, while constants are represented as 0-ary functions without parentheses.The grammar is constructed from English templates associated with functions and their types.
7 Conclusion: The toy grammar maps “Buy a green box” to the program buy(toGreen(square)).The derivation uses the command, color, and shape rules in the grammar.

A.3 Ambiguity

The SCFG introduces ambiguity through both ordinary English and recursion enabled by templates without English literals. The authors bound recursion, enumerate finite parses, and apply heuristics, while transforming representations and using macros to improve tractability and fluency.

English ambiguity can map one canonical utterance to multiple meanings, as in two interpretations of “create a meeting after the meeting with Bob.”
Templates without English literals can permit arbitrarily many applications, producing infinitely many parses for utterances such as “Bob.”
The method limits grammar recursion to produce finitely many parses, then uses heuristic rules to select among them.The authors report that this simple procedure works well in practice.
90%: heuristic rules recover the correct parse in practice after discarding programs with known structural errors.
Character-level grammar terminals accommodate differences between language-model tokenizers and grammar terminals.
An intermediate representation removes reentrancies and replaces common program fragments with macros before SCFG induction.This supports mapping between fluent canonical English and the original SMCalflow representation.

C Stratified Datasets

The stratified-dataset procedure aims to simulate small datasets that cover supported functionality. It filters rare functions, then formulates subset selection as a mixed-integer program.

Function histograms count signatures appearing in each training, validation, and test example.
Functions appearing fewer than 10 times in training are filtered, and examples containing them are removed across dataset splits.
Stratified sampling selects subsets guaranteeing that every remaining function appears in at least k examples.The selection problem is formulated as a mixed-integer program.

D.1 Our reproduction of Cao et al. (2019)

The Overnight reproduction evaluates a state-of-the-art method using the same 200 training examples as the paper’s methods, with adjusted development-set and training procedures.

200 training examples were used for each domain, matching the examples used in the paper’s own experiments.
Because Overnight lacks an official development set, a separate 20% development set was sampled from previously unused training data.
GPT-2 XL accuracy on Overnight is reported in Table 5.
The reproduction increased the maximum number of training epochs from 100 to 200 and selected the best development-set snapshot.
Meaning representations were normalized by removing the redundant string edu.stanford.nlp.sempre.overnight.SimpleWorld.

E Finetuning Experiments

The finetuning experiments use large BART and GPT-2 models trained with causal language-model loss and report additional GPT-2 XL results for Overnight and Break.

BART-large has 406 million parameters, while GPT2-XL has 1.5 billion parameters in the finetuning experiments.
Each model is trained with causal LM loss for 20,000 steps, including a 1,000-step linear learning-rate warmup.
Hyperparameters are selected by grid-searching maximum learning rates of 10^-5 and 10^-6 with decay intervals t ∈ {2, 4, 6, 8}.
Additional finetuned GPT-2 XL results are reported for Overnight and Break in Tables 5 and 6.
GPT-3 experiments use Microsoft Azure’s OpenAI API, while finetuning runs use NVIDIA DGX-2 machines with Tesla V100 GPUs.

G Further Discussion

Evaluation errors were often driven by annotation subtleties rather than model misinterpretation. On Calendar, equality strictness and annotation mistakes materially affected the reported errors.

On Calendar Overnight, 8 of the best model’s 18 errors were legitimate, while 7 were annotation errors and 3 differed only in equality strictness.The equality distinction is often ambiguous in natural language.
The system predicted “larger than” where the annotation required “at least,” illustrating how ≥ versus > can create apparent errors.The paper expects low interannotator agreement on this subtle distinction.
Calendar denotation accuracy @K saturated at 0.98 when K ≥5, with the two remaining errors caused by annotation mistakes on correctly interpreted utterances.