PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero and Few-shot Semantic Parsing with Ambiguous Inputs

Elias Stengel-Eskin, Kyle Rawlins, Benjamin Van Durme

arXiv:2306.00824v2cs.CL

TL;DR

Semantic parsing often assumes a single formal meaning for an utterance, despite natural-language ambiguity and the potential consequences of parser miscommunication. The paper introduces AmP, a templated benchmark and metrics for evaluating in-context models on ambiguity. Models struggle to compose meanings without explicit guidance but can reflect ambiguity present in mixed prompts.

  • Problem

    Semantic parsing datasets commonly assign a single interpretation to each utterance, despite natural language having multiple possible meanings and parsers being used for real-world actions.

    Source: §1 · pp. 1–2

  • Method

    The authors create AmP from templates for five ambiguity types and use three metrics to assess in-context language models in zero-shot and mixed-prompt settings.

    Source: §1 · p. 2

  • Results

    Models struggle to compose meanings without explicit guidance but are sensitive to ambiguity in mixed prompts, suggesting ambiguity in training data may be sufficient for capturing it in outputs.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The findings motivate explicitly capturing ambiguity during annotation and evaluating the distribution of possible meanings.

    Source: §6 · p. 9

    The experiments are limited to a fixed set of English-only ambiguities.

    Source: §6 · p. 9

Abstract

from arXiv · show

Despite the frequent challenges posed by ambiguity when representing meaning via natural language, it is often ignored or deliberately removed in tasks mapping language to formally-designed representations, which generally assume a one-to-one mapping between linguistic and formal representations. We attempt to address this shortcoming by introducing AmP, a framework, dataset, and challenge for translating ambiguous natural language to formal representations like logic and code. We define templates and generate data for five well-documented linguistic ambiguities. Using AmP, we investigate how several few-shot text-to-code systems handle ambiguity, introducing three new metrics. We find that large pre-trained models perform poorly at capturing the distribution of possible meanings without deliberate instruction. However, models are able to capture the distribution well when ambiguity is attested in their inputs. These results motivate a call for including ambiguity explicitly in datasets and promote considering the distribution of possible outputs when evaluating systems. Data and code: https://github.com/esteng/ambiguous_parsing

1 INTRODUCTION

Natural-language ambiguity complicates semantic parsing because people use commonsense and conversational context that models may lack. The paper introduces AmP and evaluates whether in-context models represent multiple possible parses, both without and with conflicting prompt examples.

1 INTRODUCTION

  • Semantic parsers may lack the commonsense knowledge and conversational context people use to resolve ambiguity, risking miscommunication in systems that take real-world actions.Examples include modifying calendars, sending emails, and controlling robots.Source: §1 · p. 1
  • AmP uses templates for five documented ambiguity types to generate ambiguous and unambiguous utterances, pairing each ambiguous utterance with two logical forms in FOL or Lisp.The ambiguity types include PP attachment, scope and inverse scope, pronominal coreference, and conjunctions.Source: §1 · p. 2
  • The study focuses on semantic parses rather than grounding ambiguous utterances in question answers, inferences, videos, or images.The authors cite semantic parsing’s use in interactive systems and tool use as motivations for this choice.Source: §1 · p. 2
  • The paper introduces two in-context-learning tasks and three metrics to assess how well large language models capture ambiguity in zero-shot and mixed-prompt settings.The tasks use generated AmP data to study ambiguity and provide a framework for improving prediction of multiple meanings.Source: §1 · p. 2
  • In zero-shot tests, models struggle to compose parses and generally fail to predict both possible readings, even when their preferred readings resemble people’s.Some models correctly predict each reading on different examples, but generally do not capture both readings.Source: §1 · p. 2
  • With conflicting prompt examples, some models align their output distributions with the prompt’s interpretation distribution across ambiguity types.The mixed-prompt setting examines how distributions and outputs change as the number of examples for each interpretation varies.Source: §1 · pp. 2–3

2 METHODS

The methods construct canonicalized logical forms for five ambiguity types, evaluate pretrained language models using in-context learning, and define metrics for ambiguity coverage and prompt-distribution matching.

2 METHODS

  • AmP generates FOL examples for five syntactic and semantic ambiguity types, canonicalizing logically equivalent formulas as alphabetically ordered binary trees with anonymized variables.The logical forms can be rendered in standard FOL or Lisp format.Source: §2 · p. 3
  • The model set includes Codegen models from 350M to 16B parameters, Llama-13B, instruction-tuned Vicuna-13B, and zero-shot-only gpt-3.5-turbo.Because the gpt-3.5-turbo API does not provide logits, it is used only in accuracy-based zero-shot experiments; non-API models use constrained decoding to produce valid logical statements.Source: §2 · pp. 3–4
  • For each candidate logical form, the method scores its least-probable token under forced decoding, then renormalizes the two scores into probabilities.This sequence-confidence estimate avoids the very low scores obtained by multiplying token probabilities.Source: §2 · p. 4

2.1 METRICS

  • The zero-shot metric counts examples with both logical forms in the top-k outputs, while the few-shot metrics assess whether predictions and probabilities match the prompt’s interpretation ratio.The dataset-level metric compares predicted proportions with prompt ratios; the instance-level metric measures probability error against those ratios, with lower scores better for both few-shot metrics.Source: §2.1 · p. 4

3 EXPERIMENT 1: ZERO-SHOT PARSING

The zero-shot challenge tests whether models can compose provided ingredients into both derivations without ambiguity-specific examples. Models generally fail to recover both readings, and human validation reveals item-level variation that Codegen-16B often misses.

3 EXPERIMENT 1: ZERO-SHOT PARSING

  • Prompts provide isolated ingredients but no examples of the tested ambiguity, requiring models to compose novel derivations; each ambiguity type has 200 test examples.The challenge also requires recognizing that ingredients can be combined into two different derivations.Source: §3 · p. 5

3.1 ZERO-SHOT RESULTS AND ANALYSIS

  • GPT-3.5 predicts LF1 for PP attachment and scope well and predicts both LFs for coreference and conjunction, while instruction-tuned Vicuna-13B matches Codegen-16B but not ambiguity capture.Llama-13B predicts neither LF correctly; on scope, Codegen-2B predicts LF1 correctly 18% with FOL versus 11% with Lisp.Source: §3.1 · p. 5
  • For scope, inverse scope, and conjunction, some divergence tokens receive lower confidence, but PP attachment and coreference lack interpretable confidence changes.The analysis uses Codegen-16B's first correctly predicted LF and its alternative parse.Source: §3.1 · pp. 5–6
  • Most models rarely predict both LFs correctly, and most ZM5 scores are 0.0 in most settings.Top-five outputs tend to repeat variations of the same LF.Source: §3.1 · p. 5, Table 2 · p. 6

3.2 HUMAN VALIDATION

  • Human annotators disagree on 38 examples and unanimously agree on 42; their preferences favor LF0 for bound and inverse scope and LF1 for scope, while PP attachment varies by item.Scope preferences align with some prior findings.Source: §3.2 · p. 6
  • Codegen-16B generally matches human probability direction except for PP attachment, but often assigns the same extreme probability across PP, scope, and inverse-scope examples.Bound-pronoun predictions vary more, but remain fairly extreme, suggesting poor item-level calibration for ambiguity.Source: §3.2 · pp. 6–7
  • Conjunction ambiguities were excluded from human validation because their interpretations were difficult to verbalize fluently.The validation subset otherwise sampled 20 examples per ambiguity type except conjunctions, with three annotators per example.Source: §3.2 · p. 6

4 EXPERIMENT 2: FEW-SHOT PARSING

The few-shot experiment varies the proportion of each reading in mixed prompts to test whether models track that distribution. Several models track prompt ratios fairly well, especially for scope ambiguities, though forced-decode confidence does not guarantee accurate generated parses.

4 EXPERIMENT 2: FEW-SHOT PARSING

  • The mixed prompts represent legitimate alternative interpretations rather than noisy or mislabeled examples.The experiment examines how model confidence and accuracy change with prompt ratios.Source: §4 · p. 7
  • Mixed prompts pair same-type sentences with LF0 or LF1, varying LF0 examples from 0 to 10 of 10 and shuffling their order.Each ratio uses 100 examples per ambiguity type, with a separate prompt for each test sentence.Source: §4 · p. 7
  • All FIM scores are relatively low, but low FIM does not imply low FDM because FIM uses forced gold-LF probabilities rather than the model's most probable generations.Llama-13B never produces a correct LF with constrained beam search, yet its forced-decode probabilities align fairly well with the prompt.Source: §4 · pp. 7–8
  • Prompt evidence can override zero-shot scope and inverse-scope preferences, with models especially close to the prompt distribution for those ambiguities.The authors suggest mixed prompts may make ambiguity less challenging when training data contains such examples.Source: §4 · p. 8
  • Several models achieve fairly low FDM, especially on scope and inverse scope, and some model predictions track the LF proportions in mixed prompts.Low FDM requires both matching the prompt's LF proportions and predicting the LFs accurately.Source: §4 · p. 7, Table 3 · p. 8

5 RELATED WORK

Ambiguity has long been studied across linguistics and NLP, including work on visual interpretation and semantic parsing. This paper uses synthetic data and logical forms to examine more ambiguity types with greater control.

  • Ambiguity has long been studied in linguistics and psycholinguistics as a feature arising from competing communication pressures.A systematic account of linguistic ambiguity lies beyond this section's scope.Source: §5 · p. 8
  • NLP work has modeled ambiguity using visual contexts, and related research has examined ambiguity in question-answering, inference, and coreference.Some prior work also studies parsing ambiguity using λ-calculus representations.Source: §5 · p. 8
  • Unlike video- or image-based ambiguity work, this paper represents meanings with logical forms and uses synthetic data to control and examine more ambiguity types.The authors cite easier correctness checking for logical formulas than images as one motivation.Source: §5 · p. 8

6 DISCUSSION AND CONCLUSION

Models struggle to compose symbols without explicit guidance but are sensitive to ambiguity when it appears in their inputs. The authors argue for ambiguity-aware annotation while noting limits in representations, language coverage, and comparisons with human processing.

  • Treating each input as having one correct logical form overlooks natural-language ambiguity and can reduce semantic parsing to syntactic parsing.The discussion notes that text-to-code also permits many logically equivalent programs.Source: §6 · pp. 8–9
  • When ambiguity is attested in inputs, many models capture the distribution of logical forms, supporting redundant, ambiguity-aware annotation for semantic parsing.The authors call for attention to valid disagreement and multiple judgments, at least for evaluation data.Source: §6 · p. 9
  • The authors use fully specified representations, placing the burden of resolving underspecification on the parsing model rather than the representation.They contrast this choice with representations that preserve scope or noun-phrase bracketing ambiguity.Source: §6 · p. 9
  • The results are bounded by a fixed logical-form choice and do not reflect human processing with audio or conversational context.The authors caution against taking the model results as evidence about how people process language.Source: §6 · p. 9
  • The experiments cover a fixed set of English-only ambiguities and do not optimize parsing for ambiguity-specific decoding.The authors identify broader language and ambiguity coverage as possible extensions and mention diversity-oriented decoding as a possible optimization.Source: §6 · p. 9
  • Models struggle to compose symbols without explicit guidance but are sensitive to ambiguity in mixed prompts, motivating annotation that captures ambiguity.The conclusion suggests ambiguity in training data may be sufficient for capturing it in outputs.Source: §6 · p. 9

7 ETHICS STATEMENT

The authors describe safeguards for annotators and note the work's potential for dual use.

  • Templatically generated sentences ensure that the dataset contains no harmful texts.Source: §7 · p. 10
  • The design protects annotators from exposure to harmful inputs, and annotators receive pay substantially above the US federal minimum wage and in line with living-wage estimates.Source: §7 · p. 10
  • The authors acknowledge dual-use potential but believe the work has limited negative-application potential and may contribute to safer, more reliable technology.Source: §7 · p. 10

8 REPRODUCIBILITY STATEMENT

The authors release the dataset and code, while noting that using a closed-source model may hinder reproducibility.

  • The release includes the dataset and code for AMP, which can be extended to generate new ambiguities.Source: §8 · p. 10
  • The authors use closed-source ChatGPT, which may hinder reproducibility, but most results rely on widely available open-source models.Source: §8 · p. 10
  • Prompts and code for running all experiments are included.Source: §8 · p. 10

A.1 DATASET CONSTRUCTION

The paper represents meaning with quantified first-order logic using neo-Davidsonian event semantics. Its logical forms encode events and semantic roles, with different quantification conventions for generic noun phrases and proper names.

A.1 DATASET CONSTRUCTION

  • Logical forms use quantified first-order logic and a neo-Davidsonian event-semantics approach.Events are variables with event-type predicates applied to them.Source: §A.1 · p. 14
  • Event participants are represented through agent, patient, and instrument predicates.Source: §A.1 · p. 14
  • Generic noun phrases are existentially quantified, while proper names are unquantified and refer to a single referent.Source: §A.1 · p. 14

A.1.1 LEXICAL ITEMS

Lexical choices for ambiguity examples are matched to the ambiguity type. The dataset uses different verb, noun, and argument classes across the types.

A.1.1 LEXICAL ITEMS

  • PP-attachment examples pair visual verbs with visual instruments and tactile verbs with wearable or manipulable objects.Source: §A.1.1 · p. 14
  • Scope and inverse-scope examples use common nouns with visual and tactile verbs.Source: §A.1.1 · p. 14
  • Pronoun-coreference examples use gendered names and nouns, while conjunction examples use intransitive verbs.Source: §A.1.1 · p. 14

A.1.2 EXISTENTIAL QUANTIFICATION

This appendix describes the logical-form conventions, extensible templates, and zero-shot prompts used in the dataset, alongside decoding, error-analysis, and human-annotation procedures. It reports lexical patterns in human preferences and notes limits in comparing those judgments with model outputs.

A.1.2 EXISTENTIAL QUANTIFICATION

  • Logical forms use widest-scope existential quantification for definite NPs, without distinguishing definite from indefinite NPs in AMP.The paper states that this choice has no impact on ambiguity.Source: §A.1.2 · p. 14

A.1.3 EXTENDING TO NEW TEMPLATES

  • New templates require surface-form and logical-form templates plus slot lexicons; the framework enumerates valid lexical combinations into paired inputs and LFs.Source: §A.1.3 · p. 14

A.2 ZERO-SHOT PROMPTS

  • Zero-shot prompts supply compositional ingredients for PP attachment, pronoun coreference, scope, inverse scope, and conjunction ambiguities.Examples include component structures and combinations such as separate verbs for coreference and quantifiers for scope.Source: §A.2 · pp. 14–16

A.3 CONSTRAINED DECODING

  • Locally run models use grammar-constrained decoding with a beam of 5, restricting outputs to syntactically valid formulae.This separates semantic performance from syntactic ability.Source: §A.3 · p. 16

A.4 ZERO-SHOT PARSING: QUALITATIVE ANALYSIS

  • 9/10 PP-attachment errors omitted a patient predicate; 9/10 scope errors used ∃ instead of ∀; 8/10 bound-pronoun errors reused variables; 8/10 conjunction errors used wrong connectives.The remaining sampled errors included incorrect variable use, a missing quantifier, a missing predicate, or bad connective scoping, depending on ambiguity type.Source: §A.4 · pp. 16–17

A.5 ANNOTATION TASK

  • In the main HIT, annotators rated confidence on a three-tick sliding scale; raw scores were min-max normalized, mapping lowest confidence to p(LFc) = 0.5 and highest to 1.0.Source: §A.5 · p. 17
  • Visual verbs and nouns were associated more with the instrument reading, while tactile verbs and nouns yielded a possessive interpretation.Source: §A.5 · p. 17
  • Human preferences came from choices between verbalized LFs, while grammar-constrained models could not produce forms outside AMP's expected syntax.The authors note this gives most tested models an advantage over annotators in this comparison.Source: §A.5 · pp. 17–18
Loading 2306.00824v2…