PaperlayerView exact revision on arXiv

Source-linked AI summary

Extending a Parser to Distant Domains Using a Few Dozen Partially Annotated Examples

Vidur Joshi, Matthew Peters, Mark Hopkins

arXiv:1805.06556v1cs.CL

TL;DR

Parser adaptation is costly when it requires target-domain treebanks, and partial annotations can be misaligned with models that predict full parses. The paper trains a span-focused parser directly from partial annotations and reports improved out-of-domain performance alongside a 94.3% WSJ test-set F1 result.

  • Problem

    Target-domain treebanks are expensive, while partial annotations do not directly match models that predict full parse trees.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The parser predicts constituent spans directly, allowing training from partial or full annotations without changing the training process.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    94.3% single-model F1 on the Wall Street Journal test set raises the previous state of the art from 92.6%, while the parser shows less degradation on Brown than previous parsers.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Contextualized representations and span-aligned annotations support training custom parsers with full or partial annotations using the same process.

    Source: §7 · p. 8

    A dummy-word replacement strategy can confuse parsing when the same expression takes different constituent labels in different contexts.

    Source: §4 · p. 6

Abstract

from arXiv · show

We revisit domain adaptation for parsers in the neural era. First we show that recent advances in word representations greatly diminish the need for domain adaptation when the target domain is syntactically similar to the source domain. As evidence, we train a parser on the Wall Street Jour- nal alone that achieves over 90% F1 on the Brown corpus. For more syntactically dis- tant domains, we provide a simple way to adapt a parser using only dozens of partial annotations. For instance, we increase the percentage of error-free geometry-domain parses in a held-out set from 45% to 73% using approximately five dozen training examples. In the process, we demon- strate a new state-of-the-art single model result on the Wall Street Journal test set of 94.3%. This is an absolute increase of 1.7% over the previous state-of-the-art of 92.6%.

1 Introduction

The paper revisits parser domain adaptation, arguing that contextualized representations reduce the need for adaptation in syntactically similar domains. For distant domains, it proposes training a parser from partial span annotations, alongside reporting a new WSJ single-model F1 result.

1 Introduction

  • The modified span-focused parser trains directly on labeled spans, so partial sentence bracketings require no change to the training regime.Figure 1 illustrates bracketed constituent spans.Source: §1 · p. 1, Figure 1 · p. 1
  • Dozens of partially annotated sentences can extend a newswire-trained parser to new domains without degrading its newswire performance.The authors describe the approach as a rapid extension method using examples like those in Figure 1.Source: §1 · p. 1
  • 94.3% F1 on the Wall Street Journal test set raises the single-model state of the art from 92.6%.A trained model is publicly available.Source: §1 · p. 2
  • Without domain-specific training, the parser shows much less degradation than previous parsers on newswire-adjacent domains such as Brown.The paper attributes reduced data needs to contextualized word representations.Source: §1 · pp. 1–2
  • The paper also provides an MSP version that predicts its own part-of-speech tags rather than requiring a third-party tagger.The introduction presents this as an additional contribution.Source: §1 · p. 2

2 The Reconciled Span Parser (RSP)

The Reconciled Span Parser (RSP) aligns its predictions with partial annotations by classifying spans, then combines span labels into well-formed parses.

2 The Reconciled Span Parser (RSP)

  • RSP trains directly on span classifications, aligning the model’s predictions with partial annotations and allowing training from partial or full annotations as collections of span labels.Unlike the Minimal Span Parser’s maximum-margin objective, this span-classification task supports partial annotations.Source: §2 · p. 2

2.1 Overview

  • RSP replaces the maximum-margin objective, which couples training and decoding, with span classification so it can train on partial annotations.The paper describes the maximum-margin approach as coupling training to its decoding procedure.Source: §2 · p. 2

Classification Model Parameters and Initializations

  • RSP uses contextualized word representations instead of predicted part-of-speech tags as input, while predicting tags for single-word spans.Its word representations include learned combinations of bidirectional language-model states; word dropout does not affect the ELMo component.Source: §2 · p. 2, Classification Model Parameters and… · p. 4

2.1 Overview

  • A parse maps every sentence span to a sequence of non-terminal labels, with the empty sequence marking non-constituents; an ILP selects a highest-scoring parse that forms a well-formed tree.The model scores a parse by combining its span-label probabilities.Source: §2.1 · pp. 2–3

3 Analysis of RSP

RSP raises the single-model WSJ test-set result while retaining strong performance on Brown and improving parsing in more syntactically distant domains with small amounts of annotation. Results span Brown, QuestionBank, and GENIA, with the largest reported distant-domain comparisons showing gains from 50 annotated examples.

3.1 Performance on Newswire

  • 94.3% on WSJTEST raises the single-model state of the art from 92.6%.Source: §3.1 · p. 4

The Brown Corpus

  • 90.3% average F1 on Brown verticals; most portability improvement is attributed to ELMo, while MSP with gold POS remains 3.4% below RSP.Brown has well-formed sentences and similar syntactic phenomena, with vocabulary possibly its main challenge.Source: The Brown Corpus · p. 4, Question Bank and Genia · p. 4

Question Bank and Genia

4 Rapid Parser Extension

The section contrasts a regex-and-replace pipeline for mathematical expressions with directly extending a span parser using a small number of domain-specific partial annotations. The proposed approach marks difficult constituents and can also mark spans that are not constituents.

4 Rapid Parser Extension

  • The earlier pipeline detects mathematical expressions with regular expressions, replaces them with dummy words, parses, then substitutes the expressions back.Source: §4 · pp. 5–6
  • Regex replacement assumes each expression always has the same constituent label, although “2x = 3y” is a verb phrase in one example and a noun phrase in another.Source: §4 · p. 6
  • The regex-and-replace strategy also assumes each expression is a constituent, which can erase a needed constituent such as “angle AB.”Source: §4 · p. 6
  • Right-attaching participial adjectives such as “labeled x” are frequent in geometry, but teaching the parser this construction is out of scope for regex-and-replace.Source: §4 · p. 6
  • Annotators mark domain-specific spans as constituents or non-constituents, and labels are optional when only constituent status matters.The parser can train from these partial bracketings without requiring changes to the training regime.Source: §4 · p. 6

5 Experiments

The experiments evaluate partial-annotation adaptation on geometry and biomedicine/chemistry text. Geometry results improve substantially, and a small targeted addition of examples further improves handling of right-attaching participial adjectives.

5.1 Geometry Questions

  • 87% of 185 GEODEV annotations were correct after fine-tuning (71.9% before), while error-free sentences rose from 45.2% to 72.6%.The authors report no degradation on newswire; GEOTRAIN contained 63 partially annotated sentences.Source: §5.1 · pp. 6–7, Table 6 · p. 7
  • After retraining, geometry errors clustered around math syntax (44%), PP attachment (19%), and right-attaching participial adjectives (19%).The authors note that the latter construction was unusually frequent in geometry but insufficiently marked in training examples.Source: §5.1 · p. 7
  • Adding three sentences with right-attaching participial adjectives raised GEODEV constituent accuracy to 88.6% and error-free sentences to 75.8%.Source: §5.1 · p. 7, Figure 4 · p. 7

5.2 Biomedicine and Chemistry

  • Dozens of partial annotations yielded significant improvements on BIOCHEMDEV in annotated-constituent accuracy and fully annotation-consistent parses.The experiment used 72 training and 62 development sentences.Source: §5.2 · pp. 7–8, Table 7 · p. 7

6 Related Work

Domain adaptation and learning from partial annotation each have a long NLP history. This section situates the paper’s focus on direct adaptation with partially annotated examples and its distinction from prior partial-annotation strategies.

6.1 Domain Adaptation

  • Prior parser domain-adaptation work often used co-training or self-training to augment small domain corpora, or combined models for a target domain.This paper instead focuses on the direct impact of dozens of partially annotated out-of-domain examples with a neural parser; these approaches are described as orthogonal.Source: §6.1 · p. 8

6.2 Learning from Partial Annotation

  • Prior partial-annotation strategies include completing parses before training, completing them during training, or optimizing over forests of permitted parses.Source: §6.2 · p. 8
  • Unlike much prior work, this approach trains a constituency parser from bracketings and uses partial annotations without changing the training algorithm or data.The authors also state that their approach is not tied to PCFG parsing or a specialized training algorithm.Source: §6.2 · p. 8

7 Conclusion

The conclusion argues that contextualized word representations and span-based models make custom parsers practical with few examples. Users can iteratively annotate target-domain errors and retrain to improve performance.

7 Conclusion

  • Contextualized representations help parsers learn new-domain syntax from few examples, while span-based annotation supports training from partial or full data without changing the process.Source: §7 · p. 8
  • The authors frame this as a convenient path for researchers who need a domain parser, reporting that a couple hours of effort can yield significant performance improvements.Source: §7 · pp. 8–9
  • The proposed workflow is iterative: assess target-domain errors, create partial annotations addressing them, retrain, and repeat until satisfied.Source: §7 · p. 9
Loading 1805.06556v1…