PaperlayerView exact revision on arXiv

Source-linked AI summary

Many Languages, One Parser

Waleed Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, Noah A. Smith

arXiv:1602.01595v4cs.CL

TL;DR

Dependency parsing commonly uses separate models for each language, limiting shared supervision and complicating multilingual deployment. MALOPA trains one multilingual parser with shared lexical representations and language-specific information, and it compares favorably with strong baselines across large-, small-, and no-treebank settings.

  • Problem

    Separate language-specific parsers miss supervision shared across typologically related languages and require separate model development and maintenance for each language.

  • Method

    MALOPA trains one parser on multilingual universal dependency annotations, combining multilingual word representations with token-level language information and language-specific POS features.

  • Results

    MALOPA compares favorably with strong baselines when target languages have large, small, or no treebanks, including a 5.7 absolute LAS-point average gain over a strong monolingual baseline with 3,000 annotated tokens.

  • Takeaways & Limitations

    Parameter sharing allows one parser to retain strong performance for languages with treebanks and transfer supervision effectively when target-language annotations are limited or absent.

  • Takeaways & Limitations

    The parser’s code-switching capability was not evaluated because appropriate data was unavailable.

Abstract

from arXiv · show

We train one multilingual model for dependency parsing and use it to parse sentences in several languages. The parsing model uses (i) multilingual word clusters and embeddings; (ii) token-level language information; and (iii) language-specific features (fine-grained POS tags). This input representation enables the parser not only to parse effectively in multiple languages, but also to generalize across languages based on linguistic universals and typological similarities, making it more effective to learn from limited annotations. Our parser's performance compares favorably to strong baselines in a range of data scenarios, including when the target language has a large treebank, a small treebank, or no treebank for training.

1 Introduction

Multilingual dependency annotations create an opportunity for one parser to address the theoretical and practical limitations of separate language-specific models. MALOPA is designed to perform well across both high- and low-resource settings.

  • Motivation: Separate language-specific models miss supervision shared by typologically related languages and require repeated configuration, training, tuning, monitoring, and updating.The mainstream approach also complicates deployment and distribution across many languages.
  • Motivation: Homogeneous dependency annotations across languages enable a parser that can process multiple languages and potentially replace an array of monolingual parsers.The opportunity is tied to shared annotation resources across languages.
  • Scope: Code-switching is pervasive in some genres, especially social media, but the parser’s code-switching capability was not evaluated because appropriate data was unavailable.The parser can be used on code-switched input, but this capability remains unevaluated.
  • Prior work: Prior low-resource approaches use indirect supervision from auxiliary languages, but may sacrifice accuracy on source languages with large treebanks.The paper targets a model that works in both low-resource and high-resource scenarios.
  • Contribution: MALOPA uses one multilingual parser with universal dependency annotations, multilingual representations, language information, and fine-grained POS features.It compares favorably with strong baselines when the target language has a large, small, or no treebank.

2 Overview

MALOPA trains one parsing model on the union of multilingual treebanks and adapts its behavior using language-specific representations. It is intended to retain strong performance in high-resource settings while transferring supervision in low-resource settings.

  • Architecture: MALOPA trains one parsing model on the union of source-language treebanks and applies it to target languages, including disjoint source and target sets.This replaces routing each sentence to a separately trained language-specific parser.
  • Architecture: The architecture balances language-invariant transfer with language-specific behavior through coarse POS tags, multilingual embeddings and clusters, and language-specific representations.The design supports sharing parameters while allowing different behavior across languages.
  • Resources: The parser uses universal POS annotations, bilingual dictionaries, typology information, language-specific POS annotations, and monolingual corpora as language resources.These resources are specified for each language in the combined target and source-language set.
  • Evaluation: 5.7 absolute LAS points per language on average separates MALOPA from a strong monolingual baseline with 3,000 token annotations in the small-treebank setup.The reported average is per language in the small-treebank scenario.

3 Parsing Model

The parser extends a stack-LSTM transition system with multilingual and language-specific token representations, while conditioning decisions on the input language. It jointly supports multilingual transfer, POS prediction, and robustness to predicted-tag errors.

  • Transition-based parsing: The core parser manipulates a buffer, stack, and action list using arc-standard transitions to build dependency trees.At each timestep, an action updates these structures and their vector representations; the parser greedily selects actions until a complete tree is produced.
  • Transition-based parsing: Parser states concatenate stack, buffer, and action representations before defining a distribution over possible next actions.The state is computed from [s_t; b_t; a_t] using learned parameters, then used to score categorical action choices.
  • Token representations: Multilingual parsing begins with shared coarse-POS embeddings and adds multilingual word clusters, pretrained word embeddings, language embeddings, and fine-grained POS embeddings.The representation balances language-invariant transfer features with language-specific information that can adjust parser behavior.
  • Language embeddings: Language information is injected into parser representations because typologically different languages can favor conflicting transitions in identical local configurations.The model considers language ID and word-order properties; language ID performs best in the reported experiments.
  • Predicting POS tags: The model jointly predicts POS tags and dependency parses when gold POS tags are unavailable.Tagging uses pretrained word, Brown-cluster, and language embeddings with a bidirectional LSTM and softmax, while shared embeddings connect tagging and parsing.
  • Predicting POS tags: Block dropout randomly zeros POS-tag embeddings during training to make parsing more robust to incorrect predicted POS tags.The modification targets the parser’s dependence on potentially erroneous POS predictions.

4 Experiments

The experiments evaluate MALOPA across fully supervised, small-treebank, and no-treebank scenarios, including predicted language and POS information. MALOPA benefits from multilingual lexical and language features, outperforming strong baselines in several low-resource settings while using one shared parser.

  • Experimental settings: MALOPA is evaluated when target languages have large, small, or no treebanks, using standard multilingual dependency-treebank splits.Experiments include gold and predicted POS tags in the large-treebank setting.
  • Target languages with a treebank: The coarse-POS-only parser trails monolingual baselines by 12.5 LAS points on average.This establishes the performance gap before adding lexical and language information.
  • Target languages with a treebank: Multilingual lexical embeddings recover 83% of MALOPA’s average performance gap, while language-information variants achieve LAS scores of 83.5, 83.2, and 82.5.The language-information variants are language ID, word order, and full typology, respectively.
  • Target languages with a treebank: The best MALOPA configuration beats monolingual baselines in five of seven target languages and wins by 0.3 LAS points on average.It adds fine-grained POS embeddings and uses one model trained once across all languages.
  • Low-resource scenarios: In the small-treebank setting, MALOPA exceeds Duong et al. by 1.4 LAS points and monolingual baselines by 5.7 absolute LAS points on average.The target treebank contains approximately 3,000 tokens, while English provides approximately 205,000 source tokens.
  • Low-resource scenarios: Without target-language treebanks, MALOPA beats both baselines by more than 1 LAS point on average and achieves the best result in four of six languages.This evaluates multi-source transfer with disjoint source and target language sets.

5 Related Work

The paper builds on cross-lingual model transfer and multilingual parsing, while also relating its approach to annotation projection and neural parsing models.

  • Earlier work transferred parsers from source-language treebanks to target languages, extending this to unlexicalized and typology-informed multilingual models.
  • Neural network parsing models, including Chen and Manning’s transition-based parser, precede the paper’s neural parsing approach.
  • Cross-lingual parsing has used bilingual lexicons to define lexico-syntactic features and multilingual representations to support transfer.
  • Annotation projection transfers source-language parses through parallel corpora or automatically translated sentences, sometimes combining projected annotations with unlabeled target-language data.

6 Conclusion

The conclusion presents MALOPA as a single multilingual dependency parser that shares parameters while adapting to individual languages. It reports benefits across high-resource, small-treebank, and no-treebank settings.

  • MALOPA is a single parser trained on a multilingual set of treebanks.
  • For target languages with treebanks, MALOPA equipped with language and fine-grained POS embeddings on average outperforms monolingually trained parsers.
  • Sharing is more pronounced when the target language’s training treebank is small or nonexistent, where MALOPA outperforms previous cross-lingual multi-source transfer methods.
Loading 1602.01595v4…