PaperlayerView exact revision on arXiv

Source-linked AI summary

MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark

Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, Yashar Mehdad

arXiv:2008.09335v2cs.CLcs.LG

TL;DR

Multilingual task-oriented semantic parsing lacks broad datasets and compositional annotations, making expansion to new languages expensive. The paper introduces MTOP and benchmarks multilingual pre-trained models, achieving improved existing-dataset performance and strong zero-shot transfer. Translation-based construction nevertheless can introduce errors and tokenization-related limitations.

  • Problem

    Existing multilingual task-oriented semantic parsing datasets cover few languages and domains, contain limited labeled data, and omit complex nested queries.

  • Method

    The paper releases MTOP across 6 languages and 11 domains and benchmarks multilingual pre-trained models on flat and compositional representations, including zero-shot transfer with translation and alignment.

  • Results

    67.2% exact match accuracy is achieved for zero-shot cross-lingual transfer averaged across 5 languages, compared with 77.7% for the best in-language models.

  • Takeaways & Limitations

    MTOP and the proposed methods support multilingual task-oriented semantic parsing across more languages and complex nested queries.

  • Takeaways & Limitations

    Translation-based dataset construction can introduce errors, including phrases uncommon in spoken language and tokenization-related performance drops.

Abstract

from arXiv · show

Scaling semantic parsing models for task-oriented dialog systems to new languages is often expensive and time-consuming due to the lack of available datasets. Available datasets suffer from several shortcomings: a) they contain few languages b) they contain small amounts of labeled examples per language c) they are based on the simple intent and slot detection paradigm for non-compositional queries. In this paper, we present a new multilingual dataset, called MTOP, comprising of 100k annotated utterances in 6 languages across 11 domains. We use this dataset and other publicly available datasets to conduct a comprehensive benchmarking study on using various state-of-the-art multilingual pre-trained models for task-oriented semantic parsing. We achieve an average improvement of +6.3 points on Slot F1 for the two existing multilingual datasets, over best results reported in their experiments. Furthermore, we demonstrate strong zero-shot performance using pre-trained models combined with automatic translation and alignment, and a proposed distant supervision method to reduce the noise in slot label projection.

1 Introduction

MTOP addresses limited multilingual resources for complex task-oriented semantic parsing by releasing a multilingual dataset and benchmarking cross-lingual models. The study reports strong zero-shot transfer and improved results on existing multilingual datasets.

  • Research gap: Existing multilingual datasets cover few languages and domains, provide limited labeled data, and generally omit complex nested queries.These limitations hinder systematic multilingual experimentation and evaluation.
  • Contributions: MTOP is an almost-parallel multilingual dataset covering 6 languages and 11 domains with compositional representations for complex nested queries.The paper presents it as the first multilingual dataset with such compositional representations, to the authors’ knowledge.
  • Contributions: The paper benchmarks state-of-the-art multilingual pre-trained models on both flat and compositional representations.The benchmarks cover the released MTOP dataset and existing multilingual task-oriented semantic parsing datasets.
  • Results: 67.2% exact match accuracy is achieved for zero-shot cross-lingual transfer averaged across 5 languages, versus 77.7% for the best in-language models.The zero-shot approach combines automatic translation and alignment with proposed distant supervision.

2 Related Work

Prior task-oriented semantic parsing largely uses intent classification and slot tagging for simple queries, while hierarchical representations address compositional queries. Related multilingual work includes cross-lingual pre-trained representations and a small set of multilingual benchmarks.

  • Task-oriented semantic parsing: Intent detection and slot filling frame task-oriented semantic parsing as text classification and sequence labeling for simple non-compositional queries.Hierarchical representations are used for more complex queries with recursive slots.
  • Pre-trained cross-lingual representation: Cross-lingual representation research progressed from embedding alignment to contextual and Transformer-based multilingual pre-training.The related work discusses encoder-only multilingual representations and later multilingual sequence-to-sequence pre-training.
  • Cross-lingual task-oriented semantic parsing: Existing multilingual task-oriented benchmarks contain simple non-compositional utterances suited to intent and slot detection.One cited benchmark translated ATIS utterances into Turkish and Hindi using 600 training examples.
  • Cross-lingual task-oriented semantic parsing: Table 1 summarizes MTOP dataset statistics and roughly divides the data into 70:10:20 train, evaluation, and test splits.

3 Data

MTOP is constructed as an almost-parallel multilingual corpus by translating and aligning annotated English utterances across languages. It provides flat and compositional decoupled representations, with quality control and acknowledged translation-related limitations.

  • Dataset creation: MTOP contains 100k examples across 6 languages and addresses the scarcity of multilingual data and compositional annotations.
  • Dataset creation: The dataset pipeline generates and annotates English utterances, then translates, transfers labels, post-processes, post-edits, and filters five target languages.
  • Dataset creation: Professional translators receive English slot spans and guidelines requiring each translated slot span to match its occurrence in the translated utterance.This supports slot-span alignment across languages.
  • Dataset creation: English tree structure and translated slot spans are used to reconstruct target-language annotations.The decoupled representation enables this reconstruction.
  • Quality control: 83% of translated data was marked good quality during review, and low-quality annotations were removed.Quality control included translator review and a separate lightweight annotation review.
  • Limitations: Translation-based construction can leave errors such as phrases that are uncommon in spoken language despite rigorous quality control.Each target language used three translators, and preparation took 15 to 25 days per language.
  • Data format: MTOP releases flat representations alongside compositional decoupled representations for utterances containing nested slots and intents.Flat representations retain top-level slots for comparability with existing benchmarks, while decoupled representations remove non-leaf-slot text from TOP trees.

4 Model Architecture

The paper uses separate architectures for flat and compositional semantic representations, combining multilingual pretrained encoders with sequence-to-sequence decoding for hierarchical queries. Its pointer-generator model mixes ontology generation with source-token copying.

  • Flat representation: Flat representations are modeled as intent classification and slot tagging, using an XLM-R Large encoder in place of BERT.The baseline uses a bidirectional LSTM with pretrained XLU embeddings, while the reported multilingual architecture replaces the BERT encoder with XLM-R.
  • Hierarchical representation: Compositional representations are generated with a sequence-to-sequence pointer-generator whose source is the utterance and target is the compositional decoupled representation.At each decoding step, the model can generate an ontology element or copy a token from the source sequence.
  • Hierarchical representation: The final decoder distribution mixes generation and copy distributions using a learned copying weight.The copying weight is computed from the attended vector, while the generation and copy distributions are described separately.
  • Model variants: The experiments compare a standard LSTM encoder-decoder with transformer-based multilingual pretrained models, including encoder-only and sequence-to-sequence models.The model set includes XLM-R with a randomly initialized transformer decoder, mBART, CRISS, and MARGE.

5 Experiments

The experiments evaluate multilingual task-oriented semantic parsing across existing and newly described datasets under in-language, multilingual, and zero-shot settings. Translation, alignment, multitask sampling, and distant supervision are used to support zero-shot transfer and reduce projection noise.

  • Evaluation settings: The evaluation covers Multilingual ATIS and Multilingual TOP in addition to the new dataset, whose hierarchical representations differ from the existing datasets’ flat representations.The paper notes that both existing datasets contain only flat representations, whereas its dataset contains hierarchical representations.
  • Evaluation settings: Three evaluation settings use target-language data only, all available languages jointly, or English data only, respectively.These are labeled in-language models, multilingual models, and zero-shot target language models.
  • Zero-shot transfer: Translate-train augments target-language data through machine translation and alignment to project slot annotations.The experiments use an in-house translation system and report fastalign because it produced better task performance than attention-based alignment.
  • Multilingual training: Multilingual training samples one language per batch according to a sampling ratio, allowing languages with fewer training examples to be upsampled.This multitask setting performed better than mixed-language batches in the experiments.
  • Zero-shot transfer: A second distant-supervision approach concatenates English utterances with their target-language translations and randomly masks English slot text to address noisy slot-label projection.The method is evaluated in the zero-shot setting for flat-representation models.

6 Results and Discussions

MTOP experiments show that multilingual pre-trained models perform strongly across flat and compositional semantic parsing, including zero-shot transfer and existing multilingual benchmarks. Translation-and-alignment methods consistently help zero-shot transfer, while distant supervision is effective selectively and can degrade performance under tokenization mismatch.

  • Flat representation: XLM-R models significantly outperform BiLSTM models using XLU, while multilingual training outperforms in-language models on MTOP flat representations.The gains from multilingual training are particularly large for Hindi and Thai with XLM-R.
  • Zero-shot transfer: 67.2% average exact match accuracy is achieved in zero-shot transfer across five languages using only English data, compared with 77.7% for the best in-language models.The zero-shot setup combines automatic translation and alignment with distant supervision.
  • Zero-shot transfer: Translate and align consistently improves over the XLM-R baseline across all languages, outperforming distant supervision except for German and Hindi.Distant supervision considerably improves French, German, and Hindi, but slightly drops Spanish and significantly degrades Thai.
  • Compositional representation: Multilingual pre-trained models significantly outperform the baseline for compositional decoupled representations, with XLM-R and CRISS performing best for every language after multilingual training.mBART improves substantially after translation-task fine-tuning, while XLM-R performs best on Thai because its pre-training includes Thai.
  • Existing benchmarks: XLM-R models set new state-of-the-art results on Multilingual ATIS and Multilingual TOP, with multilingual training further improving performance over reported baselines.On Multilingual ATIS, masked distant supervision and translated aligned data bring zero-shot results close to target-language training for Hindi and Turkish.
  • Existing benchmarks: Translation-based techniques degrade Multilingual TOP performance for Thai because tokenization differences create errors in slot-text boundaries.Thai is especially sensitive because it rarely uses whitespace; direct transfer is effective for Spanish.

7 Conclusion

The paper releases MTOP, a multilingual task-oriented semantic parsing dataset covering six languages and both flat and compositional representations. It also develops comprehensive multilingual pre-trained model benchmarks for zero-shot and target-language settings.

  • Dataset: MTOP covers six languages and includes both flat and compositional representations for multilingual task-oriented semantic parsing.The dataset is presented as a resource for scaling task-oriented dialog systems to more languages.
  • Benchmarks: The paper develops comprehensive benchmarks using state-of-the-art multilingual pre-trained models in zero-shot and target-language settings.The benchmarks cover both flat and compositional representations.

A Training Details

The paper specifies separate training configurations for XLM-R, XLU, and sequence-to-sequence models, including optimizer, batch-size, epoch, learning-rate, and architecture choices.

  • XLM-R: XLM-R models use Adam, batch size 16, 20 epochs, and dev-set learning-rate search over {1, 2, 3}e−5.The optimizer settings are β1 = 0.9, β2 = 0.99, and ϵ = 1e−6; runs use a single 32GB V100 GPU.
  • XLU: XLU flat models use 300-dimensional embeddings with a 2-layer, 200-dimensional BiLSTM and CRF slot decoder.The intent head uses attention pooling with dimension 128 and a 200-dimensional projection; training uses batch size 32 for 40 epochs.
  • Sequence-to-sequence models: Sequence-to-sequence models use stochastic weight averaging, Lamb optimization, exponential learning-rate decay, and 50 training epochs with dev-set learning-rate search.Marge uses batch size 4 because larger batches did not fit into 32GB memory; other pretrained models use batch size 16.
  • Sequence-to-sequence models: The sequence-to-sequence XLM-R model uses a randomly initialized 3-layer transformer decoder with hidden size 1024 and 8 attention heads.A larger decoder could not fit in GPU memory because the XLM-R encoder has 24 layers, compared with 12-layer mBART, CRISS, and MARGE encoders.
  • XLU: XLU sequence-to-sequence models use 2-layer BiLSTM and LSTM components with 256-dimensional hidden states and a single attention head.Learning rate and LSTM dropout are selected through a development-set parameter sweep.
  • Other benchmarks: Other benchmark results reuse the Table 2 settings, except multilingual ATIS uses a fixed checkpoint because it lacks a development set.The supplied passage does not specify the fixed checkpoint value.

B More Results

The paper reports additional flat-representation results using intent accuracy and Slot F1, presented in Table 5 for the models evaluated in Table 2.

  • Additional metrics: Table 5 reports intent accuracy and Slot F1 for flat-representation models.These are additional metrics for the experiments.
  • Table coverage: Table 5 covers the models listed in Table 2.The supplied table material provides the caption but no metric values or model-by-model outcomes.
Loading 2008.09335v2…