PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialog

Sebastian Schuster, Sonal Gupta, Rushin Shah, Mike Lewis

arXiv:1810.13327v2cs.CL

TL;DR

Building task-oriented dialogue models for low-resource languages is difficult because labeled data are costly and multilingual training resources are limited. The paper introduces a multilingual data set and evaluates translation-based, static-embedding, and contextual-encoder transfer methods. Cross-lingual learning improves results with limited target-language data, while the preferred transfer method depends on target-language data availability.

  • Problem

    Large amounts of labeled training data are required for intent and slot models, while multilingual data sets annotated under shared guidelines have been lacking.

  • Method

    The paper releases about 57,000 annotated English, Spanish, and Thai utterances and evaluates translated training data, cross-lingual embeddings, and multilingual contextual encoder representations.

  • Results

    Cross-lingual learning improves results over limited target-language-only training, with translation best in zero-shot settings and joint training better when some target data is available.

  • Takeaways & Limitations

    The optimal transfer method depends on the amount of annotated target-language data available.

  • Takeaways & Limitations

    The study explores only a subset of possible cross-lingual embedding models and largely omits character embeddings.

Abstract

from arXiv · show

One of the first steps in the utterance interpretation pipeline of many task-oriented conversational AI systems is to identify user intents and the corresponding slots. Since data collection for machine learning models for this task is time-consuming, it is desirable to make use of existing data in a high-resource language to train models in low-resource languages. However, development of such models has largely been hindered by the lack of multilingual training data. In this paper, we present a new data set of 57k annotated utterances in English (43k), Spanish (8.6k) and Thai (5k) across the domains weather, alarm, and reminder. We use this data set to evaluate three different cross-lingual transfer methods: (1) translating the training data, (2) using cross-lingual pre-trained embeddings, and (3) a novel method of using a multilingual machine translation encoder as contextual word representations. We find that given several hundred training examples in the the target language, the latter two methods outperform translating the training data. Further, in very low-resource settings, multilingual contextual word representations give better results than using cross-lingual static embeddings. We also compare the cross-lingual methods to using monolingual resources in the form of contextual ELMo representations and find that given just small amounts of target language data, this method outperforms all cross-lingual methods, which highlights the need for more sophisticated cross-lingual methods.

1 Introduction

The paper addresses the difficulty of building intent and slot models for low-resource languages by evaluating cross-lingual transfer with a new multilingual data set.

  • Intent identification and slot detection are early steps in parsing task-oriented user utterances.The task combines sentence classification for intents with sequence labeling for slots.
  • Large labeled-data requirements make conversational AI development resource-intensive for many languages.The paper notes this is not feasible for more than 6,500 spoken languages.
  • The work introduces a data set with English high-resource utterances and smaller Spanish and Thai low-resource sets sharing annotation guidelines.This addresses the lack of multilingual data for systematic cross-lingual transfer investigation.
  • The authors evaluate translated training data, cross-lingual pre-trained embeddings, and multilingual machine-translation encoder representations.They also compare these transfer methods with models trained only on target-language data.
  • Joint training improves target-language results, while the best transfer strategy depends on target-language data availability.Translation performs best in zero-shot settings; joint high-resource and target-language training is better when some target data exists.

2 Data

The data set combines English, Spanish, and Thai utterances across three task-oriented domains, using shared annotation procedures with language-specific adjudication differences.

  • The English data contain around 43,000 utterances from the alarm, reminder, and weather domains.Native English speakers generated domain-specific utterances before annotation.
  • Two annotators labeled each utterance’s intent and slot spans, with a third adjudicator resolving English disagreements.The adjudicator corresponded with the authors of the annotation guidelines.
  • Native Spanish and Thai speakers translated samples of English utterances, which were then annotated by two annotators.Spanish disagreements received bilingual adjudication, whereas Thai disagreements were discarded.
  • The data set is presented as a parallel word-tagging resource annotated under the same guidelines across multiple languages.The authors describe it as, to their knowledge, the first such publicly relevant data set for this task.
  • Spanish and Thai use more even train-development-test splits to retain larger development and test sets for model selection and evaluation.The English training proportion is much higher than those of Spanish and Thai.

3 NLU models

The NLU model jointly predicts intents and slots after first identifying the utterance domain, using a shared biLSTM representation with task-specific output layers.

  • The model first classifies the utterance domain, then jointly predicts intent and slots with a domain-specific model.The domains are alarm, reminder, and weather.
  • Word embeddings pass through a biLSTM shared by slot detection and intent prediction.This shared architecture supports both tasks from a common sequence representation.
  • Intent classification applies self-attention over biLSTM hidden states before a softmax projection.Slot detection uses each word’s forward-backward hidden-state concatenation before softmax labeling and subsequent processing.
  • Experiments vary token embeddings using zero embeddings, cross-lingual XLU embeddings, or encoder embeddings.XLU and encoder representations are concatenated with trainable zero embeddings.

4 Encoder models

The encoder experiments compare multilingual and monolingual sequence encoders for contextual word representations, using machine-translation objectives and validation perplexity.

  • All contextual encoders use a fixed two-layer bidirectional LSTM to generate contextual word embeddings.The experiments compare three encoder-training strategies to study multilingual embeddings.
  • CoVe trains a neural machine-translation model from the low-resource language to English.The encoder is trained for Spanish or Thai inputs.
  • Multilingual CoVe trains translation in both directions and uses decoder start tokens to encode the translation direction.Because the encoder lacks target-language access, it is expected to map similar meanings across languages into similar spaces.
  • Multilingual CoVe with an autoencoder adds reconstruction objectives alongside bidirectional translation.For each language pair, decoder start tokens select translation or same-language reproduction.
  • Table 2 reports validation perplexities for encoder models across Spanish-English and Thai-English pairs.A hyphen indicates that an encoder was not trained for the corresponding language pair; translation perplexities are generally similar across unidirectional and bidirectional MT systems.
  • Spanish experiments additionally evaluate monolingual ELMo embeddings, while ELMo and CoVe encoders are not multilingual.The ELMo resource is available for Spanish.

5 Cross-lingual learning

The study compares target-only, translated-data, and cross-lingual training strategies for Spanish and Thai intent and slot detection. Cross-lingual training generally improves target-language performance, while embedding choice has limited impact.

  • Experimental setup: Cross-lingual experiments compare target-only training, translated English data, and models trained jointly on English and target-language data.The models use different embedding types, including XLU, ELMo, and multilingual CoVe representations.
  • Results: Cross-lingual training improves results over training only on target-language data across the evaluated languages and metrics.The improvement is reported consistently for Spanish and Thai, although the magnitude varies by task and representation.
  • Results: Translation works well for Spanish domain and intent classification but performs worse for slot detection, presumably because slot annotations are projected noisily.For Spanish slot detection, ELMo gives the best results among the compared representations in exact match and slot F1.
  • Results: Thai shows the same broad pattern: translation performs worst for slot detection, while cross-lingual training consistently improves over Thai-only training.Multilingual CoVe encoders outperform the monolingual encoder in the reported Thai comparison.
  • Results: English still outperforms the other languages, but cross-lingual models substantially reduce the performance gap.The apparent advantage of Thai over Spanish may reflect the removal of more complex Thai utterances after annotation disagreements.
  • Interpretation: Embedding choice has relatively little effect when some pretrained representation is used, and Spanish monolingual ELMo achieves the best cross-lingual result.The findings suggest that sharing the biLSTM or CRF layers may contribute more than placing both languages in similar embedding spaces.

6 Zero-shot learning and learning curves

The zero-shot and learning-curve experiments examine how transfer methods behave as target-language supervision decreases. Translation is strongest without target data, whereas joint multilingual training becomes better once several hundred target-language examples are available.

  • Zero-shot learning: In zero-shot settings, multilingual CoVe outperforms models without encoder embeddings and models with monolingual CoVe across domain, intent, and slot tasks.Spanish results also place multilingual encoder embeddings above XLU embeddings.
  • Zero-shot learning: Translation performs considerably better than other transfer methods when no target-language data is available.Models using cross-lingual embeddings nevertheless perform very poorly in this setting.
  • Learning curves: Cross-lingual training improves over target-only training, especially when little target-language data is available, and pretrained representations provide further gains.Cross-lingual training also produces smaller result ranges across the repeated learning-curve runs.
  • Learning curves: Translation works best with very little target-language data, but its performance quickly plateaus; after several hundred examples, joint multilingual training performs better.The choice of embedding has little effect in these extremely low-resource scenarios.
  • Interpretation: Across languages, training sizes, and embedding types, the consistent advantage of cross-lingual training suggests that sharing the biLSTM and CRF layers is its main benefit.This interpretation is presented as a suggestion based on the combined results.

7 Related work

Related work situates the paper within cross-lingual sequence labeling, sentence representation learning, and transfer for syntactic and semantic parsing tasks.

  • Cross-lingual sequence labeling: Prior sequence-labeling studies use shared character, sentence, or cross-lingual embeddings for tasks including part-of-speech tagging, named entity recognition, and slot filling.The cited work includes multilingual transfer methods for sequence-labeling systems.
  • Cross-lingual sentence representations: Related sentence-representation research uses multilingual machine-translation encoders or aligns vector spaces after pretraining for cross-lingual sentence classification.The paper’s contextual encoder approach belongs to this broader line of work.
  • Transfer for other tasks: Cross-lingual transfer has also been studied for syntactic tasks such as part-of-speech tagging and dependency parsing, including delexicalized parsing approaches.The related literature covers both earlier and more recent multilingual parsing methods.

8 Conclusion and future work

The paper contributes a multilingual intent-and-slot dataset and finds consistent benefits from cross-lingual training, while identifying substantial room for richer multilingual representations. Future directions include character sharing and jointly trained language-model and translation objectives.

  • Conclusion: The dataset contains around 57,000 multilingual utterances, and cross-lingual learning improves over limited target-language-only training in both investigated languages.The choice between multilingual and monolingual embeddings has only a small overall effect.
  • Future work: The authors argue that the evaluated models cover only a small part of the possible cross-lingual embedding space.They frame this limitation as motivating multiple future research directions.
  • Future work: Sharing character embeddings is proposed as a potential source of improvement for languages with similar writing systems.The paper notes that character embeddings were generally absent, except for Spanish ELMo experiments.
  • Future work: A further direction is to add objectives that place translations in similar vector spaces.This follows approaches cited for multilingual sentence representations.
  • Future work: Combining multilingual CoVe with monolingual ELMo or BERT is proposed to unite large monolingual corpora with cross-lingual alignment.The proposed joint encoder would use language-model and machine-translation objectives.
  • Conclusion: The authors hope the dataset will facilitate research that ultimately improves natural-language understanding models for low-resource languages.
Loading 1810.13327v2…