Source-linked AI summary
End-to-End Slot Alignment and Recognition for Cross-Lingual NLU
Weijia Xu, Batool Haider, Saab Mansour
TL;DR
Cross-lingual NLU systems rely on machine translation and slot-label projection, making transfer sensitive to projection quality. This paper introduces MultiATIS++ and an end-to-end attention model that jointly aligns and predicts target slots, outperforming fast-align on most languages and matching a stronger projection approach with half the training time.
Problem
Machine-translation-based cross-lingual NLU depends on projecting source slot labels to translated utterances, and target performance depends strongly on projection quality.
Method
The paper introduces MultiATIS++ and an end-to-end attention model that jointly learns soft slot-label alignment and target slot recognition without external label projection.
Results
The model outperforms fast-align on most languages and achieves competitive performance with a state-of-the-art projection method using half the training time.
Takeaways & Limitations
MultiATIS++ provides a nine-language, four-family benchmark for evaluating cross-lingual NLU methods across diverse language distances.
Takeaways & Limitations
Hindi and Turkish portions of the corpus are smaller and cover only subsets of the intent and slot types.
Abstract
from arXiv · showhide
Natural language understanding (NLU) in the context of goal-oriented dialog systems typically includes intent classification and slot labeling tasks. Existing methods to expand an NLU system to new languages use machine translation with slot label projection from source to the translated utterances, and thus are sensitive to projection errors. In this work, we propose a novel end-to-end model that learns to align and predict target slot labels jointly for cross-lingual transfer. We introduce MultiATIS++, a new multilingual NLU corpus that extends the Multilingual ATIS corpus to nine languages across four language families, and evaluate our method using the corpus. Results show that our method outperforms a simple label projection method using fast-align on most languages, and achieves competitive performance to the more complex, state-of-the-art projection method with only half of the training time. We release our MultiATIS++ corpus to the community to continue future research on cross-lingual NLU.
1 Introduction
The paper addresses cross-lingual NLU by introducing a multilingual corpus and an end-to-end model that jointly aligns and predicts slot labels without external projection tools. The model outperforms fast-align on most languages and approaches a more complex projection method with half the training time.
- Existing multilingual NLU datasets support only three languages across two language families, limiting broad cross-lingual transfer study.
- MultiATIS++ expands evaluation to six additional languages and supports cross-lingual NLU across broader language distances.
- The proposed model jointly learns soft slot-label alignment and recognition using attention, intent classification, slot filling, and an augmented reconstruction objective.
- The model achieves significantly higher slot F1 than fast-align on most languages using the same training time.
- The model achieves competitive performance with a more complex linguistic-feature projection method while using half the training time.
2 Related Work
Prior cross-lingual NLU methods transfer through multilingual models or machine translation, with effectiveness depending on language similarity and slot-label projection quality.
- Cross-lingual transfer methods broadly use multilingual models or machine translation to adapt sequence-tagging systems.
- Shared character encoders help related languages but are less effective for dissimilar languages lacking common lexical features.
- Machine-translation transfer requires projecting source-language labels onto translated target sentences, commonly using word alignments or neural attention.
3 Data
MultiATIS++ extends multilingual ATIS with professionally translated and slot-annotated data across nine languages and four language families. The corpus preserves spoken-language characteristics while documenting narrower coverage for Hindi and Turkish.
- MultiATIS++ extends English ATIS to nine languages spanning Indo-European, Sino-Tibetan, Japonic, and Altaic language families.
- Professional native translators translated utterances and annotated aligned slot segments for each new language.
- Figure 1 illustrates one English flight-intent example alongside its translations and BIO slot labels in eight other languages.
- The translated data preserves spoken modality phenomena and original formality, then uses BIO tags for tokenized utterances.
- Hindi and Turkish data are smaller and cover only subsets of the intent and slot types.
4 Cross-Lingual NLU
The section presents a joint intent-and-slot NLU architecture that learns soft alignment between source slot labels and target utterance representations without external label projection. During training it uses attention and reconstruction objectives, while inference directly applies the encoder to target intent and slot classification.
- Joint Intent Detection and Slot Filling: The encoder represents the input sequence, using either a bidirectional LSTM or multilingual BERT for intent and slot prediction.Multilingual BERT is pretrained on monolingual corpora from 104 languages and fine-tuned on NLU data.
- Joint Intent Detection and Slot Filling: The model jointly predicts intent and slot labels while learning soft alignment between target utterance representations and source slot labels.It avoids external slot-label projection during training.
- End-to-End Slot Alignment and Recognition via Attention: Attention computes source-position hidden states as weighted averages of target representations, with weights linking source words to translation words.The attention uses projected query and key vectors, with τ controlling the softmax temperature.
- End-to-End Slot Alignment and Recognition via Attention: A reconstruction module recovers the source utterance from attention outputs, and the final training loss combines intent, slot, and reconstruction losses.The model is jointly trained on translated target data and supervised source data.
- End-to-End Slot Alignment and Recognition via Attention: During inference, the attention and reconstruction modules are removed and target encoder representations feed directly into the intent and slot classification layers.The translated utterance is assumed to retain the source utterance’s intent during training.
5 Multilingual NLU
The section evaluates supervised multilingual NLU strategies with multilingual BERT and compares target-only and multilingual training. Multilingual BERT improves intent accuracy and slot F1, with especially large gains for Hindi and Turkish, while multilingual training adds further improvements on those languages.
- Training Strategies: The experiments compare target-only and multilingual training strategies using multilingual BERT for multilingual NLU.Target-only training uses one language’s data, whereas multilingual training concatenates data from all languages.
- Setup: Evaluation uses intent accuracy and slot F1, with models trained for 20 epochs and selected by development-set performance.Results are averaged over 5 runs for the MultiATIS++ full-training-data comparison.
- Results: 1–6% intent-accuracy and 1–11% slot-F1 improvements result from multilingual BERT over LSTM target-only models.The largest improvements occur on the low-resource Hindi and Turkish languages.
- Results: 2–5% intent-accuracy and 4–11% slot-F1 improvements result from multilingual training on Hindi and Turkish for both LSTM and BERT models.These gains are reported relative to the corresponding target-only training strategies.
- Comparison with SOTA: +6.1% Hindi and +7.1% Turkish slot-F1 improvements over prior bilingual-training results are achieved by the multilingual BERT model.The prior reported slot F1 values were 80.6% for Hindi and 78.9% for Turkish.
6 Cross-Lingual Transfer
The study compares zero-shot and few-shot cross-lingual transfer methods using machine translation, label projection, and soft alignment across languages. Soft alignment is generally more robust than fast-align and competitive with MT+TMP while requiring less training time.
- 6.1 Zero-Shot Results: The experiments compare No MT, MT+fast-align, MT+TMP, and MT+soft-align under zero-shot and few-shot transfer settings.The MT-based approaches use translated English data, while few-shot experiments add small amounts of labeled target-language data.
- 6.1 Zero-Shot Results: 14–32% intent accuracy and 29–61% slot F1 are the gains from adding multilingual BERT to MT+fast-align over LSTM on most languages.Turkish is the exception for these gains, and MT+fast-align improves intent accuracy over corresponding no-MT models by 13–33% with LSTM and 1–24% with BERT.
- 6.1 Zero-Shot Results: 20–44% slot F1 gains occur with BERT plus MT+fast-align on Chinese, Japanese, and Hindi, whereas French declines by around 6%.The authors attribute the French decline possibly to strong multilingual BERT representations and noise from projected labels.
- 6.1 Zero-Shot Results: Our method outperforms MT+fast-align on five of eight languages and achieves on-par-or-higher intent accuracy than MT+TMP on all languages.For slot F1, it is on par or better than MT+TMP on four of eight languages and close on the remainder, while using half the training time.
- 6.1 Zero-Shot Results: +16.9% slot F1 is the margin by which our method outperforms MT+TMP on Turkish, where fast-align and TMP projection accuracy are 20% and 39%.Projection accuracy exceeds 70% on other languages, while Turkish-English morphological differences make hard word alignment difficult.
- 6.1 Zero-Shot Results: Comparable or higher slot F1 than BERT+MT+TMP is achieved with a few hundred target examples, respectively on French or Chinese.BERT+NoMT is comparable to or better than the best MT-based approach on French but trails it by around 5% in Chinese intent accuracy.
- 6.3 Ablation Study: 3.6% and 8.4% slot F1 drops result from ablating reconstruction loss and supervised source-data joint training, while intent accuracy changes little.The ablation scores are averaged over eight target languages.
7 Conclusion
The paper contributes MultiATIS++, a broad multilingual corpus, and an end-to-end soft-alignment model for cross-lingual NLU. The model avoids external label projection, outperforms fast-align on most languages, and matches a stronger projection method with half the training time.
- 7 Conclusion: MultiATIS++ extends Multilingual ATIS to nine languages across four language families.The languages span Indo-European, Sino-Tibetan, Japonic, and Altaic families.
- 7 Conclusion: The proposed model jointly aligns and recognizes slot labels without external label projection.The corpus is used to evaluate multilingual BERT, machine translation, and label-projection transfer methods.
- 7 Conclusion: The model outperforms fast-align on most languages and achieves competitive performance with the state-of-the-art projection approach using half the training time.The corpus is released to facilitate future research on cross-lingual NLU.
A Model and Training Details
The experiments train LSTM and BERT models with Adam for 20 epochs on four NVIDIA V100 GPUs. Model-size and training-time tables document the computational setup across multilingual and transfer experiments.
- A Model and Training Details: All models are trained with Adam for 20 epochs on 4 NVIDIA V100 Tensor Core GPUs.The initial learning rates are 10^-3 for LSTM and 10^-5 for BERT.
- A Model and Training Details: The LSTM uses 256-dimensional embeddings, 128 hidden units, and dropout of 0.1 on embeddings and encoder hidden states.Both LSTM and BERT use WordPiece tokenization.
- A Model and Training Details: The parameter-count table includes extra attention-module and reconstruction-loss output-layer parameters for the proposed model.The cited training-time table reports total training time for all languages in multilingual NLU experiments.
B Evaluation
Evaluation uses intent accuracy and slot F1, with slot segmentation merged before scoring. Slot F1 is computed using conlleval.pl.
- B Evaluation: Intent accuracy and slot F1 are the evaluation metrics for all NLU models.Slot F1 measures slot-labeling performance after preprocessing adjustments.
- B Evaluation: Segmented slots are merged to their original spans before slot F1 is computed with conlleval.pl.This preprocessing step aligns the scoring units with the original slot segments.
C Validation Performance
Validation reports average intent accuracy and slot F1 on development sets for multilingual NLU experiments. Zero-shot models are selected at the last epoch because target-language development sets are assumed unavailable.
- Zero-shot models are selected at the last epoch because target-language development sets are assumed unavailable.
- Validation uses average intent accuracy and slot F1 on development sets for the multilingual NLU experiments.
- Target-only models average scores over all nine languages, whereas multilingual models are validated only on the English development set.