Source-linked AI summary
XNLI: Evaluating Cross-lingual Sentence Representations
Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R. Bowman, Holger Schwenk, Veselin Stoyanov
TL;DR
Multilingual NLP lacks annotated data and standardized evaluation for cross-lingual sentence understanding, especially across many and lower-resource languages. The paper introduces XNLI by extending NLI data to 15 languages and evaluates translation-based and parallel-data-based approaches. XNLI provides a practical evaluation suite, while translation-based baselines outperform the aligned encoder approaches.
Problem
Annotated data is usually available in one language, and standardized benchmarks for difficult cross-lingual understanding tasks such as NLI are scarce.
Method
The paper extends NLI development and test data to 15 languages and evaluates machine translation and parallel-data-based multilingual sentence encoders.
Results
Machine translation baselines outperform parallel-data alignment, although the alignment mechanism achieves very competitive results.
Takeaways & Limitations
XNLI provides a practical, challenging benchmark for evaluating cross-lingual sentence understanding and multilingual sentence encoders.
Takeaways & Limitations
Because XNLI is obtained by translation, it does not capture cultural differences between languages, and translation can rarely alter semantic relations.
Abstract
from arXiv · showhide
State-of-the-art natural language processing systems rely on supervision in the form of annotated data to learn competent models. These models are generally trained on data in a single language (usually English), and cannot be directly used beyond that language. Since collecting data in every language is not realistic, there has been a growing interest in cross-lingual language understanding (XLU) and low-resource cross-language transfer. In this work, we construct an evaluation set for XLU by extending the development and test sets of the Multi-Genre Natural Language Inference Corpus (MultiNLI) to 15 languages, including low-resource languages such as Swahili and Urdu. We hope that our dataset, dubbed XNLI, will catalyze research in cross-lingual sentence understanding by providing an informative standard evaluation task. In addition, we provide several baselines for multilingual sentence understanding, including two based on machine translation systems, and two that use parallel data to train aligned multilingual bag-of-words and LSTM encoders. We find that XNLI represents a practical and challenging evaluation suite, and that directly translating the test data yields the best performance among available baselines.
1 Introduction
XNLI addresses the lack of multilingual evaluation data for sentence understanding by extending NLI evaluation to 15 languages. It also compares cross-lingual encoders and translation-based approaches for transferring English-trained NLI systems.
- Annotated training data is usually available in one language, while multilingual applications must handle inputs across many languages.
- XNLI extends NLI development and test data to 15 languages, including lower-resource Swahili and Urdu, for cross-lingual sentence-understanding evaluation.The corpus contains 7,500 annotated examples per language and 112,500 annotated pairs overall.
- The benchmark evaluates models trained in one language and tested in different languages, focusing on development and test rather than training data.
- Parallel data can align multilingual sentence encoders so an English-trained NLI classifier can classify sentence pairs in other languages.
- XNLI also supports evaluation of pretrained multilingual sentence encoders and the development of multilingual text embedding spaces.
2 Related Work
Prior work developed multilingual word, sentence, and document representations, but evaluation resources remained limited for large-scale, sentence-level cross-lingual understanding. XNLI fills this gap with broader multilingual NLI coverage.
- Multilingual representation research includes aligned word embeddings learned with bilingual supervision and sentence encoders built from bag-of-words, compositional, autoencoding, and sequence-to-sequence methods.
- The Reuters benchmark evaluates cross-lingual document classification, but its document-level setup, unbalanced classes, and missing development set complicate sentence-embedding comparisons.
- Other multilingual resources include semantic textual similarity datasets in four languages and smaller RTE datasets created by translation or parallel-corpus annotation.
- XNLI is described as the first large-scale corpus for evaluating sentence-level representations across that many languages.
- XNLI does not capture cultural differences between languages because its corpus was obtained by translation.
3 The XNLI Corpus
XNLI constructs new English NLI examples and professionally translates them into multiple languages, preserving shared source content and labels. The resulting corpus largely resembles MultiNLI, though translation can rarely alter semantic relations.
- The corpus begins with 7,500 newly collected English examples, generated and validated using the MultiNLI crowdsourcing procedure.
- Translation keeps data distributions similar across languages, reuses the trusted worker pool, and provides corresponding hypotheses across languages.The paper notes that the resulting cross-language combinations are not evaluated in this work.
- 3.2 The Resulting Corpus: Translation can change semantic relations, including an observed Chinese case where entailment became contradiction, although such cases appear rare.
- Professional translators translate premises and hypotheses separately into 15 languages, while labels are copied from the English source text.
- 3.2 The Resulting Corpus: Bilingual annotators recovered English consensus labels 85% of the time on English data and 83% of the time on translated French data.
- 3.2 The Resulting Corpus: Across languages, premises average about twice as many tokens as hypotheses, and negation words such as no, not, or never are among the top two contradiction cues.
4 Cross-Lingual NLI
This section compares translation-based and sentence-embedding approaches for cross-lingual NLI. It presents aligned multilingual encoders that let an English-trained classifier operate in other languages without translation at inference time.
- 4.1 Translation-Based Approaches: Translation-based XLU uses either translated training data or test-time translation, but each approach has distinct practical costs.TRANSLATE TRAIN requires separate classifiers for each language, whereas TRANSLATE TEST requires computationally intensive translation during inference.
- 4.2 Multilingual Sentence Encoders: Language-universal sentence embeddings offer an alternative in which an English-trained classifier can classify other languages without an inference-time translation system.The approach relies on embeddings of translated sentences being close enough for a shared classifier to interpret them equivalently.
- 4.2 Multilingual Sentence Encoders: The evaluated encoders include pretrained multilingual CBOW representations and MultiNLI-trained bidirectional LSTM encoders, covering transfer-learning and in-domain settings.Both approaches align target-language sentence spaces to a fixed English encoder using parallel data.
- 4.2 Multilingual Sentence Encoders: The alignment pipeline trains an English encoder and classifier, aligns a target encoder with parallel data, and then applies the shared classifier in the other language.Figure 1 illustrates this process for Spanish, with the target encoder trained to mimic the English encoder.
- 4.2.3 Aligning Sentence Embeddings: The proposed alignment loss combines source-target distance with a contrastive negative-sampling term, using the L2 norm as the distance measure.The loss is designed to make parallel sentence embeddings sufficiently close for the shared classifier, whereas the ranking-loss alternative produced very poor results in this setting.
- 4.2.3 Aligning Sentence Embeddings: The ranking loss performs poorly because it only orders translation pairs relative to negative pairs rather than forcing them close enough for the shared classifier to recognize equivalent meaning.The authors use Lalign in the X-CBOW and X-BILSTM alignment baselines.
5 Experiments and Results
The experiments compare translation-based and aligned multilingual encoder approaches on XNLI. Translation at test time performs best, while aligned encoders provide competitive, cheaper transfer with performance shaped by encoder strength and alignment quality.
- Cross-lingual encoders: BiLSTM-max consistently outperforms BiLSTM-last and pretrained CBOW across languages, indicating that NLI requires more than word-level information.The same superiority over CBOW holds even when CBOW word embeddings are fine-tuned on MultiNLI.
- Translation baselines: TRANSLATE TEST achieves the best cross-lingual results for all evaluated language directions.It consistently outperforms TRANSLATE TRAIN and serves as the strongest translation baseline.
- Translation baselines: Above 70% XNLI performance is reached for three of the four languages with the best translation systems, versus 73.7% English NLI accuracy.The remaining gap may reflect translation errors, style changes, or machine-translation artifacts.
- Cross-lingual encoders: 67.7% French accuracy is obtained with an English classifier and multilingual sentence encoder, while aligned encoders remain competitive with TRANSLATE TRAIN.TRANSLATE TEST exceeds the multilingual encoder approach by up to 6% on Swahili.
- Cross-lingual encoders: A two-point accuracy advantage of X-BiLSTM-max over X-BiLSTM-last is maintained across languages.The authors suggest that a stronger English encoder also improves transfer performance in other languages.
- Alignment analysis: Alignment loss and XNLI accuracy are strongly correlated during training for the French, Arabic, and Urdu X-BiLSTM encoders.As English-Arabic parallel embeddings become closer in L2 distance, the English classifier improves on Arabic embeddings; Urdu shows some over-fitting with 64k parallel sentences.
- Alignment analysis: Fine-tuning target embeddings does not significantly affect BiLSTM-max validation accuracy, while the negative alignment term yields at most a 1.6% improvement in Chinese.The negative term is generally not critical to performance.
6 Conclusion
XNLI extends MultiNLI development and test sets to 15 languages to address limited standardized evaluation for cross-lingual understanding. Machine translation obtains the best results, while cross-lingual encoders offer an encouraging and efficient alternative that remains below translation-based methods.
- Contribution: XNLI extends MultiNLI development and test sets to 15 languages, including low-resource Swahili and Urdu.The benchmark contains 112,500 annotated sentence pairs in total.
- Results: Machine translation baselines obtain the best results in the experiments but require computationally intensive translation models at training or test time.This establishes a strong performance baseline with a substantial computational requirement.
- Implications: Cross-lingual encoder baselines provide an encouraging and efficient alternative, although further work is needed to match translation-based methods.Their efficiency follows from avoiding the computational cost of machine translation at inference.