PaperlayerView exact revision on arXiv

Source-linked AI summary

Translation Artifacts in Cross-lingual Transfer Learning

Mikel Artetxe, Gorka Labaka, Eneko Agirre

arXiv:2004.04721v4cs.CLcs.LG

TL;DR

Cross-lingual transfer systems and multilingual benchmarks mix original, human-translated, and machine-translated text, but the resulting artifacts are insufficiently understood. The paper analyzes these artifacts through translated and back-translated training data, showing that they affect model behavior and improve XNLI performance for TRANSLATE-TEST and ZERO-SHOT approaches. It concludes that translation procedures can distort cross-lingual comparisons and previous interpretations.

  • Problem

    Cross-lingual models are often trained and tested on different types of original, human-translated, and machine-translated text, but the impact of this mismatch has been overlooked.

  • Method

    The paper compares original, back-translated, and machine-translated training data across translation-based and zero-shot transfer settings, including original and translated test sets.

  • Results

    The analysis finds that translation artifacts alter superficial patterns such as lexical overlap and improves XNLI state of the art for TRANSLATE-TEST and ZERO-SHOT approaches.

  • Takeaways & Limitations

    Translation artifacts can materially affect cross-lingual model evaluation, so previous transfer-learning findings and cross-lingual gaps may need reconsideration.

  • Takeaways & Limitations

    The evaluation implications do not have a simple solution because translation may introduce artifacts or mitigate artifacts already present in original datasets.

Abstract

from arXiv · show

Both human and machine translation play a central role in cross-lingual transfer learning: many multilingual datasets have been created through professional translation services, and using machine translation to translate either the test set or the training set is a widely used transfer technique. In this paper, we show that such translation process can introduce subtle artifacts that have a notable impact in existing cross-lingual models. For instance, in natural language inference, translating the premise and the hypothesis independently can reduce the lexical overlap between them, which current models are highly sensitive to. We show that some previous findings in cross-lingual transfer learning need to be reconsidered in the light of this phenomenon. Based on the gained insights, we also improve the state-of-the-art in XNLI for the translate-test and zero-shot approaches by 4.3 and 2.8 points, respectively.

1 Introduction

Cross-lingual benchmarks and transfer systems commonly rely on human or machine translation, creating mismatches between training and test text. The paper shows that translation artifacts affect model performance and can change conclusions about cross-lingual transfer.

  • 1 Introduction: Translation creates mismatches between training and test text across TRANSLATE-TEST, ZERO-SHOT, and TRANSLATE-TRAIN systems.TRANSLATE-TEST trains on original data but evaluates on machine-translated text, while ZERO-SHOT and TRANSLATE-TRAIN evaluate on human-translated text.
  • 1 Introduction: Back-translation produces training data that better resembles translated test sets, substantially improving TRANSLATE-TEST and ZERO-SHOT approaches.The paper argues that the effect is not ordinary data augmentation because it appears for translated, but not original, test sets.
  • 1 Introduction: Independent translation of NLI premises and hypotheses alters lexical-overlap patterns, severely affecting current models’ generalization.The paper attributes this behavior to subtle artifacts arising from translation itself.
  • 1 Introduction: These findings require reconsidering previous conclusions in cross-lingual transfer learning and motivate improved XNLI results.The paper specifically reports improvements for TRANSLATE-TEST and ZERO-SHOT approaches.

2 Related work

Prior work spans multilingual transfer methods, translated and separately annotated benchmarks, annotation artifacts, and translationese. This paper distinguishes its translation-artifact phenomenon from general translationese by focusing on inconsistencies caused by translating without sufficient context.

  • Cross-lingual transfer learning: Cross-lingual transfer methods include multilingual pre-training, parallel-data training, and transfer from English data to other languages.The related work places the paper within research on multilingual representations and cross-lingual model transfer.
  • Multilingual benchmarks: Multilingual benchmarks commonly use translation, including XNLI, PAWS-X, XQuAD, and MLQA, while TyDi QA is separately annotated across languages.Other datasets leverage existing multilingual resources rather than the same translation-based construction.
  • Annotation artifacts: NLI datasets contain exploitable annotation artifacts, including hypothesis-only cues and a tendency to predict entailment for high-overlap pairs.These findings motivate adversarial datasets and analyses of spurious patterns.
  • Translationese: Translationese includes features such as simplification, explicitation, normalization, and interference, whereas this paper studies translation inconsistencies caused by limited context.The paper focuses on effects in cross-lingual transfer learning rather than machine-translation evaluation.

3 Experimental design

The experiments compare original, back-translated, and machine-translated training data using representative English and multilingual models across NLI and QA benchmarks. Evaluation includes original, human-translated, and machine-translated settings, with repeated runs and XNLI comparisons.

  • 3.1 Models and transfer methods: The study compares ROBERTA and XLM-R using original, back-translated, and machine-translated training variants.Back-translation uses Spanish or Finnish pivots, while machine-translated variants are produced in Spanish or Finnish.
  • 3.3 Tasks and evaluation procedure: XNLI evaluates NLI models across 15 languages using premises and hypotheses independently translated by professional translators.TRANSLATE-TEST additionally uses machine-translated versions of the test set.
  • 3.3 Tasks and evaluation procedure: QA evaluation uses XQuAD and MLQA, whose translated contexts and questions provide complementary multilingual test settings.XQuAD translates a subset of SQuAD, while MLQA contains parallel contexts with questions translated into multiple languages.
  • 3.3 Tasks and evaluation procedure: BT-XX and MT-XX consistently outperform ORIG in XNLI development results.Table 1 reports accuracy comparisons across the evaluated settings.

4 NLI experiments

The NLI experiments show that matching training text to translated test text improves cross-lingual transfer, while translation artifacts alter model behavior and class predictions. These findings yield state-of-the-art gains for TRANSLATE-TEST and ZERO-SHOT.

  • 4.2 ZERO-SHOT results: Approximately 2 points: Back-translated and machine-translated training data outperform ORIG for ZERO-SHOT transfer on XNLI.The result suggests translated training data resembles human-translated test sets better than original training data, even when the target language differs.
  • 4.3 Original vs. translated test sets: BT-XX and MT-XX outperform ORIG on translated test sets, but ORIG performs best on original English and neither translated variant clearly improves Spanish.This confirms that the phenomenon is limited to translated test sets.
  • 4.4 NLI Stress Tests: BT-FI and MT-FI are weaker than ORIG on competence tests but stronger on distraction tests, especially for lexical overlap and negation.Independent translation of premise and hypothesis reduces lexical overlap and changes the superficial patterns models encounter.
  • 4.5 Output class distribution: ORIG underpredicts entailment and overpredicts neutral in transfer languages, whereas translated-data systems are less affected by this class-distribution bias.The independently translated premise and hypothesis reduce overlap, while BT-FI and MT-FI encounter the same pattern during training.
  • 4.6 Comparison with the state-of-the-art: 4.3 and 2.8 points: The best method improves the XNLI state of the art for TRANSLATE-TEST and ZERO-SHOT, respectively.It also achieves the best overall published results while using one English–Finnish translation system rather than systems for all 14 target languages.

5 QA experiments

QA experiments find consistent but modest gains from back-translated training for TRANSLATE-TEST on MLQA, while ZERO-SHOT and XQuAD results remain inconclusive. Differences across QA benchmarks partly reflect whether questions and answer-bearing contexts were translated together.

  • 5 QA experiments: Back-translated training outperforms ORIG for TRANSLATE-TEST on MLQA consistently across languages, models, and runs.The improvement is modest, while machine-translated training results are less conclusive because answer-span mapping can introduce noise.
  • 5 QA experiments: ZERO-SHOT shows no clear improvement on MLQA, and XQuAD results are more positive but still inconclusive.The QA findings therefore provide weaker support than the NLI results for translated-data training.
  • 5 QA experiments: QA benchmark translation procedures may explain the weaker effects: XQuAD translates questions and contexts together, while MLQA uses parallel contexts and exposes answer-bearing sentences.These procedures produce more consistent translations than independently translating XNLI premises and hypotheses.

6 Discussion

The discussion argues that translation artifacts change how cross-lingual methods and benchmarks should be interpreted. It reports that translated-data effects, rather than target-language training alone, explain key results and that XNLI understates XLM-R’s cross-lingual ability.

  • XNLI overestimates the cross-lingual transfer gap because original-to-translated generalization loss can exceed English-to-other-language loss.The results suggest XLM-R’s real cross-lingual generalization ability is considerably better than XNLI accuracy reflects.
  • Training on translated data, rather than training on the target language, is key to TRANSLATE-TRAIN outperforming ZERO-SHOT.
  • Improvements attributed to data augmentation should instead be reconsidered as effects of dataset artifacts that alter superficial patterns between premises and hypotheses.The experimental design keeps the number of unique sentences and examples constant, preventing a data-augmentation explanation.
  • TRANSLATE-TEST narrowed its XNLI gap to the state of the art from 4.6 points to 0.8 points by addressing translation artifacts.This approach is especially affected because human translation followed by machine back-translation doubles the phenomenon, while equivalent mitigation was not previously applied.
  • Future multilingual benchmarks should control translation artifacts by providing consistent English and non-English test sets, preferably through original annotations or document-level translation.The authors note that artifacts may originate in datasets and be altered or mitigated by translation, making evaluation design a broader unresolved issue.

7 Conclusions

The paper concludes that human and machine translation can alter superficial data patterns, requiring reconsideration of cross-lingual transfer findings. It improves XNLI substantially for TRANSLATE-TEST and ZERO-SHOT and finds a less pronounced version of the phenomenon in QA.

  • Translation artifacts affect cross-lingual transfer findings, requiring reconsideration of prior conclusions.
  • The paper improves state-of-the-art XNLI performance for TRANSLATE-TEST and ZERO-SHOT by a substantial margin.
  • The phenomenon also affects QA, although it is less pronounced because of the translation procedure used in those benchmarks.
Loading 2004.04721v4…