PaperlayerView exact revision on arXiv

Source-linked AI summary

UXLA: A Robust Unsupervised Data Augmentation Framework for Zero-Resource Cross-Lingual NLP

M Saiful Bari, Tasnim Mohiuddin, Shafiq Joty

arXiv:2004.13240v4cs.CLcs.LG

TL;DR

UXLA addresses scarce target-language labels and weaker transfer across structurally dissimilar languages with unsupervised augmentation and self-training. It achieves XNER SoTA across tested languages and, with 5% source labels, matches full-label baselines on XNLI and PAWS-X.

  • Problem

    Target-task annotations are scarce in low-resource languages, and cross-lingual transfer is constrained by structural differences between source and target languages.

    Source: §1 · p. 1

  • Method

    UXLA augments unlabeled target examples and source/target vicinal samples, then uses simultaneous self-training with multilingual co-distillation.

    Source: §1 · p. 2

  • Results

    UXLA sets XNER SoTA across tested languages and, with 5% source labels, achieves results comparable to full-label baselines on XNLI and PAWS-X.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Experiments across three cross-lingual tasks demonstrate UXLA’s effectiveness in zero-resource task adaptation.

    Source: §6 · p. 9

    The study does not explore distillation features beyond model prediction scores, such as sequence length or language.

    Source: §4.1 · p. 8

Abstract

from arXiv · show

Transfer learning has yielded state-of-the-art (SoTA) results in many supervised NLP tasks. However, annotated data for every target task in every target language is rare, especially for low-resource languages. We propose UXLA, a novel unsupervised data augmentation framework for zero-resource transfer learning scenarios. In particular, UXLA aims to solve cross-lingual adaptation problems from a source language task distribution to an unknown target language task distribution, assuming no training label in the target language. At its core, UXLA performs simultaneous self-training with data augmentation and unsupervised sample selection. To show its effectiveness, we conduct extensive experiments on three diverse zero-resource cross-lingual transfer tasks. UXLA achieves SoTA results in all the tasks, outperforming the baselines by a good margin. With an in-depth framework dissection, we demonstrate the cumulative contributions of different components to its success.

1 Introduction

The introduction frames zero-shot cross-lingual transfer as a way to reuse multilingual pretrained models without target-language labels, while noting that annotations are scarce and language-structure gaps constrain transfer.

  • Cross-lingual transfer is more difficult when source and target languages are structurally dissimilar, especially when the target is also low-resource.Source: §1 · p. 1
  • Text augmentation is challenging because smooth embedding changes may not yield plausible, meaning-preserving linguistic units, while back-translation requires parallel data and task-compatible alignments.Source: §1 · p. 2
  • UXLA combines target-language and source/target vicinal augmentation with simultaneous self-learning and two-stage multilingual co-distillation to support transfer to dissimilar or low-resource languages.Source: §1 · p. 2
  • The experiments cover XNER, XNLI, and PAWS-X across 14 language pairs spanning similar, dissimilar, and low-resource languages.Source: §1 · p. 2
  • 28.54%, 16.05%, and 9.25% absolute improvements for Urdu, Burmese, and Arabic, respectively, accompany UXLA’s XNER SoTA across all tested languages.Source: §1 · p. 2
  • On XNLI, 5% source labels yield results comparable to the all-labeled baseline; PAWS-X shows similar findings.Source: §1 · p. 2

2 UXLA Framework

UXLA adapts task models to unlabeled target-language data through staged augmentation, self-training, and sample filtering. Three task models exchange pseudo-labels and selected samples while training on source, target, and generated vicinal data.

2.2 Sentence Augmentation

  • UXLA augments unlabeled target examples and masked-LM-generated vicinal sentences, using successive max for single sequences and successive cross for paired tasks.The pretrained masked LM generates the virtual examples around source and target sentences.Source: §2 · p. 2, §2.2 · p. 4

2 UXLA Framework

  • Training progresses from source plus target data in epoch 1, to virtual target data in epoch 2, then all four datasets in epoch 3.The four datasets are source, original target, virtual target, and virtual source data.Source: §2 · pp. 2–3, §2.4 · p. 5

2.1 Warm-up: Training Task Models

  • Three independently seeded XLM-R-large task models are warm-started on labeled English data, with a negative-entropy confidence penalty to counter overly confident predictions.The models share an architecture but use different random seeds.Source: §2.1 · p. 3

2.3 Co-labeling through Co-distillation

  • Single-model confidence or a two-component GMM over pseudo-label loss selects samples, then agreement between two peer models trains the third model.The GMM assigns goodness probabilities from the lower-loss component; the method is designed to filter noisy augmented data.Source: §2.3 · p. 5

2.4 Data Samples Manipulation

  • Balanced multinomial sampling uses dataset sizes and α to control each dataset’s contribution and reduce overfitting to any one dataset.The dataset sizes can differ as augmentation and distillation produce variable-sized sample sets.Source: §2.4 · p. 5

3 Experiments

Experiments evaluate zero-resource transfer from English across XNER, XNLI, and PAWS-X, with XNLI and PAWS-X also tested using only 5% of source training data. UXLA improves over baselines across the reported tasks, including SoTA results for full-data XNLI and gains in low-resource or structurally dissimilar settings.

3.2 Results

  • On XNER, UXLA improves over prior baselines by 3.76%, 4.34%, 6.94%, 8.31%, and 4.18% for es, nl, de, ar, and fi, respectively.It surpasses supervised LSTM-CRF for nl and de without target-language labels and produces comparable results for es.Source: §3.2 · p. 6
  • For XNER, UXLA is particularly effective in structurally dissimilar or low-resource settings, with absolute gains of 28.54%, 16.05%, and 9.25% for ur, my, and ar.The WikiANN results are reported in Table 2; Arabic and Finnish results from Table 1 also inform the stated pattern.Source: §3.2 · pp. 6–7, Table 2 · p. 6
  • With 5% source labels, UXLA on XNLI performs comparably to the 100%-label XTREME baseline, lagging by about 0.7% on average, and beats the standard 5% baseline by 4.2% on average.The authors report that gains are relatively higher for low-resource or dissimilar languages despite a weak base model.Source: §3.2 · p. 7, Table 3 · p. 7
  • UXLA achieves XNLI SoTA across languages with a 2.55% average absolute improvement over the XTREME baseline using 100% of source labels.The reported language-specific improvements are 1.95%, 1.68%, 4.30%, 3.50%, 3.24%, and 1.65% for es, de, ar, sw, hi, and ur, respectively.Source: §3.2 · p. 7, Table 3 · p. 7
  • In PAWS-X, UXLA improves over baselines by 5.94% on average with 5% source data and 3.25% with 100%.With 5% source data, it outperforms 100% XLM-R baselines for es, ja, and zh; with 100%, its ensemble approaches supervised XLM-R trained on target-language data.Source: §3.2 · p. 7

4 Analysis

The analysis tests UXLA’s distillation choices, augmentation stages, confidence penalty, ensemble effect, and robustness. It reports benefits from distillation and staged augmentation, while target-language adaptation leaves English XNER performance nearly unchanged.

4.1 Analysis of distillation methods

  • Model-confidence and clustering distillation perform similarly, with model confidence slightly better; clustering could support other features, which the paper does not explore.The main experiments and subsequent analysis use model confidence.Source: §4.1 · pp. 7–8
  • The best XNER results for every language use a distillation factor η other than 100%, indicating that distillation is effective.Here, 100% means no single-model distillation based on model confidence.Source: §4.1 · p. 8, Table 5 · p. 8
  • Distillation by model agreement improves results in all reported cases on top of single-model distillation.The comparison is against results without agreement; the η = 100, agreement condition is the tri-training baseline.Source: §4.1 · p. 8, Table 5 · p. 8

4.2 Augmentation in Stages

  • Each augmentation epoch significantly boosts validation F1 across languages, with larger relative gains for structurally dissimilar Arabic, especially in epoch 2.Figure 2 presents validation F1 for different types of augmented data across epochs.Source: §4.2 · p. 8, Figure 2 · p. 8

4.3 Effect of Confidence Penalty & Ensemble

  • Confidence penalty improves XNER by 0.56%, XNLI-5% by 0.74%, PAWS-X-5% by 1.89%, and PAWS-X-100% by 1.18%.XNLI-100% improvements are marginal and inconsistent, which the authors suspect is due to balanced class distribution.Source: §4.3 · p. 9
  • UXLA outperforms ensemble baselines, and ensembling UXLA models improves performance further.The authors say this indicates its co-teaching and co-distillation capability goes beyond the ensemble effect.Source: §4.3 · p. 9

4.4 Robustness & Efficiency

  • UXLA adaptation leaves English XNER F1 almost unchanged, and adapting on another language improves Arabic relative to its 50.88 score.The authors interpret this as evidence that UXLA augmentation does not overfit to a target language.Source: §4.4 · p. 9, Table 6 · p. 9

5 Related Work

The related work situates UXLA among multilingual transfer and text augmentation approaches. It distinguishes UXLA’s unsupervised use of a pretrained language model from prior methods that rely on labeled data for augmentation.

5 Related Work

  • Multilingual pretrained models support zero-shot transfer, while prior work differs on whether shared subwords or structural similarity drives successful cross-lingual transfer.Pires et al. (2019) identify structural similarity as important; K et al. (2020) find shared subwords contribute minimally and structural similarity matters more.Source: §5 · p. 9
  • Prior contextualized-LM augmentation methods use labels for label-constrained augmentation, unlike UXLA’s purely unsupervised LM augmentation.UXLA does not fine-tune the pretrained vicinity model, allowing replacement with a stronger monolingual LM for a target language.Source: §5 · p. 9
  • Earlier augmentation approaches used distributional clusters, while later methods use contextualized LMs.The passage names Täckström et al. (2012), Kobayashi (2018), Wu et al. (2018), Shi et al. (2019), Ding et al. (2020), and Liu et al. (2021).Source: §5 · p. 9

6 Conclusion

The conclusion presents UXLA as a framework for zero-resource cross-lingual task adaptation. Across three tasks, it reports SoTA XNER results and comparable XNLI and PAWS-X performance using 5% of source labels versus a baseline using all labels.

6 Conclusion

  • UXLA combines simultaneous self-training, data augmentation, and unsupervised sample selection for zero-resource cross-lingual task adaptation.Source: §6 · p. 9
  • Across three cross-lingual tasks, UXLA sets XNER SoTA in all tested languages and, with 5% source labels, matches the 100%-labeled baseline on XNLI and PAWS-X.Source: §6 · p. 9

A FAQ: Justifications for design methodology of UXLA

The FAQ gives design rationales and qualifications for UXLA’s modeling and training choices. It contrasts task-data language-model finetuning with UXLA’s robustness result and explains the roles of model diversity, staged datasets, and joint training.

A FAQ: Justifications for design methodology of UXLA

B.1 Effect of confidence penalty in classification

On Spanish XNER classification, the warm-up confidence penalty improves class separation in target-language representations and supports more robust predictions.

B.1 Effect of confidence penalty in classification

  • The warm-up confidence penalty yields better inter-class separation in Spanish target-language XNER logits, indicating greater robustness to unseen target-language data.The visualization uses t-SNE plots of final logits after training on English NER.Source: §B.1 · p. 13
  • The confidence penalty improves predictions on unseen out-of-distribution target-language data.Source: §B.1 · p. 13
  • The resulting predictions provide a better prior for self-training with pseudo labels.Source: §B.1 · p. 13

B.2 Effect of confidence penalty in loss distribution

In Spanish XNER, confidence-penalty training improves the separation of clean and noisy pseudo-labeled samples and supports more robust distillation across sentence lengths.

B.2 Effect of confidence penalty in loss distribution

  • The confidence penalty produces more scattered per-sentence losses, while longer sentences show more wrong predictions.The scatter plot sorts sentences by length on the x-axis and plots loss on the y-axis.Source: §B.2 · pp. 13–14
  • With confidence-penalty training, confidence-based distillation selects more short sentences while still covering the full range of lengths.The passage notes that selecting only short sentences risks overfitting.Source: §B.2 · p. 14
  • Confidence-penalty training improves pseudo-labeling accuracy and helps distillation filter noisy samples.Without the penalty, many sentences with more than two wrong NER labels have small loss.Source: §B.2 · pp. 13–14

C Details on distillation by clustering

Clustering-based distillation models pseudo-label loss, optionally alongside task-specific features, to separate good samples from noisy ones and select samples by estimated goodness.

C Details on distillation by clustering

  • Unlike confidence-based single-model distillation, clustering can incorporate task-specific signals such as sequence length or fluency in addition to classifier confidence.Longer sequences may be noisier in sequence labeling, and fluency may be estimated with a pretrained conditional language model.Source: C Details on distillation by clustering · p. 14
  • The method fits a mixture model to per-sample loss and other task-specific features using EM, computing loss from model-predicted pseudo labels rather than gold labels.The clustering goal is to separate good from noisy samples based on their goodness.Source: C Details on distillation by clustering · p. 14
  • For loss-only clustering, the method uses a two-component GMM, with extension to features such as sequence length or fluency score described as straightforward.The account introduces the one-dimensional loss and cluster-id variables for the model.Source: C Details on distillation by clustering · pp. 14–15
  • In EM, the E-step computes each cluster's responsibility for a sample, and the M-step optimizes the expected complete-data log likelihood.Source: C Details on distillation by clustering · p. 15
  • At inference, the goodness probability is the posterior probability of the lower-mean component, and η thresholds that probability to select samples.Per-sample loss and model confidence are deterministically related and use the same source of information, though clustering can incorporate additional features.Source: C Details on distillation by clustering · p. 15

D Hyperparameters

The appendix reports training and evaluation settings for XNER and XNLI and reviews related work on contextual transfer, augmentation, and vicinal risk minimization.

D Hyperparameters

  • For XNER and XNLI, the framework is trained by steps, with target-language development data used for cross-lingual model selection.The appendix identifies learning rate and effective batch size as crucial hyperparameters.Source: D Hyperparameters · pp. 15–16
  • The reported setup uses 550M-parameter models and V100 GPUs; runtime varies with augmentation volume and training settings.On average, one million augmentation samples require 0.5–2 days.Source: Table 8 · p. 16
  • Experiments average scores across three differently seeded task models and use a single GPU with float32 precision.Maximum sequence lengths are 280 for XNER and 128 tokens for XNLI.Source: D Hyperparameters · p. 16

E Additional Related Work

  • The related-work discussion traces contextual representations from static embeddings to contextual language models, including multilingual BERT's cross-lingual transfer setting.It describes mBERT as jointly trained on 102 languages with a shared 110K-token subword vocabulary.Source: E Additional Related Work · pp. 16–17
  • The discussion presents vicinal risk minimization as an augmentation-based alternative to empirical risk minimization for generalization beyond training distributions.Source: E Additional Related Work · p. 17
  • Prior augmentation approaches use image transformations or interpolation, while text augmentation remains challenging because text is discrete.Source: E Additional Related Work · p. 17

Related papers

Loading 2004.13240v4…