PaperlayerView exact revision on arXiv

Source-linked AI summary

Curriculum Learning for Domain Adaptation in Neural Machine Translation

Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul McNamee, Marine Carpuat, Kevin Duh

arXiv:1905.05816v1cs.CL

TL;DR

NMT adaptation is difficult when domains differ and in-domain parallel data is scarce. This paper ranks unlabeled-domain samples by similarity and schedules them through curriculum learning; across four tasks, the approach improves over standard continued training by up to 3.22 BLEU.

  • Problem

    NMT performance often drops when training and test domains differ and in-domain data is scarce, while selected samples can vary in similarity.

    Source: §1 · p. 1

  • Method

    The approach ranks unlabeled-domain samples by similarity to in-domain data and uses a probabilistic curriculum to present more similar samples earlier and more frequently.

    Source: §1 · p. 1, §6 · p. 8

  • Results

    Across four tasks, curriculum learning improves over standard continued training by up to 3.22 BLEU points.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The curriculum models can take better advantage of distant and noisy data.

    Source: §6 · p. 9

    Evaluation focuses on in-domain test sets; performance on general-domain test sets and potential catastrophic forgetting remain unmeasured.

    Source: §6 · p. 9

Abstract

from arXiv · show

We introduce a curriculum learning approach to adapt generic neural machine translation models to a specific domain. Samples are grouped by their similarities to the domain of interest and each group is fed to the training algorithm with a particular schedule. This approach is simple to implement on top of any neural framework or architecture, and consistently outperforms both unadapted and adapted baselines in experiments with two distinct domains and two language pairs.

1 Introduction

The paper addresses the difficulty of adapting NMT when in-domain parallel data is scarce and selected data vary in similarity. It orders samples by domain similarity and reports consistent gains over standard continued training.

1 Introduction

  • NMT performance often drops when training and test domains differ and in-domain training data is scarce.High-quality parallel data is unavailable for some domains.Source: §1 · p. 1
  • Data selection can identify bitext similar to in-domain data, which can be combined with in-domain bitext for continued training.This increases the in-domain training set with pseudo in-domain samples.Source: §1 · p. 1
  • Continued training lacks a clear threshold for deciding whether selected samples are sufficiently similar, leaving the model to train on samples with varied similarity.The paper presents its approach as addressing this challenge.Source: §1 · p. 1
  • The method orders training samples by similarity, presenting more similar examples earlier and more frequently.It applies curriculum learning to domain adaptation.Source: §1 · p. 1
  • Up to 3.22 BLEU improvement over standard continued training was reported across TED Talks and patent abstracts for German-English and Russian-English.Experiments used Moore-Lewis and cynical data selection.Source: §1 · p. 1

2 Curriculum Learning for Adaptation

The approach scores unlabeled-domain samples for similarity to in-domain data, then uses a probabilistic curriculum to introduce increasingly less similar samples while retaining randomized training.

2 Curriculum Learning for Adaptation

  • Curriculum learning for adaptation estimates sample difficulty from distance to in-domain data and modifies sampling probabilistically.The strategy retains standard practices such as bucketing and mini-batching.Source: §2 · p. 2

2.1 Domain Similarity Scoring

  • Moore-Lewis scores sentences using cross-entropy difference, with lower values indicating greater similarity to in-domain data.The two cross-entropies come from language models trained on in-domain data and a random unlabeled-domain sample of roughly equal size.Source: §2.1 · p. 2
  • Cynical data selection iteratively adds the sentence expected to most decrease cross-entropy between the selected corpus and in-domain data.The selected corpus is constructed to approximately model the in-domain corpus.Source: §2.1 · p. 2

2.2 Curriculum Learning Training Strategy

  • The probabilistic curriculum samples easier examples more often in earlier phases, unlike a deterministic fixed-order curriculum.Randomization allows minibatches and multiple epochs, which neural methods benefit from.Source: §2.2 · p. 2
  • Samples are ranked and divided into similarity-based shards, then training phases add shards from easiest to less easy.Shards are shuffled, and samples are bucketed by length before batches are drawn randomly.Source: §2.2 · p. 2

3 Experiments and Results

The experiments test curriculum continued training across German–English and Russian–English TED and patent tasks, using selected Paracrawl alongside limited in-domain data. Curriculum methods outperform standard continued training, with results varying by domain and data-selection method.

3.1 Data and Setup

  • The evaluation uses TED and patent domains, samples 15k parallel sentences as in-domain bitext, and includes up to 4096k German–English or 2048k Russian–English Paracrawl pairs, filtered at Zipporah threshold 1.Source: §3.1 · p. 3

3.3 Results

  • 1.94 BLEU is lost on TED(de) and 2.43 BLEU on patent(de) when standard continued training increases CDS data from 64k to 4096k; curriculum models remain at least as good as their initial model.For patent(ru), CL ML rises from 32.41 to 34.18 as Paracrawl data increases.Source: Figure 2 · p. 4, §3.3 · p. 5

3.2 Experimental Comparison

  • The comparison tests random-order and curriculum schedules with Moore–Lewis and Cynical Data Selection, alongside generic, in-domain, in-domain continued-training, and random-subset references.Both schedules use selected Paracrawl concatenated with in-domain data.Source: §3.2 · p. 4

3.3 Results

  • Across the four tasks, curriculum methods improve over standard continued-training baselines by up to 3.22 BLEU, while standard training with Paracrawl underperforms IN CT.The table reports BLEU for unadapted and adapted models, with Δ denoting the improvement of curriculum learning over standard training.Source: Table 1 · p. 4, §3.3 · pp. 4–5
  • On patent, CDS raises BLEU from 2.18 to 4.05 for standard models and from 2.20 to 4.25 for curriculum models, whereas Moore–Lewis slightly outperforms CDS on TED.Patent is more distant from Paracrawl than TED; the authors suggest testing different selection methods on new tasks.Source: §3.3 · p. 5

4 Analysis

The analysis finds that curriculum learning performs well across tested schedules and learning curves, while reducing lexical errors; selected data differ by method and domain.

4.1 Comparison of Curriculum Strategies

  • CL outperformed CL reverse and CL noshuffle in 5 of 7 cases and CL scrambled in 6 of 7 cases.The authors suggest that presenting the most in-domain-like examples first and using a probabilistic curriculum are beneficial.Source: Figure 3 · p. 5, §4.1 · p. 5

4.2 Learning Curves

  • Curriculum models started at high performance, oscillated less than IN CT, and gradually achieved the highest performance.IN CT heavily oscillated without reaching its initial performance, while standard ML and standard random sampling did not improve final performance over IN CT.Source: Figure 4 · p. 5, §4.2 · p. 6

4.4 Characteristics of Selected Data

  • Top-64k ML and CDS selections overlapped by 3.69% for TED and 8.43% for patent; even at top 4096k, 46.25% and 65.40% differed, respectively.These overlap measurements are for German-English Paracrawl selections.Source: §4.4 · p. 6
  • CDS subsets covered in-domain vocabulary better, while ML better matched the in-domain unigram distribution; patent data were farther from Paracrawl than TED.CDS dominated ML for distant patent data, while ML could do slightly better for the less-distant TED domain.Source: §4.4 · pp. 6, 8, Figure 8 · p. 8

4.3 Impact of Curriculum Learning on Lexical Choice: S4 Analysis

  • With 4096k CDS-selected Paracrawl, curriculum continued training translated 554 more German-English TED words correctly than standard continued training, mainly correcting 1.75% of SCORE errors.SEEN and SENSE errors also fell by 0.02% and 0.026%, respectively, but curriculum learning did not help much on SEEN errors.Source: §4.3 · p. 6, Figure 5 · p. 7

4.4 Characteristics of Selected Data

  • Moore-Lewis scores favored longer sentences, whereas CDS selected sentences closer to the in-domain average length.The near-linear TED ML curve may be a side effect of sentence-length normalization.Source: §4.4 · p. 6, Figure 6 · p. 7

5 Related Work

Related work frames curriculum learning as ordering examples by difficulty and distinguishes this domain-adaptation use from prior fine-tuning schedules. The approach also connects data selection with instance weighting and combines data- and model-centric adaptation.

5 Related Work

  • For domain adaptation, sample difficulty is measured by distance from in-domain data; this work targets translation quality rather than convergence speed.Prior curriculum-learning work discusses guiding sample order from easier to harder examples, while NMT difficulty has no single criterion (Zhang et al., 2018).Source: §5 · p. 8
  • The workflow is a hybrid of data-centric and model-centric adaptation because it uses additional unlabeled-domain data and modifies continued training.This characterization follows the taxonomy of Chu and Wang (2018).Source: §5 · p. 8
  • Curriculum presentation is related to instance weighting because examples are dynamically up- or down-weighted across training phases; prior NMT methods typically modify the objective function.Gradual fine-tuning instead starts with all data and removes less-similar sentences over time, which may waste computation when much Paracrawl data is irrelevant.Source: §5 · p. 8

6 Conclusion

The conclusion presents a probabilistic curriculum for domain adaptation and reports gains over standard continued training across four tasks. It attributes the gains to better lexical scoring and to training with diverse, similarity-ranked data.

6 Conclusion

  • The method ranks unlabeled-domain samples by similarity to in-domain data, then uses a probabilistic curriculum so more similar samples appear earlier and more often.The conclusion describes the approach as a way to adapt neural machine translation models to new domains.Source: §6 · p. 8
  • Up to 3.22 BLEU points over standard continued training, the curriculum models improve on four tasks.The reported S4 analysis attributes the improvement mainly to better scoring of words with a new SENSE or a different SCORE distribution in the new domain.Source: §6 · p. 9
  • The curriculum accommodates varied sample similarities, retains previously visited shards when adding new ones, and builds on continued training and common NMT practices.The paper says these features help exploit diverse webcrawl data and avoid forgetting what the model just learned.Source: §6 · p. 9
  • The study focuses on in-domain testing; evaluating general-domain performance and relative catastrophic forgetting is left for future work.Source: §6 · p. 9

A In-domain Data Details

The appendix describes experiments using 15,000 in-domain samples per dataset to examine low-resource adaptation, and reports that the method also scales to more in-domain data with smaller gains.

A In-domain Data Details

  • 15,000 in-domain samples per dataset were used to evaluate an extreme low-resource domain-adaptation setting.The amount is a relatively small percentage of the original corpora.Source: A In-domain Data Details · p. 12
  • Pilot experiments found that curriculum learning still outperformed standard training with more in-domain data, but by less.Source: A In-domain Data Details · p. 12
  • With enough in-domain data, continued training on that data alone can perform well, reducing the need to add unlabeled-domain data.Source: A In-domain Data Details · p. 12

B Data Sharding

The sharding experiment finds that performance first improves and then declines as the number of shards grows; at either extreme, the curriculum reduces to standard continued training.

B Data Sharding

  • Putting all data into one shard is an extreme case equivalent to standard continued training.Source: B Data Sharding · p. 12
  • Using as many shards as samples is also equivalent to standard continued training.Source: B Data Sharding · p. 12
  • Performance first improves, then degrades as the number of shards increases.Figure 9 examines shard counts for a curriculum model trained with German-English data augmented by 1024k Paracrawl data.Source: Figure 9 · p. 12, B Data Sharding · p. 12

C S4 Error Analysis Algorithm

Algorithm 1 counts S4 word-translation error categories using source sentences and aligned words from test references, system outputs, and training references.

C S4 Error Analysis Algorithm

D Bilingual Criterion

The section compares source-side, target-side, and bilingual similarity scores for curriculum learning. Moore–Lewis results vary across curriculum variants, while cynical data selection favors source-side scores.

D Bilingual Criterion

  • For Moore–Lewis selection, bilingual scores sum the scores from both sides; target-only scores were also tested for both selection methods.The comparison used source-side, target-side, and bilingual scores.Source: D Bilingual Criterion · pp. 12–13
  • For cynical data selection, curriculum models preferred scores from the source side of the corpus.Moore–Lewis curriculum results did not follow a consistent ordering: CL en had strong scores, CL de sometimes outperformed it, and CL bi could be better or worse than both.Source: D Bilingual Criterion · p. 13
  • 39.37 BLEU at 4096k Moore–Lewis-selected sentences made CL bi the best German-English TED test result reported.This was the best score among the reported results for German-English TED.Source: Figure 10 · p. 13, D Bilingual Criterion · p. 13

E Perplexity Selection

Perplexity-based selection did not identify the data cutoffs associated with high BLEU scores. The authors recommend 1024k ranked unlabeled-domain examples as a first choice when computational resources are limited.

E Perplexity Selection

  • The preferred data size could only be identified from decoding results after training models on different amounts of Paracrawl data.The section examines perplexity selection as a way to determine the selected-data cutoff beforehand and potentially save computation.Source: E Perplexity Selection · p. 13
  • Perplexity-selected thresholds were inconsistent with the cutoffs that achieved high BLEU scores, so perplexity selection may not identify the optimal unlabeled-data amount for NMT.Perplexity was evaluated on the in-domain set using language models trained on selected subsets of different sizes.Source: E Perplexity Selection · p. 13, Figure 11 · p. 13
  • 1024k is the recommended first cutoff choice for curriculum-based NMT domain adaptation when computational resources are limited.The recommendation concerns ranked unlabeled-domain data.Source: E Perplexity Selection · p. 13
Loading 1905.05816v1…