PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey

Danielle Saunders

arXiv:2104.06951v2cs.CL

TL;DR

NMT can struggle on text from new domains, while fine-tuning depends on relevant data and can risk overfitting or forgetting. This survey organizes robust domain and multi-domain adaptation approaches and concludes that adaptation can improve target-domain performance without retraining from scratch.

  • Problem

    NMT models may struggle with new domains, and fine-tuning requires relevant bilingual data while risking overfitting and forgetting previously learned behavior.

    Source: §1 · p. 1, §4.2 · p. 13

  • Method

    The survey organizes domain-adaptation techniques around data selection or generation, model architecture, parameter adaptation, and inference.

    Source: §1 · p. 3, §5 · p. 14, §5.3 · p. 16

  • Results

    Domain adaptation can achieve good performance on language of interest with limited training data and without retraining from scratch, and may outperform from-scratch training on that domain.

    Source: §10 · p. 42

  • Takeaways & Limitations

    The survey presents adaptation techniques as relevant to other ongoing machine-translation challenges, including low-resource translation, gender handling, and document-level translation.

    Source: §10 · p. 42

    Discrete domain-specific subnetworks may not use multi-domain data optimally when domains overlap or training domains benefit one another.

    Source: §6.4 · p. 24

Abstract

from arXiv · show

The development of deep learning techniques has allowed Neural Machine Translation (NMT) models to become extremely powerful, given sufficient training data and training time. However, systems struggle when translating text from a new domain with a distinct style or vocabulary. Fine-tuning on in-domain data allows good domain adaptation, but requires sufficient relevant bilingual data. Even if this is available, simple fine-tuning can cause overfitting to new data and `catastrophic forgetting' of previously learned behaviour. We concentrate on robust approaches to domain adaptation for NMT, particularly where a system may need to translate across multiple domains. We divide techniques into those revolving around data selection or generation, model architecture, parameter adaptation procedure, and inference procedure. We finally highlight the benefits of domain adaptation and multi-domain adaptation techniques to other lines of NMT research.

1. Introduction

NMT systems can perform poorly outside their training domain, while retraining from scratch for each new domain is often impractical. This survey reviews adaptation methods, their trade-offs and pitfalls, and how adaptation framing applies to other NMT challenges.

1. Introduction

  • Domain adaptation repurposes a trained system for new language types faster than training from scratch and can improve performance on the language of interest.The article notes that training sets often contain millions of sentence pairs and training can take multiple days.Source: §1 · p. 1
  • Domains can overlap, and provenance alone is not an exclusive domain marker because topic and genre may also identify language characteristics.The provenance of unseen test data may be unknown.Source: §1 · p. 2
  • The survey covers domain adaptation and multi-domain translation, emphasizing approaches that avoid forgetting, brittleness, overfitting, and failure beyond selected adaptation sentences.It also examines the benefits of framing other NMT research challenges as adaptation problems.Source: §1 · pp. 2–3
  • The survey emphasizes trade-offs among methods and discusses applying adaptation techniques to NMT challenges not usually treated as domain problems.It contrasts its broader scope with earlier surveys focused on a handful of methods.Source: §1 · p. 3
  • The survey provides an up-to-date overview of popular domain-adaptation techniques for machine translation.It also distinguishes domain adaptation from multi-domain adaptation and describes techniques for each.Source: §1 · p. 3
  • Its taxonomy follows stages of NMT system development so practitioners can identify techniques relevant to the elements they can control.For example, customers of a commercial system may only control adaptation data, while researchers may adjust architecture or adaptation procedures.Source: §1 · p. 3

2. What Constitutes a Domain Adaptation Problem?

The survey treats domains as potentially overlapping combinations of provenance, topic, and genre. It frames adaptation around improving a trained system for identifiable language characteristics without retraining, and multi-domain adaptation around serving multiple domains with one system.

2.1 Exploring the Definition of Domain

  • Domains can be described through provenance, topic, and genre, which capture different characteristics of language and need not identify wholly distinct data.Topic relates to subject and vocabulary; genre includes function, register, syntax, and style.Source: §2.1 · p. 4

2.2 Domain Adaptation and Multi-Domain Adaptation

  • Domain adaptation targets better translation for sentences with identifiable characteristics while avoiding retraining the existing system.Retraining can require substantial time and computation and may be impractical without original data or resources.Source: §2.2 · pp. 4–5
  • Multi-domain adaptation seeks good performance across more than one domain using the same system, with domain membership sometimes known, inferred, or unseen at inference.The survey also treats low-resource translation, gender handling, and document-level translation as possible adaptation problems.Source: §2.2 · p. 5

3. A Brief Overview of Neural Machine Translation

The section outlines NMT representations and model components relevant to domain adaptation, from tokenization and tags to Transformer architecture and depth. These choices shape how language is represented and translated.

3.1 Representing Language for Neural Machine Translation

  • NMT represents each training example as paired source and target token sequences, with vocabulary choices balancing useful information, computational cost, OOVs, and sequence length.Vocabulary and terminology can indicate domain.Source: §3.1 · p. 6

3.1.1 Word Vocabularies; 3.1.2 Subword Vocabularies

  • Word vocabularies can leave rare words out, while subword methods such as BPE represent rare words as sequences of more frequent units.BPE repeatedly merges frequent token pairs; its granularity affects the balance between subword frequency and sequence length.Source: §3.1.1 · p. 6, §3.1.2 · pp. 6–7

3.1.3 Tags

  • Tags can add domain, speaker, formality, linguistic, or terminology information to sentences, but usually require reliable tags across training data.New tags can also be introduced at adaptation time.Source: §3.1.3 · p. 7

3.1.4 Representing Extra-Sentence Context

  • Context may improve lexical cohesion and anaphora resolution in self-attention models, but its effects on overall translation quality are inconsistent.The survey focuses on sentence-level NMT and discusses document-level approaches separately.Source: §3.1.4 · p. 7

3.2.1 Continuous Token Embeddings

  • Continuous embeddings make large vocabularies tractable and may generalize across words with similar contexts, including tokens from the same domain.The section notes typical embedding dimensions of d ≤ 1024.Source: §3.2.1 · p. 8

3.2.2 Transformers: Attention-Based Encoder-Decoder Networks

  • The Transformer is a self-attention encoder-decoder model; its encoder represents the source, while its decoder generates the target using encoder-decoder attention.Its parallelizability and quality improved over earlier RNN-based NMT models, and it remained state-of-the-art at the time of writing.Source: §3.2.2 · pp. 8–9, Figure 1 · p. 8
  • Transformer self-attention relates positions within the same sequence, and its encoder, decoder, or attention network can be made domain-specific or duplicated.This provides architectural components that can be adapted by domain.Source: §3.2.2 · p. 9

3.2.3 Increasing Model Depth

  • Deeper models can improve NMT, but low-resource NMT often performs better with shallower models; residual connections can mitigate training difficulties in deep networks.Residual networks are also needed for some domain-specific adapter subnetworks.Source: §3.2.3 · p. 9

3.3 Training Neural Machine Translation Models

The section describes NMT training through maximum likelihood and cross-entropy, then surveys objective and output-distribution regularization relevant to domain adaptation.

3.3.1 Cross-Entropy Loss

  • Maximum likelihood estimation maximizes the likelihood of training examples, and with one reference label it is equivalent to minimizing cross-entropy loss.The loss uses reference tokens as targets.Source: §3.3.1 · pp. 9–10
  • The survey covers variations on maximum-likelihood loss and adaptation-specific uses of non-MLE objectives.These approaches are discussed as domain adaptation methods.Source: §3.3.1 · p. 10

3.3.2 Objective Function Regularization

  • Objective regularization adds a term to the loss, including auxiliary translation objectives or dropout, and is often used to avoid catastrophic forgetting during adaptation.Examples include coverage, right-to-left translation, future-cost, and target-language-modeling objectives.Source: §3.3.2 · p. 10

3.3.3 Output Distribution Regularization

  • Label smoothing regularizes the output-label distribution to mitigate overfitting, using a smoothed target distribution instead of a single reference label.Smoothing may use a uniform distribution, a teacher model, or a unigram distribution; related distribution regularization is applied to domain adaptation.Source: §3.3.3 · pp. 10–11

3.4 Inference with Neural Machine Translation Models

At inference, NMT generates translations from the source without reference targets, commonly using autoregressive decoding and beam search. Ensembles combine model behavior but carry efficiency costs and assumptions that domain-specific approaches may relax.

3.4 Inference with Neural Machine Translation Models

  • Autoregressive inference generates one token at a time, conditioned on the source and previously generated tokens; exhaustive search over possible partial translations is impractical.The survey focuses on this common inference approach and on practical approximations.Source: §3.4 · p. 11

3.4.1 Beam Search

  • Beam search tracks the top N partial hypotheses by log likelihood, expands them at each step, and stops when they terminate or exceed a maximum length.Greedy search is the special case N = 1.Source: §3.4.1 · p. 11

3.4.2 Ensembling

  • Ensembling can improve performance over individual models by combining their token-expansion decisions, but multiplies softmax computation and may require all models in memory.Many approaches assume a shared generation scheme and fixed model weighting; domain-specific methods relax these assumptions.Source: §3.4.2 · p. 12

4. Fine-Tuning as a Domain Adaptation Baseline, and its Difficulties

Fine-tuning is a straightforward, efficient adaptation baseline, but its success depends on in-domain data and can trade performance across domains or overfit to narrow or irregular data.

4. Fine-Tuning as a Domain Adaptation Baseline, and its Difficulties

  • Fine-tuning continues training a pre-trained model on in-domain data with its original loss, often gaining substantial adaptation improvement at a fraction of full retraining time.The approach usually requires bilingual parallel data for end-to-end NMT.Source: §4 · p. 12
  • The survey presents changes to data, architecture, tuning procedure, and inference as alternatives or complements to simple fine-tuning.Approaches may combine synthetic data, domain-specific subnetworks, and objective regularization.Source: §4 · p. 12, §4.3 · p. 14

4.1 Difficulty: Not Enough In-Domain Data

  • In-domain parallel data may be unavailable or insufficient, motivating data-centric methods that expand the corpus available for tuning.Previous-year shared-task test sets are one possible source of stylistically similar adaptation data.Source: §4.1 · pp. 12–13

4.2 Difficulty: Forgetting Previous Domains

  • Fine-tuning on domain B can improve B while sharply degrading performance on previously learned domain A, problematic when test domains vary or training order matters.Some forgetting may be acceptable when adapting only to a small amount of highly specific data.Source: §4.2 · p. 13

4.3 Difficulty: Overfitting to New Domains

  • Small or repetitive tuning sets can cause overfitting, so slight differences between the test text and adaptation domain may lead to poor translations.High learning rates or too many epochs can worsen degradation in per-sentence adaptation, and even small amounts of systematically misaligned data can trigger undesirable outputs.Source: §4.3 · pp. 13–14

5. Data-Centric Adaptation Methods

Data-centric adaptation selects or generates relevant in-domain data, including from monolingual sources, and can also filter, perturb, or synthesize examples. The survey covers these options alongside their practical benefits and risks.

5.1 Selecting Additional Natural Data for Adaptation

  • Diversity-aware rarity or Feature Decay selection outperforms TF-IDF; unsupervised domain clusters match or outperform adaptation on correctly labelled data.The cited comparisons favor selection that accounts for diversity and clustering rather than relying only on corpus labels.Source: §5.1 · pp. 15–16
  • Natural-data retrieval uses token overlap, sentence representations, or external relevance scores; imperfect per-input matching can overfit, while selected examples can improve terminology use.The survey also describes n-gram and fuzzy matching, which sometimes performs better than fuzzy matching for NMT adaptation.Source: §5.1 · pp. 14–15

5.2 Filtering Existing Natural Data

  • Filtering existing corpora can remove unrepresentative or noisy examples, but filtering must preserve adequate training coverage.The survey describes feature-space novelty filtering and preserving n-gram coverage.Source: §5.2 · p. 16

5.3 Generating Synthetic Bilingual Adaptation Data from Monolingual Data

  • When in-domain parallel data is unavailable, forward or back translation of in-domain monolingual text can create aligned, partially synthetic adaptation data.Back translation uses target-side monolingual text to generate synthetic source sentences; domain mismatch can make those sources poor, while domain repair or filtering can improve adaptation.Source: §5.3 · pp. 16–17, §5.3.1 · p. 17

5.3.1 Back Translation

  • Back-translated data can constitute a translationese domain, so apparent gains on back-translated test sets may not transfer to natural test sets.Interposing authentic natural parallel data during tuning is proposed as a mitigation when such data is available.Source: §5.3.1 · pp. 17–18

5.3.2 Forward Translation

  • Forward translation can use the actual test source as a few-shot seed, but adaptation may reinforce its translation errors or produce translationese.Self-learning avoids needing an additional NMT model; the outcome of tuning on forward translations depends heavily on the data-generating system's quality.Source: §5.3.2 · p. 18

5.4 Artificially Noising and Simplifying Natural Data

  • Noising and simplification create variants from small in-domain sets, potentially reducing overfitting; noising can improve robustness, while fragmentation may limit performance on longer sentences.Simplifying sources can also be applied to test inputs, and simplifying targets can specify output-language complexity.Source: §5.4 · pp. 18–19

5.5 Synthetic Bilingual Adaptation Data

  • Purely synthetic data can support adaptation when in-domain natural monolingual data or a sufficiently strong translation model is unavailable; lexicons can complement back translation.The survey reports lexicons aid isolated-word translation, while back-translated data aids overall in-domain fluency.Source: §5.5 · pp. 19–20

6. Architecture-Centric Adaptation

Architecture-centric approaches add domain signals or parameters to NMT systems, sometimes after pre-training. The survey considers their control mechanisms, parameter costs, and suitability for multi-domain translation.

6.1 Domain Labels and Control

  • Source tags or domain features can improve in-domain translation over fine-tuning alone, with embedded features slightly outperforming discrete tags; multiple tags can represent overlap.Target-side tags avoid requiring explicit domain labels for new user-generated sources, but make control of the desired output domain more complicated.Source: §6.1 · p. 21, Figure 2 · p. 21

6.2 Domain-Specific Subnetworks

  • Domain-specific subnetworks can specialize embeddings, representations, or encoder-decoder components, but duplicating encoders or decoders quickly increases model size and computational and energy costs.Some approaches use domain classifiers or shared components alongside domain-specific components.Source: §6.2 · p. 22

6.3 Lightweight Added Parameters

  • Adapters add a small domain-specific parameter set in encoder and decoder layers while freezing the original model, avoiding forgetting; they can outperform full fine-tuning on a noisy domain.Adapter size and the number of tuning steps require careful selection.Source: §6.3 · p. 23, Figure 3 · p. 23

6.4 Architecture-Centric Multi-Domain Adaptation

  • Multi-domain systems with shared parameters are most effective when domains are close or overlapping, but generally underperform single-domain fine-tuning and can be hampered by wrong or unknown tags.Keeping original parameters unchanged can avoid forgetting-related degradation by reverting to those parameters.Source: §6.4 · pp. 23–24

7. Training Schemes for Adaptation

Simple fine-tuning can cause forgetting of older domains and overfitting to new ones. The survey reviews training schemes intended to mitigate these problems.

7. Training Schemes for Adaptation

  • Simple fine-tuning can cause catastrophic forgetting of old domains and overfitting to new domains.The surveyed alternatives include regularized training, curriculum learning, instance weighting, and non-MLE training.Source: §7 · p. 24
  • The section considers training schemes designed to address forgetting and overfitting during domain adaptation.These schemes include regularization, curriculum learning, instance weighting, and non-MLE training.Source: §7 · p. 24
  • Fine-tuning adapts the model using the same MLE objective function used during pre-training.The survey presents it as a straightforward approach before discussing alternatives.Source: §7 · p. 24

7.1 Objective Function Regularization

Objective-function regularization approaches seek to preserve pre-training behavior while adapting to new domains. The section covers freezing parameters, regularizing parameter changes, and knowledge distillation.

7.1.1 Freezing Parameters

  • Freezing selected parameters can reduce forgetting, though some approaches trade off new-domain performance; other parameter-efficient schemes achieve performance similar to full fine-tuning.Sparse-area methods also support adaptation to multiple domains while significantly reducing forgetting on the generic domain, and retaining only a small parameter set can save memory for many personalized models.Source: §7.1.1 · pp. 24–25

7.1.2 Regularizing Parameters

  • L2 and EWC regularization constrain changes from pre-trained parameters, with EWC scaling the constraint by parameter importance; EWC mitigates forgetting, including during sequential multi-domain adaptation.The regularization weight can balance relative importance of old and new domains; parameter regularization has higher computational load than freezing but may behave more predictably across settings.Source: §7.1.2 · pp. 25–26

7.1.3 Knowledge Distillation

  • Knowledge distillation regularizes adaptation by encouraging the in-domain model to match a teacher's output distribution, including in multi-domain and continual-learning settings.It requires two models to operate on the data, making parameter regularization potentially more practical when model sizes remain the same.Source: §7.1.3 · pp. 26–27

7.2 Curriculum Learning

Curriculum learning ranks training examples and controls their presentation during training or fine-tuning, often by difficulty. For domain adaptation, curricula can gradually shift toward in-domain data or retain generic examples to reduce forgetting and overfitting.

7.2 Curriculum Learning

  • Competence-based curricula use an example only when its difficulty falls within the model’s current competence.Perceived difficulty may also depend on model competence.Source: Figure 5 · p. 27, §7.2 · p. 27
  • Curriculum learning ranks examples to guide their training order, commonly presenting easier examples before more complex ones.Difficulty can reflect sentence length, linguistic complexity, or word rarity; both easy-first and hardest-first schedules have been reported to improve speed.Source: §7.2 · p. 27
  • Domain-adaptation curricula can gradually transition from generic toward target-domain data, using similarity to target or generic data as a difficulty signal.Examples include rearranging pseudo-in-domain data by similarity and emphasizing more in-domain data over successive epochs.Source: §7.2 · p. 28
  • Multi-domain curricula include model stacking across increasingly close domains, difficulty scores based on multidimensional domain features, and uncertainty-guided sampling across corpora.Source: §7.2 · p. 28
  • Mixed fine-tuning retains domain-tagged generic data during adaptation and is reported to reduce catastrophic forgetting versus simple fine-tuning.Multi-stage mixed fine-tuning improves over single-stage mixed fine-tuning; replay buffers can also help avoid forgetting without hard domain boundaries.Source: §7.2 · pp. 28–29

7.3 Instance Weighting

Instance weighting changes the loss contribution of each training example according to its relevance to the target domain. It effectively simulates over- or under-sampling while usually maintaining a constant bias toward in-domain data during training.

7.3 Instance Weighting

  • Instance weighting assigns each source-target example a weight in the cross-entropy loss to represent its target-domain relevance.Source: §7.3 · p. 29
  • Higher weights emphasize examples for target-domain learning, while low or zero weights effectively ignore them during tuning.This simulates over- or under-sampling without reordering data.Source: §7.3 · p. 29
  • Weights may be assigned by domain label or sentence-level measures such as n-gram similarity, cross-entropy difference, domain classification, or embedding similarity.Learning rates and adaptation epochs can likewise be adjusted according to domain similarity.Source: §7.3 · p. 29
  • Unlike curricula, domain-adaptation instance weights usually remain fixed during training, continuously biasing learning toward in-domain examples.Both approaches emphasize selected examples and can affect fitting, forgetting, or overfitting.Source: §7.3 · p. 29

7.4 Non-MLE Training

The section reviews minimum risk training and meta-learning as alternatives or complements to maximum-likelihood fine-tuning for domain adaptation. The surveyed evidence describes settings where each can help, alongside conditions that limit or shape those benefits.

7.4.1 Minimum Risk Training

  • MRT fine-tunes an already converged maximum-likelihood model and optimizes expected sequence-level risk using sampled hypotheses and a sentence-level BLEU-based cost.Samples may be generated with beam search or autoregressive sampling.Source: §7.4.1 · p. 30
  • MRT may reduce exposure bias by optimizing model samples rather than tuning-set targets, with reported benefits for domain-mismatched tests and noisy or unreliable adaptation data.The cited studies report mitigation in those settings; the passage frames the broader effect as an indication rather than a guarantee.Source: §7.4.1 · p. 30

7.4.2 Meta-Learning

  • Meta-learning inserts a training phase between pre-training and target-domain fine-tuning to prepare parameters for rapid adaptation across domains with few examples.Each meta-training task uses temporary updates on training examples and evaluates the updated model on task test examples.Source: §7.4.2 · pp. 30–31
  • Adapter-based meta-learning outperforms fine-tuning on the same small samples when many simulated domains are used, while fewer larger domains may overfit and full-parameter meta-training can forget the original domain.Dynamic learning-rate adjustment particularly helps when target domains differ greatly in size.Source: §7.4.2 · p. 31
  • Meta-learning has shown strong improvements over simple and mixed fine-tuning in unsupervised adaptation using monolingual data, back-translation, and language-modeling objectives.Source: §7.4.2 · p. 31
  • Reported meta-learning gains over fine-tuning are strongest for domain variations whose test data is well represented by tuning data; infrequent words are an exception in one study.Curriculum sampling similar domains before divergent ones yielded small improvements on domains seen during meta-learning.Source: §7.4.2 · p. 31

8. Inference Schemes for Adaptation

Inference-time adaptation can address unknown-domain inputs through adaptive ensembles, retrieval, rescoring, constraints, or terminology cues. These approaches can change domain-specific behavior without always requiring a domain-specific model or model retraining.

8. Inference Schemes for Adaptation

  • For unknown-domain inputs, inference schemes combine separately adapted models or use rescoring and terminology techniques to choose or adjust outputs.Some terminology approaches bypass domain-specific NMT tuning altogether.Source: §8 · p. 31

8.1.1 Domain Adaptive Ensembling

  • Domain-adaptive ensembling conditions model weights on the source sentence or partial hypothesis instead of manually tuning weights for a known test domain.Bayesian interpolation updates weights from task priors and the task posterior as decoding proceeds.Source: §8.1 · p. 32, §8.1.1 · pp. 32–33
  • Source-conditioned adaptive ensembles can handle unknown domains that vary within an input, and are most beneficial when component models specialize in non-overlapping domains.Similar multi-domain models may perform well with uniform weighting.Source: §8.1.1 · p. 33, Figure 6 · p. 33

8.1.2 Retrieval-Based Ensembling

  • Retrieval-based kNN ensembling interpolates NMT predictions with a distribution over target tokens retrieved from a domain-specific datastore; adaptive k and datastore adaptation have improved in-domain translation without another NMT model.Source: §8.1.2 · pp. 33–34

8.2 Constrained Inference and Rescoring

  • Multi-pass rescoring adjusts initial hypotheses and can avoid holding multiple models in memory simultaneously; constrained inference instead restricts NMT decoding to a lattice of permitted translations.Domain-specific lattices can improve adequacy, while permitted-change constraints have been reported to mitigate forgetting in gender adaptation.Source: §8.2 · pp. 34–35, Figure 7 · p. 34

8.3.1 Terminology Tagging

  • Terminology tagging injects or replaces specified terms before or after translation, but post-inference injection depends on an effective alignment model.Tagging may be less effective at introducing terminology the model already handles poorly.Source: §8.3.1 · pp. 35–36
  • Many terminology-injection approaches can support unseen-domain terms even when the baseline model has not seen or produced them.Source: §8.3.1 · p. 36

8.3.2 In-Domain Priming

  • In-domain priming supplies similar sentence pairs or translations as inference-time cues, adapting domain specificity without domain-specific retraining.The surveyed approaches range from source-side cues to forcing an in-domain target-language cue before translation.Source: §8.3.2 · p. 36

9. Case Studies for (Multi-)Domain Adaptation

The case studies show how domain-adaptation framing applies beyond conventional topic or genre adaptation, while illustrating approaches across low-resource translation, gender handling, and document-level translation. The survey presents these examples to demonstrate the framing’s wider applicability, not as comprehensive surveys of those fields.

9. Case Studies for (Multi-)Domain Adaptation

  • The case studies apply domain-adaptation techniques to low-resource translation, gender handling, and document-level translation, while offering guidance rather than comprehensive surveys of these fields.The authors assess whether each problem fits single- or multi-domain adaptation and place selected approaches in the context of broader research.Source: §9 · pp. 36–37

9.1.1 Can Low Resource Language Translation Be Treated as a Domain Adaptation Problem?

  • Low-resource translation can be treated as single-domain adaptation, while multilingual translation may pose a multi-domain problem when performance across language pairs must be maintained.The low-resource language pair can be treated as its own domain; multilingual NMT seeks good performance across distinct text sets without compromising any of them.Source: §9.1 · p. 37, §9.1.1 · p. 38

9.1.2 Improving Low Resource Language Translation by Adaptation

  • Low-resource approaches include tuning an existing model, creating synthetic or pseudo-parallel data, and using multilingual models in which language pairs can share representations or components.The survey notes that forgetting the original model’s abilities may be less concerning when users can pre-determine which model translates each language pair.Source: §9.1.2 · pp. 38–39

9.2.1 Can Gender Bias be Treated as a Domain Adaptation Problem?

  • Gender errors can reflect social stereotypes, and the survey frames gender handling as controlling an implicitly inferred ‘gender’ domain, including for ambiguous source sentences.The source may leave gender ambiguous even as the model associates gendered terms with masculine or feminine target-language grammar.Source: §9.2 · pp. 39–40, §9.2.1 · p. 40

9.2.2 Mitigating the Effects of Gender Bias By Adaptation

  • Gender adaptation approaches include tuning on pronoun examples or gender-balanced data, with methods such as discriminative losses, EWC, constrained retranslation, and general-data mixing to address overfitting or forgetting.The survey also describes gender tags introduced during adaptation and prompts that control translation gender at inference time without model changes.Source: §9.2.2 · pp. 40–41

9.3 Document-Level Translation as a Domain Adaptation Problem

  • Document-level translation includes both document-specific terminology or style and general use of extra-sentential context, and incorporating context into an existing model often involves adaptation-adjacent methods.Document-aligned bilingual datasets are usually much smaller than sentence-aligned datasets.Source: §9.3 · p. 41

9.3.1 Can Document NMT Be Treated as a Domain Adaptation Problem?

  • Document NMT fits domain adaptation when improving a document without retraining from scratch; using one model across multiple documents can make it a multi-domain problem for efficiency.A document or document-aligned dataset is very small compared with typical generic training data, while the survey notes that often only one document is of interest at a time.Source: §9.3.1 · pp. 41–42

9.3.2 Improving Document-Level Translation By Adaptation

  • Document adaptation can use the target document as seed text, encode document information in tags, adjust model architecture for context, or incorporate document context at inference.Examples include extracting a lexicon of novel test-document words, inline document-content tags, context-aware parameters, and document-context reranking.Source: §9.3.2 · p. 42

10. Conclusions and Future Directions

The survey describes domain adaptation as a way to improve NMT for language of interest with limited data and without retraining from scratch, and discusses its use beyond conventional domains. It identifies fine-grained and unsupervised adaptation, efficiency, and intentional forgetting as future directions, while noting that whether unlearning improves NMT remains open.

Conclusions

  • Domain adaptation can improve NMT for language of interest with limited training data and without retraining from scratch, and may outperform from-scratch training on a given domain.The survey also frames ongoing NMT challenges as potential adaptation problems and notes existing use of adaptation for pseudo-domains beyond provenance, topic, or genre.Source: §10 · pp. 42–43

Future Directions

  • The survey highlights extremely fine-grained adaptation, unsupervised adaptation, efficiency, and intentional forgetting as areas for future NMT research.It recommends considering abilities to maintain or abandon, not only individual domains of immediate interest.Source: §10 · pp. 43–44

Extremely Fine-Grained Adaptation

  • Fine-grained adaptation may improve lexical choice but can be slow and risk overfitting to irrelevant data, motivating work on relevant-data extraction or per-sentence adaptation without model changes.Examples include adaptation for individual users or sentences.Source: §10 · p. 43

Unsupervised Adaptation

  • Unsupervised adaptation without parallel data can use in-domain monolingual data to produce pseudo-parallel data, but extracting data and producing in-domain pseudo-parallel examples remain open questions.The survey predicts growing interest as sufficient in-domain parallel data becomes increasingly scarce for smaller domains.Source: §10 · p. 43

Efficiency

  • Efficiency matters as fine-grained adaptation and larger models increase the cost of training and storing many models, prompting interest in fewer tuning steps or adapting less than the entire model.The survey notes that aggregate personalization costs may outweigh reduced per-model training time.Source: §10 · p. 43

Intentional Forgetting

  • Intentional forgetting may target obsolete domains, outdated translations, demographic biases, or specific examples, but whether unlearning can improve NMT remains an open question.The survey notes potential value in redacting translation examples in privacy-sensitive settings such as biomedical translation.Source: §10 · pp. 43–44

Related papers

Loading 2104.06951v2…