Source-linked AI summary
Simple, Scalable Adaptation for Neural Machine Translation
Ankur Bapna, Naveen Arivazhagan, Orhan Firat
TL;DR
The paper addresses the parameter and maintenance costs of fine-tuning separate NMT models for different languages and domains. It inserts small task-specific adapters into a frozen pre-trained model and evaluates them for domain adaptation and massively multilingual NMT. Adapters achieve comparable or better results than full fine-tuning or bilingual baselines across the evaluated settings, while supporting adaptation within a single model.
Problem
Full fine-tuning for new languages and domains requires training and maintaining a separate model for every task, creating a parameter-inefficient adaptation problem.
Method
The method freezes a pre-trained NMT model, inserts lightweight task-specific adapter layers, and fine-tunes those adapters for multiple domains and languages.
Results
Adapters yield comparable or better results than full fine-tuning or bilingual baselines across domain-adaptation and multilingual NMT evaluations, without hyper-parameter tuning across varying dataset sizes and model capacities.
Takeaways & Limitations
A single model with globally shared parameters and small task-specific layers can be trained and adapted for many languages and domains.
Takeaways & Limitations
Multilingual adapters still show a minor regression for high-resource languages translating into English relative to bilingual baselines.
Abstract
from arXiv · showhide
Fine-tuning pre-trained Neural Machine Translation (NMT) models is the dominant approach for adapting to new languages and domains. However, fine-tuning requires adapting and maintaining a separate model for each target task. We propose a simple yet efficient approach for adaptation in NMT. Our proposed approach consists of injecting tiny task specific adapter layers into a pre-trained model. These lightweight adapters, with just a small fraction of the original model size, adapt the model to multiple individual tasks simultaneously. We evaluate our approach on two tasks: (i) Domain Adaptation and (ii) Massively Multilingual NMT. Experiments on domain adaptation demonstrate that our proposed approach is on par with full fine-tuning on various domains, dataset sizes and model capacities. On a massively multilingual dataset of 103 languages, our adaptation approach bridges the gap between individual bilingual models and one massively multilingual model for most language pairs, paving the way towards universal machine translation.
1 Introduction
The paper targets poor NMT performance on out-of-domain data and low-resource languages, where full fine-tuning requires separate, costly models for each task. It proposes lightweight task-specific adapters to adapt one pre-trained model across domains and languages.
- NMT performance remains poor on out-of-domain data and low-resource languages despite major gains for high-resource sentence-level translation.
- Full fine-tuning requires training and maintaining a separate model for every language and domain, becoming parameter-inefficient as tasks and model capacities grow.
- Adapter capacity can be adjusted to target-task complexity and corpus size, while separating task and original parameters enables simultaneous adaptation without catastrophic interference.
- The proposed approach inserts lightweight task-specific adapter layers into a pre-trained network and fine-tunes only those layers for each adaptation corpus.
- Domain-adaptation experiments show adapters match full fine-tuning at a fraction of per-domain parameter cost, while massively multilingual training covers 103 languages and improves low-resource transfer without huge high-resource regression.
- The results support progress toward a flexible universal translation model spanning languages and domains.
2 Related Work
Prior work reduces adaptation cost through domain-specific parameters or selective fine-tuning, but these approaches impose capacity or scalability constraints. Residual adapter formulations provide a related basis for multi-task adaptation, while this work adds tunable capacity.
- Existing adaptation methods include domain-specific output biases, selective component fine-tuning, and domain-specific gates controlling hidden-unit contributions.
- Domain-specific gates provide limited per-domain capacity that does not scale well when substantial domain-specific data is available.
- Residual adapters originated in vision, while later work adapted BERT to multiple tasks simultaneously using a different adapter formulation.
- The proposed NMT formulation is motivated by prior adapters but differs in several respects, including its tunable capacity based on target-domain requirements.
3 Approach
The approach freezes a converged pre-trained NMT model and inserts small trainable adapters after encoder and decoder layers. Each adapter normalizes inputs, projects through a tunable hidden dimension, returns to the original dimension, and uses a residual connection.
- The method has two phases: train a generic base model, then adapt it to new tasks with added small network modules.
- After pre-training convergence, all base-model parameters are frozen and task-specific adapters are inserted after every encoder and decoder layer.
- Adapter Modules: Each adapter is a single-hidden-layer feed-forward network with a nonlinear activation between two projection layers.
- Adapter Modules: The adapter’s inner dimension is its only tunable capacity control, allowing adjustment to target-task complexity and adaptation-corpus size.
- Formulation: Adapters apply parameterized input normalization before projecting to dimension b, which may form a bottleneck or exceed the input dimension.
- Formulation: The inner representation is projected back to input dimension d and combined with a residual connection, allowing the module to represent a no-op.
- The self-contained adapter can be inserted between network layers without disrupting the original operation.
4 Domain Adaptation
The study evaluates lightweight adapters for English–French domain adaptation using IWSLT and JRC-Acquis, comparing them with full fine-tuning and LHUC. Results examine parameter capacity, training-corpus size, and adaptation dynamics.
- 4.1 Dataset: The experiments adapt WMT-trained English–French models to IWSLT and JRC-Acquis using domain-specific adapters, full fine-tuning, and LHUC.The WMT corpus contains 36M sentence pairs; IWSLT contains 237k and JRC-Acquis 797k training pairs.
- 4.4 Results and Analysis: Adapters match full fine-tuning on IWSLT within error margins while adding less than 0.11% of the original model parameters.Adapters train newly added parameters without updating existing model parameters.
- 4.4 Results and Analysis: On JRC-Acquis, adapters recover around 90% of fine-tuning improvements while adding around 13.5% additional parameters.This result is achieved without updating any existing parameters.
- 4.4 Results and Analysis: Adapter capacity trades off against adaptation performance: IWSLT reaches within 0.5 BLEU of fine-tuning at 0.03% capacity, whereas JRC-Acquis requires 13.5%.These capacities correspond to bottleneck dimensions of 4 and 2048, respectively.
- 4.4 Results and Analysis: With varying amounts of IWSLT adaptation data, capacity-tuned adapters match and outperform fine-tuning on almost all evaluated datapoints.LHUC is competitive with extremely small data fractions but becomes limited as more adaptation data is available.
- 4.4 Results and Analysis: Adapters converge gradually and remain steady with little over-fitting, while full fine-tuning over-fits rapidly and makes model selection more difficult.Careful learning-rate or batch-size tuning can remedy fine-tuning’s behavior but must be done separately for domains, models, and corpus sizes.
5 Massively Multilingual Machine Translation
The paper trains one multilingual NMT model across 204 language pairs, then refines high-resource language pairs with task-specific adapters. This improves low-resource transfer while recovering much of the high- and medium-resource performance lost during global training, though a minor high-resource-to-English regression remains.
- Dataset: The dataset covers 102 languages and 204 language pairs to and from English, with training sizes ranging from tens of thousands to nearly 2 billion sentence pairs.Figure 6 shows an approximately 10^4-to-10^9 range in examples per language pair.
- Training approach: The multilingual procedure first trains a fully shared model globally, then fine-tunes language-pair-specific adapters for high-resource languages.The refinement stage is applied only to language pairs included during global training.
- Training approach: Temperature-based sampling balances high- and low-resource language pairs, using T = 5 instead of either the raw data distribution or equal sampling.Raw sampling favors high-resource languages, whereas equal sampling substantially harms their performance.
- Results: The globally trained multilingual model substantially outperforms bilingual baselines for extremely low-resource languages, with especially amplified gains when translating into English.The same global model causes significant deterioration for high-resource languages.
- Results: Adapter refinement produces large gains for high- and medium-resource languages and converges within 20–50k steps for most language pairs.Adapters let high-resource languages see more training data and can use larger capacities for further improvements, especially into English.
- Limitations: Adapters bridge most of the bilingual–multilingual gap, but high-resource languages translating into English retain a minor regression versus bilingual baselines.The paper leaves larger adapters, more expressive adapter architectures, and joint fine-tuning for future work.
6 Conclusion
The paper presents lightweight task-specific adapters inserted between frozen NMT layers to adapt one model to multiple tasks without forgetting the original parameters. Across domain adaptation and multilingual NMT, adapters achieve comparable or better results than full fine-tuning or bilingual baselines while using a small parameter fraction and avoiding hyper-parameter tuning across varied settings.
- Contribution: Lightweight adapters are inserted between frozen base-model layers so one NMT network can adapt to multiple target tasks without forgetting original parameters.Only the small task-specific layers are trained during adaptation.
- Findings: Across domain adaptation and multilingual NMT, adapters yield comparable or better results than full fine-tuning or bilingual baselines without hyper-parameter tuning across dataset sizes and model capacities.The conclusion characterizes the approach as flexible and scalable.
- Implication: Shared global parameters plus small task-specific layers allow a single model to be trained and adapted for many languages and domains.The paper connects this design to further research on massively multitask and universal translation models.