Source-linked AI summary
Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation
Mozhdeh Gheini, Xiang Ren, Jonathan May
TL;DR
The paper asks whether cross-attention can efficiently adapt pretrained machine-translation Transformers to new language pairs, rather than updating every parameter. It compares selective fine-tuning strategies and finds that cross-attention-only updates approach full fine-tuning while producing aligned embeddings with practical transfer benefits.
Problem
The paper examines the missing transfer-learning evidence on cross-attention’s importance when adapting pretrained machine-translation models to new language pairs.
Method
The authors compare full, cross-attention-only, embedding-only, and scratch fine-tuning while transferring between language pairs that share a source or target language.
Results
Cross-attention-only fine-tuning achieves quality close to full fine-tuning, depends on pretrained cross-attention values, and aligns new embeddings with corresponding pretrained embeddings.
Takeaways & Limitations
Aligned embeddings support catastrophic-forgetting mitigation and zero-shot translation, while updating only cross-attention reduces storage overhead for new language pairs.
Takeaways & Limitations
The transfer formulation assumes that the child and parent language pairs share either their source or target side.
Abstract
from arXiv · showhide
We study the power of cross-attention in the Transformer architecture within the context of transfer learning for machine translation, and extend the findings of studies into cross-attention when training from scratch. We conduct a series of experiments through fine-tuning a translation model on data where either the source or target language has changed. These experiments reveal that fine-tuning only the cross-attention parameters is nearly as effective as fine-tuning all parameters (i.e., the entire translation model). We provide insights into why this is the case and observe that limiting fine-tuning in this manner yields cross-lingually aligned embeddings. The implications of this finding for researchers and practitioners include a mitigation of catastrophic forgetting, the potential for zero-shot translation, and the ability to extend machine translation models to several new language pairs with reduced parameter storage overhead.
1 Introduction
The paper investigates whether cross-attention can support transfer to new machine-translation language pairs while other Transformer components remain frozen. Experiments show competitive adaptation, aligned new embeddings, and practical benefits for forgetting, zero-shot translation, and storage.
- The study asks how powerful cross-attention alone is for adapting to new language pairs, whether pretrained cross-attention values matter, and how representations change.
- The experiments compare training from scratch with regular fine-tuning, cross-attention-plus-new-embedding tuning, and new-embedding-only tuning.
- Cross-attention-only fine-tuning achieves MT quality close to full fine-tuning while keeping the encoder and decoder bodies fixed.
- Fine-tuning only cross-attention aligns newly learned embeddings with corresponding pretrained-language embeddings, unlike full-body fine-tuning.
- Aligned embeddings can help mitigate catastrophic forgetting and support zero-shot translation.
- Updating only cross-attention reduces storage overhead because adapted language pairs require copies of fewer parameters than full models.
2 Cross-Attention Fine-Tuning for MT
This section formulates transfer learning for machine translation at the level of Transformer parameter groups and evaluates cross-attention through controlled freezing comparisons. The analysis tests performance, pretrained-value importance, and knowledge from different pretraining objectives.
- Problem formulation: Fine-tuning initializes a child model from a parent model trained on a different language pair sharing either its source or target language.
- Parameterization: The parameter groups separate source and target embeddings, encoder parameters, decoder parameters, and cross-attention with its corresponding layer norms.
- Analysis framework: The study freezes selected pretrained components and measures the resulting translation quality with BLEU.
- Comparative experiments: Experiments compare embedding-only, full-body, and cross-attention fine-tuning to determine whether cross-attention adapts to child pairs with encoder and decoder stacks frozen.
- Comparative experiments: The study compares pretrained and randomly initialized cross-attention to assess whether transferred cross-attention knowledge contributes to adaptation.
- Comparative experiments: Translation-trained and language-model-trained cross-attention are contrasted to test whether the pretraining objective affects transfer behavior.
3 Experimental Setup
The experiments use controlled transfer from a French–English parent model to six child language pairs, with randomly initialized new-language embeddings and several fine-tuning variants. The setup also includes scratch training, an mBART ablation, and fixed Transformer-base modeling choices.
- Transfer setup: New-language embeddings are randomly initialized, while other parameters are imported from the parent and selectively frozen or updated.
- Fine-tuning settings: The evaluated settings update embeddings alone, embeddings plus the full Transformer body, embeddings plus cross-attention, or embeddings plus randomly initialized cross-attention.
- Baselines: Each transfer setting has a scratch-training variant to assess transfer effectiveness on the child dataset.
- Pretraining ablation: An mBART ablation tests cross-attention inherited from a denoising, self-supervised pretraining objective rather than a translation model.
- Transfer setup: The study transfers a French–English parent model to Ro–En, Ja–En, De–En, Ha–En, Fr–Es, and Fr–De child pairs.The parent training data comprise 5,251,875 sentences.
- Model: All models use the Transformer-base architecture with six encoder and decoder layers, model dimension 512, and eight attention heads.
4 Results and Analysis
Across transfer experiments, updating cross-attention with new-language embeddings substantially improves adaptation and approaches full-body fine-tuning, while preserving aligned cross-lingual representations. These benefits depend on pretrained translation knowledge in cross-attention and reduce the parameters that must be stored for new language pairs.
- Translation Quality: Cross-attention fine-tuning substantially improves upon language-and-embedding transfer in all but one case and is competitive with body fine-tuning.The exception is Ha–En; gains are especially pronounced when transferring to a new target language.
- Storage: 124,430,336 parameters are stored for six new language pairs with cross-attention fine-tuning, versus 313,583,616 with body fine-tuning.The reported storage fraction averages updated parameters across six language pairs and includes new-vocabulary embeddings.
- Pretrained and Random Values: Randomly initialized cross-attention performs worse than pretrained cross-attention and falls behind training from scratch in two cases.This comparison isolates the importance of translation knowledge already encoded in pretrained cross-attention.
- Translation Quality: With mBART, exclusive embedding fine-tuning fails because its pretrained cross-attention lacks translation knowledge.Embedding-only fine-tuning beats scratch only for Ja–En, and only by 0.1 BLEU.
- Learned Representations Properties: For German–French bilingual lexicon induction, cross-attention fine-tuning reaches 55% accuracy versus 19.7% with body fine-tuning.A manual sample inspection likewise places the two settings in ranges of 80% and 30%, respectively; the exact-match evaluation is strict.
- Learned Representations Properties: Under cross-attention-only settings, newly learned language embeddings align with corresponding parent embeddings, unlike embeddings learned with body fine-tuning.The alignment also supports indirect child–child bilingual dictionary induction through a shared parent language.
5 Utilities of Aligned Embeddings
Aligned embeddings from cross-attention-based transfer support mitigating catastrophic forgetting and constructing zero-shot translation systems without direct training for the new pair.
- Aligned Embeddings: Cross-attention-based transfer yields embeddings aligned with corresponding parent embeddings, unlike full fine-tuning.This alignment is linked to using pretrained encoder and decoder translation knowledge without updating them.
- Mitigating Forgetting: 35.0 BLEU is the original Fr–En parent model’s score on the Fr–En test set after transfer evaluation.Transferred models are evaluated after plugging the original Fr–En embeddings back into them.
- Mitigating Forgetting: {src,tgt}+xattn constantly outperforms {src,tgt}+body on the original Fr–En task despite comparable child-task performance.The comparison evaluates De–En, Ro–En, and Fr–Es models transferred from Fr–En.
- Mitigating Forgetting: tgt+xattn is much more robust against forgetting than tgt+body, which remembers close to nothing at 0.2 BLEU.The source-transferred models outperform the target-transferred model relative to the original Fr–En model.
- Zero-Shot Translation: Cross-attention transfer forms De–Es, Ro–Es, and Ro–De zero-shot systems by replacing aligned language embeddings across transferred models.The De–Es construction requires no De–Es training or direct De–Fr alignment.
- Zero-Shot Translation: 9.2 BLEU is the De–Es zero-shot score, exceeding the 50,000-sentence scratch model’s 7.2 BLEU but below the 100,000-sentence model’s 12.0 BLEU.These results are reported on the newstest2013 De–Es test set.
6 Related Work
The paper relates cross-attention-only transfer to studies of attention importance, module freezing, lightweight fine-tuning, and cross-lingual embedding construction.
- Studying Cross-attention: Prior pruning and hard-coding studies find cross-attention heads relatively more important than self-attention heads.These studies examine Transformer models trained from scratch.
- Module Freezing: Module-freezing work transfers lexical embeddings or selected input and normalization parameters while keeping most Transformer parameters frozen.The cited studies cover encoder-only and decoder-only architectures.
- Lightweight Fine-tuning: Adapter methods add trainable language-specific or monolingual modules to pretrained Transformer architectures for multilingual or zero-shot adaptation.These approaches update newly inserted adapter parameters while retaining the rest of the model.
- Lightweight Fine-tuning: Prompt-tuning-inspired methods update small sets of additional task-specific parameters, whereas this work updates a subset of the pretrained model’s own parameters.The contrast concerns whether fine-tuning adds new parameters or modifies existing ones.
- Cross-lingual Embeddings: Other cross-lingual embedding approaches use direct parallel data, shared translation-reconstruction models, or dictionary-based linear transformations.The paper reports obtaining aligned embeddings through transfer without dictionaries or direct parallel corpora.
7 Conclusion
The paper concludes that cross-attention can adapt pretrained translation models comparably to full-body fine-tuning while aligning new embeddings with parent-language embeddings.
- 7 Conclusion: Cross-attention can single-handedly achieve comparable performance to fine-tuning the entire Transformer body.The encoder and decoder body can remain frozen while cross-attention adapts to the child task.
- 7 Conclusion: Pretrained cross-attention values provide translation knowledge that supports alignment between new embeddings and corresponding parent-language embeddings.The conclusion describes this alignment as the mechanism behind cross-attention-only transfer.
- 7 Conclusion: Aligned embeddings can be used for catastrophic-forgetting mitigation and zero-shot transfer.These applications are presented as uses of the learned cross-lingual representations.
A Manual Bilingual Dictionary Evaluation
Table 4 presents sampled German words and their embedding-based equivalents from each model, with correct translations highlighted and manually checked.
- A Manual Bilingual Dictionary Evaluation: Table 4 compares sampled German words with their equivalents based on embeddings learned by each model.Correct translations are highlighted, and each pair was manually checked using an automatic translator.