Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation

• The study asks how powerful cross-attention alone is for adapting to new language pairs, whether pretrained cross-attention values matter, and how representations change.Source: §1 · p. 1
• The experiments compare training from scratch with regular fine-tuning, cross-attention-plus-new-embedding tuning, and new-embedding-only tuning.Source: §1 · pp. 1–2
• Cross-attention-only fine-tuning achieves MT quality close to full fine-tuning while keeping the encoder and decoder bodies fixed.Source: §1 · p. 2
• Fine-tuning only cross-attention aligns newly learned embeddings with corresponding pretrained-language embeddings, unlike full-body fine-tuning.Source: §1 · p. 2
• Aligned embeddings can help mitigate catastrophic forgetting and support zero-shot translation.Source: §1 · pp. 2–3
• Updating only cross-attention reduces storage overhead because adapted language pairs require copies of fewer parameters than full models.Source: §1 · p. 2

Problem formulation

• Fine-tuning initializes a child model from a parent model trained on a different language pair sharing either its source or target language.Source: §2.1 · p. 3

Parameterization

• The parameter groups separate source and target embeddings, encoder parameters, decoder parameters, and cross-attention with its corresponding layer norms.Source: §2.1 · p. 3

Analysis framework

• The study freezes selected pretrained components and measures the resulting translation quality with BLEU.Source: §2.2 · p. 3

Comparative experiments

• Experiments compare embedding-only, full-body, and cross-attention fine-tuning to determine whether cross-attention adapts to child pairs with encoder and decoder stacks frozen.Source: §2.2 · p. 3
• The study compares pretrained and randomly initialized cross-attention to assess whether transferred cross-attention knowledge contributes to adaptation.Source: §2.2 · pp. 3–4
• Translation-trained and language-model-trained cross-attention are contrasted to test whether the pretraining objective affects transfer behavior.Source: §2.2 · p. 4

Transfer setup

• New-language embeddings are randomly initialized, while other parameters are imported from the parent and selectively frozen or updated.Source: §3.1 · p. 4

Fine-tuning settings

• The evaluated settings update embeddings alone, embeddings plus the full Transformer body, embeddings plus cross-attention, or embeddings plus randomly initialized cross-attention.Source: §3.1 · p. 4

Baselines

• Each transfer setting has a scratch-training variant to assess transfer effectiveness on the child dataset.Source: §3.1 · p. 4

Pretraining ablation

• An mBART ablation tests cross-attention inherited from a denoising, self-supervised pretraining objective rather than a translation model.Source: §3.1 · p. 4

Transfer setup

• The study transfers a French–English parent model to Ro–En, Ja–En, De–En, Ha–En, Fr–Es, and Fr–De child pairs.The parent training data comprise 5,251,875 sentences.Source: §3.1 · p. 4, §3.2 · p. 5

Model

• All models use the Transformer-base architecture with six encoder and decoder layers, model dimension 512, and eight attention heads.Source: §3.2 · p. 5

Translation Quality

• Cross-attention fine-tuning substantially improves upon language-and-embedding transfer in all but one case and is competitive with body fine-tuning.The exception is Ha–En; gains are especially pronounced when transferring to a new target language.Source: §4.1 · p. 5

Storage

• 124,430,336 parameters are stored for six new language pairs with cross-attention fine-tuning, versus 313,583,616 with body fine-tuning.The reported storage fraction averages updated parameters across six language pairs and includes new-vocabulary embeddings.Source: §4.1 · p. 5

Pretrained and Random Values

• Randomly initialized cross-attention performs worse than pretrained cross-attention and falls behind training from scratch in two cases.This comparison isolates the importance of translation knowledge already encoded in pretrained cross-attention.Source: §4.1 · p. 5

Translation Quality

• With mBART, exclusive embedding fine-tuning fails because its pretrained cross-attention lacks translation knowledge.Embedding-only fine-tuning beats scratch only for Ja–En, and only by 0.1 BLEU.Source: §4.1 · p. 6, Figure 2 · p. 7

Learned Representations Properties

• For German–French bilingual lexicon induction, cross-attention fine-tuning reaches 55% accuracy versus 19.7% with body fine-tuning.A manual sample inspection likewise places the two settings in ranges of 80% and 30%, respectively; the exact-match evaluation is strict.Source: §4.2 · p. 6
• Under cross-attention-only settings, newly learned language embeddings align with corresponding parent embeddings, unlike embeddings learned with body fine-tuning.The alignment also supports indirect child–child bilingual dictionary induction through a shared parent language.Source: §4.2 · pp. 6–7, Figure 3 · p. 7

Aligned Embeddings

• Cross-attention-based transfer yields embeddings aligned with corresponding parent embeddings, unlike full fine-tuning.This alignment is linked to using pretrained encoder and decoder translation knowledge without updating them.Source: §5 · p. 7

Mitigating Forgetting

• 35.0 BLEU is the original Fr–En parent model’s score on the Fr–En test set after transfer evaluation.Transferred models are evaluated after plugging the original Fr–En embeddings back into them.Source: §5.1 · p. 7, Figure 4 · p. 7
• {src,tgt}+xattn constantly outperforms {src,tgt}+body on the original Fr–En task despite comparable child-task performance.The comparison evaluates De–En, Ro–En, and Fr–Es models transferred from Fr–En.Source: §5.1 · p. 7, Figure 4 · p. 7
• tgt+xattn is much more robust against forgetting than tgt+body, which remembers close to nothing at 0.2 BLEU.The source-transferred models outperform the target-transferred model relative to the original Fr–En model.Source: §5.1 · p. 7

Zero-Shot Translation

• Cross-attention transfer forms De–Es, Ro–Es, and Ro–De zero-shot systems by replacing aligned language embeddings across transferred models.The De–Es construction requires no De–Es training or direct De–Fr alignment.Source: §5.2 · p. 8, Table 3 · p. 8
• 9.2 BLEU is the De–Es zero-shot score, exceeding the 50,000-sentence scratch model’s 7.2 BLEU but below the 100,000-sentence model’s 12.0 BLEU.These results are reported on the newstest2013 De–Es test set.Source: §5.2 · p. 8, Table 3 · p. 8

Studying Cross-attention

• Prior pruning and hard-coding studies find cross-attention heads relatively more important than self-attention heads.These studies examine Transformer models trained from scratch.Source: §6 · p. 8

Module Freezing

• Module-freezing work transfers lexical embeddings or selected input and normalization parameters while keeping most Transformer parameters frozen.The cited studies cover encoder-only and decoder-only architectures.Source: §6 · p. 8

Lightweight Fine-tuning

• Adapter methods add trainable language-specific or monolingual modules to pretrained Transformer architectures for multilingual or zero-shot adaptation.These approaches update newly inserted adapter parameters while retaining the rest of the model.Source: §6 · p. 8
• Prompt-tuning-inspired methods update small sets of additional task-specific parameters, whereas this work updates a subset of the pretrained model’s own parameters.The contrast concerns whether fine-tuning adds new parameters or modifies existing ones.Source: §6 · p. 8

Cross-lingual Embeddings

• Other cross-lingual embedding approaches use direct parallel data, shared translation-reconstruction models, or dictionary-based linear transformations.The paper reports obtaining aligned embeddings through transfer without dictionaries or direct parallel corpora.Source: §6 · pp. 8–9

7 Conclusion

• Cross-attention can single-handedly achieve comparable performance to fine-tuning the entire Transformer body.The encoder and decoder body can remain frozen while cross-attention adapts to the child task.Source: §7 · p. 9
• Pretrained cross-attention values provide translation knowledge that supports alignment between new embeddings and corresponding parent-language embeddings.The conclusion describes this alignment as the mechanism behind cross-attention-only transfer.Source: §7 · p. 9
• Aligned embeddings can be used for catastrophic-forgetting mitigation and zero-shot transfer.These applications are presented as uses of the learned cross-lingual representations.Source: §7 · p. 9

A Manual Bilingual Dictionary Evaluation

• Table 4 compares sampled German words with their equivalents based on embeddings learned by each model.Correct translations are highlighted, and each pair was manually checked using an automatic translator.Source: Table 4 · p. 12