Source-linked AI summary
Jointly Learning to Align and Translate with Transformer Models
Sarthak Garg, Stephan Peitz, Udhyakumar Nallasamy, Matthias Paulik
TL;DR
Neural translation models outperform statistical approaches for translation, but statistical models remain strong for word alignment. This paper jointly trains a Transformer for both tasks using attention-derived or external alignment supervision, achieving stronger alignment performance while maintaining translation performance.
Problem
Neural machine translation is strong for translation, but traditional statistical alignment models remain commonly used for word alignment.
Method
The paper uses a multi-task objective that combines translation training with supervision of an attention head using derived or external alignment labels and fuller target context.
Results
The framework outperforms previous neural and statistical alignment approaches without degrading translation accuracy, and external IBM alignments produce significantly better alignment results than GIZA++ while maintaining translation performance.
Takeaways & Limitations
A single Transformer can jointly produce translations and alignments, including alignment improvements from external GIZA++ supervision without changing translation performance.
Takeaways & Limitations
The self-training approach requires two training runs, and no alignment validation set exists for selecting an alignment checkpoint.
Abstract
from arXiv · showhide
The state of the art in machine translation (MT) is governed by neural approaches, which typically provide superior translation accuracy over statistical approaches. However, on the closely related task of word alignment, traditional statistical word alignment models often remain the go-to solution. In this paper, we present an approach to train a Transformer model to produce both accurate translations and alignments. We extract discrete alignments from the attention probabilities learnt during regular neural machine translation model training and leverage them in a multi-task framework to optimize towards translation and alignment objectives. We demonstrate that our approach produces competitive results compared to GIZA++ trained IBM alignment models without sacrificing translation accuracy and outperforms previous attempts on Transformer model based word alignment. Finally, by incorporating IBM model alignments into our multi-task training, we report significantly better alignment accuracies compared to GIZA++ on three publicly available data sets.
1 Introduction
Transformer-based neural machine translation is strong at translation but does not automatically yield traditional word alignments. This paper trains Transformers to produce both translations and interpretable alignments through multi-task learning.
- Attention probabilities can differ from traditional word alignments, while Transformer depth and multiple heads complicate extracting discrete alignments.
- Word alignments support bilingual lexicon construction, dictionary-assisted translation, annotation transfer, and user-facing alignment information.
- The proposed approach teaches Transformer models to produce translations and interpretable alignments simultaneously.
- The model combines NLL translation loss with an alignment loss supervising one attention head.
- Different objective components use different amounts of target context to preserve autoregressive translation while improving alignment.
- The system outperforms previous neural and statistical alignment approaches without degrading translation accuracy.
2 Preliminaries
Word alignment represents source-target correspondence as a discrete relation, while Transformer encoder-decoder attention computes contextual representations through multiple attention heads.
- 2.1 Word Alignment Task: An alignment A is a subset of source-target word-position pairs representing correspondence between sentence words.
- 2.2 Transformer Model: The Transformer is an encoder-decoder model whose encoder and decoder use multi-layer self-attention and feed-forward sub-layers.
- 2.2 Transformer Model: Decoder encoder-decoder attention uses a target-side query together with source-side key and value matrices.
- 2.2 Transformer Model: Each attention head projects inputs into a subspace, computes attention, aggregates head outputs, and projects them back.
- 2.2 Transformer Model: For each target token, an attention vector gives probabilities over source tokens; vectors across targets form an attention matrix.
3 Baseline Methods
Baseline alignment methods extract or supervise attention-based alignments using different sources of supervision and context, often combining bidirectional models.
- A common baseline averages attention matrices across layers and heads, then aligns each target word to its highest-probability source word.
- Peter et al. guide attention toward statistical-toolkit alignments and provide the current target word as additional context.
- Zenkel et al. add an attention layer and optimize its activations to predict the target word without external alignment labels.
- Existing approaches train models in both translation directions and merge their alignments using the grow diagonal heuristic.
4 Proposed Method
The method jointly trains Transformer models for translation and interpretable alignment by supervising an attention head while preserving autoregressive translation. It supports self-training from averaged attention or stronger supervision from GIZA++ alignments, with separate context handling for the two objectives.
- 4.1 Averaging Layer-wise Attention Scores: The layer average baseline uses penultimate-layer attention probabilities because they naturally produce better alignments than averaging across all layers.Attention matrices are averaged across heads within each layer before evaluating the resulting alignments.
- 4.2 Multi-task Learning: The alignment labels are formed by row-normalizing a 0-1 alignment matrix for target tokens aligned to at least one source token.The resulting matrix Gp is a probability distribution over source tokens for each target token.
- 4.2 Multi-task Learning: One penultimate-layer attention head is supervised to match the labeled alignment distribution using Kullback-Leibler divergence, equivalently a cross-entropy loss.The selected head's attention matrix is trained against Gp.
- 4.2 Multi-task Learning: The alignment loss is combined with the standard negative log-likelihood translation loss in a multi-task objective weighted by a hyperparameter.Supervising only one head leaves other heads available for representations useful to translation.
- 4.3 Providing Full Target Context: Because past-only target context limits alignment, the model uses different context amounts for translation and alignment, requiring separate masked and unmasked decoder passes.The masked pass computes translation loss, while the unmasked pass computes alignment loss.
- 4.3 Providing Full Target Context: The full-context alignment formulation improves alignment accuracy without degrading translation quality, and the framework can alternatively use GIZA++ alignments as labels.GIZA++ supervision is presented for settings where alignment accuracy is paramount.
5 Experiments
Experiments evaluate alignment and joint alignment-translation performance across three language pairs and two translation setups. The proposed multi-task models improve alignment quality while preserving translation performance, with full target context and GIZA++ supervision providing the strongest results.
- Evaluation: The evaluation reports alignment error rate (AER), tests AER differences with a two-sided Wilcoxon signed-rank test, and uses BLEU for translation quality.AER differences are tested at α = 0.1%, and translation quality is computed with sacreBLEU.
- Experimental setups: Experiments cover German→English, Romanian→English, and English→French alignment datasets, plus a WMT’18 English-German align-and-translate setup.The alignment datasets include gold alignments, while the WMT setup evaluates both AER and BLEU.
- Averaging attention: Across language pairs, attention-based alignments improve substantially in higher Transformer layers, while initial layers do not produce meaningful alignments.The penultimate layer outperforms the final layer, suggesting that the final layer uses alignment-based features from the penultimate layer.
- Alignment task: Full target context gives the largest AER gains, while GIZA++ supervision enables the multi-task model to outperform GIZA++ and Peter et al. (2017).The full-context comparison is directly comparable to Zenkel et al. (2019), and the authors attribute the advantage partly to access to the full target context.
- Align and translate task: In the joint task, full target context makes the model comparable to GIZA++, and GIZA++ supervision achieves the best AER without degrading BLEU relative to the layer-average baseline.The layer-average baseline has high precision (94.2%) but low recall (29.6%), providing mostly correct alignments for supervision.
6 Analysis
The analysis examines alignment behavior relative to GIZA++ and training dynamics for attention-based alignment extraction. It finds contextual pronoun–noun links that GIZA++ misses, while earlier checkpoints can provide better supervision.
- Alignment behavior: Our model aligns pronouns with contextually related nouns, including German Voraussetzungen and Haus, whereas GIZA++ misses both links.The authors speculate that contextual encoding, rather than co-occurrence counting alone, may support these links.
- Training dynamics: 42.7% AER is reached by the fifth epoch on WMT’18 English→German.The result comes from plotting test AER and validation NLL across epochs.
- Training dynamics: An earlier checkpoint could improve alignment supervision, but no alignment validation set exists for this task.This prevents direct checkpoint selection using alignment validation performance.
7 Related Work
Related work has guided recurrent-model attention with statistical alignments, while this paper studies multi-task alignment supervision in Transformers. It also contributes a quantitative way to evaluate how Transformer attention models alignments.
- Guided attention: Earlier studies supervised recurrent-model attention using alignments from statistical MT toolkits, but reported limited alignment gains over those toolkits at larger data scales.The present work transfers the multi-task idea to the Transformer architecture.
- Transformer supervision: The paper injects alignment information through one Transformer attention head, following related work that adds linguistic information to attention heads.The cited precedent used parse-tree information for semantic role labeling.
- Attention analysis: The paper presents a quantitative method for evaluating and analyzing attention probabilities with respect to word alignment quality.This extends prior efforts to understand attention mechanisms.
8 Conclusions
The paper jointly trains a Transformer to produce translations and alignments, extends the framework with GIZA++ supervision, and maintains translation performance while improving alignment results. Its self-training approach currently requires two training runs.
- Conclusions: A single Transformer jointly produces translations and alignments through a multi-task objective with full target-sentence context for the alignment module.The framework is designed to improve alignment quality without separating translation and alignment models.
- Conclusions: GIZA++ supervision yields significantly better alignment results than GIZA++ while maintaining the same translation performance.This conclusion concerns the framework extended with external alignments.
- Limitations: The self-training approach currently requires two training runs.The paper proposes alternating alignment extraction and model training as a possible single-run alternative.