Adding Interpretable Attention to Neural Translation Models Improves Word Alignment

Thomas Zenkel; Joern Wuebker; John DeNero

Source-linked AI summary

Adding Interpretable Attention to Neural Translation Models Improves Word Alignment

Thomas Zenkel, Joern Wuebker, John DeNero

arXiv:1901.11359v1Jan 31, 2019cs.CL

TL;DR

Meaningful word alignments are difficult to extract from multi-layer, multi-head Transformer attention. The paper adds an encoder-only alignment layer and target-conditioned SGD inference, achieving alignment quality comparable to Giza++ on two of three datasets without word-alignment supervision.

Problem
Extracting accurate word alignments from Transformer attention is difficult because multiple layers and heads make relevant source context hard to identify.
Method
The paper adds an encoder-only alignment layer and optimizes its attention activations with stochastic gradient descent toward a specified target word.
Results
The resulting alignments are comparable to Giza++ on two of three datasets and improve French-English alignment quality threefold over the naive baseline.
Takeaways & Limitations
The method extracts useful alignments from a Transformer while using the same bilingual training data without word-alignment supervision.
Takeaways & Limitations
The alignment layer is trained with a single attention head, which the authors report performs slightly better than alternatives tested.

Abstract

from arXiv · show

Multi-layer models with multiple attention heads per layer provide superior translation quality compared to simpler and shallower models, but determining what source context is most relevant to each target word is more challenging as a result. Therefore, deriving high-accuracy word alignments from the activations of a state-of-the-art neural machine translation model is an open challenge. We propose a simple model extension to the Transformer architecture that makes use of its hidden representations and is restricted to attend solely on encoder information to predict the next word. It can be trained on bilingual data without word-alignment information. We further introduce a novel alignment inference procedure which applies stochastic gradient descent to directly optimize the attention activations towards a given target word. The resulting alignments dramatically outperform the naive approach to interpreting Transformer attention activations, and are comparable to Giza++ on two publicly available data sets.

1 Introduction

The paper extends the Transformer to extract meaningful word alignments without alignment supervision and introduces target-conditioned attention optimization. Its alignments improve substantially over naive attention extraction and approach Giza++ quality on two of three datasets.

Motivation: Word alignments support applications including lexicon injection for rare words and projection of annotations or formatting between languages.These uses motivate extracting alignments directly from translation models.
Method: The model adds an encoder-only alignment layer without skip connections, encouraging attention toward source words relevant to each target word.The alignment layer predicts the next target word from a linear combination of encoder information and is ignored during translation inference.
Method: Stochastic gradient descent directly optimizes attention activations to maximize the likelihood of a specified target word.This inference procedure uses the target word that is unavailable to ordinary forward-pass attention extraction.
Contributions: The alignment component can be trained on the same bilingual data as the translation model without word-aligned supervision.The method is trained unsupervised and does not inject IBM-model alignments into the training pipeline.
Contributions: The extension combines alignment and translation in one network and can be obtained by fine-tuning an existing translation model.The paper identifies this integration and reduced overall training cost as practical properties of the approach.
Results: The approach achieves alignment quality comparable to Giza++ on two of three datasets after bidirectional alignment merging.It is evaluated on three hand-aligned public datasets against a naive baseline, FastAlign, and Giza++; the French-English alignment quality improves threefold over the naive baseline.

2 Machine Translation Model

The paper describes the Transformer encoder-decoder and its multi-head attention, then uses averaged attention activations as a straightforward but inadequate alignment baseline.

Transformer architecture: The Transformer encoder maps source sentences to continuous representations, while the decoder uses them to define target-language sentence probabilities.Unlike recurrent models, the architecture incorporates context entirely through attention.
Transformer architecture: Decoder layers combine masked self-attention, encoder-attention, and feed-forward sublayers, with encoder-attention using the final encoder representation.The decoder self-attention is restricted to left-hand target context and excludes the end-of-sentence token in this work.
Attention mechanism: Multi-head attention linearly projects queries and key-value pairs, concatenates the resulting head outputs, and feeds the projection into the next layer.Each head computes a lower-dimensional context vector before concatenation and output projection.
Average Transformer Attentions: The baseline averages all attention matrices into a source-target soft-attention matrix and selects the maximum source unit for each target unit to form hard alignments.Word alignments are recovered when any pair of their subwords is aligned.
Average Transformer Attentions: The paper argues that averaged attention is not an ideal alignment method because it does not encourage soft attentions to correspond to useful alignments.This motivates the dedicated alignment layer and target-conditioned optimization introduced later.

3 Related Work

Prior alignment systems rely mainly on statistical IBM models, while neural approaches either require complicated training or supervision derived from those models. The paper instead uses source and target context to infer alignments.

Statistical Models: FastAlign reparameterizes IBM Model 2 for usability and speed, whereas Giza++ is based on IBM Model 4 and provides a strong AER benchmark.The experiments use MGIZA++ and FastAlign with default parameters as baselines.
Statistical Models: IBM lexical models estimate source-word likelihood from a single target word independently of broader source and target context.The paper identifies this conditional-independence assumption as a disadvantage relative to neural representations.
Neural alignment methods: Neural attention modifications have targeted either translation quality or alignment generation as their primary objective.The paper distinguishes these goals when organizing prior work.
Neural alignment methods: Nguyen and Chiang (2018) improve source attention by adding source embeddings to decoder output, while Alkhouli et al. (2018) supervise a Transformer alignment head with Giza++ data.Peter et al. (2017) obtain their best results by bootstrapping attention with Giza++ alignments and using target foresight.
Neural alignment methods: Neural alignment methods either use complicated training procedures or supervised data generated by existing alignment tools.Examples include recurrent models trained with specialized objectives and Transformer heads trained on Giza++-generated alignments.

4 Alignment Layer

The alignment layer adds an encoder-only prediction path to the Transformer, encouraging attention toward source representations relevant to each target word. It is trained without word-alignment labels while the original translation path remains available.

4 Alignment Layer: The alignment component is trained unsupervised by fine-tuning it on top of a fixed, fully trained Transformer, without word-aligned training data.The paper uses a single attention head for the alignment layer; a footnote reports slightly better results with this choice than with an alternative.
4 Alignment Layer: The alignment layer predicts each next target word from a decoder query attending directly to encoder representations, with the same encoder input used as keys and values.The model therefore adds a second next-word distribution alongside the original decoder output.
4 Alignment Layer: Removing self-attention and skip connections forces the alignment prediction to rely solely on a linear combination of encoder-side information.The decoder output is masked and encodes only left-hand target context.
4 Alignment Layer: Experiments vary the encoder representation supplied as key and value, including word embeddings, encoder outputs, and their average.Table 1 also summarizes initialization schemes for learning attention weights with SGD.

5 Attention Optimization based on Target Word

The method optimizes attention activations for the known target word rather than relying only on forward-pass attention. It treats the activations as adjustable weights and updates them by gradient descent to increase the correct word's probability.

5 Attention Optimization based on Target Word: Forward-pass attention cannot use the identity of the target word, so the method optimizes activations specifically toward predicting that word.This addresses the mismatch between extracting alignments during translation and aligning when both source and target sentences are known.
5 Attention Optimization based on Target Word: The probability distribution depends on fixed transformed values and attention activations, allowing the attention weights to be optimized while the attention sub-network is evaluated.The optimized activations are treated as a weight matrix during this procedure.
5 Attention Optimization based on Target Word: Using two attention heads performed best among the reported head-count variants, while four heads performed considerably worse.
5 Attention Optimization based on Target Word: For each target word, stochastic gradient descent updates the attention weights to maximize its probability, with all other alignment-layer parameters fixed.The updates can be performed in parallel across target positions using cross-entropy loss.
5 Attention Optimization based on Target Word: The optimization relaxes the requirement that attention weights sum to one and uses RLU to keep activations non-negative before applying attention.The authors found this easier to optimize than applying softmax during optimization.
5 Attention Optimization based on Target Word: Three gradient-descent steps with learning rate 1 produced surprisingly good target-word prediction and alignment quality during validation tuning.Initialization can be random, uniform, or derived from an existing alignment hypothesis converted into attention weights.

6 Experimental Setup

The evaluation compares neural and statistical alignment methods on three publicly available language-pair datasets using standardized preprocessing and hand-aligned test data.

6 Experimental Setup: The study evaluates German-English, Romanian-English, and English-French alignment in both directions and with grow-diagonal symmetrization.The comparison uses the same training data and standardized preprocessing across approaches.
6 Experimental Setup: Training corpora contain 0.4–1.9 million parallel sentences, with Europarl used for German-English and additional Europarl data expanding Romanian-English training.Training data are tokenized with Moses and consistently lowercased.
6 Experimental Setup: Parallel sentences are preprocessed with byte pair encoding, producing subword units for both source and target sequences.The preprocessing example shows words split into subword units before alignment.
6 Experimental Setup: The training-data statistics are summarized in Table 3.The supplied passage identifies the table but does not provide its individual values.
6 Experimental Setup: Hand-aligned test sets omit null-token links, allow unaligned target words, and distinguish probable from sure alignments.The datasets are publicly available for all three language-pair settings.

7 Results

The alignment layer and attention optimization improve alignment quality over naïve Transformer attention, with results approaching statistical baselines across several language-pair evaluations. Qualitative analyses show that the learned layer attends to meaningful source words and that forward-pass initialization is important for successful optimization.

7.1 German-English: 27.1% AER is achieved with combined word embeddings and encoder outputs as keys and values, roughly matching FastAlign at 27.0%.Word embeddings alone achieve 31.4%, while encoder outputs alone achieve 28.6%.
7.1 German-English: The naïve average-attention baseline exceeds 50% AER for unidirectional models and reaches 50.9% after combining directions.Adding the alignment layer consistently improves these results.
7.1 German-English: Optimizing attention from random initialization fails to produce useful alignments, whereas initializing from forward-pass activations improves most symmetrized alignments and yields AER similar to Giza++.The optimization targets the reference translation through SGD.
7.2 Qualitative Analysis: In a German-English example, the alignment layer attends to meaningful source words, unlike the Transformer, which mainly attends to sentence-final punctuation.Figure 2 visualizes the attention activations and resulting word alignments.
7.3 English-French and Romanian-English: On English-French and Romanian-English, adding the alignment layer and optimizing attention consistently improve AER.Neural approaches appear to benefit more from bidirectional symmetrization because they lack future target context.

8 Conclusion

The paper extracts meaningful word alignments from Transformer translation models using an alignment layer and target-conditioned optimization. Symmetrized alignments are comparable to IBM Model 4 on two of three tasks and can be trained without IBM-derived supervision.

8 Conclusion: The proposed alignment layer removes skip connections around encoder attention, encouraging attention toward source words corresponding to each target word.The layer predicts the next target word from encoder information, making source-focused activations useful for alignment extraction.
8 Conclusion: A novel inference procedure directly optimizes attention activations for a given target word using stochastic gradient descent.Figure 3 compares random and forward-path initializations after attention optimization, but the supplied caption does not state a winning initialization.
8 Conclusion: Comparable alignment quality to IBM Model 4 as implemented in Giza++ is achieved on two of three tasks through symmetrizing both translation directions.This is the paper’s main reported comparison.
8 Conclusion: Unlike prior approaches that rely on IBM-model outputs, the model is trained unsupervised without injecting external IBM knowledge into the training pipeline.The alignment and translation models are incorporated into a single network.