Source-linked AI summary
On The Alignment Problem In Multi-Head Attention-Based Neural Machine Translation
Tamer Alkhouli, Gabriel Bretschner, Hermann Ney
TL;DR
The paper studies how to obtain useful source-target alignments from multi-head transformer NMT, especially for translations constrained by user dictionary entries. It adds alignment information and proposes alignment pruning, achieving competitive translation performance, better dictionary-guided results, and faster alignment-based decoding.
Problem
Transformer NMT does not use explicit alignments, making it harder to identify source words behind generated targets when user-specific translation constraints must be enforced.
Source: §1 · p. 1Method
The paper augments transformer multi-head attention with alignment information, uses self-attentive alignment models, extracts alignments, and prunes alignment hypotheses during decoding.
Source: §1 · p. 2, §3.2 · p. 4, §5 · p. 5Results
Alignment-assisted systems achieve competitive performance with transformer baselines, outperform standard transformers in dictionary-guided translation, and reach a 1.8 speed-up without translation-quality loss through alignment pruning.
Source: §6.1 · p. 6, §6.2 · p. 7, §8 · p. 8Takeaways & Limitations
Explicit alignment can support dictionary-guided translation while retaining competitive transformer performance and reducing alignment-based decoding cost.
Source: §1 · p. 2, §6.2 · p. 7, §8 · p. 8
Abstract
from arXiv · showhide
This work investigates the alignment problem in state-of-the-art multi-head attention models based on the transformer architecture. We demonstrate that alignment extraction in transformer models can be improved by augmenting an additional alignment head to the multi-head source-to-target attention component. This is used to compute sharper attention weights. We describe how to use the alignment head to achieve competitive performance. To study the effect of adding the alignment head, we simulate a dictionary-guided translation task, where the user wants to guide translation using pre-defined dictionary entries. Using the proposed approach, we achieve up to $3.8$ % BLEU improvement when using the dictionary, in comparison to $2.4$ % BLEU in the baseline case. We also propose alignment pruning to speed up decoding in alignment-based neural machine translation (ANMT), which speeds up translation by a factor of $1.8$ without loss in translation performance. We carry out experiments on the shared WMT 2016 English$\to$Romanian news task and the BOLT Chinese$\to$English discussion forum task.
1 Introduction
The paper addresses unreliable alignment extraction in multi-head transformer NMT, where alignments matter for user-constrained translation. It augments transformers with alignment information, preserves competitive baseline performance, improves dictionary-guided translation, and introduces pruning for faster decoding.
Motivation
- Alignment extraction is difficult because transformer NMT uses multiple attention components, while maximum attention can point to the wrong source word.This makes attention-based alignments non-trivial for applications that require identifying which source words generate target words.Source: §1 · p. 1
Approach
- The proposed alignment-assisted transformer integrates external alignment information into multi-head attention while maintaining strong baseline performance.The method also describes training procedures for using external alignments and self-attentive alignment models for faster evaluation.Source: §1 · p. 2
Applications
- Alignment-assisted systems outperform standard transformer systems in dictionary-guided translation tasks.The task models users enforcing translations from predefined dictionary entries during decoding.Source: §1 · p. 2
Efficiency
- Alignment pruning speeds alignment-based decoding by a factor of 1.8 without affecting translation quality.The paper presents pruning as a search-time method for reducing evaluation cost.Source: §1 · p. 2
2 Related Work
Prior alignment-based neural translation models provide explicit alignment information but have been studied mainly with recurrent or smaller architectures. This work applies transformer-style self-attention to alignment-based NMT and evaluates it for online dictionary-guided translation.
Alignment-based NMT
- Earlier alignment-based neural models used explicit alignment information, but prior work focused on feedforward or recurrent models and did not establish competitiveness with strong transformer systems.Earlier CPU-based experiments were limited to small 200-node layers and recurrent architectures.Source: §2 · p. 2
Transformer alignment models
- This work explores transformer models for alignment-based NMT instead of recurrent models.The change targets standalone neural machine translation using transformer-style architectures.Source: §2 · pp. 2–3
Model design
- The proposed alignment models use non-monotonous alignment and self-attentive layers rather than recurrent layers, enabling parallelized computation.The self-attentive alignment model replaces recurrent encoder and decoder components.Source: §2 · p. 3
Constrained translation
- For online dictionary-guided translation, alignment-assisted transformer NMT outperforms standard transformer models when source words match predefined dictionary entries.The constraints can be activated during decoding without requiring them during training.Source: §2 · p. 3
3 Alignment-Based Neural Machine Translation
The section replaces recurrent alignment and lexical components with transformer-based self-attentive models, while conditioning lexical translation on explicit alignment information through an added alignment head.
3 Alignment-Based Neural Machine Translation
- The alignment-based model separates translation into alignment and word-generation steps, with lexical scores conditioned on hypothesized source positions and alignment history.Both models use full source and target context; the lexical model additionally depends on the current aligned source position.Source: §3 · p. 3
- Transformer self-attentive layers replace recurrent structures in the lexical and alignment models, enabling parallelizable computation.The alignment model predicts source-position jumps and uses a single-head hard attention layer based on the previous aligned position.Source: §3 · p. 3, §3.1 · p. 4, §3.2 · p. 4
3.1 Transformer-Based Lexical Model
- The lexical model conditions translation on alignment information by concatenating a one-hot alignment head with the other source-to-target attention heads.The one-hot distribution selects the encoder representation at the aligned source position.Source: §3.1 · p. 4
- Figure 2 shows the alignment block added to multi-head source-to-target attention; removing it recovers the default transformer attention component.The component combines encoder states at all source positions with the state at the aligned position and the previous decoder state.Source: Figure 2 · p. 4
- Training the alignment-assisted lexical model from scratch was suboptimal, whereas initializing from a converged transformer baseline produced better systems and improved perplexity after alignment information was introduced.The initialization strategy reduces the risk that the model over-relies on alignment information while other attention heads become ineffective.Source: §3.3 · pp. 4–5
4 Alignment Pruning
Alignment pruning accelerates alignment-based decoding by pruning unlikely source positions during beam search while retaining alignment hypotheses for lexical scoring.
4 Alignment Pruning
- During beam search, each lexical hypothesis retains an underlying alignment hypothesis that is used to compute its translation score.The alignment model is evaluated in batch for all beam entries before source positions are hypothesized and pruned.Source: §4 · p. 5
5 Alignment Extraction
The paper extracts alignments from attention by selecting the source position with the maximum accumulated attention, while alignment-assisted models prioritize the hypothesized aligned position.
5 Alignment Extraction
- Attention-based extraction selects the source word with the maximum accumulated attention weight at each decoding step.For transformer models, the weights are aggregated across attention heads and decoder layers.Source: §5 · p. 5
- In the alignment-assisted transformer, the extraction score adds an alignment indicator that favors the source position hypothesized during search.The indicator equals 1 at the hypothesized position and 0 elsewhere, and those positions are scored by the alignment model.Source: §5 · p. 5
6 Experiments
Experiments evaluate alignment-assisted neural translation on WMT English→Romanian and BOLT Chinese→English, comparing RNN, transformer, and alignment-based systems. The proposed alignment-assisted transformer achieves competitive baseline performance and supports faster alignment-based decoding through pruning.
6.1 Performance Comparison
- 1.6% BLEU and 2.0% TER improvements over previously published alignment-based RNN results follow from increasing layer size to 1000 nodes.Source: §6.1 · p. 6
- 3.1% BLEU improvement over the RNN alignment-based attention system is achieved on Chinese→English, while matching transformer-baseline performance on both tasks.Source: §6.1 · p. 6
- Development perplexity drops from 6.2 to 5.0 on English→Romanian and from 6.0 to 4.7 on Chinese→English, indicating use of alignment information.Source: §6.1 · pp. 6–7
- 1.8x speedup is obtained at pruning threshold 0.15 without translation-quality loss; higher thresholds prune more aggressively but degrade BLEU.Source: §6.2 · p. 7, Figure 3 · p. 7
7 Dictionary Suggestions
The dictionary-suggestion experiment tests whether extracted attention alignments help incorporate user-provided translation pairs. Alignment-assisted transformers improve more than transformer baselines, consistent with sharper attention weights.
7 Dictionary Suggestions
- Maximum accumulated attention weights are used to extract a source word for each decoding step, enabling relaxed dictionary-guided translation.Source: §7 · p. 7
- The simulated dictionary uses IBM4-aligned, one-to-one source–target pairs from development references, excludes English stop words, and includes up to four entries per sentence.Source: §7 · p. 7
- 3.8% BLEU improvement is achieved with dictionary suggestions on English→Romanian, compared with 2.4% for the transformer baseline.The simulated dictionaries cover 11.6% of the English→Romanian reference set and 9.9% of the Chinese→English reference set.Source: §7 · p. 7
- Sharper accumulated attention weights in alignment-assisted systems can identify translated source words more accurately, allowing more dictionary entries to help translation.Source: §7 · pp. 7–8
8 Conclusion
The paper augments transformer attention with an alignment head, evaluates alignment-assisted translation, and introduces pruning for faster alignment-based decoding. The systems remain competitive with transformer baselines and improve dictionary-guided translation.
8 Conclusion
- The proposed alignment head supports alignment extraction, while alignment pruning yields a 1.8x speedup without loss in translation quality.Source: §8 · p. 8
- Alignment-assisted systems outperform standard transformers on dictionary-guided translation across two tasks while maintaining competitive overall performance.Source: §8 · p. 8