Source-linked AI summary
Incorporating Structural Alignment Biases into an Attentional Neural Translation Model
Trevor Cohn, Cong Duy Vu Hoang, Ekaterina Vymolova, Kaisheng Yao, Chris Dyer, Gholamreza Haffari
TL;DR
Traditional neural translation models omit structural biases found in word-based alignment models. This paper adds positional, Markov, fertility, and bilingual-agreement biases to an attentional model, finding consistent improvements across challenging low-resource language pairs in perplexity and BLEU, including up to 3.5 BLEU points in phrase-based reranking.
Problem
Attentional neural translation models omit several structural biases incorporated by traditional word-based alignment models.
Method
The paper incorporates positional bias, Markov conditioning, fertility, and agreement between translation directions into an attentional translation model.
Results
Up to 3.5 BLEU points are gained when the model reranks candidate translations generated by a state-of-the-art phrase-based model.
Takeaways & Limitations
The structural extensions improve perplexity and reranking evaluations across several challenging language pairs in a low-resource setting.
Takeaways & Limitations
The fertility formulation is constrained because enforcing exactly one use per word is inappropriate when words may be dropped or translated multiple times.
Abstract
from arXiv · showhide
Neural encoder-decoder models of machine translation have achieved impressive results, rivalling traditional translation models. However their modelling formulation is overly simplistic, and omits several key inductive biases built into traditional models. In this paper we extend the attentional neural translation model to include structural biases from word based alignment models, including positional bias, Markov conditioning, fertility and agreement over translation directions. We show improvements over a baseline attentional model and standard phrase-based model over several language pairs, evaluating on difficult languages in a low resource setting.
1 Introduction
The paper extends attentional neural translation with structural biases from traditional word-based alignment models, then evaluates these extensions in low-resource translation. The resulting models improve over neural baselines in perplexity and BLEU, including gains when reranking phrase-based translations.
- Neural translation models achieve strong results but use simpler formulations with fewer assumptions than traditional statistical translation models.
- The attentional model dynamically represents the source and attends to different source positions while generating each target sentence.
- The proposed extensions add absolute and relative positional biases, fertility, Markov conditioning, and agreement between translation directions.
- The study evaluates these structural biases across four language pairs in a low-resource setting against encoder-decoder and attentional baselines.
- Up to 3.5 BLEU points are gained when the proposed model reranks candidate translations from a state-of-the-art phrase-based model.
2 The attentional model of translation
The attentional translation model encodes the source bidirectionally and uses learned attention scores to condition a left-to-right decoder. These scores form alignment-like distributions that aggregate source information into a fixed-length context for predicting each target word.
- Encoder: The encoder uses left-to-right and right-to-left recurrent networks to represent each source word together with its surrounding context.
- Decoder: The decoder generates the target sentence left-to-right with a recurrent network conditioned on an attention-derived source representation.
- Decoder: The context, decoder state, and target word are transformed through hidden and output layers, then normalized to predict the next target word.
- Implementation: Although the exposition uses recurrent networks generally, the implemented model uses LSTM units, which were found empirically to perform best.
- Attention: A neural compatibility function scores each target hidden state against every source encoding, producing attention values over source positions.
- Attention: Softmax-normalized attention values can be interpreted as alignment probabilities and reweight source encodings into a fixed-length context vector.
3 Incorporating Structural Biases
The paper augments attentional translation with structural biases from traditional alignment models, including positional, Markov, fertility, and bilingual agreement mechanisms.
- Traditional alignment models supply structural components omitted by attentional translation, motivating their integration into the neural model.
- 3.1 Position bias: Position bias adds source and target position features to attention, enabling learned relative-position and near-diagonal preferences without using unknown target length.Log-transformed source position, target position, and source length features avoid numerical instability and support interactions with other inputs.
- 3.2 Markov condition: Markov conditioning summarizes nearby previous attention positions to model monotone, one-to-many, and local-inversion alignment patterns.The implementation uses local offsets of ±k because attention vectors vary with source sentence length.
- 3.3 Fertility: Fertility models how consistently a source word is translated into a number of target words, using local features or a contextual global model.The contextual model can represent omitted words and multiword translations while using source context to learn fertility.
- 3.3 Fertility: The global fertility formulation avoids forcing every source word to align exactly once, because translation may drop some words or translate others multiple times.The paper notes that an exactly-once constraint is inappropriate for determiners, function words, and multiword target phrases.
- 3.4 Bilingual Symmetry: Joint bilingual training combines directional translation objectives with a trace bonus that encourages the two attention matrices to agree.The trace bonus compares αs→t with the transpose of αt←s through matrix multiplication.
4 Experiments
The experiments evaluate structural biases in low-resource translation across four language pairs using perplexity and BLEU, including attention-pattern analyses and reranking. The proposed models generally improve over neural baselines and the phrase-based system, with benefits depending on how fertility and symmetry are combined.
- Experimental setup: Experiments use four language pairs in a simulated low-resource setting, evaluating neural models with test perplexity and phrase-based reranking with BLEU.The languages include English paired with Romanian, Estonian, Russian, and Chinese; Chinese-English uses BTEC and the other pairs use Europarl.
- Analysis of alignment biases: 4.56 perplexity results from adding positional bias, local fertility, and Markov structure to the attentional model, versus 4.77 for vanilla attention and 5.35 for encoder-decoder.The comparison is reported for Chinese→English translation on BTEC.
- Analysis of alignment biases: 5.20 perplexity from adding global fertility directly was detrimental, while pre-training reduced perplexity to 4.31 when global fertility refined an already trained model.Global fertility also slowed early training when added to the alignment-feature model, but produced a sizeable gain during fine-tuning.
- Analysis of alignment biases: Attention visualizations assign more weight to diagonal and 1-to-many alignments with alignment features, while global fertility shifts attention toward sentence-boundary symbols.The visualizations cover attentional, +align, +glofer, and +sym variants for Estonian↔English.
- Analysis of alignment biases: Joint symmetric training produces more confident attention values, especially for consistent 1-to-many alignments between English and Estonian.The example compares both translation directions and shows stronger confidence for difficult↔raskeid.
- Full results: Across four language pairs and both directions, the proposed models achieve lower perplexity than vanilla attention and encoder-decoder baselines.The authors attribute the lower perplexities to the incorporated linguistic constraints.
- Full results: At least 1 BLEU point above the phrase-based baseline is achieved in every reranking setting, with almost 3 points for Chinese-English using an ensemble.The reranker is significantly better for Chinese and Estonian and on par for Russian and Romanian→English relative to the other neural rerankers.
5 Related Work
Related work established neural encoder-decoder translation, attention, and refinements that constrain or exploit source context. These models motivate combining neural attention with alignment information from earlier translation approaches.
- Neural translation models: Early neural translation used convolutional or recurrent encoders with recurrent decoders, while attention enabled dynamic source context during decoding.The progression runs from Kalchbrenner and Blunsom through Sutskever and Bahdanau.
- Attention mechanisms: Luong et al. refined attention by constraining it to a local text span whose word representations are averaged.This refinement is presented as a way to make attention more local.
- Connection to alignment models: The paper’s approach builds on this neural attention literature while incorporating structural information associated with traditional alignment models.The supplied related-work passages provide the neural-model lineage; the structural-bias connection is stated in the paper’s framing.
6 Conclusion
The paper concludes that attentional translation models omit established properties of word-based models. Adding structural biases improves performance across challenging language pairs in low-resource settings.
- Structural biases were proposed to encode properties of traditional word-based translation models that attentional models do not capture.
- Improvements were observed across several challenging language pairs in a low-resource setting.
- Evaluation showed gains in both perplexity and re-ranking performance.