PaperlayerView exact revision on arXiv

Source-linked AI summary

Effective Approaches to Attention-based Neural Machine Translation

Minh-Thang Luong, Hieu Pham, Christopher D. Manning

arXiv:1508.04025v5cs.CL

TL;DR

Attention-based neural machine translation lacked well-explored architectures. This paper evaluates global and local attention, achieving up to 5.0 BLEU gains and new English-to-German state-of-the-art results.

  • Problem

    Attention-based neural machine translation had received little exploration of useful architectural alternatives.

  • Method

    The paper evaluates global attention over all source positions and local attention over subsets of source positions.

  • Results

    Up to 5.0 BLEU gains were achieved over non-attentional systems, while English-to-German ensembles exceeded previous state-of-the-art systems by more than 1.0 BLEU.

  • Takeaways & Limitations

    Attention-based NMT models were superior to non-attentional models in many cases, including translating names and handling long sentences.

  • Takeaways & Limitations

    The global attention model uses only the top portion of its alignment vector for short sentences and ignores words near the end for long sentences.

Abstract

from arXiv · show

An attentional mechanism has lately been used to improve neural machine translation (NMT) by selectively focusing on parts of the source sentence during translation. However, there has been little work exploring useful architectures for attention-based NMT. This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time. We demonstrate the effectiveness of both approaches over the WMT translation tasks between English and German in both directions. With local attention, we achieve a significant gain of 5.0 BLEU points over non-attentional systems which already incorporate known techniques such as dropout. Our ensemble model using different attention architectures has established a new state-of-the-art result in the WMT'15 English to German translation task with 25.9 BLEU points, an improvement of 1.0 BLEU points over the existing best system backed by NMT and an n-gram reranker.

1 Introduction

The paper introduces two simple attention architectures for NMT: global attention over all source words and local attention over a subset. Both approaches improve English–German translation, with gains of up to 5.0 BLEU and new state-of-the-art results.

  • Motivation: NMT offers simple end-to-end training, strong generalization to long sequences, a small memory footprint, and easier decoder implementation than standard MT.Unlike standard MT, it does not require explicitly storing gigantic phrase tables and language models.
  • Attention architectures: The paper proposes global attention over all source words and local attention over only a subset at each time.The local approach is computationally less expensive than global attention and blends aspects of hard and soft attention.
  • Experimental findings: Both attention approaches are effective on WMT English–German translation tasks in both directions.The experiments evaluate the proposed attentional models against non-attentional systems incorporating techniques such as dropout.
  • Experimental findings: 5.0 BLEU is the maximum improvement achieved over non-attentional systems that already incorporate techniques such as dropout.For English-to-German translation, the models also achieve new state-of-the-art results for WMT’14 and WMT’15, outperforming previous systems by more than 1 BLEU point.

2 Neural Machine Translation

Neural machine translation directly models the conditional probability of a target sentence given a source sentence. Its basic encoder–decoder architecture uses neural representations and recurrent decoding, with systems differing in how they encode and repeatedly access source information.

  • Core architecture: NMT directly models p(y|x), translating a source sequence x_1, ..., x_n into a target sequence y_1, ..., y_m.A basic system consists of an encoder that computes a source representation and a decoder that generates target words sequentially.
  • Core architecture: The basic NMT architecture has an encoder for the source sentence and a decoder that generates one target word at a time.The decoder decomposes the conditional probability over the target sequence.
  • RNN architectures: Recent NMT systems commonly use recurrent neural networks, differing in decoder architecture and in how the encoder computes the source representation.The surveyed architectures include standard RNNs, convolutional encoders, stacked LSTMs, and LSTM-inspired recurrent variants.
  • Source representations: Some systems use a single source representation only to initialize the decoder, whereas attention-based systems consult a set of source hidden states throughout decoding.The recurrent hidden-state function may use a vanilla RNN, GRU, or LSTM unit.
  • This work: This work uses a stacked-LSTM NMT architecture with the LSTM unit defined by Zaremba et al. (2015).The training objective is formulated over a parallel training corpus D.

3 Attention-based Models

The paper develops global and local attention mechanisms for deriving source-side context during decoding, then introduces input feeding to incorporate past alignment information into subsequent decisions.

  • Shared Decoding: Both attention types combine the decoder hidden state and source context into an attentional hidden state before producing the target-word distribution.Their shared decoding path is ht → at → ct → ˜ht, which the paper contrasts with the more complex path in Bahdanau et al. (2015).
  • Global Attention: Global attention compares the current target hidden state with every encoder hidden state and computes context as their weighted average.The alignment vector length equals the number of source time steps; for short or long sentences, only the usable portion is considered.
  • Local Attention: Local attention addresses global attention’s cost by selecting a small source window for each target word and forming context from states within that window.The model first predicts an aligned position, then computes a weighted average over the surrounding source hidden states.
  • Local Attention: Predictive local attention estimates a real-valued aligned position and applies a Gaussian centered there to favor nearby source positions.The predicted position lies within the source sentence, and the model remains differentiable almost everywhere.
  • Input Feeding: Input feeding concatenates each attentional vector with the next input so alignment decisions can incorporate past alignment information rather than being made independently.The approach is applicable to general stacking recurrent architectures, including non-attentional models.

4 Experiments

The experiments evaluate English–German translation in both directions using WMT data and BLEU on newstest2014 and newstest2015. Attention, input feeding, unknown replacement, and ensembling yield progressive gains, including new state-of-the-art results for English→German.

  • Experimental setup: Experiments evaluate English–German translation in both directions, tuning hyperparameters on newstest2013 and reporting case-sensitive BLEU on newstest2014 and newstest2015.The development set contains 3000 sentences; newstest2014 and newstest2015 contain 2737 and 2169 sentences, respectively.
  • Experimental setup: Models train on 4.5M WMT’14 sentence pairs while restricting each language vocabulary to the top 50K words and mapping others to <unk>.The training data contains 116M English words and 110M German words.
  • English–German results: +2.8 BLEU from global attention, followed by +1.3 BLEU from input feeding, makes the system outperform Bahdanau et al.’s attentional baseline.Reversing the source sentence contributes +1.3 BLEU, and dropout contributes +1.4 BLEU before these attention improvements.
  • English–German results: +1.9 BLEU from unknown replacement and 23.0 BLEU from an ensemble of 8 models produce a new English–German state-of-the-art result.The ensemble outperforms the existing best system by +1.4 BLEU.
  • WMT’15 evaluation: 25.9 BLEU establishes a new WMT’15 English–German state-of-the-art performance, outperforming the existing NMT system with a 5-gram LM reranker by +1.0 BLEU.The models were trained on WMT’14 data but generalized to the newstest2015 test set.
  • German–English results: +2.2 BLEU from attention and up to +1.0 BLEU from input feeding demonstrate progressive gains for German→English, although the systems do not match the state-of-the-art system.The experiments also examine improved alignment functions in this direction.

5 Analysis

The analysis evaluates learning behavior, long-sentence handling, attention architectures, translation examples, and alignment quality on English-German newstest2014 or RWTH data. Attentional models learn robustly, preserve quality on longer sentences, and produce alignment errors comparable to established aligners, with local attention generally outperforming global attention on AER.

  • Learning behavior: Attentional models separate clearly from non-attentional models in learning curves, while input feeding and local attention drive test costs lower.The non-attentional dropout model learns more slowly initially but becomes more robust at minimizing test errors over time.
  • Length analysis: Attentional models handle long sentences more effectively because translation quality does not degrade as sentence length increases.The best model outperforms every other system in all length buckets.
  • Attentional architectures: Dot works well for global attention, general works better for local attention, and concat yields poor performance in the tested configurations.The location-based function learns poor alignments, while the local attention model with the strongest reported configuration continues the architecture comparison.
  • Alignment quality: Local attention models achieve lower AERs than the global model, while the ensemble is not better than local-m despite having good AER.The attentional models achieve AER scores comparable to the Berkeley aligner’s one-to-many alignments on 508 gold-aligned English-German Europarl sentences.
  • Sample translations: Attentional models correctly translate names and doubly negated phrases that non-attentional models often miss because they lack direct source-side connections.The attentional model also demonstrates superiority on the long-sentence example.

6 Conclusion

The paper proposes and evaluates global and local attention mechanisms for neural machine translation. Local attention improves BLEU substantially, while ensemble and alignment analyses demonstrate broader advantages over non-attentional systems.

  • Attention mechanisms: Global attention considers all source positions, whereas local attention attends to only a subset at each time.Both mechanisms were evaluated on English–German WMT translation tasks in both directions.
  • Translation performance: Up to 5.0 BLEU improvement comes from local attention over non-attentional models incorporating techniques such as dropout.This result establishes the effectiveness of local attention over a strong non-attentional baseline.
  • Limitations: Both models fail to translate “passenger experience,” despite one reference using the more elaborate translation “im Widerspruch zu etwas stehen” for “incompatible”.This example identifies a remaining translation limitation.
  • Translation performance: More than 1.0 BLEU separates the English-to-German ensemble from existing best systems on both WMT’14 and WMT’15.The competing systems used NMT models and n-gram language-model rerankers.
  • Analysis: Attention-based NMT outperforms non-attentional systems in many cases, including name translation and handling long sentences.The paper also compares alignment functions and identifies which functions work best for different attentional models.

A Alignment Visualization

The paper visualizes attention-based alignment weights and finds that local attention produces sharper patterns than global attention. Compared with prior English–French visualizations, the alignments are less sharp, possibly because English–German translation is harder.

  • Alignment Visualization: Local attention produces sharper alignment visualizations than global attention, consistent with focusing on a subset of source words at each step.The visualizations compare global and local attention models, including local-m and local-p variants.
  • Alignment Visualization: The global model shows white strides at “reality” and “.” when translating reversed English into German.These gaps reveal an interesting access pattern in the global attention visualization.
  • Alignment Visualization: Compared with Bahdanau et al.’s English–French visualizations, these alignment patterns are less sharp, possibly because English–German translation is harder.The paper identifies this difference as an interesting direction for future examination.
Loading 1508.04025v5…