PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Machine Translation by Jointly Learning to Align and Translate

Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio

arXiv:1409.0473v7cs.CLcs.LGcs.NEstat.ML

TL;DR

Fixed-length encoder–decoder representations may bottleneck neural translation, especially for long sentences. This paper jointly learns soft alignment and translation, improving performance over the basic model and approaching phrase-based translation on English–French.

  • Problem

    Fixed-length encoder–decoder vectors may make neural translation difficult for long sentences, whose performance deteriorates as input length increases.

  • Method

    The model encodes source sentences as vector sequences and adaptively soft-searches relevant positions while jointly learning alignment and translation.

  • Results

    The proposed model significantly outperforms the basic encoder–decoder, is more robust to sentence length, and achieves phrase-based translation performance on English–French.

  • Takeaways & Limitations

    Joint soft alignment and translation improves neural machine translation, particularly its ability to produce good results on longer sentences.

  • Takeaways & Limitations

    The approach requires scoring every source word for each target word and may have limited applicability beyond typical 15–40-word translation sentences.

Abstract

from arXiv · show

Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder-decoders and consists of an encoder that encodes a source sentence into a fixed-length vector from which a decoder generates a translation. In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition.

1 INTRODUCTION

The paper introduces a neural machine translation model that jointly learns to align and translate, replacing the fixed-length sentence representation with adaptive source-position selection. This addresses the encoder–decoder bottleneck and improves performance, especially for longer sentences, reaching performance comparable or close to a conventional phrase-based system on English-to-French translation.

  • Neural machine translation trains a single large neural network to read a sentence and output its translation, unlike separately tuned phrase-based systems.
  • Encoder–decoder models encode a source sentence into a fixed-length vector, from which a decoder generates the translation.
  • Fixed-length encoding can hinder translation of long sentences, and basic encoder–decoder performance deteriorates rapidly as input length increases.
  • The proposed model jointly learns alignment and translation by soft-searching relevant source positions before predicting each target word.It predicts using context vectors associated with selected source positions and previously generated target words.
  • The model encodes the source sentence as a sequence of vectors and adaptively selects among them during decoding instead of compressing the entire sentence into one vector.This reduces the need to squash all source information into a fixed-length representation regardless of sentence length.
  • The approach significantly improves over the basic encoder–decoder, with larger gains for longer sentences and English-to-French performance comparable or close to a conventional phrase-based system.The improvement is also observed for sentences of any length, using a single model.

2 BACKGROUND: NEURAL MACHINE TRANSLATION

Neural machine translation models translation as maximizing the conditional probability of a target sentence given a source sentence. The standard Encoder–Decoder framework encodes the source into a vector and uses a decoder to generate the target sequentially, motivating architectures that learn alignment and translation jointly.

  • 2 BACKGROUND: NEURAL MACHINE TRANSLATION: Translation is framed as finding the target sentence y that maximizes p(y | x), with model parameters learned from parallel sentence pairs.Given a learned conditional distribution, translation is generated by searching for a target sentence.
  • 2 BACKGROUND: NEURAL MACHINE TRANSLATION: The typical neural machine translation architecture has an encoder that reads source sentence x and a decoder that generates target sentence y.This encoder–decoder design is commonly implemented with two recurrent neural networks.
  • 2 BACKGROUND: NEURAL MACHINE TRANSLATION: Prior neural machine translation results were promising, with RNNs using LSTM units achieving close to conventional phrase-based state-of-the-art performance on English-to-French translation.The cited state-of-the-art comparison excludes neural-network components from the conventional phrase-based system.
  • 2 BACKGROUND: NEURAL MACHINE TRANSLATION: The encoder maps source vectors x = (x_1, · · ·, x_Tx) to a context vector c generated from recurrent hidden states.For example, an LSTM can serve as the encoder, with c set to the final hidden state h_T.
  • 2 BACKGROUND: NEURAL MACHINE TRANSLATION: The decoder predicts each next word y_t′ from context c and previously predicted words, decomposing the translation probability into ordered conditionals.With an RNN, each conditional probability is modeled using the preceding target word, decoder hidden state, and context.

3 LEARNING TO ALIGN AND TRANSLATE

The proposed translation architecture combines a bidirectional RNN encoder with a decoder that searches source annotations while generating each target word. Its jointly trained soft-alignment mechanism computes target-specific context vectors, allowing attention to replace the fixed-length bottleneck.

  • Architecture: The architecture uses a bidirectional RNN encoder and a decoder that emulates searching through the source sentence during translation.The encoder and decoder are described in Sections 3.2 and 3.1, respectively.
  • Target-specific context: Each target word is conditioned on a distinct context vector rather than the single fixed context used by the existing encoder–decoder approach.The context vector depends on source annotations produced by the encoder.
  • Soft alignment: The context vector is a weighted sum of source annotations, with weights representing the probabilities that target word yi aligns to source word xj.This weighted sum can be interpreted as the expected annotation over possible alignments.
  • Soft alignment: An alignment model scores how well source inputs around position j match the output at position i using the decoder’s preceding hidden state and the j-th annotation.The score is eij = a(si−1, hj).
  • Attention and training: The decoder’s attention mechanism selects source regions to emphasize, while differentiable soft alignments let gradients train the alignment and translation models jointly.The alignment is directly computed rather than treated as a latent variable.
  • Bidirectional encoding: The bidirectional encoder summarizes preceding and following words in each annotation, which remains focused on nearby source words because RNNs better represent recent inputs.Forward and backward hidden states are combined to form the annotation sequence used by the decoder and alignment model.

4 EXPERIMENT SETTINGS

The experiments evaluate the proposed RNNsearch model against RNNencdec on English-to-French translation using WMT ’14 parallel data. Models are trained and evaluated under matched data, preprocessing, architecture, and optimization settings.

  • Data and evaluation: The evaluation uses ACL WMT ’14 bilingual English-French parallel corpora, comparing RNNsearch with Cho et al.’s RNN Encoder–Decoder under the same training procedures and dataset.The combined corpus is reduced from 850M to 348M words using Axelrod et al.’s data-selection method, with no additional monolingual data.
  • Data and evaluation: The development set combines news-test-2012 and news-test-2013, while news-test-2014 provides a 3003-sentence test set absent from training.The test set is from WMT ’14.
  • Data and evaluation: After tokenization, training uses 30,000 most frequent words per language; all other words map to [UNK], without lowercasing or stemming.No other special preprocessing is applied.
  • Models: The study trains RNNencdec and RNNsearch variants on sentences up to 30 and 50 words, producing RNNencdec-30, RNNsearch-30, RNNencdec-50, and RNNsearch-50.RNNencdec uses 1000-unit encoder and decoder; RNNsearch uses 1000-unit forward and backward encoders plus a 1000-unit decoder.
  • Training and decoding: Each model is trained with minibatch SGD and Adadelta using minibatches of 80 sentences for approximately 5 days, then decoded with beam search.Beam search approximately maximizes the conditional translation probability.

5 RESULTS

RNNsearch outperforms RNNencdec, matches Moses on sentences containing only known words, and is substantially more robust on long sentences. Its soft alignments also capture both monotonic and non-monotonic word correspondences, supporting more reliable translation.

  • Translation performance: RNNsearch outperforms conventional RNNencdec across the reported cases and matches Moses when evaluating sentences containing only known words.Moses additionally uses a separate 418M-word monolingual corpus.
  • Long-sentence translation: RNNencdec performance drops sharply as sentence length increases, whereas RNNsearch-30 and RNNsearch-50 are more robust, especially RNNsearch-50.The proposed model avoids encoding an entire long sentence perfectly into one fixed-length vector.
  • Soft alignments: Visualized annotation weights provide an interpretable view of which source positions influence each generated target word.Each matrix row represents the source-word weights used when generating a target word.
  • Soft alignments: The learned alignments are mostly monotonic but also handle non-monotonic reordering and phrase-level context, including European Economic Area translated as zone économique européen.Soft alignment can attend jointly to words such as the and man, unlike a hard one-to-one alignment.
  • Long-sentence translation: RNNsearch preserves the meaning and details of long test sentences that RNNencdec mistranslates after deviating from the source.The examples include semantic substitutions and omissions in RNNencdec outputs, while RNNsearch produces correct translations.

6 RELATED WORK

Earlier neural approaches mainly served as components or features within existing statistical machine translation systems, whereas this paper pursues a standalone neural translation system. Its alignment mechanism also differs from prior handwriting-synthesis work by supporting reordering, though computing weights for every source word may limit applicability beyond typical translation sentence lengths.

  • Alignment mechanisms: Graves (2013) similarly aligned output symbols with input symbols for handwriting synthesis, using predicted Gaussian-kernel locations, widths, and mixture coefficients to compute annotation weights.The prior work generated handwriting for a given character sequence.
  • Alignment mechanisms: Unlike Graves’s alignment, whose weight modes move only in one direction, this paper’s approach accommodates long-distance reordering needed for translations such as English-to-German.The passage identifies one-directional movement as a severe limitation for machine translation.
  • Alignment mechanisms: Computing an annotation weight for every source word at each target word is not severe for translation sentences of 15–40 words but may limit use in other tasks.The limitation arises from the approach’s per-source-word, per-target-word computation.
  • Neural machine translation: Earlier neural machine translation work largely used neural networks as features or re-rankers within existing statistical machine translation systems.This included scoring source–target phrase pairs and using neural networks as sub-components of existing translation systems.
  • Neural machine translation: This paper instead targets a completely new translation system based on neural networks that operates independently of an existing statistical machine translation system.The authors characterize this approach as a radical departure from earlier neural components.

7 CONCLUSION

The proposed attention-based architecture removes the fixed-length sentence bottleneck by softly selecting relevant source information for each target word. RNNsearch significantly outperforms the conventional encoder–decoder, matches phrase-based translation performance, and remains limited by rare-word handling.

  • Motivation: The conventional encoder–decoder compresses each entire source sentence into a fixed-length vector, which is problematic for translating long sentences.The paper motivates this concern using prior empirical studies.
  • Proposed approach: The proposed architecture softly searches encoder annotations for relevant input words while generating each target word, avoiding whole-sentence fixed-length encoding.This also lets the model focus on information relevant to the next target word.
  • Results: RNNsearch significantly outperforms RNNencdec regardless of sentence length and is more robust to source-sentence length.Qualitative analysis also found that its generated soft alignments correctly align target words.
  • Results: RNNsearch achieves translation performance comparable to existing phrase-based statistical machine translation.The paper characterizes this as striking given the recent emergence of the architecture and neural machine translation family.
  • Future work: Better handling of unknown or rare words remains necessary for wider use and state-of-the-art performance across all contexts.The paper identifies this as a key future challenge.

A MODEL ARCHITECTURE … A.2.1 ENCODER

The paper specifies architectural choices for RNNsearch, including gated recurrent units, a computationally efficient alignment model, and a bidirectional encoder that maps source words to annotations for translation.

  • A.1 ARCHITECTURAL CHOICES: The framework permits independent choices of the RNN activation function and alignment model, with the paper describing the choices used in its experiments.
  • A.1.1 RECURRENT NEURAL NETWORK: The RNN uses gated hidden units that, like LSTM units, are designed to better model and learn long-term dependencies than conventional simple units.
  • A.1.1 RECURRENT NEURAL NETWORK: The gated RNN uses update gates to preserve previous activations and reset gates to control which information from the previous state is reset.
  • A.1.1 RECURRENT NEURAL NETWORK: At each decoder step, output probabilities are computed with a single hidden layer of maxout units and normalized with a softmax function.
  • A.1.2 ALIGNMENT MODEL: The alignment model is a single-layer multilayer perceptron, chosen because it is evaluated Tx × Ty times for each sentence pair and can pre-compute terms independent of i.
  • A.2 DETAILED DESCRIPTION OF THE MODEL: The detailed RNNsearch architecture takes source sentences as 1-of-K coded word vectors and produces translated sentences in the same coding format.Kx and Ky are the source- and target-language vocabulary sizes, while Tx and Ty are the corresponding sentence lengths.
  • A.2.1 ENCODER: The encoder computes forward and backward recurrent states, sharing the word embedding matrix E between directions but not their weight matrices.The forward and backward states are concatenated to form annotations (h1, h2, · · ·, hTx).

A.2.2 DECODER · A.2.3 MODEL SIZE

The decoder combines the previous target word, recurrent state, and dynamically recomputed context to generate each target word. Its architecture uses gated recurrent updates, an alignment-based context, a deep maxout output, and fixed dimensions across models.

  • A.2.2 DECODER: The decoder computes its hidden state from the previous target word, recurrent state, and context vector supplied by the encoder annotations.The update uses a tanh candidate state together with reset and update gates.
  • A.2.2 DECODER: The target-language word embedding matrix E and weight matrices W, U, and C parameterize the decoder’s gated recurrent computation.The dimensions are W, Wz, Wr ∈ R^n×m; U, Uz, Ur ∈ R^n×n; and C, Cz, Cr ∈ R^n×2n.
  • A.2.2 DECODER: The context vector ci is recomputed at every decoding step by an alignment model over the source annotations.The alignment model uses annotations hj and parameters va, Wa, and Ua.
  • A.2.2 DECODER: If ci is fixed to the final forward encoder state, the model reduces to the RNN Encoder–Decoder architecture.The cited formulation identifies this fixed-context condition explicitly.
  • A.2.2 DECODER: The probability of each target word yi is defined from the preceding decoder state, context vector, and previously generated word.The decoder’s output is further described as a deep output with a single maxout hidden layer.
  • A.2.2 DECODER: The deep output uses weight matrices Wo, Uo, Vo, and Co together with a single maxout hidden layer.The matrix dimensions include Wo ∈ R^Ky×l, Uo ∈ R^2l×n, Vo ∈ R^2l×m, and Co ∈ R^2l×2n.
  • A.2.3 MODEL SIZE: All models use hidden-layer size n = 1000, target-word embedding dimensionality m = 620, maxout-layer size l = 500, and alignment-model size n′ = 1000.These dimensions apply across the models used in the paper.

B TRAINING PROCEDURE · B.1 PARAMETER INITIALIZATION

Parameter initialization uses distinct schemes for recurrent matrices, attention-related parameters, biases, and other weight matrices. Recurrent matrices are orthogonal, while remaining parameters are initialized with specified Gaussian distributions or zeros.

  • B.1 PARAMETER INITIALIZATION: Recurrent weight matrices are initialized as random orthogonal matrices.This includes U, Uz, Ur, ←− U, ←− Uz, ←− Ur, −→ U, −→ Uz, and −→ Ur.
  • B.1 PARAMETER INITIALIZATION: Wa and Ua are initialized elementwise from a Gaussian distribution with mean 0 and variance 0.0012.
  • B.1 PARAMETER INITIALIZATION: All elements of Va are initialized to zero.
  • B.1 PARAMETER INITIALIZATION: All bias vectors are initialized to zero.
  • B.1 PARAMETER INITIALIZATION: Any other weight matrix is initialized elementwise from a Gaussian distribution with mean 0 and variance 0.012.
  • B.1 PARAMETER INITIALIZATION: The initialization procedure distinguishes recurrent matrices from attention parameters, biases, and other weight matrices.The cited schemes are orthogonal initialization, Gaussian sampling, or zero initialization, depending on the parameter group.

B.2 TRAINING · C TRANSLATIONS OF LONG SENTENCES

Training used SGD with adaptive learning rates, gradient clipping, minibatches, and length-based batching to reduce wasted computation. The paper also presents long-sentence translations from two neural models alongside gold-standard and Google Translate outputs.

  • B.2 TRAINING: SGD training used Adadelta to automatically adapt each parameter’s learning rate, with ϵ = 10^-6 and ρ = 0.95.The optimization algorithm was stochastic gradient descent.
  • B.2 TRAINING: Gradients were normalized to an L2-norm threshold of 1 whenever the norm exceeded that value.This normalization was applied to the gradient of the cost function at each update.
  • B.2 TRAINING: Each SGD update direction was computed from a minibatch of 80 sentences.
  • B.2 TRAINING: Every 20th update, 1600 sentence pairs were retrieved, sorted by length, and split into 20 minibatches to reduce computation wasted on padding.The implementation’s update time was proportional to the longest sentence in each minibatch.
  • B.2 TRAINING: The training data was shuffled once before training and then traversed sequentially through the length-organized minibatches.
  • C TRANSLATIONS OF LONG SENTENCES: Table 3 presents translations from RNNenc-50 and RNNsearch-50 for test-set source sentences containing 30 words or more.Each example also includes the gold-standard translation and a Google Translate output produced on 27 August 2014.
Loading 1409.0473v7…