PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Machine Translation and Sequence-to-sequence Models: A Tutorial

Graham Neubig

arXiv:1703.01619v1cs.CLcs.LGstat.ML

TL;DR

The tutorial addresses how neural machine translation and sequence-to-sequence models can represent and transform sequential data, including challenges from long-distance dependencies and fixed-size representations. It develops the subject from statistical and neural foundations through recurrent, LSTM, encoder-decoder, and attention-based methods. The tutorial concludes by giving readers tools to apply these models to their own sequential-data applications.

  • Problem

    Sequence-to-sequence systems must handle long-distance dependencies and the challenge of storing arbitrarily long sentences in fixed-size encoder-decoder representations.

  • Method

    The tutorial progresses from statistical machine translation and language modeling to recurrent networks, LSTMs, encoder-decoder models, attention, and training methods such as automatic differentiation.

  • Results

    Encoder-decoder models can perform translation with similar accuracy to heavily engineered machine-translation systems, while recurrent networks can pass information across arbitrary consecutive time steps.

  • Takeaways & Limitations

    The tutorial provides readers with tools to apply neural sequence-to-sequence models to language and other sequential-data applications.

  • Takeaways & Limitations

    Greedy search is not guaranteed to find the highest-probability translation, and large vocabularies make rare-word translation difficult with limited data.

Abstract

from arXiv · show

This tutorial introduces a new and powerful set of techniques variously called "neural machine translation" or "neural sequence-to-sequence models". These techniques have been used in a number of tasks regarding the handling of human language, and can be a powerful tool in the toolbox of anyone who wants to model sequential data of some sort. The tutorial assumes that the reader knows the basics of math and programming, but does not assume any particular experience with neural networks or natural language processing. It attempts to explain the intuition behind the various methods covered, then delves into them with enough mathematical detail to understand them concretely, and culiminates with a suggestion for an implementation exercise, where readers can test that they understood the content in practice.

1 Introduction

The tutorial presents machine translation as a representative sequence-to-sequence task and progressively develops neural methods for modeling sequential data. It moves from language-modeling foundations through recurrent networks to encoder-decoder and attention-based models.

  • Background: Machine translation converts a source-language word sequence into a target-language word sequence across languages and content.The broader goal is accurate conversion over varied languages and material.
  • Background: Sequence-to-sequence models are a broader class that maps one sequence to another, covering machine translation and other tasks.The tutorial notes that even computer programs can be viewed as sequence-to-sequence mappings, though this is not always natural.
  • Background: The tutorial uses machine translation because it is useful, widely recognized, central to model development, and informed by techniques from other tasks.These motivations support using translation examples to explain sequence-to-sequence difficulties and methods.
  • Tutorial progression: Language-modeling foundations progress from n-gram and log-linear models to feed-forward and recurrent neural networks.The sequence introduces probability modeling, feature-based learning, neural combination of information, and memory across time steps.
  • Sequence-to-sequence models: Encoder-decoder models encode an input sequence into a vector and decode it into an output sentence, while attention focuses on different input parts during generation.Attention is presented as more efficient, intuitive, and often more effective than the simpler encoder-decoder approach.

2 Statistical MT Preliminaries

Statistical machine translation formalizes translation as selecting the most probable target sentence given a source sentence. The framework separates model design, parameter learning from aligned data, and search for the best hypothesis.

  • Formal framework: Statistical machine translation models the probability P(E | F; θ) and selects the target sentence E that maximizes it for source sentence F.The parameters θ specify the probability distribution.
  • Formal framework: Training learns model parameters θ from aligned source-target sentences called parallel corpora.These data provide the basis for estimating the translation model.
  • Core problems: Building a translation system requires solving modeling, learning, and search problems.Modeling chooses the probability structure, learning estimates parameters, and search finds the best hypothesis through decoding.

3 n-gram Language Models

This section introduces language models for assigning probabilities to target-language sentences, then develops n-gram approximations and smoothing for unseen word sequences.

  • Language-model goals: Language models estimate P(E) to assess sentence naturalness and generate text by sampling from the target distribution.They can support fluency assessment, grammar checking, error correction, and later neural translation models.
  • Word-by-word probability computation: Directly modeling full-sentence probabilities is difficult because sentence length varies and the number of possible word combinations is large.The decomposition turns the problem into calculating the next word given preceding words.
  • Word-by-word probability computation: Sentence probability can be decomposed into a product of conditional next-word probabilities, including an end-of-sentence symbol.The end symbol identifies when the sentence terminates and determines its final length.
  • Count-based n-gram models: Unsmoothed count-based models assign zero probability to unseen word strings, causing the probability of an entire unseen sentence to become zero.This prevents useful assessment of whether new sentences are natural and limits generation of new outputs.
  • Count-based n-gram models: n-gram models approximate next-word probabilities using a fixed window of n −1 previous words, with unigram, bigram, and trigram models corresponding to n = 1, 2, and 3.Their parameters are learned from training data, commonly using maximum likelihood estimation based on word-string counts.
  • Smoothing: Interpolation combines estimates from different n-gram orders so every vocabulary word receives nonzero probability when α > 0.The unigram distribution receives the held-out probability mass, improving robustness to low-frequency phenomena.

4 Log-linear Language Models

Log-linear language models predict the next word from flexible context features rather than count-based n-grams. They convert feature-based scores into probabilities and learn parameters by minimizing training loss with stochastic optimization.

  • Model formulation: A feature function maps the context to a real-valued vector x ∈ R^N describing it with N features.The tutorial focuses on features over the context rather than the current word.
  • Model formulation: A one-hot feature vector represents a word by setting the element associated with its vocabulary ID to one and all others to zero.Multiple such vectors can be concatenated to represent several preceding words.
  • Model formulation: Log-linear language models calculate next-word probabilities from features of the context, offering more flexible feature design than standard n-gram models.Features may represent one or multiple preceding words and other context properties.
  • Model formulation: Model parameters produce a score for every vocabulary word, with a bias capturing each word’s overall likelihood.For sparse features, scores can be computed by adding weight-matrix columns for active features.
  • Model formulation: The softmax function exponentiates and normalizes arbitrary scores so they become probabilities between 0 and 1 that sum to 1.Applying softmax creates the mapping from context features to language-model probabilities.
  • Learning model parameters: Training defines negative log likelihood as the loss and uses stochastic gradient descent to update parameters toward higher likelihood on the training data.Learning-rate choices, decay, AdaGrad, shuffling, and early stopping help keep optimization stable and limit overfitting.

5 Neural Networks and Feed-forward Language Models

Neural networks learn more sophisticated functions for language modeling, improving probability estimates with less feature engineering. Feed-forward models address combination features through nonlinear transformations, while neural networks also offer more generalizable context representations and efficient training.

  • Neural networks learn more sophisticated functions to improve probability estimates with less feature engineering.
  • 5.1 Potential and Problems with Combination Features: Combining context words helps resolve cases where separate features cannot distinguish natural from unnatural phrases, but expands parameters from O(|V|^2) to O(|V|^3).
  • 5.2 A Brief Overview of Neural Networks: Multi-layer perceptrons represent nonlinear functions by transforming inputs into a hidden space where the output can be computed linearly.
  • 5.2 A Brief Overview of Neural Networks: Differentiable nonlinearities such as tanh and ReLU make gradient-based training more practical than the step function.tanh has a continuous gradient, while ReLU avoids tanh saturation for large-magnitude inputs.
  • 5.3 Training Neural Networks: Automatic differentiation represents computations as graphs and provides separate graphs for prediction and loss calculation during training.Nodes represent inputs or operations such as multiplication, addition, tanh, and squared error.
  • 5.5 Neural-network Language Models: Neural-network language models generalize across similar words, represent combinations of words feature-efficiently, and naturally handle skipped context positions.

6 Recurrent Neural Network Language Models

Recurrent neural networks extend neural language models with recurrent state, allowing information to pass across arbitrary time steps and capture long-distance dependencies. The section also explains vanishing and exploding gradients and motivates LSTMs as a remedy for diminishing gradients.

  • Motivation: RNN language models extend feed-forward models with mechanisms that capture long-distance dependencies in language.They are presented as more powerful and generalizable than n-gram models for sequential modeling.
  • Long-distance dependencies: Finite-history models cannot reliably capture grammatical agreement, selectional preferences, or topic and register consistency across intervening words.Examples include gender agreement, verb conjugation, commonsense role compatibility, and document-level consistency.
  • Recurrent neural networks: RNNs pass information from h_t−1 to h_t, enabling features such as sentence-subject gender to propagate across arbitrarily many consecutive time steps.The recurrent connection is represented by the W_hh h_t−1 term, and the network can be unrolled to expose information flow through time.
  • Recurrent language models: RNN language models feed the previous word into a recurrent state that is expected to retain information about earlier words.This makes directly feeding a longer fixed context unnecessary when the recurrent state successfully stores that information.
  • Training difficulties: Vanishing and exploding gradients arise when repeated recurrent transformations exponentially diminish or amplify gradients during backpropagation.Diminishing gradients can leave early parameters with too little influence from losses received later in a sequence.
  • Long short-term memory: LSTMs add a memory cell whose recurrent derivative is exactly one, preventing stored information from suffering vanishing gradients and improving long-distance dependency capture.The tutorial notes that the cell is designed specifically to address diminishing gradients, while a footnote qualifies the derivative as not exactly one in all respects.

7 Neural Encoder-Decoder Models

The tutorial returns from language modeling to neural machine translation by modeling the probability of a target sequence E given a source sequence F.

  • Neural encoder-decoder models: Neural encoder-decoder translation models the conditional probability P(E | F) of an output sequence given an input sequence.Earlier sections instead focused on calculating the probability P(E) of a target sequence without conditioning on a source sentence.

7.1 Encoder-decoder Models

Encoder-decoder models encode the source sentence with one RNN and use its final state to initialize a decoder RNN that generates the target sentence. This provides a straightforward model of P(E | F) and can achieve translation accuracy similar to heavily engineered systems with additional techniques.

  • Encoder-decoder Models: An encoder RNN processes the source sentence F and represents its information in a final hidden-state vector.The final encoder state is intended to encode all information from the source sentence.
  • Encoder-decoder Models: The encoder-decoder computation graph combines source encoding, decoder recurrence, and softmax output into a model of P(E | F).The model is illustrated in the encoder-decoder computation graph.
  • Encoder-decoder Models: A decoder RNN is initialized with the encoder’s final state, conditioning target-word probabilities on the source sentence.At each step, the decoder also uses the previous target word before applying a softmax to produce probabilities.
  • Encoder-decoder Models: A basic encoder-decoder model can perform translation with similar accuracy to heavily engineered machine-translation systems.The cited result uses additional techniques, including beam search, a different encoder, and ensembling.

7.2 Generating Output

Generating output from an encoder-decoder model can involve sampling, greedy or n-best search, and beam search. Greedy search is not guaranteed to find the highest-probability translation, while beam search broadens the hypotheses considered but can introduce a bias toward short sentences.

  • Generating Output: Output generation may randomly sample from P(E | F), select the 1-best sequence, or return the n highest-probability outputs.The preferred criterion depends on the application, such as seeking varied responses in dialogue or a best translation.
  • Generating Output: Ancestral sampling generates outputs by sampling each next word from the conditional distribution given previously sampled inputs.The process can also accumulate word-level log probabilities to compute the sampled sentence’s overall score without numerical precision problems.
  • 7.2.2 Greedy 1-best Search: Greedy search selects the highest-probability next word at every time step, but it is not guaranteed to find the translation with the highest probability.The tutorial illustrates this failure with a search graph and contrasts it with exact sampling from P(E | F).
  • Beam Search: Beam search retains the b best hypotheses at each time step instead of only one, expanding and pruning candidate continuations.The tutorial illustrates beam search with b = 2 and compares hypotheses using log probabilities.
  • Search Biases: Larger beam sizes often strengthen a length bias toward shorter sentences because each additional word multiplies another probability into the sentence score.Proposed remedies include a length prior and normalizing log probability by target length.

7.3 Other Ways of Encoding Sequences

The tutorial surveys alternative sequence encoders beyond linear recurrent encoding, including reverse, bidirectional, convolutional, and tree-structured networks. These approaches alter information flow to improve learning, capture local features, or follow syntactic structure, while retaining distinct assumptions and limitations.

  • Reverse and Bidirectional Encoders: Reversing the encoder shortens dependencies for words at the beginnings of sentences, helping bootstrap training and enabling effective encoder-decoder models.For aligned beginning words, dependency distance can be reduced to 1, with later pairs at distance 2t−1.
  • Reverse and Bidirectional Encoders: Reverse encoding assumes that source and target word orders are similar, whereas bidirectional encoding is described as more robust to ordering differences.The assumption is especially appropriate for languages such as English and French with subject-verb-object ordering.
  • Convolutional Neural Networks: Convolutional networks combine information from local word-sequence segments and pool variable-width representations into a fixed-size vector.Pooling options include average, max, and k-max pooling.
  • Convolutional Neural Networks: CNNs offer simple feature detection, sentence-wide accumulation, and reduced exposure to vanishing gradients, but are less expressive for patterns beyond their filter width.They have been found effective for text classification, where identifying indicative features is more important than modeling an overall view of the content.
  • Tree-structured Networks: Tree-structured networks combine word representations according to syntactic structure, recursively computing parent representations from left and right child states.This bottom-up composition follows coherent grammatical phrases and exploits language’s compositional structure.
  • Tree-structured Networks: Recursive networks inherit the vanishing-gradient problem, while tree LSTMs are presented as an adaptation that fixes it.The tutorial also notes that different tree-composition functions suit different NLP tasks.

7.4 Ensembling Multiple Models

Ensembling combines predictions from multiple independently trained encoder-decoder models. During search, their next-word probabilities are averaged to smooth individual model errors.

  • Ensembling Multiple Models: Ensembling combines multiple independently trained models because their different errors can be smoothed when their predictions are combined.The models may differ through independent random initialization before training.
  • Ensembling Multiple Models: At each decoding step, the ensemble uses the average probability from N models to search among output hypotheses.The averaged probability is used directly for selecting or extending hypotheses during search.

7.5 Exercise

The exercise asks readers to build an encoder-decoder translation model, train it, generate translations, and evaluate the outputs. Suggested extensions compare beam and greedy search, alternative encoders, and ensembling.

  • Exercise: The exercise builds an encoder-decoder translation model that generates translations.It is intended to test whether readers understood the chapter’s methods in practice.
  • Exercise: Implementation requires extending an RNN language model to encode a source sentence and calculate the initial hidden state.The exercise also requires implementing the training loss and parameter updates.
  • Exercise: Readers generate development-set translations with greedy search and compare them with reference translations.Automatic evaluation can use BLEU in addition to qualitative inspection.
  • Exercise: Potential improvements include implementing beam search, trying an alternative encoder, and adding ensembling.The exercise explicitly encourages comparing beam search with greedy search.

8 Attentional Neural MT

Attention addresses encoder-decoder difficulties caused by long dependencies and the need to compress arbitrarily long sentences into one fixed-size vector. It instead retains representations for individual source words and dynamically combines them while decoding, improving translation accuracy and interpretability.

  • Problems of Representation in Encoder-Decoders: Standard encoder-decoder models retain long-distance dependencies and compress sentences of arbitrary length into a fixed-size hidden vector.Limited data and large parameter counts can also make these models harder to learn without overfitting.
  • Attention: Attention keeps a vector for every source word and references these vectors at each decoding step, providing a variable-length sentence representation.Longer inputs therefore provide more vectors to reference than shorter inputs.
  • Attention: A bidirectional RNN creates a representation for each source word, which is assembled into a matrix whose columns correspond to input words.This matrix preserves a separate encoded representation for every source position.
  • Calculating Attention: The attention vector weights the columns of the source-representation matrix to produce a context vector for the current decoding step.Its elements are generally between zero and one and sum to one; larger values indicate greater focus on a source word.
  • Calculating Attention: Attention scores are derived from the decoder state and source-word representations, normalized with softmax, and used with the decoder state to predict the next target word.The resulting context vector makes source encodings directly available when calculating output probabilities.

8.4 Ways of Calculating Attention Scores

The tutorial presents dot-product, bilinear, and multi-layer perceptron functions for calculating attention scores, each trading off parameterization, flexibility, and representational constraints.

  • Dot product: Dot-product attention calculates similarity without additional parameters but requires encoder and decoder representations to share the same space.The score can also be computed efficiently across all source words using the concatenated source-encoding matrix.
  • Implementation: The tutorial notes that attention operations can be combined for more efficient implementation, especially on GPUs.The same combined calculation can be applied to the other attention functions.
  • Bilinear functions: Bilinear attention applies a learned linear transform before the dot product, allowing encoder and decoder vectors to have different dimensions.This flexibility introduces |h(f)| × |h(e)| additional parameters that may be difficult to train properly.
  • Multi-layer perceptrons: A multi-layer perceptron provides a more flexible attention score than the dot product, typically with fewer parameters than the bilinear function.The tutorial identifies this approach as generally producing good results.

8.5 Copying and Unknown Word Replacement

Attention supports interpretable word correspondence and unknown-word replacement, while several priors and alternative mechanisms impose alignment structure or provide additional ways to access input information.

  • Attention and copying: Attention increases translation accuracy and makes source-target word correspondences easier to inspect through alignment visualizations.These visualizations can aid error analysis.
  • Unknown word replacement: Unknown-word replacement copies the source word receiving the highest attention weight when the decoder emits the unknown token.A translation dictionary can instead map that source word to its most probable target-language counterpart.
  • Intuitive priors on attention: Attention priors can encode position, local movement, fertility, and bilingual symmetry assumptions about cross-language alignments.These assumptions encourage diagonal alignments, local source movements, appropriate translation counts, or agreement between translation directions.
  • Fertility and coverage: Coverage-related methods address repeated or omitted words by penalizing excessive or insufficient attention and incorporating coverage into training or decoding.The tutorial connects these failures to violations of fertility assumptions.
  • Intuitive priors on attention: Bilingual symmetry is particularly effective among the tested approaches and is enforced by jointly training models whose alignment matrices are similar in both directions.The constraint links alignments for translation from F to E with those for translation from E to F.
  • Further directions: Further attention extensions include hard binary attention, supervised alignment training, and memory networks for reading or writing processing state.The chapter exercise asks readers to implement attentional translation, train it, generate translations, and evaluate them.

9 Conclusion

The tutorial progresses from n-gram models to attention-based sequence-to-sequence models and surveys applications, practical challenges, and directions beyond its introductory scope.

  • Summary: The tutorial moves from n-gram language models through increasingly sophisticated methods, culminating in attention as state-of-the-art for many sequence-to-sequence tasks.It frames attention as the endpoint of the tutorial’s progression.
  • Scope: The tutorial covers foundational material while noting that advanced topics in this active research field remain beyond its scope.It aims to give readers tools for applying these models to their own sequential-data problems.
  • Challenges: Large vocabularies remain difficult because rare-word translation is data-limited and computation becomes burdensome.Suggested responses include character or subword units and broad-coverage translation dictionaries.
  • Challenges: The tutorial distinguishes maximizing target-sentence likelihood P(E | F) from optimizing the accuracy of generated translations.It points to methods that directly consider generated translation quality during training.
  • Future directions: Multilingual learning can use data from multiple languages jointly or transfer a model from one language pair to another through fine-tuning.These approaches extend the tutorial’s earlier two-language setup.
  • Applications: Sequence-to-sequence models are applied across tasks including dialogue, summarization, speech recognition, speech synthesis, image captioning, and image generation.These examples illustrate the breadth of applications discussed in the conclusion.
Loading 1703.01619v1…