PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Machine Translation: A Review and Survey

Felix Stahlberg

arXiv:1912.02047v2cs.CL

TL;DR

Early NMT models faced limitations including poor long-sentence translation, inefficient decoding, and difficulty extending systems with new vocabulary. The paper surveys NMT’s foundations, architectures, design choices, and research trends, concluding that attention addresses the limited capacity of fixed context vectors while broader challenges remain.

  • Problem

    NMT research must address long-sentence translation errors, costly or sequential decoding, vocabulary extension, and mismatches between training objectives and evaluation metrics.

  • Method

    The paper traces NMT from embeddings and encoder-decoder networks, reviews recurrent, convolutional, and attention-based architectures, and surveys design choices and advanced topics.

  • Results

    Attention provides different source context vectors at each target position, addressing the limited capacity of a fixed context vector in NMT.

  • Takeaways & Limitations

    NMT has become the de facto standard for large-scale machine translation, with attention-based context selection addressing a central weakness of fixed-length encodings.

  • Takeaways & Limitations

    Fully trained NMT systems cannot be extended with new words, limiting customization to new domains and vocabularies.

Abstract

from arXiv · show

The field of machine translation (MT), the automatic translation of written text from one natural language into another, has experienced a major paradigm shift in recent years. Statistical MT, which mainly relies on various count-based models and which used to dominate MT research for decades, has largely been superseded by neural machine translation (NMT), which tackles translation with a single neural network. In this work we will trace back the origins of modern NMT architectures to word and sentence embeddings and earlier examples of the encoder-decoder network family. We will conclude with a survey of recent trends in the field.

1. Nomenclature

The paper establishes notation for source and target sentences, vocabularies, indices, projections, and matrices, then situates NMT in embeddings and encoder-decoder architectures.

  • Nomenclature: The source sentence is x, its translation is y, and source and target vocabularies are represented as indexed subword or word sets.Tokens are represented by natural-number indices, with vocabulary size defined by the cardinality of each vocabulary.
  • Nomenclature: A projection function π_k selects the k-th entry of a tuple or vector.The notation is defined by π_k(z_1, . . ., z_k, . . ., z_n) = z_k.
  • Word Embeddings: Continuous word embeddings map vocabulary items to lower-dimensional real-valued vectors and can capture morphological, syntactic, and semantic similarity.The embedding dimension d is normally much smaller than the vocabulary size, and embedding matrices are typically trained jointly with the network.
  • Word Embeddings: Contextualized word embeddings depend on the entire input sentence rather than a single embedding matrix and have advanced several NLP benchmarks.Common approaches use LSTM or Transformer sequence models, with differing ways of generating word representations.
  • Phrase Embeddings: Phrase and sentence embeddings compose word representations recursively or convolutionally to produce fixed-dimensional representations of larger spans.Recurrent autoencoders merge parent representations according to a binary tree, while convolutional models produce n-gram representations at increasing depths.
  • Encoder-Decoder Networks: Encoder-decoder NMT models encode a source sentence and generate the target sentence by factorizing the target distribution into conditional probabilities.The encoder may produce a fixed-length representation, while the decoder uses recurrent states and previous target-token embeddings to model successive outputs.

6. Attentional Encoder-Decoder Networks

Attentional encoder-decoder networks address fixed-length encoding limitations by selecting source information dynamically during decoding. The section presents attention, recurrent RNNsearch, multi-head attention, positional encodings, batching, and three broad NMT architecture families.

  • Attention: Attention weights are normalized similarity scores, and each query’s output is a weighted sum of value vectors.The score matrix is normalized across columns so the weights for each query sum to one.
  • Attention: Attention replaces a fixed-length source representation with time-dependent context vectors computed from decoder queries and encoder states.Decoder states serve as queries, while encoder states provide keys and values; the resulting context represents relevant source information.
  • Multi-head attention: Multi-head attention performs H attention operations, concatenates their outputs, and typically divides head dimensionality by H to control parameter growth.The query, key, and value vectors are linearly transformed for each head.
  • Interpretability: Multi-head attention makes a single attention matrix harder to derive, reducing interpretability compared with a single-head visualization.The section also notes that attention matrices should not be treated as soft traditional SMT alignments.
  • RNNsearch: RNNsearch uses decoder state, previously generated token, and attention context to compute the next target-token distribution.Its context vector is a distributed representation of the relevant parts of the source sentence.
  • Architecture comparison: NMT architectures are recurrent, convolutional, or self-attention-based, while sharing an encoder-decoder structure with decoder attention to the encoder.The architectures generate output probabilities through a linear projection followed by softmax; self-attention models additionally use positional encodings to make representations position-sensitive.

7. Neural Machine Translation Decoding

NMT decoding is the inference problem of finding the most likely target translation from a source sentence. It is difficult because the search space grows exponentially and model likelihood can diverge from translation quality.

  • The Search Problem in NMT: Decoding finds the most likely translation ˆy for a given source sentence x, since translation probabilities alone do not generate the target sentence.This task is also called inference.
  • The Search Problem in NMT: With a vocabulary of 32,000 tokens, translations of 20 words or fewer already exceed 10^82 possibilities, making complete enumeration impossible.The search space grows exponentially with sequence length.
  • The Search Problem in NMT: NMT decoding is additionally complicated because the most likely translation may differ from the best translation due to common model errors.The passage identifies this mismatch as having implications for search.
  • Decoding algorithms: Greedy search and beam search build translations left to right, scoring partial prefixes with next-token conditional probabilities.Both procedures follow NMT’s left-to-right factorization and operate synchronously by target position.

Algorithm 1 OneStepRNNsearch(sprev, yprev, h)

OneStepRNNsearch performs one recurrent decoding step by computing attention, updating the decoder state, and producing the next-token distribution.

  • Attention and state update: The procedure first computes attention weights over encoder states from the previous decoder state.The weights are obtained by applying softmax to attention scores.
  • Attention and state update: It then updates the recurrent decoder state using the previous state, previous target token, and attention context.The updated state summarizes the information used for the current decoding step.
  • Output distribution: Finally, it computes and returns the distribution over the next target token together with the updated decoder state.The distribution is represented as p ∈ R^|Σtrg|.

Algorithm 2 GreedyRNNsearch(sinit, h)

NMT decoding generates translations incrementally, either greedily or with beam search, while ensembling combines predictions from multiple models to improve accuracy at substantial computational cost.

  • Search trade-offs: Greedy search can suffer from the garden-path problem because an early locally best choice may lead to poor later continuations.Beam search mitigates this by comparing multiple partial hypotheses before selecting the next set of expansions.
  • Greedy decoding: Greedy decoding repeatedly selects the highest-probability target token until emitting the end-of-sentence symbol.It uses the posterior component for each candidate token and iteratively calls the recurrent decoder.
  • Beam search: Beam search retains n promising partial hypotheses instead of only the single best expansion at each time step.Each hypothesis stores a translation prefix, accumulated score, and decoder state.
  • Ensembling: Ensembling combines predictions from K NMT networks using arithmetic or geometric averaging, replacing the single-model conditional probabilities.The networks may be trained independently or share some training iterations.
  • Ensembling: Ensembling consistently outperforms single NMT by a large margin, but decoding becomes slower because every time step requires K model passes and softmax computations.It can also increase CPU/GPU switching, communication overhead, and implementation difficulty.
  • Checkpoint averaging: Checkpoint averaging produces one model without increasing decoding time, but it cannot be applied to independently trained models.It smooths training fluctuations rather than combining independently learned model parameters.

8. Open Vocabulary Neural Machine Translation

Word-based NMT is constrained by fixed vocabularies, creating parameter, training, and out-of-vocabulary problems. Character and subword units address these issues, but their relative advantages and linguistic adequacy remain unsettled.

  • Vocabulary constraints: Fixed-shape embedding matrices require NMT to use a fixed, predefined vocabulary.Vocabulary size directly affects the embedding and model dimensions.
  • Vocabulary constraints: Larger vocabularies inflate models, reduce feasible batch sizes, and can produce noisier gradients, slower training, and worse performance.Embedding matrices make up most parameters in a standard RNNsearch model.
  • Vocabulary constraints: Word-based NMT struggles with OOV words because test distributions vary across genres, corpora, and time, while rare words may be mapped to the same UNK token.Translation-specific methods replace UNKs afterward, whereas model-specific methods modify training or output estimation.
  • Vocabulary constraints: Word-based approaches remain limited because UNK replacement cannot distinguish hypotheses differing only in OOV words, and trained systems cannot easily add new domain-specific words.These limitations are especially relevant for commercial domain customization and proper names.
  • Alternative units: Character and subword models decompose words into finer-grained units, with subwords currently serving as the most common NMT translation units.Character-based systems can avoid closed-vocabulary restrictions, while BPE and related methods offer a compromise between characters and full words.
  • Alternative units: There is no conclusive agreement on whether characters or subwords are better for NMT, although character systems may outperform subwords while remaining harder to deploy.Automatically learned subwords also do not necessarily correspond to linguistic units, and linguistically motivated alternatives do not always improve performance.

9. Using Monolingual Training Data

Because parallel MT data is scarce while monolingual text is abundant, NMT research incorporates monolingual data through decoder fusion, data augmentation, and modified training objectives.

  • Motivation: Parallel training data is difficult and expensive to acquire, whereas untranslated monolingual data is usually abundant.This imbalance motivates methods that incorporate monolingual text into NMT.
  • Language-model integration: Language-model integration combines a separately trained RNN language model with NMT through shallow or deep fusion.Deep fusion uses a controller network to dynamically adjust the relative weights of the language-model and NMT signals.
  • Data augmentation: Data augmentation adds target-language monolingual sentences to parallel training data by constructing synthetic or modified source sides.Back-translation is described as the most successful strategy among these approaches.
  • Data augmentation: Back-translation requires balancing synthetic data with real parallel data, so it can use only a small fraction of available monolingual text.Over-sampling real data can partially correct the imbalance, but very high rates often perform poorly.
  • Data augmentation: Adding noise to back-translated sentences can improve translation quality and make training more robust to high synthetic-to-real data ratios.This addresses part of the balancing problem in synthetic data augmentation.
  • Training objectives: Other methods incorporate monolingual data by adding autoencoder or reconstruction terms to the NMT training objective.Reconstruction error is also central to unsupervised dual-learning approaches.

10. NMT Model Errors

NMT assigns strong translation scores but faces search and model errors, especially a tendency toward overly short translations. Length-oriented corrections and coverage mechanisms address this deficiency, though their applicability varies by architecture.

  • Sentence length: Beam size 10 maximizes BLEU on the cited Transformer test set, while wider beams reduce performance as translations become too short.The model assigns excessive probability mass to short hypotheses found through more exhaustive search.
  • Sentence length: The length deficiency can reflect locally normalized training, which underestimates the margin between correct and shorter translations.The cited explanation also concerns the difficulty of estimating the probability budget for longer continuations.
  • Sentence length: Small-beam decoding can find good translations quickly, but beam size must be tuned across training techniques, architectures, and language pairs.The cited discussion notes that wider-beam search gains may be obscured by NMT length deficiency.
  • Length and coverage remedies: Model-agnostic remedies add score corrections favoring longer outputs, including length normalization, tunable penalties, and word rewards.Other approaches modify attention or add explicit coverage and fertility models to reduce under- and over-translation.
  • Length and coverage remedies: Attention-based remedies are readily applicable to single encoder-decoder attention models but not directly to architectures with multiple attention modules.The limitation specifically concerns models such as ConvS2S and Transformer.

11. NMT Training

NMT training relies mainly on gradient-based cross-entropy optimization, but training remains variable and imperfectly aligned with decoding and evaluation. Regularization and alternative objectives address these issues while introducing their own limitations.

  • Training objectives: NMT models are trained with backpropagation and gradient-based optimization, commonly using cross-entropy loss.Modern recurrent, convolutional, and Transformer architectures help address vanishing-gradient problems, though optimization remains incomplete.
  • Training reliability: NMT training is variable: models with identical architecture, data, and iteration counts can differ by up to 1 BLEU.Ensembling consistently outperforms single models by a large margin, suggesting difficulties in training individual models.
  • Training objectives: Cross-entropy training can be interpreted equivalently as maximizing data likelihood or estimating cross-entropy to sequence-level, token-level, and Dirac target distributions.The survey emphasizes the equivalence of these interpretations.
  • Regularization: Over-parameterized NMT models can overfit, motivating regularizers such as early stopping, dropout, and label smoothing.A standard subword Transformer may contain 200–300 million parameters.
  • Regularization: Label smoothing produces smoother distributions but has objectionable sequence-level effects because its target distribution does not assign a fixed probability to the correct sequence.The resulting distribution is sharper for short correct sequences and smoother for long ones.
  • Training and decoding mismatch: Standard maximum-likelihood training exposes a mismatch between training and decoding, called exposure bias, because training feeds correct previous labels while decoding feeds model outputs.A second mismatch arises between word-level cross-entropy training and sentence- or document-level BLEU evaluation.

12. Explainable Neural Machine Translation

Explainable NMT seeks to make complex sequence-to-sequence predictions interpretable through post-hoc analyses, hidden-state inspection, confidence estimates, and alignment information. Evidence indicates that attention can be informative but is not consistently a faithful explanation.

  • Interpretability: Explaining deep NMT predictions is difficult because the models contain tens of thousands of neurons and millions of parameters.Interpretability therefore remains an open research question.
  • Interpretability: Post-hoc approaches analyze perturbed inputs or changes in predictive distributions, while other methods inspect hidden neurons, layers, activities, and gradients.These methods provide black-box relevance estimates or probe internal representations.
  • Confidence and quality: NMT probabilities are candidate confidence scores, but disagreement remains about how well NMT models are calibrated.Quality estimation instead seeks metrics more accepted by users and more correlated with real-world usefulness than BLEU.
  • Word alignment: Traditional SMT produces word alignments as part of phrase-based translation, whereas introducing explicit alignment information into NMT remains an open problem.Proposed approaches include alignment supervision, explicit alignment layers, hard attention, and additional alignment heads.
  • Limits of attention: Several studies caution that attention weights often do not provide meaningful explanations and should not be treated as decision justifications.Other findings characterize attention as useful for source-target connections but insufficient for deep interpretation of target-word generation.
  • Word alignment: Attention agrees with traditional alignments to a high degree for nouns but captures additional information rather than only translational equivalents for verbs.This supports using attention as informative evidence while avoiding an interpretation as a complete alignment explanation.

13. Alternative NMT Architectures

Alternative NMT architectures extend or depart from standard encoder-decoder and attention designs to address limitations such as token-level focus, diffuse attention, and difficult sequence computation. The survey covers Transformer variants, advanced attention, external memory, and departures from the encoder-decoder structure.

  • Transformer extensions: The Transformer has become the de facto NMT architecture because of superior translation quality across language pairs.
  • Transformer extensions: Transformer variants modify attention, positional representations, context masks, head weighting, or inter-layer connections, but none has been widely adopted.Relative positioning increases computational complexity because keys and values must be recomputed at each decoding step.
  • Advanced attention models: Standard attention is token-based and lacks an explicit mechanism for attending to full phrases or multi-word expressions.Phrase-based NMT addresses this limitation by enabling attention to full phrases or multi-word expressions.
  • Advanced attention models: Regular attention can spread over many elements in long sequences, producing noisier averaged outputs that impede information propagation.Proposed remedies include hard attention, learned attention temperature, and GRU-gated attention outputs.
  • External memory: External memory structures were motivated by the practical difficulty of training RNNs to solve basic sequence-to-sequence tasks such as copying and reversal.Research on neural data structures has mainly focused on synthetic algorithmic tasks, with limited early applications to NMT.
  • Departures from encoder-decoder: NMT architectures generally use encoder-decoder networks, while initial alternatives include methods that define a distribution over representations.The encoder produces a continuous source representation and the decoder defines a target-sentence probability distribution conditioned on it.

14. Data Sparsity

Data sparsity limits NMT, especially in low-resource settings, while web-mined parallel data introduces substantial noise. The survey reviews filtering, domain adaptation, monolingual-data use, and unsupervised methods as responses.

  • Scope: NMT is data hungry, and traditional statistical MT can outperform it when training data is scarce.The section examines noise reduction, cross-domain data, and methods using less or no parallel data.
  • Corpus filtering: Web-crawled MT data commonly contains fragments, wrong languages, misaligned pairs, or machine-translated text.Studies report that NMT is not robust against naturally occurring noise during training and testing.
  • Corpus filtering: The most effective WMT18 NMT corpus-filtering approaches combined likelihood scores from neural translation and clean-data language models.These criteria favor sentence pairs likely to be translations of one another.
  • Domain adaptation: Domain adaptation commonly selects or weights samples from large out-of-domain corpora, uses back-translation, or fine-tunes on in-domain data.Fine-tuning can cause catastrophic forgetting and over-fitting.
  • Domain adaptation: Elastic weight consolidation reduces catastrophic forgetting and can yield gains on the general domain when fine-tuning on a related domain.
  • Low-resource and unsupervised NMT: Back-translation is particularly effective for low-resource MT, while unsupervised NMT learns from unrelated monolingual data without cross-lingual data.Unsupervised NMT often begins with cross-lingual word embeddings mapped into a joint embedding space.

15. Multilingual NMT

Multilingual NMT uses one model for multiple translation directions and can exploit similarities across language pairs. Sharing components reduces the number of systems needed for all-way translation.

  • Multilingual models: Multilingual NMT covers translation directions between multiple languages with a single model.
  • Multilingual models: A multilingual system can reduce the number of systems required for all-way translation from quadratic to linear or even one.Systems are categorized by which components they share across language directions, ranging from the entire encoder-decoder architecture to partial sharing.

16. NMT Model Size

NMT models often contain hundreds of millions of parameters, creating GPU, memory, storage, and computational burdens. The survey reviews architecture search, reduced precision, vector quantization, and pruning to improve efficiency.

  • Efficiency challenges: NMT models usually have hundreds of millions of parameters, making efficient execution dependent on expensive, memory-limited GPUs.
  • Efficiency challenges: Smaller NMT models can reduce computational complexity and make better use of GPU parallelism by enabling larger batch sizes.Large model files also create storage problems on mobile platforms.
  • Model compression: Efficiency methods include neural architecture search, 8- or 16-bit arithmetic, vector quantization, and pruning neural-network connections.These approaches target model space efficiency, translation speed, or compactness.

17. NMT with Extended Context

Extended-context NMT moves beyond isolated sentence translation by incorporating broader linguistic, document, multimodal, and graph-based information. These approaches target ambiguities and structural limitations that sentence-level or sequence-only systems may not capture.

  • Multimodal NMT supplements the source sentence with image information that can provide clues for translation.
  • NMT commonly uses characters or subwords because open-vocabulary modeling is difficult, despite translation involving larger linguistic structures.
  • Tree-based NMT introduces syntactic constituency or dependency structures on the source, target, or both sides, often by linearizing them.
  • Lattice-based NMT represents uncertainty from upstream speech recognition or tokenization and can incorporate external knowledge such as knowledge graphs.
  • Document-level NMT incorporates intersentential context through state initialization, multi-source encoders, decoder inputs, memory networks, language models, or hierarchical attention.Cross-sentence context is relevant to ambiguities such as pronoun prediction and lexical coherence.

18. NMT-SMT Hybrid Systems

Hybrid NMT-SMT systems combine neural and statistical translation components in several ways because the paradigms have different strengths. The survey reviews component borrowing, system combination, cascades, and dynamic score integration alongside broader NMT developments.

  • NMT is now the prevalent machine-translation approach, while comparative studies report superior overall quality in most settings and complementary strengths between NMT and SMT.
  • Some hybrids borrow SMT components or ideas, including log-linear feature combination, lexical translation tables, and alignment concepts such as fertility and relative distortion.
  • System-combination hybrids independently train SMT and NMT systems, then combine them through rescoring or reranking.
  • NMT and SMT can be cascaded in either direction, with one system supplying input to the other for post-processing.
  • The survey concludes that NMT has become the de facto standard and reviews recurrence, convolution, attention, architectures, design choices, explainability, and data sparsity.
Loading 1912.02047v2…