PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Deep Learning Techniques for Neural Machine Translation

Shuoheng Yang, Yuxin Wang, Xiaowen Chu

arXiv:2002.07526v1cs.CL

TL;DR

Machine translation research has moved through rule-based and statistical methods toward NMT, creating a need to organize the rapidly expanding literature. This survey traces NMT’s origins and development, categorizes its major branches and mechanisms, and reviews advanced models and future directions.

  • Problem

    The rapid growth of NMT research has left limited work examining the development process and relationships among its major technologies.

  • Method

    The paper conducts a literature survey that traces NMT’s origin, categorizes models by structural orientation, and reviews attention, vocabulary coverage, advanced models, and future research.

  • Results

    The survey provides a comprehensive interpretation of mainstream NMT technology and summarizes strengths, weaknesses, performance, and speed across model types.

  • Takeaways & Limitations

    NMT is presented as a major emerging direction whose simple architecture and long-dependency modeling offer substantial potential beyond conventional methods.

Abstract

from arXiv · show

In recent years, natural language processing (NLP) has got great development with deep learning techniques. In the sub-field of machine translation, a new approach named Neural Machine Translation (NMT) has emerged and got massive attention from both academia and industry. However, with a significant number of researches proposed in the past several years, there is little work in investigating the development process of this new technology trend. This literature survey traces back the origin and principal development timeline of NMT, investigates the important branches, categorizes different research orientations, and discusses some future research trends in this field.

I. INTRODUCTION

This introduction situates neural machine translation within machine translation’s historical development and explains the survey’s scope, organization, and limitations. It contrasts NMT’s motivation and potential with weaknesses in rule-based and phrase-based statistical approaches.

  • A. Introduction of Machine Translation: Machine translation evolved through rule-based, statistical, and neural approaches, with NMT emerging as a deep-learning method that offers simple architecture and long-dependency modeling.The survey presents these stages as the primary waves in MT development.
  • B. Related Work and Our Contribution: The survey reviews NMT’s origins, development timeline, model categories, design trade-offs, attention and vocabulary coverage mechanisms, advanced models, and future directions.It aims to interpret mainstream NMT technologies through design principles and translation quality and speed analyses.
  • B. Related Work and Our Contribution: The survey excludes several less-emphasized NMT areas and acknowledges that expert selection bias makes it a snapshot rather than a complete account of current research.Examples of excluded areas include robustness, domain adaptation, and applications such as speech and document translation.
  • 1) Rule-based Machine Translation:: Rule-based translation struggles with computational inefficiency, difficult grammar-rule organization, conflicting syntax rules, and missing contextual information.The paper illustrates the context problem using ambiguous word meanings in syntactically similar sentences.

2) Statistical Machine Translation:

Statistical machine translation decomposes translation into statistical phrase or word substitutions, but its context and engineering complexity motivate neural alternatives. Neural language models address limited context representation and support the transition toward end-to-end NMT.

  • Statistical Machine Translation: PBSMT builds translation from bilingual phrase pairs and multiple preprocessing, alignment, extraction, feature, and language-modeling components.Its phrase-based lexicon uses bilingual training data and context within phrases, improving over simple word-to-word translation.
  • Neural Machine Translation: NMT emerged after early neural translation research was constrained by poor performance and limited computing hardware.The survey identifies Kalchbrenner and Blunsom's DNN-based model as an early successful NMT proposal.
  • Motivation: Traditional machine-learning systems depend on hand-crafted features, whereas DNNs require less domain knowledge and avoid some preprocessing steps.The survey contrasts linguistic assumptions in context selection and bag-of-words representations with neural modeling.
  • Motivation: PBSMT can miss dependencies beyond phrase boundaries, causing inconsistencies, while its many interacting components complicate further development.The survey identifies the language model as a central bottleneck because it estimates word or phrase probabilities conditioned on prior words.
  • Language Models: Conventional language models ignore dependencies beyond a fixed context window, and larger windows suffer exponential data sparsity known as the curse of dimensionality.The survey notes that modest context sizes of generally 4–6 words were considered acceptable in experiments.
  • Neural Language Models: RNN language models read sentences sequentially and model conditional probabilities without a fixed context-size limitation, improving perplexity and practical-task performance.The survey presents RNLMs as a step toward neural translation and reports that direct NLM integration into SMT produced promising preliminary results.

2) Formulation of NMT Task:

Modern NMT maps a source sequence directly to a target sequence through encoding and decoding, with target generation conditioned on source and previously generated words. Its development progresses from shallow and hybrid systems to deep, attention-based, and fully attention-based architectures.

  • 2) Formulation of NMT Task:: Modern NMT is an end-to-end task that directly maps source data to target data through separate encoder and decoder functions.The encoder represents the source as a semantic vector, while the decoder predicts the target sentence.
  • 2) Formulation of NMT Task:: NMT selects a target sequence by maximizing its conditional probability given a source sequence of length n and target sequence of length m.At each step, prediction uses the previously generated target word and the source sentence.
  • 2) Formulation of NMT Task:: The task can be viewed as a neural language model constrained by a given source sequence.This formulation connects NMT's sequence generation objective to the language-model discussion.
  • C. The Recent Development in NMT: The survey divides recent NMT development into shallow NMT, NLM-assisted SMT, DNN-based NMT, attention-based NMT, and fully attention-based NMT.The stages trace a progression from early neural designs and hybrid systems toward architectures dominated by attention.
  • C. The Recent Development in NMT: Early NMT work used shallow architectures, while later NLM-assisted SMT combined a statistical translation system with a neural language model.Primitive early models established the field's original idea, and hybrid work opened a path toward pure neural translation.
  • C. The Recent Development in NMT: Pure DNN-based NMT replaced the SMT framework with an end-to-end model intended to represent the entire translation process.The survey states that such models could capture subtle irregularities in both languages more efficiently.
  • C. The Recent Development in NMT: Attention was introduced to provide word-alignment information and mitigate the fixed-length representation weakness of initial RNN-based NMT.The survey reports considerable improvement after attention was added, especially for long sentences.
  • C. The Recent Development in NMT: Transformer made attention the dominant feature-extraction mechanism, using self-attention with feed-forward connections instead of the prior RNN or CNN framework.The survey identifies Transformer as the representative fully attention-based model and attributes its progress to the attention structure.

III. DNN BASED NMT

DNN-based NMT developed from neural language modeling into end-to-end translation, with recurrent models initially dominant and attention-enabled systems surpassing SMT. CNN-based alternatives improved speed and parallelism but faced dependency and representation challenges.

  • III. DNN BASED NMT: End-to-end DNN-based NMT emerged from neural language modeling as a pure neural translation model.The model learns translation directly through a single neural architecture rather than combining separately engineered components.
  • RNN-based NMT: RNN-based NMT became the dominant design and achieved state-of-the-art performance.Its adoption was motivated partly by earlier RNN language-modeling work on sequence data.
  • RNN-based NMT: Attention significantly improved RNN-based translation and exceeded the best SMT model, with GNMT marking an industry-level milestone.The survey identifies GNMT as a milestone in recurrent NMT development.
  • CNN-based NMT: CNN-based NMT enabled parallel computation and faster training, but initially struggled to match RNN translation quality.Its limitations included local dependency capture and weaknesses caused by fixed-size sentence representations.
  • CNN-based NMT: Attention-equipped CNN models later addressed key shortcomings and achieved better results than RNN-based models in reported work.The survey describes this as a later development in CNN-based NMT.

B. Encoder-Decoder Structure

The classic NMT encoder-decoder reads a source sequence into hidden representations and generates the target sequence step by step. Its design varies by directionality, activation function, and depth, with deeper and bidirectional configurations trading capacity against training and efficiency costs.

  • B. Encoder-Decoder Structure: The encoder reads the source sentence sequentially, compresses it into hidden representations, and passes the resulting vector to the decoder.The decoder then uses this representation to generate the target sequence.
  • B. Encoder-Decoder Structure: RNN-based NMT designs differ mainly in directionality, activation function, and layer depth.The survey discusses these as the principal structural choices within recurrent encoder-decoder models.
  • Depth: Deeper recurrent networks generally support competitive performance, but stacking layers can make models slower and harder to train.The survey cites four encoder layers as optimal for a specific dataset without auxiliary methods.
  • Directionality: Bidirectional RNNs capture context from both reading directions and generally improve experimental performance, but they are difficult to parallelize.Implementations therefore commonly restrict bidirectionality to the encoder’s bottom layer.
  • Activation Function Selection: LSTM and GRU units are more robust than vanilla RNNs against gradient exploding and vanishing problems.The survey also reports better sequence-processing performance for GRU and LSTM units.
  • C. Training method: Training maps bilingual sentence pairs to target sequences by updating randomly initialized parameters toward minimum neural-network loss.Inputs are represented as word-vector sequences and processed in mini-batches.

D. Inference method

NMT inference generates translations from the source sentence without target-side ground truth. Greedy decoding selects the highest-probability word at each step, whereas beam search retains multiple candidates for better results while increasing decoding cost.

  • D. Inference method: At inference time, the decoder has access to the source sentence representation but not the target sentence used during training.Decoding strategies include sampling, greedy search, and beam search.
  • Greedy search: Greedy decoding emits the highest-probability word at each step, feeds it back as the next input, and stops at < EOS >.The next prediction is conditioned on the previously generated word.
  • Beam-search: Beam search retains the top-k partial translations at each step and selects the highest-scoring completed candidate.The beam width k controls how many candidate translations remain active.
  • Beam-search: Beam search generally produces better results than greedy search and is used by most NMT models for best performance.The survey describes beam search as useful but not necessary for NMT.
  • Beam-search: Length normalization and coverage penalties address beam search’s tendency toward short outputs and insufficient source-word coverage.These refinements adjust candidate scoring during decoding.
  • Beam-search: Beam widths of 5 to 10 were reported as best-performing experimentally, reflecting a trade-off between decoding efficiency and translation accuracy.Larger beams require evaluating more translation candidates and therefore increase decoding time.

IV. NMT WITH ATTENTION MECHANISM

Attention was introduced to reduce the information loss caused by compressing long source sentences into one fixed-length vector. It dynamically aligns decoder states with encoder states to construct context vectors from source-side information.

  • A. Motivation of Attention Mechanism: Original NMT performance deteriorates on long sentences because a fixed-length encoding vector cannot fully represent the source sequence.The resulting information loss particularly affects long-range word dependencies.
  • 1) basic structure: Attention supplements decoding with dynamic word-alignment information and alleviates information reduction for long inputs.It acts as an intermediate component between the encoder and decoder.
  • 1) basic structure: Luong and Bahdanau attention use different score-function formulations for measuring decoder–source-state relevance.The survey presents these as two classic definitions of the attention score.
  • 1) basic structure: Attention mechanisms provide alignment information beyond the fixed-length-vector representation used by the original NMT model.This distinction is the central conceptual motivation for the mechanism’s role in NMT.
  • 1) basic structure: The mechanism normalizes alignment scores, computes a weighted average of source states, and combines that context with the decoder hidden state.The resulting attention vector can be fed into the decoder at the next time step.
  • Global Attention: Global attention compares the current decoder state with all source states instead of relying on a single compressed sentence vector.Each top encoder-layer unit represents one source time step.

2) Global Attention & Local Attention:

Global attention considers all source hidden states for alignment, whereas local attention restricts computation to a source subset. GNMT uses global attention in a decoder structure designed to retain parallelism and improve inference speed.

  • Global Attention & Local Attention:: Global attention can slow decoding on long sequences because its score computation grows linearly with encoder time steps.The limitation becomes more consequential for compound sentences or paragraphs.
  • Global Attention & Local Attention:: Local attention balances translation performance and computing speed by limiting its relevance calculation to a fixed source-sentence scope.Its context vector is derived from hidden states within [p_t − D, p_t + D].
  • Global Attention & Local Attention:: Global attention calculates alignment using all source-side hidden states, while local attention uses only a subset.Local attention fixes an attention scope, reducing the computation required for context-vector construction.
  • Global Attention & Local Attention:: GNMT achieved state-of-the-art performance at the time and improved inference speed through its elaborate architecture.Its attention calculation follows global attention with a one-hidden-layer feed-forward score function.
  • Global Attention & Local Attention:: GNMT drives attention from the decoder’s bottom layer and shares the resulting context vector across higher decoder layers to preserve parallelism.GNMT also removes direct encoder–decoder connections, allowing more flexible layer dimensions.

5) Self-attention:

Self-attention represents dependencies within a sequence using query, key, and value vectors, while later refinements add structural and alignment-related biases. These variants seek more accurate attention and improved translation performance.

  • Self-attention:: Self-attention calculates dependencies within a single sequence rather than between source and target sequences.It produces an attention-based sequence representation and is associated with the emergence of the Transformer.
  • Self-attention:: Self-attention derives query, key, and value vectors from the original embedding before calculating attention weights.The calculation can be implemented with matrix multiplication to obtain word-relation matrices.
  • Self-attention:: The scaling factor 1/√d_k stabilizes gradients associated with dot-product operations in attention calculation.The factor is applied before the attention weighting operation.
  • Other related work: Attention refinements model relationships involving previous and subsequent attention, recurrent alignment, and structural biases such as position, Markov conditioning, fertility, and bilingual symmetry.These approaches extend the basic attention structure with linguistic or alignment-oriented assumptions.
  • Other related work: Fertility prediction adds coverage information before decoding, while agreement-based and supervised alignment methods impose additional alignment constraints.The cited supervised Chinese-to-English experiments reported benefits for translation performance and alignment.

V. VOCABULARY COVERAGE MECHANISM

NMT vocabulary coverage is limited by modest predefined vocabularies, creating OOV problems that affect rare words and translation quality. Proposed solutions trade computation, context-based copying, or representation granularity against coverage.

  • V. VOCABULARY COVERAGE MECHANISM: NMT commonly uses modest vocabularies, leaving proper nouns, rare verbs, and other words outside coverage as OOV items.Reported vocabulary sizes generally range from 30k–80k per language, with one 500k exception using approximate softmax.
  • V. VOCABULARY COVERAGE MECHANISM: Replacing OOV words with a uniform UNK symbol can reduce semantic completeness and introduce ambiguity when crucial words are lost.NMT performs worse on sentences containing more OOV words, and related difficulties include multi-word alignment, transliteration, and spelling.
  • V. VOCABULARY COVERAGE MECHANISM: Vocabulary methods are organized around computation speedup, context information, and finer-grained representations of words.The first category seeks larger vocabularies, while later categories address unknown words through contextual or subword information.
  • V. VOCABULARY COVERAGE MECHANISM: Softmax approximations, hierarchical models, importance sampling, noise-contrastive estimation, and vocabulary subsampling reduce training computation but retain OOV limitations.The initial hierarchical method was comparable with a traditional trigram LM but did not exceed the original NLM.

2) methods by using context information:

Context-based methods handle OOV words through alignment, copying, dictionaries, or additional linguistic information, while subword and character methods represent internal word structure. These approaches improve coverage but differ in generalization and linguistic scope.

  • methods by using context information:: Copy Mechanism maps each OOV word to a source position and uses a dictionary or source word during post-processing.The approach achieved a considerable BLEU improvement and effectively provides an unlimited vocabulary for copied items.
  • methods by using context information:: Context-based methods also use lexicons, memory-augmented attention, and contextual information to assist low-frequency or ambiguous-word translation.These methods incorporate dictionary knowledge or address homographs alongside copying approaches.
  • methods by using context information:: Copy-based approaches have limited generalization because they handle sophisticated cross-linguistic scenarios poorly, especially in morphologically rich languages such as Finnish and Turkish.Their alignment-based replacement strategy is described as too crude for these settings.
  • Methods in fine grit level:: Subword methods represent OOV words as sequences of units, allowing frequent patterns to remain merged while rare words retain unmerged subwords.BPE repeatedly merges the most frequent adjacent character pairs until reaching a specified vocabulary size.
  • Methods in fine grit level:: Character-level approaches model word-internal information with CNNs or recurrent networks, including fully character-level translation without segmentation.Reported variants include character-based source representations, character decoders, and fully character-level NMT.

VI. ADVANCED MODELS

Advanced NMT models achieve state-of-the-art performance across distinct structural categories, with ConvS2S representing a CNN-based design that combines deep convolutional layers, gated units, residual connections, and multi-step attention.

  • VI. ADVANCED MODELS: Advanced models from different structural categories can achieve similar state-of-the-art performance while offering different advantages.The survey presents these models as structurally distinct but comparably strong in overall performance.
  • A. ConvS2S: ConvS2S uses an entirely CNN-based encoder and decoder with 15 layers and fixed kernel width 3.Its deep structure is intended to mitigate weaknesses in capturing context information.
  • A. ConvS2S: Gated Linear Units control which convolution inputs are relevant, while residual connections link different convolution layers.The GLU operates on a 2d-dimensional convolution output split into A and B vectors, applying σ(B) to gate A.
  • A. ConvS2S: ConvS2S applies multi-step attention separately in each decoder layer, combining decoder state summaries with final encoder outputs.The attention vector uses a dot product between the decoder summary and encoder output, followed by a weighted-average context vector.
  • A. ConvS2S: ConvS2S is illustrated as a successful CNN-based NMT model with competitive performance to the state of the art.

B. RNMT+

RNMT+ enhances the GNMT recurrent architecture with deeper bidirectional decoding, multi-head additive attention, synchronous training, and per-gate normalization, while Transformer replaces recurrence and convolution with self-attention and positional encoding.

  • B. RNMT+: RNMT+ extends GNMT with six bidirectional LSTM decoder layers, trading computation efficiency for extreme performance.GNMT instead uses one bidirectional layer and seven unidirectional layers.
  • B. RNMT+: RNMT+ replaces conventional single-head attention with multi-head additive attention inspired by Transformer.
  • B. RNMT+: Synchronous training improves convergence speed with model performance based on empirical results, while per-gate layer normalization helps stabilize training.
  • 1) Structure of the model:: Transformer abandons standard RNN and CNN structures for multi-layer self-attention blocks combined with positional encoding.The architecture uses attention to represent sequences and positional encoding to retain word-order information.
  • 1) Structure of the model:: Transformer represents inputs with query, key, and value vectors derived from learned matrix transformations of embeddings.Multi-head self-attention computes scaled dot-product attention in parallel and combines the resulting outputs.
  • 1) Structure of the model:: Transformer stacks six encoder components and six decoder components, using residual connections, layer normalization, and masked decoder attention.Masked attention prevents positions from attending to subsequent target positions during prediction.

2) Transformer based NMT variants:

Transformer-based NMT variants address limitations of the vanilla architecture through deeper or adaptive computation, refined attention, and architecture search, while the survey identifies long-sentence translation, vocabulary coverage, low-resource settings, and applications as continuing challenges and directions.

  • 2) Transformer based NMT variants:: Vanilla Transformer refinements target missing recurrence, limited position modeling, non-Turing-completeness, and high model complexity.These weaknesses are presented as obstacles to further translation-performance improvement.
  • 2) Transformer based NMT variants:: Deeper Transformer variants refine attention or normalization to make substantially deeper encoder stacks easier to optimize.Reported examples include 2–3x deeper models and a 25-layer encoder.
  • 2) Transformer based NMT variants:: Universal Transformers combine recurrent inductive bias with adaptive computation-time halting instead of stacking a fixed number of layers.Under certain assumptions, this adjustment has been shown to make the model Turing-complete.
  • 2) Transformer based NMT variants:: The Evolved Transformer uses neural architecture search to combine basic blocks into a model matching Transformer-Big quality with 37.6% fewer parameters.
  • VII. FUTURE TREND: NMT still performs poorly on long sentences because of engineering limits and model learning limitations, motivating trade-offs between sequence modeling and computational efficiency.The survey also identifies attention, vocabulary coverage, subword or character modeling, low-resource translation, and broader applications as future directions.
Loading 2002.07526v1…