PaperlayerView exact revision on arXiv

Source-linked AI summary

Variational Neural Machine Translation

Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, Min Zhang

arXiv:1605.07869v2cs.CL

TL;DR

Neural machine translation commonly relies on discriminative encoder-decoders whose semantic representations are implicit. VNMT introduces a continuous latent variable, approximates its posterior with neural networks, and uses reparameterization for end-to-end training; experiments report significant gains on Chinese-English and English-German tasks.

  • Problem

    Conventional attention-based encoder-decoders learn sentence semantics implicitly, while latent-variable NMT introduces intractable posterior inference and large-scale training challenges.

  • Method

    VNMT uses a continuous latent variable for sentence-pair semantics, a neural posterior approximator conditioned on source and target representations, and reparameterization of the variational lower bound.

  • Results

    VNMT achieves significant improvements over several strong baselines on Chinese-English and English-German translation tasks.

  • Takeaways & Limitations

    The latent variable provides global semantic guidance complementary to attention and makes VNMT better at translating long sentences.

  • Takeaways & Limitations

    The model uses a sentence-level latent variable, motivating future exploration of more fine-grained latent variables.

Abstract

from arXiv · show

Models of neural machine translation are often from a discriminative family of encoderdecoders that learn a conditional distribution of a target sentence given a source sentence. In this paper, we propose a variational model to learn this conditional distribution for neural machine translation: a variational encoderdecoder model that can be trained end-to-end. Different from the vanilla encoder-decoder model that generates target translations from hidden representations of source sentences alone, the variational model introduces a continuous latent variable to explicitly model underlying semantics of source sentences and to guide the generation of target translations. In order to perform efficient posterior inference and large-scale training, we build a neural posterior approximator conditioned on both the source and the target sides, and equip it with a reparameterization technique to estimate the variational lower bound. Experiments on both Chinese-English and English- German translation tasks show that the proposed variational neural machine translation achieves significant improvements over the vanilla neural machine translation baselines.

1 Introduction

VNMT extends the discriminative encoder-decoder framework with a continuous latent variable that explicitly represents sentence-pair semantics and guides translation. Neural posterior approximation and reparameterization make this variational model trainable end-to-end, with improvements reported across translation tasks.

  • Motivation: NMT uses a unified end-to-end encoder-decoder, while conventional models learn bilingual semantics implicitly and rely heavily on attention alignments.The paper identifies potentially insufficient attention-based context as a source of undesirable translation phenomena.
  • Variational model: VNMT introduces a continuous latent variable z to explicitly model underlying semantics of bilingual sentence pairs and guide generation with x.The latent variable is intended to provide a global semantic signal complementary to the attention-based context vector.
  • Training: The model addresses intractable posterior inference and large-scale training through neural prior and posterior distributions plus reparameterization of the variational lower bound.These techniques permit standard stochastic-gradient optimization while retaining end-to-end training.
  • Architecture: VNMT comprises a variational encoder, inferer, and decoder that respectively represent sentences, infer z, and integrate z into target generation.The inferer uses source representations for pθ(z|x) and source-target representations for qφ(z|x,y).
  • Results: Experiments on Chinese-English and English-German translation tasks report significant improvements over several strong baselines.The paper presents these results as evidence for the effectiveness of the proposed end-to-end variational model.

2 Background: Variational Autoencoder

A variational autoencoder introduces a continuous latent variable and combines a generative model with an approximate posterior. Neural approximation and reparameterization make the variational objective compatible with end-to-end backpropagation.

  • VAE formulation: A VAE introduces a continuous latent variable z and assumes the observed variable x is generated from z.The latent variable is used to capture variations in the observed variable.
  • VAE formulation: The generative model pθ(x|z) describes how x is generated from z, while pθ(z) serves as the prior distribution.The paper notes that the prior may be a simple Gaussian and that the conditional distribution is typically modeled by a deep neural network.
  • Inference: Neural approximation models qφ(z|x) as a diagonal Gaussian whose mean and variance are parameterized by deep neural networks.This approximates posterior inference while allowing the latent distribution to be learned from the observed input.
  • Optimization: Reparameterization expresses z as a function of the Gaussian location and scale parameters plus noise, enabling gradient-based optimization.The noise variable is standard Gaussian, and the operation uses an element-wise product.
  • Optimization: Together, the generative model and posterior inference model form an end-to-end neural network optimized with standard backpropagation through a variational lower bound.These techniques address posterior inference and large-scale learning challenges associated with latent variables.

3 Variational Neural Machine Translation

VNMT introduces a continuous latent semantic variable and integrates it into an end-to-end encoder-decoder translation model. Its inferer, decoder, and encoder jointly support posterior approximation, latent-guided generation, and differentiable training.

  • Model formulation: VNMT introduces z as a global latent signal for translation and formulates the variational lower bound using a prior, posterior approximator, and latent-guided decoder.The prior is pθ(z|x), the posterior is qφ(z|x,y), and the decoder is pθ(y|z,x).
  • Variational neural encoder: The variational encoder maps source and target sequences into continuous representations using a bidirectional RNN with GRU units and annotation vectors.Annotation vectors combine forward and backward hidden states and encode each word relative to surrounding words.
  • Variational neural inferer: The variational neural inferer models both prior and posterior distributions as diagonal-covariance Gaussian distributions with different conditioning information.The posterior uses x and y, while the prior is modeled from x alone; both use neural networks to produce their parameters.
  • Neural posterior approximator: The posterior approximator uses neural networks to produce the mean and standard deviation from observed source-target variables, addressing intractable posterior inference.The approximation is motivated by the limitation of conventional mean-field approaches in capturing the true posterior.
  • Latent representation: Reparameterization connects inference and generation by sampling z during training while using the prior mean during decoding.Noise in the reparameterized representation makes it vary across uses of the same source sentence and is described as helping avoid overfitting.
  • Decoder and training: VNMT incorporates the latent representation into the decoder alongside attention, allowing z to influence hidden states and provide semantic guidance when attention is undesirable.The training objective combines a Gaussian KL divergence with an expected translation log-likelihood; with L=1, the latter reduces to conventional NMT.

4 Experiments

Experiments evaluate VNMT against SMT, attention-based NMT, and a no-KL variant across Chinese-English, English-German, and increasingly long inputs. VNMT consistently improves BLEU, especially for long sentences, while qualitative examples suggest better preservation of source meaning.

  • 4.1 Setup: VNMT is compared with Moses, GroundHog, and a VNMT variant without the KL objective.Moses is phrase-based SMT, GroundHog is attention-based NMT, and VNMT w/o KL removes the KL component.
  • 4.2 Results on Chinese-English Translation: 0.86 and 1.35 BLEU points are VNMT’s average gains over Moses and GroundHog, respectively, on Chinese-English tasks.VNMT w/o KL performs worse than GroundHog, supporting the reported importance of the KL objective.
  • 4.3 Results on Long Sentences: 3.55 BLEU points is VNMT’s largest improvement over GroundHog for the longest source-sentence group.VNMT’s BLEU curve remains above GroundHog’s across all six length-based groups.
  • 4.3 Results on Long Sentences: All new test sets show significant VNMT improvements when synthetic source sentences average more than 50 words.The synthetic inputs concatenate neighboring source sentences, producing lengths almost twice those of the original data.
  • 4.4 Results on English-German Translation: 0.73 BLEU points is VNMT’s significant gain over GroundHog on English-German translation (p < 0.01).With unknown-word replacement, VNMT reaches a performance level comparable to previous state-of-the-art NMT results.
  • 4.5 Translation Analysis: On a long Chinese-English example, VNMT preserves source fragments that GroundHog omits and conveys the entire sentence’s meaning.GroundHog misses the opening phrase and loses the final clause, whereas VNMT does not miss or mistake these fragments.

5 Related Work

Related work spans sequence-to-sequence and attention-based NMT, methods for rare words and improved attention, and variational neural models. The paper distinguishes VNMT by adapting variational modeling to bilingual neural translation, which it presents as previously uninvestigated.

  • 5.1 Neural Machine Translation: Sequence-to-sequence NMT encodes a source sentence before decoding, while attention-based NMT concentrates on relevant source parts during prediction.Attention addresses limitations from encoding all source information into a fixed-length vector and improves handling of long sentences.
  • 5.1 Neural Machine Translation: Recent NMT work targets rare words and large vocabularies, better attentional structures, SMT integration, and memory-network approaches.These approaches form distinct categories organized by their motivations.
  • 5.2 Variational Neural Model: Variational neural networks approximate intractable posteriors and optimize reparameterized variational lower bounds with stochastic gradients.This framework supports efficient inference and learning in directed probabilistic models on large-scale datasets.
  • 5.2 Variational Neural Model: Prior variational work includes semi-supervised generative models, latent recurrent states, spatial attention with sequential variational autoencoding, and variational text models.These studies apply variational methods across supervised, recurrent, image, and text-generation settings.
  • 5.2 Variational Neural Model: VNMT differs from Bowman et al. by adapting variational modeling from monolingual language modeling to bilingual translation.The paper states that variational neural models had not previously been investigated for neural machine translation, to its knowledge.

6 Conclusion and Future Work

The paper presents VNMT, a variational neural machine translation model that uses a continuous latent variable to represent sentence-pair semantics. Experiments verify its effectiveness, while future work targets finer-grained latent variables.

  • Conclusion: VNMT incorporates a continuous latent variable to model the underlying semantics of sentence pairs.The model is trained end-to-end by approximating the posterior distribution and reparameterizing the variational lower bound.
  • Conclusion: The latent variable improves translation of long sentences compared with conventional attention-based NMT.The paper also reports benefits from a special regularization term introduced by the latent variable.
  • Conclusion: Experiments on Chinese-English and English-German translation tasks verified the effectiveness of the proposed model.
  • Future Work: Future work will explore more fine-grained latent variables because the current latent variable operates at the sentence level.The paper gives the Recurrent Latent Variable Model as an example and also proposes applying VNMT to similar tasks.
Loading 1605.07869v2…