Variational Recurrent Neural Machine Translation

1. Introduction

• VNMT adds a global latent semantic signal, but its deterministic recurrent transition may inadequately model language variability and complex dependencies.Source: §1 · p. 1
• VRNMT introduces a continuously generated latent sequence, with each z_j participating in the generation of target word y_j and hidden state s_j+1.Source: §1 · p. 1
• The latent variables are intended to capture strong dependencies between adjacent target words that conventional NMT or VNMT may not model sufficiently.Source: §1 · p. 1
• Neural prior and posterior approximators address intractable inference, while reparameterization supports efficient end-to-end stochastic optimization.Source: §1 · pp. 1–2
• VRNMT adapts variational recurrent neural networks to NMT, combining high-level latent variables with recurrent modeling of target-word dependencies.Source: §1 · p. 2
• VRNMT significantly outperforms conventional NMT and VNMT on Chinese-English and English-German translation tasks.Source: §1 · p. 2

2.1 Attention-based NMT Model

• Attention-based NMT uses a bidirectional RNN encoder and an attention-based forward decoder to generate translations sequentially.The encoder produces source annotations, while the decoder combines its state with attention-derived source context.Source: §2.1 · p. 2
• The decoder computes each hidden state recurrently from the previous state, previous target word, and source context, and can use RNN variants beyond GRUs.The work uses GRUs for both encoder and decoder but states that the approach applies to other RNN types.Source: §2.1 · p. 2

2.2 VRNN

• VRNN extends the VAE recurrently by generating a latent variable from the previous hidden state, using it to generate the current hidden state and observation.This recurrent structure exploits temporal dependencies in sequential data.Source: §2.2 · pp. 2–3
• Unlike a standard VAE, VRNN uses timestep-dependent Gaussian latent priors and conditions observation generation on both the latent variable and prior hidden state.The hidden state is updated through a recurrent function that incorporates the latent variable.Source: §2.2 · p. 3
• VRNN inference uses an approximate posterior whose parameters depend on observed inputs and hidden states, then optimizes a timestep-wise variational lower bound.Neural approximation and reparameterization are used because latent-variable integration makes posterior inference and large-scale learning difficult.Source: §2.2 · p. 3

3. Our Model

• VRNMT extends VNMT by adapting VRNN to NMT, using latent variables to capture dependencies between adjacent target words and refine translation.Its variational lower bound combines latent-distribution matching with expected target-word likelihood.Source: §3 · p. 3

3.1 Neural Encoder

• The model contains a conventional source encoder, a variational neural inferer for prior and posterior distributions, and a variational decoder for target generation.The encoder is the same as in conventional NMT.Source: §3 · p. 3, §3.1 · p. 3

3.2 Variational Neural Inferer

• The posterior qφ(zj|x, y≤j) is modeled as a diagonal-covariance Gaussian whose parameters are computed from source, target, decoder-state, and attention representations.Nonlinear projection followed by linear regressions produces the posterior parameters, and injected noise yields non-fixed latent representations.Source: §3.2 · pp. 3–4
• Reparameterization samples zj as zj = µj + σj ⊙ϵ with ϵ ∼N(0, I), bridging posterior inference and decoder generation for end-to-end learning.The prior uses a similar neural model but omits the current target word.Source: §3.2 · p. 4

3.3 Variational Neural Decoder

• The decoder first produces zj from the previous target word, decoder state, and source context, then uses zj with those representations to generate the next-word distribution.The latent variable also affects the next hidden state through the GRU gates and temporary state, creating indirect dependencies across adjacent timesteps.Source: §3.3 · p. 4

Model Training

• Training maximizes a joint objective containing KL divergence and expected log likelihood, with Monte Carlo sampling and differentiable terms enabling joint optimization of θ and φ.The sampled latent noise also makes VRNMT a regularized version of NMT in the authors’ formulation.Source: Model Training · p. 4

4. Experiments

• Experiments assess VRNMT on Chinese-English and English-German translation tasks.The Chinese-English data contain 1.25M sentence pairs, while the English-German data contain 4.46M sentence pairs.Source: §4 · p. 4, §4.1 · pp. 4–5

4.1 Setup

• Chinese-English uses NIST MT02 for validation and NIST MT03/04/05/06 for testing, whereas English-German uses news-test 2013 for validation and news-test 2015 for testing.English-German words are segmented into subwords with byte pair encoding.Source: §4.1 · pp. 4–5
• BLEU evaluates translation quality, and paired bootstrap sampling with the Moses script tests statistical significance.Source: §4.1 · p. 5
• Training truncates sentences at 50 words, uses corpus-specific 30K or 50K vocabularies, and replaces other words with UNK.The stated vocabularies cover approximately 97.4%/100.0% and 99.3%/98.2% of source/target words for the two corpora.Source: §4.1 · p. 5
• Models are trained with RMSprop and decoded with beam search using beam size 10, while VRNMT is initialized from a trained conventional NMT model.The implementation uses word embeddings of size 620, hidden layers of size 1000, dropout 0.3, and one latent sample.Source: §4.1 · p. 5

• VRNMT is compared with Moses, DL4MT, VNMT, and VRNMT(-TD), spanning SMT, conventional NMT, variational NMT, and an ablated model.Source: §4.2 · p. 5
• The VRNMT(-TD) variant removes temporal dependencies by excluding previous outputs and decoder states from the posterior input.Source: §4.2 · p. 5
• Table 1 reports case-insensitive BLEU scores for Chinese-English translation and marks significance against VNMT and VRNMT(-TD).Source: Table 1 · p. 5
• Figure 3 examines BLEU scores across different translated-sentence lengths.Source: Figure 3 · p. 5

• VRNMT gains 5.66, 1.42, 0.78 and 1.0 BLEU points over Moses, DL4MT, VNMT and VRNMT(-TD), respectively, across the Chinese-English test sets.Source: §4.3 · p. 5
• Table 2 marks fluent, incorrect, over-translated, and under-translated words in example translations from different systems.Source: Table 2 · p. 6
• The over-translation evaluation compares systems using a lower-is-better score, providing a separate measure of repetition-related translation problems.Source: Table 3 · p. 6
• VRNMT significantly outperforms VRNMT(-TD), indicating that modeling temporal dependencies between latent variables benefits NMT.Source: §4.3 · pp. 5–6

Results on Source Sentences with Different Lengths

• VRNMT outperforms the other systems across test sentences grouped by different source-length spans.Source: Results on Source Sentences with… · p. 6

Analysis on Over Translation

• N-GRR measures the portion of repeated n-grams, and comparison with reference scores indicates the severity of over-translation.Source: Analysis on Over Translation · p. 6
• Table 4 reports word-alignment quality with lower scores indicating better alignments.Source: Table 4 · p. 6
• VRNMT better handles over-translation than the other models according to the final N-GRR results.Source: Analysis on Over Translation · p. 6

Analysis on Attention Results

• In a manually aligned set of 900 Chinese-English sentence pairs, latent variables improve the attention model’s alignment performance under AER and SAER.Source: Analysis on Attention Results · p. 6

Case Study

• Table 5 presents case-sensitive BLEU scores for English-German translation alongside results from prior systems.Source: Table 5 · p. 7
• The case study finds VRNMT produces a more complete, fluent, and accurate translation than Moses, DL4MT, VNMT, and VRNMT(-TD).Source: Case Study · pp. 6–7
• The compared NMT systems leave some source phrases untranslated, while VRNMT benefits from modeling long-distance dependencies among target words.Source: Case Study · p. 7

• The English-German experiments include comparisons with previous work by Chung, Cho, and Bengio, Yang et al., and Gehring et al.Source: §4.4 · p. 7
• VRNMT outperforms Moses, DL4MT, VNMT, and VRNMT(-TD) by 5.39, 1.05, 0.44, and 0.59 BLEU points, respectively.Source: §4.4 · p. 7
• VRNMT reaches a performance level competitive to or higher than several recent NMT systems.Source: §4.4 · p. 7

NMT

• Standard NMT uses encoder-decoder networks, with attention-based architectures becoming dominant after fixed-length encoding limitations were identified.Attention-based NMT addressed the challenge of compressing all source information into a fixed-length vector, though attention failures remained a concern.Source: §5 · p. 7
• Later NMT research improved attention, neural architectures, or external knowledge, but remained within discriminative frameworks that leave explicit semantic modeling open.VNMT addressed this gap by incorporating a continuous latent variable to model sentence-pair semantics.Source: §5 · p. 7

Variational Neural Networks

• Variational neural networks use neural posterior approximators and reparameterized variational lower bounds for scalable inference and learning in directed probabilistic models.Related work also incorporated latent variables into recurrent states and developed sequential variational models for complex generative processes.Source: §5 · p. 7
• VRNMT extends VNMT into a recurrent framework with a sequence of recurrent latent variables for semantic modeling in NMT, a combination not previously investigated in this setting.The work differs from Chung et al. (2015), which applied recurrent latent variables to speech and handwriting generation rather than machine translation.Source: §5 · p. 7

Conclusions

• VRNMT introduces a sequence of continuous latent variables, neural posterior inference, and reparameterization to enable end-to-end stochastic-gradient optimization.The recurrent latent variables capture global semantic contexts and dependencies among generated words at different timesteps.Source: §6 · p. 7
• Experiments on Chinese-English and English-German translation tasks demonstrate the effectiveness of VRNMT compared with dominant NMT and VNMT models.Source: §6 · p. 7

Future Work

• Future work will improve latent-variable use in NMT and apply the model to encoder-decoder tasks such as neural summarization and dialogue generation.Source: §6 · p. 7