PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior

Raphael Shu, Jason Lee, Hideki Nakayama, Kyunghyun Cho

arXiv:1908.07181v5cs.CLcs.LG

TL;DR

Autoregressive NMT limits simultaneous token decoding and increases latency, while non-autoregressive models must capture target-token dependencies to avoid inconsistent sequences. LaNMT models target-sentence uncertainty with continuous latent variables and uses deterministic iterative inference to update the approximate posterior and target tokens. The models outperform the autoregressive baseline on ASPEC Ja-En with 8.6x decoding speedup and bring the WMT’14 En-De performance gap down to 2.0 BLEU points with 12.5x decoding speedup.

  • Problem

    Autoregressive NMT limits simultaneous token decoding and increases latency, while non-autoregressive models must capture target-token dependencies to avoid inconsistent sequences.

    Source: Non-Autoregressive NMT · p. 2

  • Method

    LaNMT models target-sentence uncertainty with continuous latent variables and uses deterministic iterative inference to update the approximate posterior and target tokens.

    Source: §1 · p. 1

  • Results

    The models outperform the autoregressive baseline on ASPEC Ja-En with 8.6x decoding speedup and bring the WMT’14 En-De performance gap down to 2.0 BLEU points with 12.5x decoding speedup.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Translation length can adapt during latent-variable refinement, allowing an initially incorrect length prediction to be corrected alongside target tokens.

    Source: §1 · p. 1

Abstract

from arXiv · show

Although neural machine translation models reached high translation quality, the autoregressive nature makes inference difficult to parallelize and leads to high translation latency. Inspired by recent refinement-based approaches, we propose LaNMT, a latent-variable non-autoregressive model with continuous latent variables and deterministic inference procedure. In contrast to existing approaches, we use a deterministic inference algorithm to find the target sequence that maximizes the lowerbound to the log-probability. During inference, the length of translation automatically adapts itself. Our experiments show that the lowerbound can be greatly increased by running the inference algorithm, resulting in significantly improved translation quality. Our proposed model closes the performance gap between non-autoregressive and autoregressive approaches on ASPEC Ja-En dataset with 8.6x faster decoding. On WMT'14 En-De dataset, our model narrows the gap with autoregressive baseline to 2.0 BLEU points with 12.5x speedup. By decoding multiple initial latent variables in parallel and rescore using a teacher model, the proposed model further brings the gap down to 1.0 BLEU point on WMT'14 En-De task with 6.8x speedup.

1 Introduction

LaNMT uses continuous latent variables and deterministic iterative inference to improve non-autoregressive translation while adapting output length. It reports strong speedups and competitive translation quality on ASPEC Ja-En and WMT’14 En-De.

1 Introduction

  • LaNMT introduces continuous latent variables to capture uncertainty in the target sentence, motivated by iterative refinement and discrete-latent non-autoregressive NMT.The authors conjecture that lower-dimensional continuous variables are easier to refine than high-dimensional discrete variables; the prior and posterior are jointly trained to maximize the evidence lowerbound.Source: §1 · p. 1
  • Deterministic iterative inference alternates updates to the approximate posterior and target tokens, improving the lowerbound and BLEU while allowing translation length to adapt.An initially incorrect length prediction can be corrected alongside target tokens.Source: §1 · p. 1
  • The authors’ models outperform the autoregressive baseline on ASPEC Ja-En with 8.6x decoding speedup and reduce the WMT’14 En-De performance gap to 2.0 BLEU points with 12.5x speedup.By decoding multiple latent variables sampled from the prior and rescoring with an autoregressive teacher model, the proposed model further narrows the WMT’14 En-De gap to 1.0 BLEU point with 6.8x speedup.Source: §1 · p. 1
  • The model learns as many latent vectors as input tokens and uses a length transformation mechanism to match the target length.Source: §1 · p. 1
  • The authors report that deterministic inference converges rapidly in practice and improves translation quality by around 2.0 BLEU points.Source: §1 · p. 2

2 Background

Autoregressive NMT generates target tokens sequentially, limiting parallel decoding, while non-autoregressive NMT predicts tokens in parallel. The latter faces a challenge in maintaining consistency across target tokens.

Autoregressive NMT

  • Autoregressive decoding selects target tokens sequentially, and large beam sizes significantly decrease translation speed.Beam search is often used to obtain better translations than greedy decoding.Source: Autoregressive NMT · p. 2

Non-Autoregressive NMT

  • Autoregressive modeling prevents simultaneous token selection, resulting in inefficient computation and increased translation latency.Source: Non-Autoregressive NMT · p. 2
  • Non-autoregressive NMT predicts target tokens without depending on preceding tokens, allowing predictions to be parallelized.Each token prediction depends on the source sentence and its position.Source: Non-Autoregressive NMT · p. 2
  • Independent token predictions can produce inconsistent sequences with duplicated or missing words, making target-token consistency a central challenge.Source: Non-Autoregressive NMT · p. 2

3 Latent-Variable Non-Autoregressive NMT

LaNMT uses continuous latent variables and an ELBO objective for non-autoregressive translation, with a length predictor and deterministic length transformation. Training uses KL-budget annealing and sequence-level distillation to address posterior collapse and support prediction.

3 Latent-Variable Non-Autoregressive NMT

  • LaNMT represents each source with an equally long sequence of continuous latent variables and trains its prior, approximate posterior, and decoder using an ELBO.The prior and posterior are spherical Gaussians trained end-to-end with the reparameterization trick.Source: §3 · p. 2

A Modified Objective Function with Length Prediction

  • The objective factors target generation into predicting target length from the latent variables and generating tokens conditioned on that length.The length predictor models the difference between target and source lengths.Source: A Modified Objective Function with… · pp. 2–3

Model Architecture

  • The model parameterizes a prior, approximate posterior, decoder, and length predictor, using an unmasked attention decoder for target-conditioned latent inference.The decoder also uses source representations computed for the prior.Source: Model Architecture · p. 3
  • The length predictor estimates the target–source length difference, while monotonic location-based attention transforms source-length latents into target-length vectors weighted toward nearby positions.Figure 2 illustrates this length transformation.Source: Model Architecture · p. 3, Figure 2 · p. 3

Training

  • To address posterior collapse, training applies a per-latent KL budget that stays at 1 for the first half and then anneals to 0, alongside sequence-level knowledge distillation.Without intervention, the KL divergence can drop to zero and the model may stop using its latent variables.Source: Training · pp. 3–4

4 Inference with a Delta Posterior

Inference seeks a translation that maximizes the ELBO without stochastic posterior sampling. It alternates between updating a delta posterior and choosing target tokens until the sequence stabilizes.

4 Inference with a Delta Posterior

  • The inference algorithm searches for a translation that maximizes the ELBO, avoiding stochasticity when selecting a single target sentence.Exact optimization is intractable because its first expectation cannot be computed directly.Source: §4 · p. 4
  • A Dirac delta proxy is fitted to the approximate posterior, then initialized from the prior to make inference deterministic.The proxy is set to minimize KL divergence against the approximate posterior.Source: §4 · p. 4
  • Inference alternates between estimating latent variables from the current target and selecting the most likely target under those latents.The procedure repeats until successive target sequences match.Source: §4 · p. 4

5 Related Work

LaNMT differs from related non-autoregressive approaches through its continuous latent variables, jointly trained posterior and prior, non-autoregressive prior, and latent-space refinement. Its deterministic inference also contrasts with an EM-like approach for variational NMT.

5 Related Work

  • This work differs from Kaiser et al. and Roy et al. (2018; 2018): it jointly trains a continuous-latent posterior and prior, uses a non-autoregressive prior, and refines in latent space.Both approaches first train a discrete autoencoder on the target sentence, then train an autoregressive prior to predict its discrete latent variables from the source sentence.Source: §5 · p. 4
  • Masked-prediction refinement and insertion-based NMT operate on discrete tokens, whereas LaNMT refines in a low-dimensional latent space.The cited examples include Ghazvininejad et al. (2019) and Gu, Liu, and Cho; Stern et al.; Welleck et al. (2019; 2019; 2019).Source: §5 · p. 4
  • Unlike the EM-like inference of Shah and Barber (2018), LaNMT uses a deterministic algorithm to remove inference-time nondeterminism.The comparison concerns variational NMT methods that model source–target joint distributions.Source: §5 · p. 4

6 Experimental Settings

Experiments use ASPEC Ja-En and WMT’14 En-De with dataset-specific tokenization and subword preprocessing. The settings specify model selection, evaluation, and a multi-latent search procedure rescored by an autoregressive model.

6 Experimental Settings

  • Experiments use ASPEC Ja-En with 3M sentence pairs and WMT’14 En-De with 4.5M sentence pairs.Source: §6 · p. 5
  • ASPEC uses Moses tokenization for English and Kytea for Japanese, followed by BPE with 40K-token vocabularies; WMT uses SentencePiece with 32K-token vocabularies.Source: §6 · p. 5
  • Hyperparameters are selected by validation ELBO, and each latent variable has dimension 8, with similar performance observed for dimensions from 4 to 32.The models use 6-layer priors and decoders and a 3-layer approximate posterior.Source: §6 · p. 5
  • ASPEC is evaluated with tokenized BLEU and WMT’14 with Sacre-BLEU, after removing repeated tokens from translations.Source: §6 · p. 5
  • For latent search, multiple prior samples are decoded independently, rescored by the autoregressive baseline, and selected by log probability; sampling temperature is halved and the random seed fixed.Source: §6 · p. 5

7 Result and Analysis

The model’s deterministic inference improves translation quality, with its strongest reported WMT’14 En-De result narrowing the gap to the autoregressive baseline while retaining a decoding speedup. Refinement also improves the ELBO, can adjust target length, and has a remaining weakness on some poor initial translations.

Quantitative Analysis

  • ASPEC Ja-En beats the autoregressive baseline at 8.6x speedup; WMT’14 En-De moves from 24.1 BLEU (-2.0) at 12.5x to 25.1 (-1.0) with latent search at 6.8x.The WMT’14 En-De autoregressive baseline is 1.0 BLEU point below the original paper’s result; latency is averaged over test samples on one NVIDIA V100 GPU.Source: Quantitative Analysis · pp. 5–6
  • Deterministic inference adds 2.3 BLEU points on ASPEC Ja-En and 1.9 BLEU points on WMT’14 En-De; knowledge distillation significantly reduces the translation-quality gap.Source: Quantitative Analysis · p. 5

Non-autoregressive NMT Models

  • On WMT’14 En-De, the model’s 1.0-BLEU gap is relatively small among existing non-autoregressive models and offers a better speed-accuracy tradeoff than other refinement-based models.Source: Non-autoregressive NMT Models · p. 6, Table 2

Analysis of Deterministic Inference

Qualitative Analysis

  • Refinement can change target length as well as tokens, especially for long sentences; one example inserts a whole clause.Source: Qualitative Analysis · p. 7, Table 3 · p. 7
  • Some final translations retain duplicated words, typically when the initial translation guess is considerably worse than average.The authors identify a high-quality initial guess as crucial for good translations.Source: Qualitative Analysis · p. 7

8 Conclusion

LaNMT uses continuous latent variables and a deterministic delta-posterior inference algorithm for non-autoregressive translation. Its experiments report faster decoding and narrower performance gaps with autoregressive baselines.

8 Conclusion

  • The model introduces continuous latent vectors to capture target-sentence uncertainty, then adapts their number to match target length.It trains by maximizing the lowerbound of log p(y|x).Source: §8 · p. 7
  • The deterministic inference algorithm uses a delta posterior and improves the evidence lowerbound of predicted target sequences; BLEU scores converge in one refinement step.It alternates between updating the delta posterior and target tokens.Source: §8 · p. 7
  • 8.6x speedup closes the performance gap on ASPEC Ja-En; on En-De, multiple prior-sampled latent variables reduce the gap to 1.0 BLEU with 6.8x speedup.The En-De result uses multiple latent variables sampled from the prior.Source: §8 · p. 7
Loading 1908.07181v5…