PaperlayerView exact revision on arXiv

Source-linked AI summary

Non-Autoregressive Machine Translation with Latent Alignments

Chitwan Saharia, William Chan, Saurabh Saxena, Mohammad Norouzi

arXiv:2004.07437v3cs.CLcs.LG

TL;DR

Non-autoregressive translation must address token repetition and target-length constraints while preserving translation quality. This paper applies CTC and Imputer latent-alignment models trained with dynamic programming, achieving strong single-step and constant-step results without target length prediction or autoregressive re-scoring. On WMT’14 En→De, Imputer reaches 27.5 BLEU with 2 steps and 28.0 BLEU with 4 steps, while the paper reports 25.8 BLEU for single-step CTC.

  • Problem

    Non-autoregressive models face token repetitions from conditional independence and often require target length prediction before decoding.

  • Method

    The paper adapts CTC and Imputer latent-alignment models, using monotonic discrete alignments and dynamic programming to marginalize them during training.

  • Results

    25.8 BLEU on WMT’14 En→De sets a new single-step non-autoregressive result, while Imputer reaches 27.5 BLEU with 2 steps and 28.0 BLEU with 4 steps.

  • Takeaways & Limitations

    Latent alignment models provide simple non-autoregressive translation systems that avoid target length prediction and autoregressive candidate re-scoring.

  • Takeaways & Limitations

    The approach cannot generate a target sequence longer than the source sequence, relying on the assumption |x| ≥ |y|.

Abstract

from arXiv · show

This paper presents two strong methods, CTC and Imputer, for non-autoregressive machine translation that model latent alignments with dynamic programming. We revisit CTC for machine translation and demonstrate that a simple CTC model can achieve state-of-the-art for single-step non-autoregressive machine translation, contrary to what prior work indicates. In addition, we adapt the Imputer model for non-autoregressive machine translation and demonstrate that Imputer with just 4 generation steps can match the performance of an autoregressive Transformer baseline. Our latent alignment models are simpler than many existing non-autoregressive translation baselines; for example, we do not require target length prediction or re-scoring with an autoregressive model. On the competitive WMT'14 En$\rightarrow$De task, our CTC model achieves 25.7 BLEU with a single generation step, while Imputer achieves 27.5 BLEU with 2 generation steps, and 28.0 BLEU with 4 generation steps. This compares favourably to the autoregressive Transformer baseline at 27.8 BLEU.

1 Introduction

The paper addresses key limitations of non-autoregressive translation by modeling latent alignments with CTC and Imputer. These methods achieve strong translation quality with single-step or constant-step generation.

  • Motivation: Non-autoregressive models often struggle with token repetitions from conditional independence and require separate target length prediction.These limitations can require stochastic or iterative decoding, heuristic repetition collapsing, and fixed-length inference.
  • Approach: Latent alignment models use discrete monotonic alignments and dynamic programming to marginalize alignment variables during training.The paper studies CTC and Imputer as two instances of this approach.
  • Results: 25.8 BLEU on WMT’14 En→De establishes a new state-of-the-art for single-step non-autoregressive translation.The paper attributes the improvement over earlier CTC work primarily to distillation during training.
  • Results: 27.5 BLEU with 2 steps, 28.0 BLEU with 4 steps, and 28.2 BLEU with 8 steps set new constant-step non-autoregressive translation results.All reported results are on WMT’14 En→De.

2 Latent Alignment Models

Latent alignment models represent monotonic mappings between parallel predictions and target tokens, marginalizing alignments with dynamic programming. CTC enables one-step parallel decoding, while Imputer models dependencies across a constant number of iterative steps.

  • Alignment formulation: Alignments map predicted-token sequences to target sequences, using blank tokens and collapse operations to recover the target.The alignment length matches the source sequence length, and repeated tokens are collapsed before blanks are removed.
  • Alignment formulation: The model assumes a monotonic mapping and a source sequence at least as long as the target sequence.Formally, the method requires |x| ≥ |y|.
  • Latent-alignment computation: Dynamic programming makes latent-alignment likelihoods tractable despite the combinatorial number of possible alignments.The paper applies this strategy to both CTC and Imputer.
  • Connectionist Temporal Classification: CTC factorizes alignment-token predictions conditionally independently, enabling exact marginalization and parallel generation in a single step.Inference recovers the output with greedy decoding or beam search; the experiments use greedy decoding.
  • Imputer: Imputer conditions each parallel generation step on a partially predicted alignment, modeling dependencies across steps while retaining a constant number of iterations.Its likelihood lower-bound is solved efficiently with dynamic programming, and the iteration count is independent of sequence length.
  • Advantages: Latent alignment models address token repetition and target-length prediction issues identified in prior non-autoregressive translation work.The paper presents these as key advantages over earlier approaches.

3 Latent Alignment Models for Machine Translation

The paper adapts latent alignment models to machine translation, using monotonic alignments and dynamic programming while addressing repetition and target-length constraints. CTC enables single-step parallel generation, whereas Imputer supports a constant number of decoding iterations.

  • Latent alignment adaptation: Latent alignment models adapt CTC and Imputer to translation by marginalizing discrete alignment variables with dynamic programming.The adaptation assumes a monotonic mapping between alignment predictions and targets and addresses the associated length constraint.
  • Fewer Token Repetitions: Token repetitions arise from conditional independence, but collapsing repeated alignment tokens can remove many erroneous repetitions.The paper applies the collapsing function β−1 after generating the target alignment.
  • Generation procedure: CTC generates the entire alignment in one step, while Imputer uses a constant number of decoding iterations independent of sequence length.CTC makes a stronger conditional-independence assumption; Imputer relaxes this through iterative decoding.
  • No Target Length Prediction Needed: Latent alignment models infer target length through the alignment, avoiding separate target-length prediction and length-candidate search during inference.The alignment has source-sequence length, so the target length is determined implicitly when blanks are collapsed.
  • No Target Length Prediction Needed: A length caveat remains: the unmodified construction cannot generate a target sequence longer than the source sequence.The paper addresses this restriction by upsampling the source representation.

4 Related Work

The work extends prior non-autoregressive translation methods by combining latent alignments with a unified conditional architecture and both single-step and iterative generation. It differs from related approaches in its use of alignment-based superpositioning and conditional p(y|x) modeling.

  • Prior non-autoregressive methods: The paper relates to iterative refinement, insertion-based methods, and conditional masked language models for non-autoregressive translation.Like insertion-based models, it does not commit to a fixed target length.
  • Unified architectures: Unlike KERMIT’s concatenation approach, the paper superposes input and output sequences through latent alignment.Both use unified architectures without conventional encoder-decoder separation.
  • Modeling objective: The work focuses on conditional p(y|x) modeling, whereas KERMIT emphasizes generative p(x, y) modeling.This distinguishes the modeling objective even though the architectures are similar.
  • CTC translation: Compared with prior CTC translation work, this paper uses data distillation and reports a significant performance boost from distillation.The CTC models are closely related to Libovicky and Helcl’s single-step generation models.
  • Concurrent work: Unlike concurrent AXE CMLM work focused on single-step generation, this paper applies latent alignment models to both single-step and iterative generation.Both approaches use latent alignments and dynamic programming for learning.

5 Experiments

Experiments evaluate latent alignment models on WMT’14 En↔De and WMT’16 En-Ro using BLEU, with distilled data and comparisons against autoregressive and non-autoregressive systems. CTC performs strongly in one step, while Imputer approaches or exceeds autoregressive Transformer performance with few iterations.

  • Experimental setup: Experiments use WMT’14 En↔De and WMT’16 En-Ro, evaluate with BLEU, and report newstest2013 development and newstest2014 test results for WMT’14.Models use SentencePiece shared subword vocabularies and Transformer-based configurations.
  • Distillation: Distillation uses autoregressive base Transformers generally, with big Transformer distillation additionally reported for iterative-generation comparisons.The experiments compare distillation from base and big Transformers in WMT’14 En-De.
  • Single Step Decoding: 25.7 BLEU is achieved by CTC with single-step generation on WMT’14 En→De, while Imputer achieves 25.8 BLEU.Both single-step models outperform the autoregressive GNMT baseline at 24.6 BLEU.
  • Iterative Decoding: 28.2 BLEU is achieved by Imputer with 8 decoding iterations on En→De, slightly exceeding the autoregressive Transformer’s 27.8 BLEU.On De→En, Imputer reaches 31.3 BLEU, on par with the autoregressive Transformer.
  • Iterative Decoding: 27.5 BLEU is achieved on En→De with only 2 Imputer iterations, demonstrating robustness when decoding iterations are reduced from 8 to 2.The corresponding De→En result is 30.2 BLEU.
  • Iterative Decoding: Imputer matches the autoregressive Transformer on En↔Ro with 4 decoding iterations.The table comparison covers autoregressive and non-autoregressive models under the stated distillation settings.

6 Analysis

The analysis examines repetition, decoding-iteration trade-offs, distillation, and target-length effects for CTC and Imputer. Imputer remains robust at fewer iterations, while distillation and additional iterations improve performance.

  • 6.1 Token Repetitions: CTC produces significantly fewer repeated tokens than single-step Mask-Predict and AXE CMLM.This supports the hypothesis that β−1 helps remove spurious token repetitions.
  • 6.2 Impact of Number of Decoding Iterations: 0.6 BLEU points are lost when Imputer reduces decoding iterations from 8 to 2.The model is therefore robust to reducing iterations, trading a small quality decrease for faster inference.
  • 6.2 Impact of Number of Decoding Iterations: 4 decoding iterations match Imputer’s autoregressive teacher, while autoregressive O(n) decoding reaches 28.3 BLEU.Performance increases consistently with more iterations and slightly exceeds the teacher in the autoregressive extreme.
  • 6.3 Impact of Distillation: Distilled training data significantly improves every evaluated model compared with original training data.The performance gap is largest for CTC and decreases as Imputer uses more decoding iterations.
  • 6.4 Impact of Target Length for Imputer: Increasing decoding iterations consistently improves BLEU across all target-length buckets.The analysis computes BLEU separately for examples grouped by target sentence length.

7 Conclusion

The paper applies CTC and Imputer latent alignment models to non-autoregressive machine translation using dynamic programming. These models avoid target length prediction and autoregressive candidate rescoring while supporting single-step or constant-step generation.

  • 7 Conclusion: CTC and Imputer are latent alignment models adapted to non-autoregressive machine translation.They use dynamic programming to marginalize latent alignments during training.
  • 7 Conclusion: CTC generates in one step, whereas Imputer uses only a constant number of generation steps.Both models are designed for parallel translation without requiring sequence-length-dependent generation.
  • 7 Conclusion: The models do not require target length prediction or autoregressive rescoring of candidates.They also use a simplified neural architecture without the cross-attention mechanism found in many prior encoder-decoder architectures.
Loading 2004.07437v3…