Source-linked AI summary
Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement
Jason Lee, Elman Mansimov, Kyunghyun Cho
TL;DR
Autoregressive sequence models make decoding difficult to solve exactly and difficult to parallelize. This paper proposes a conditional non-autoregressive model based on iterative refinement, latent-variable modeling, and denoising autoencoders. Across machine translation and image captioning, it reports substantial decoding speedups while retaining comparable, though lower, generation quality.
Problem
Autoregressive decoding requires approximate search and sequential computation, creating computational overhead and latency.
Method
The paper uses a conditional non-autoregressive model with iterative refinement, hybrid latent-variable and denoising-autoencoder learning, and adaptive inference steps.
Results
Across translation and image captioning, the model decodes substantially faster than autoregressive counterparts while maintaining 90–95% translation quality and 85% caption quality.
Takeaways & Limitations
The results support non-autoregressive iterative refinement as a broadly applicable approach for reducing decoding latency across sequence generation tasks.
Takeaways & Limitations
The proposed non-autoregressive model is outperformed by its autoregressive counterpart in generation quality, motivating better marginal-log-probability approximations and further study of corruption processes and architectures.
Abstract
from arXiv · showhide
We propose a conditional non-autoregressive neural sequence model based on iterative refinement. The proposed model is designed based on the principles of latent variable models and denoising autoencoders, and is generally applicable to any sequence generation task. We extensively evaluate the proposed model on machine translation (En-De and En-Ro) and image caption generation, and observe that it significantly speeds up decoding while maintaining the generation quality comparable to the autoregressive counterpart.
1 Introduction
The paper proposes a broadly applicable deterministic non-autoregressive sequence model based on iterative refinement. Across translation and image captioning, it reports substantially faster decoding while retaining much of autoregressive quality.
- Motivation: Autoregressive decoding is sequential, difficult to solve exactly, and incurs substantial latency because approximate decoding cannot be easily parallelized.These limitations motivate non-autoregressive sequence modeling.
- Proposed approach: The proposed model uses iterative refinement and is designed for sequence generation tasks beyond machine translation.The approach is presented as generally applicable rather than task-specific.
- Proposed approach: Its learning algorithm combines latent-variable lower-bound maximization with conditional denoising-autoencoder reconstruction-error minimization.The model is interpreted through both latent-variable-model and denoising-autoencoder principles.
- Evaluation: On machine translation, decoding is approximately 2–3× faster than autoregressive beam search while retaining 90–95% of translation quality.The comparison covers IWSLT’16 En↔De, WMT’16 En↔Ro, and WMT’14 En↔De on GPU and CPU.
- Evaluation: On image caption generation, decoding is approximately 3× faster on GPU and 5× faster on CPU while retaining 85% of caption quality.The reported speedup is larger on CPU for this task.
2 Non-Autoregressive Sequence Models
Autoregressive models provide strong sequence modeling but make exact decoding intractable and sequential. Non-autoregressive factorization enables parallel exact decoding, although it can introduce a modeling-performance gap.
- Autoregressive modeling: Autoregressive sequence models factorize target probability into next-symbol conditionals given preceding symbols.The conditional distribution is parameterized by a neural network.
- Autoregressive modeling: Exact autoregressive decoding has no known polynomial-time algorithm, so practitioners use approximate methods such as beam search.Beam search generally performs better than greedy decoding but adds substantial computational overhead.
- Non-autoregressive modeling: Non-autoregressive models factorize target distributions into conditionally independent per-step distributions.Each per-step distribution is produced from the source sentence and spans the target vocabulary.
- Non-autoregressive modeling: This independence makes maximum decoding exact and parallel by selecting arg max_y_t p(y_t|X) independently for each position.The factorization bypasses autoregressive decoding overhead and suboptimality.
- Trade-offs: The benefit of exact parallel decoding may come with performance degradation because non-autoregressive factorization can enlarge the modeling gap.The modeling gap is described as the difference between the underlying true model and the neural sequence model.
3 Iterative Refinement for Deterministic Non-Autoregressive Sequence Models
The model introduces discrete latent variables and interprets their conditionals as deterministic iterative refinement, combining latent-variable and denoising-autoencoder objectives. Shared refinement parameters support adaptive inference, while decoding repeatedly updates a predicted target until a stopping criterion is met.
- Latent Variables: The model introduces L intermediate latent target sequences to capture dependencies among target symbols without autoregression.Each conditional models a target distribution from the source and a preceding latent sequence.
- Deterministic Approximation: Deterministic inference selects the most likely intermediate sequence instead of sampling, turning latent-variable marginalization into iterative refinement.The deterministic approximation uses maximum-probability latent values, while stochastic approximation samples from the corresponding distributions.
- Latent Variables: Shared parameters across refinement steps allow the model to adapt the iteration count for each input.The shared network models p(Y^l|Ŷ^{l−1}, X) and enables dynamic iteration counts.
- Training: Training combines lower-bound maximization with denoising reconstruction by stochastically replacing generated intermediate sequences with corrupted targets.The corruption process can copy, randomly replace, or swap target tokens, and the two objectives are mixed with probability pDAE.
- Distillation: Knowledge distillation replaces each reference target with a target generated by a well-trained autoregressive model, leaving the architecture and mixed cost function unchanged.The procedure follows prior non-autoregressive sequence modeling work.
- Inference: Deterministic decoding first predicts target length, generates an initial sequence, and repeatedly refines it until consecutive outputs satisfy a stopping criterion.The experiments stop when Jaccard distance between consecutive target sequences is sufficiently small.
4 Related Work
The paper situates iterative refinement among non-autoregressive, post-editing, and infusion-training approaches. It distinguishes its method through deterministic refinement and a stochastic mixture of denoising and lower-bound objectives for discrete text.
- Non-Autoregressive Sequence Modeling: Earlier non-autoregressive approaches targeted translation, waveform generation, or short phrase reranking, with some substantially behind autoregressive variants.The proposed method instead applies iterative refinement to sequence generation beyond those restricted settings.
- Non-Autoregressive Neural Machine Translation: Unlike Gu et al. (2017), the proposed method removes stochastic behavior by interpreting latent variables as deterministic iterative refinement rather than using supervised inference alignments.Gu et al. used a word-alignment tool for supervised inference and stochastic latent-variable inference for their best result.
- Parallel WaveNet: Unlike Parallel WaveNet, the proposed approach is intended for discrete as well as continuous target variables.Parallel WaveNet is restricted to continuous targets.
- Post-Editing for Machine Translation: Post-editing systems iteratively modify translations, whereas this approach can edit an intermediate translation with a higher degree of freedom.The comparison concerns the flexibility of edits during refinement.
- Infusion Training: For discrete text, the paper replaces infusion training’s undefined weighted mixture of sequences with a stochastic mixture of denoising and lower-bound objectives.The mixture is designed for sequences of discrete tokens.
5 Network Architecture
The architecture uses three transformer blocks: an encoder, an initial decoder, and a decoder shared across refinement steps. Decoder 2 repeatedly consumes the current predicted target and prior-step activations.
- Architecture: Three transformer blocks implement the model: Encoder, Decoder 1, and Decoder 2.The blocks are composed side-by-side in the architecture.
- Architecture: The Encoder encodes the input, Decoder 1 models p(Y^0|X), and shared Decoder 2 models p(Y^l|Ŷ^{l−1}, X).The encoder follows the original Transformer, while the decoders use the specified non-autoregressive decoder design.
- Architecture: Before Decoder 1, the input is padded or shortened to the reference target length.This length adjustment is part of the initial decoding path.
- Architecture: Decoder 2 takes the previous predicted target sequence and the previous step’s final activation vectors at each refinement step.This provides the inputs used to produce the next refined sequence.
- Evaluation View: Figure 1 relates BLEU quality to refinement steps and decoding latency to competing approaches using logarithmic axes.Panel (a) covers WMT’14 En-De; panel (b) covers IWSLT’16 En→De.
6 Experimental Setting
Experiments compare the non-autoregressive model with an autoregressive Transformer on translation and image captioning, measuring BLEU and decoding efficiency. The evaluation spans three translation datasets and MS COCO, with task-specific model and inference settings.
- Evaluation Metrics: The evaluation measures generation quality with BLEU and efficiency with tokens per second for translation or images per second for captioning.The comparison is against an autoregressive counterpart and is measured sentence-by-sentence for decoding efficiency.
- Machine Translation: Machine translation covers IWSLT’16 En↔De, WMT’16 En↔Ro, and WMT’14 En↔De, containing 196k, 610k, and 4.5M sentence pairs respectively.The datasets represent different sizes and use Moses tokenization with BPE subword segmentation.
- Machine Translation: The experiments use a small model for IWSLT’16 En-De and the base Transformer for WMT’14 En-De and WMT’16 En-Ro.The configurations differ in model dimensions, layers, and attention heads across task sizes.
- Image Caption Generation: Image captioning uses MS COCO with 113,287 training images, 5k validation images, and 5k test images.Image features are extracted from ResNet-18 representations.
- Target Length Prediction: Target length is separately predicted because the non-autoregressive model does not naturally model target length without an arbitrary upper bound.The predictor is trained from reference lengths and used only during testing.
- Training and Inference: Inference predicts target length first, then applies refinement using deterministic or stochastic strategies selected from validation performance.The deterministic strategy is used for IWSLT’16 En-De, WMT’16 En-Ro, and MS COCO; stochastic inference is used for WMT’14 En-De.
7 Results and Analysis
Across machine translation and image captioning, iterative refinement improves generation quality while offering faster, parallel decoding and controllable speed–quality trade-offs. Results also show that training choices, approximation strategy, and repeated refinement materially affect performance.
- Results and Analysis: Generation quality improves with additional refinement steps, including beyond the four iterations used during training.On WMT’14, quality continued improving when decoding was extended to 100 iterations.
- Results and Analysis: Adaptive decoding reaches near-best generation quality with lower computational overhead than using many fixed refinement steps.It increases refinement steps with sentence length, while latency grows less severely than for autoregressive decoding.
- Results and Analysis: The model’s speedup is clearer on GPU than CPU because its highly parallel computation is better suited to GPU hardware.On WMT’14 En-De, it outperforms the best model from Gu et al. (2017) by two points while using no external tool.
- Results and Analysis: The approach generalizes beyond translation to image caption generation, where successive iterations capture increasingly specific image details.Examples refine descriptions such as “yellow bus” to “yellow and black bus” and add a tennis court and racquet.
- Results and Analysis: Using four refinement iterations during training improves BLEU by approximately 1.5 points in both translation directions.The hybrid learning strategy is necessary to maximize the benefit of additional training iterations, while removing repeated consecutive symbols adds approximately +1 BLEU.
- Results and Analysis: Knowledge distillation is crucial for narrowing the gap with autoregressive models, and stochastic approximation is especially important on the large WMT’14 corpus.Deterministic approximation works as well or better on the smaller IWSLT’16 corpus; both strategies benefit from distillation.
- 7.1 Qualitative Analysis: Translation refinements add missing words and remove unnecessary ones, although individual iterations need not monotonically improve the output.The examples show gradual insertion of “at the time” and correction of unnecessary or awkward words.
8 Conclusion
The paper presents a deterministic non-autoregressive sequence model using iterative refinement, evaluated on machine translation and image caption generation. It achieves substantial decoding speedups while performing closely to autoregressive models, though generation quality remains lower.
- 8 Conclusion: The proposed model combines latent-variable modeling with denoising-based iterative refinement for deterministic non-autoregressive generation.The learning algorithm interprets the model as a latent variable model and each refinement step as denoising.
- 8 Conclusion: The approach was implemented with the Transformer and evaluated on machine translation and image caption generation.
- 8 Conclusion: On both tasks, the non-autoregressive model performed closely to the autoregressive counterpart with significant decoding speedup.
- 8 Conclusion: Qualitative analyses showed that iterative refinement gradually improves target sequences over multiple steps.
- 8 Conclusion: The non-autoregressive model was outperformed by the autoregressive counterpart in generation quality.The paper identifies better marginal-log-probability approximation, corruption-process analysis, and improved architectures as future directions.
A Impact of Length Prediction
Target-length prediction materially affects translation and captioning performance. The proposed predictor outperforms a training-set length-statistics baseline in exact and near-exact prediction accuracy.
- A Impact of Length Prediction: Approximately 1 BLEU score improvement was observed across datasets when reference target length was used during inference.The comparison is against the reported results using predicted target length.
- A Impact of Length Prediction: The baseline predicts target length from average target lengths associated with training sources of the same source length.
- A Impact of Length Prediction: The proposed length prediction model predicted target length correctly twice as often as the baseline, 16% vs. 8%.
- A Impact of Length Prediction: The model predicted target length within five tokens more accurately than the baseline, 83% vs. 69%.