PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond

Yisheng Xiao, Lijun Wu, Junliang Guo, Juntao Li, Min Zhang, Tao Qin, Tie-yan Liu

arXiv:2204.09269v2cs.CLcs.LG

TL;DR

Parallel non-autoregressive generation speeds up machine translation but sacrifices accuracy relative to autoregressive generation, with missing target-side dependency identified as a central challenge. This survey organizes NAT methods, compares their approaches, and reviews applications beyond translation; it concludes with research directions and guidance for choosing suitable strategies.

  • Problem

    NAT parallel decoding accelerates translation but sacrifices accuracy relative to autoregressive generation, and missing target-side dependency is identified as a central challenge.

    Source: §1 · p. 1, §2.2 · p. 4

  • Method

    The survey compares AT and NAT and organizes NAT research across data manipulation, modeling, training criteria, decoding, pretrained models, and applications beyond machine translation.

    Source: §1 · pp. 1–2, §10 · p. 15

  • Results

    The survey synthesizes methods for improving NAT and reviews their application across diverse tasks beyond machine translation.

    Source: §1 · p. 2, §8 · p. 11, §10 · p. 15

  • Takeaways & Limitations

    The survey aims to help researchers and engineers understand NAR techniques and choose strategies suited to their application tasks.

    Source: §10 · p. 15

    NAT latency comparisons can be unfair across batch sizes, hardware, and AR baselines, while translation-quality evaluation also has defects.

    Source: §8 · p. 11

Abstract

from arXiv · show

Non-autoregressive (NAR) generation, which is first proposed in neural machine translation (NMT) to speed up inference, has attracted much attention in both machine learning and natural language processing communities. While NAR generation can significantly accelerate inference speed for machine translation, the speedup comes at the cost of sacrificed translation accuracy compared to its counterpart, autoregressive (AR) generation. In recent years, many new models and algorithms have been designed/proposed to bridge the accuracy gap between NAR generation and AR generation. In this paper, we conduct a systematic survey with comparisons and discussions of various non-autoregressive translation (NAT) models from different aspects. Specifically, we categorize the efforts of NAT into several groups, including data manipulation, modeling methods, training criterion, decoding algorithms, and the benefit from pre-trained models. Furthermore, we briefly review other applications of NAR models beyond machine translation, such as grammatical error correction, text summarization, text style transfer, dialogue, semantic parsing, automatic speech recognition, and so on. In addition, we also discuss potential directions for future exploration, including releasing the dependency of KD, reasonable training objectives, pre-training for NAR, and wider applications, etc. We hope this survey can help researchers capture the latest progress in NAR generation, inspire the design of advanced NAR models and algorithms, and enable industry practitioners to choose appropriate solutions for their applications. The web page of this survey is at \url{https://github.com/LitterBrother-Xiao/Overview-of-Non-autoregressive-Applications}.

1 INTRODUCTION

Autoregressive translation generates target tokens sequentially, while non-autoregressive translation generates them in parallel for faster inference but struggles to capture target-side dependencies. This survey compares AT and NAT, reviews methods addressing that challenge, and covers applications beyond machine translation and future directions.

Motivation

  • NAT’s central challenge is capturing target-side dependencies, which autoregressive models retain through previously generated tokens.Without that information, NAT also lacks target-side signals such as decoding length.Source: §1 · p. 1

Data manipulation

  • Data manipulation methods reduce training-data complexity to make the task easier for NAT models lacking target dependency.The survey also reviews training-data learning strategies.Source: §1 · p. 1

Modeling

  • Modeling approaches include iteration with partial target information, latent variables, and enhancements to decoder input, output, or intermediate states.These architectures are proposed to better capture target dependency.Source: §1 · pp. 1–2

Survey scope

  • The survey compares AT and NAT training objectives, architectures, and inference schedules, then organizes methods intended to address NAT’s main challenge.Its outline proceeds from the comparison and challenge to methods, applications, and open problems.Source: Figure 1 · p. 2, §1 · pp. 2–3

Training criterion

  • Some work replaces standard cross-entropy with CTC-based, n-gram-based, or order-based criteria to improve NAT training.These alternatives are motivated by claims that traditional cross-entropy is not optimal for NAT.Source: §1 · p. 2

Decoding, pretraining, and broader scope

  • The survey reviews decoding strategies and pretrained-model information, and extends its coverage to other NAR applications and future research directions.Applications include speech recognition, text generation, summarization, grammatical error correction, dialogue, and semantic parsing; proposed directions include reducing reliance on knowledge distillation, improved objectives, and pretraining for NAR.Source: §1 · p. 2

2 OVERVIEW OF AT AND NAT MODELS

The survey contrasts autoregressive, non-autoregressive, and intermediate translation approaches, then identifies missing target-side information as NAT’s central challenge. It organizes proposed improvements into five categories and traces representative developments.

2.1 Comparison

  • NAT removes the preceding-target-token context used by AT, enabling parallel translation and faster inference.AT predicts left to right with teacher forcing; NAT predicts target tokens independently, using a predicted target length at inference.Source: §2.1 · p. 3
  • SAT generates target-token groups in parallel but autoregressively across groups, whereas iteration-based NAT refines parallel predictions over successive iterations.Fully NAT models decode in one step; iteration-based models feed the previous iteration’s translation back for refinement.Source: §2.1 · pp. 3–4

2.2 The Main Challenge of NAT Models

  • Without target information in the decoder, NAT relies on the source and may combine fragments of valid translations or repeat or omit tokens.The survey attributes these errors to difficulty capturing target-side dependency.Source: §2.2 · p. 4

2.3 Overview of Improving Methods

  • The survey groups NAT improvements into data manipulation, modeling, training criteria, decoding, and use of pre-trained models.Modeling approaches include iterative refinement, latent variables, and enhancements that supply stronger target-side information.Source: §2.3 · p. 4
  • The development timeline presents SAT as a bridge between AT and NAT, followed by representative modeling, criterion, and pre-training advances.The survey reports that NAT’s performance gap with AT models is narrowing.Source: Figure 2 · p. 5, §2.3 · p. 5

3 DATA MANIPULATIONS

The survey reviews data manipulation as a way to help NAT models capture target-side dependency, covering knowledge distillation and learning strategies. It describes both the benefits and drawbacks of distillation, alongside alternative approaches to training data and learning.

3.1 Knowledge Distillation

  • Sequence-level knowledge distillation trains NAT on target sentences decoded by a pre-trained AT teacher, using greedy decoding or beam search.The survey describes distilled data as less noisy and more deterministic than the original corpus.Source: §3.1 · p. 5
  • Distillation can simplify data and reduce target-token dependency, but may harm real-data density and reduce low-frequency-word accuracy.Teacher capacity should align with NAT capacity; the survey also describes using raw, distilled, and reverse-distilled data to rejuvenate low-frequency words.Source: §3.1 · pp. 5–6

3.2 Data Learning Strategies

  • Learning strategies include curricula, consistency training, and contrastive learning to improve how NAT models learn from training data.Examples include gradually switching decoder inputs from AT to NAT, glancing from fragments toward whole sentences, and requiring consistent predictions for the same masked position across contexts.Source: §3.2 · p. 6

4 MODELING

The modeling section organizes NAT approaches around iterative refinement, latent variables, and enhancements that supply target-side information across decoder stages.

4.1 Iteration-Based Methods

  • Iteration-based NAT refines partially observed targets; insertion/deletion models edit tokens, while CMLM masks and predicts target positions using uniform masking.Insertion approaches vary in generation order and may infer it during decoding; CMLM predicts target tokens during training.Source: §4.1 · pp. 6–7

4.2 Latent Variable-Based Methods.

  • Latent-variable NAT first predicts a latent sequence used to represent linguistic information, alignments, target positions, or syntax and semantics; parser-dependent syntactic supervision works only in limited scenarios.The latent variables may be word chunks or other prompt information.Source: §4.2 · pp. 7–8

4.3 Other Enhancements-based Methods

  • Enhancement methods add target information at decoder input, supervise intermediate states, or improve outputs through regularization, recovery, DAG representations, and rephrased training targets.Input enhancements include phrase-table lookup, embedding mapping, and decoder-input transformation.Source: §4.3 · p. 8

5 CRITERION

The section surveys training criteria that address independent token prediction and sensitivity to alignments by modeling sequence matches, n-grams, or alternative token orderings.

5 CRITERION

  • Traditional cross-entropy is widely used, but its token-level objective can be suboptimal for NAT and sensitive to inconsistent prediction-reference alignments.The section frames criterion alternatives as ways to address these issues.Source: §5 · pp. 8–9
  • CTC criteria use dynamic programming over prefix and suffix log-probabilities to alleviate misalignment, while related criteria marginalize monotonic alignments.Bipartite and n-gram matching also explore non-monotonic latent alignments.Source: §5 · p. 8
  • N-gram criteria minimize the difference between model and reference n-gram counts, addressing word-level CE’s lack of global correctness and repetitive translations.Edit-Invariant Sequence Loss instead focuses on n-gram matching under inconsistent source-target order.Source: §5 · pp. 8–9
  • AXE uses the best monotonic alignment to soften penalties for word-order errors, whereas OAXE finds a best alignment and focuses on lexical matching.Extensions allow phrase-level reordering or relax strict matching, and MgMO uses feedback across translation-segment granularities.Source: §5 · p. 9

6 DECODING

NAT decoding requires a predicted target length and offers schedules ranging from one-step parallel generation to iterative and hybrid strategies.

6.1 Length Prediction

  • Because parallel NAT lacks start and end tokens to determine termination, it must predict target length before generating the target content.Length estimates can come from fertility, classification, linear functions, a special [LENGTH] token, or CTC-based alignment.Source: §6.1 · p. 9
  • LPD generates candidates within a window around a predicted length and selects with an AR scorer; NPD selects among top predicted lengths using average token log-probabilities.Both approaches address deviation between predicted and true length.Source: §6.1 · p. 10

6.2 Decoding Strategy

  • Fully NAT’s one-step decoding is faster but lower quality, and iteration-based models generally perform better, indicating that one-step decoding captures target-side dependency inadequately.The comparison is reported in Table 1.Source: §6.2 · p. 10
  • Decoding strategies include semi-autoregressive generation, insertion/deletion, mask-and-predict refinement, and mixed schemes that combine AR, semi-AR, or iterative generation.Aggressive decoding drafts tokens non-autoregressively and verifies them autoregressively; drafting and verification can run in parallel.Source: §6.2 · p. 10

7 BENEFITING FROM PRE-TRAINED MODELS

The section reviews ways to use autoregressive models and pretrained language models to improve NAT training and modeling.

7.1 AT Models

  • Autoregressive models support NAT through decoding-state supervision, representation hints, curriculum fine-tuning, joint AT-NAT optimization, and shared-encoder or weak-AR-decoder designs.One curriculum shifts training from AT to semi-autoregressive and then NAT.Source: §7.1 · pp. 10–11

7.2 Pre-Trained Language Models

  • Pretrained language models are incorporated into NAT through BERT-initialized mask-predict models, BERT backbones with CRF output, aligned code-switching masking, and XLM-R adaptation.The aligned code-switching model can be fine-tuned on both NAT and AT tasks.Source: §7.2 · p. 11

8 SUMMARY OF NON-AUTOREGRESSIVE NMT

NAT methods address missing target-side dependency through data, model, training, and decoding strategies, while also drawing on stronger models. Reported efficiency and translation-quality comparisons have important evaluation caveats.

8 SUMMARY OF NON-AUTOREGRESSIVE NMT

  • Data simplification, learning strategies, model structures, and criteria such as CTC-, n-gram-, and order-based losses target NAT’s dependence on target tokens.Data distillation reduces corpus complexity; alternative criteria address misalignment, global context, or reasonable translations with mismatched token positions.Source: §8 · p. 11
  • Decoding methods use length prediction, partial target information, or combinations of decoding modes; information from autoregressive and pretrained models can also improve NAT.The survey compares existing methods’ effectiveness and inference speed in an appendix.Source: §8 · p. 11
  • Single-sentence, single-GPU latency measurements can overstate NAT’s efficiency advantage: larger batches narrow it, and hardware and decoder choices complicate comparisons.The survey also notes that BLEU may under-penalize unusual NAT errors and calls for broader quality metrics and fine-grained evaluation.Source: §8 · p. 11

9 EXTENSIVE APPLICATIONS BEYOND NMT

The survey reviews NAR generation beyond machine translation across text and speech tasks, diffusion models, and other applications. Across these areas, it describes task-specific approaches to target-side information, length prediction, and parallel generation.

9.1 Text Generation

  • General-purpose text-generation work includes pretrained, iterative, and editing frameworks; task-specific methods address ASR, summarization, grammatical correction, and dialogue.ASR work includes iteration-based, CTC-based, and pretrained-model approaches,, [132]; summarization methods address length control,.Source: §9.1 · pp. 12–13
  • Text-style transfer explores alignment, iterative decoding, and distillation,, while controllable generation, image captioning, and question answering study task-specific constraints and parallel decoding,, [183].Other examples include AutoTemplate for lexical constraints [172], discrete mode learning for captioning, and KECP for extractive question answering [184].Source: §9.1 · p. 13

9.2 Semantic Parsing

  • For semantic parsing, accurate length prediction is especially important because small errors can change the result; proposed methods use CNN-based length estimation or span prediction.The survey notes that iterative refinement offers little benefit for task-oriented semantic parsing, so one decoding step is used.Source: §9.2 · p. 13

9.3 Text to Speech

  • NAR text-to-speech work addresses alignment, over-smoothing, and diversity, with approaches including latent-variable models, duration models, and prosody control.Some methods still use a teacher model, while bidirectional inference VAE is proposed to rely less on one without reducing performance.Source: §9.3 · pp. 13–14

9.4 Speech Translation

  • Speech translation adapts NAR ASR and machine-translation strategies, including CTC auxiliaries, shared NAR/AR decoders, rescoring, and Mask-CTC intermediates.Fast-MD also uses sampling prediction to reduce training–testing mismatch.Source: §9.4 · p. 14

9.5 Diffusion Models

  • Diffusion generation adds noise through a forward process and denoises from Gaussian noise to target data non-autoregressively; text methods include SUNDAE and Diffusion-LM.The survey connects denoising iterations to iterative NAR decoding and describes diffusion applications in translation, summarization, and style transfer.Source: §9.5 · pp. 14–15, Figure 3 · p. 14

9 EXTENSIVE APPLICATIONS BEYOND NMT

  • Beyond NMT, the survey covers NAR methods for information extraction, video generation, voice conversion, lip-to-speech, speech representations, code completion, classification, and time-series forecasting.Examples include MacroIE’s parallel fact-set prediction, DIGAN’s parallel frame generation, and MANF for forecasting [227].Source: §9.6 · p. 15

9.6 Others

  • Other NAR applications include unordered fact extraction, parallel video generation, speech representation learning, code completion, extreme classification, and time-series forecasting.The survey describes local-dependency non-autoregressive predictive coding for speech representations [224] and a latent-variable model for extreme classification [226].Source: §9.6 · p. 15

10 CONCLUSION AND OUTLOOKS

The survey organizes NAT improvements and applications while identifying unresolved challenges in distillation dependence, training objectives, decoding speed, application scope, and pretraining.

10 CONCLUSION AND OUTLOOKS

  • The survey reviews NAT solutions across data manipulation, modeling, criteria, decoding, and pretrained models, then surveys applications beyond NMT and offers future directions.It aims to help readers understand NAR techniques and choose strategies for application tasks.Source: §10 · p. 15
  • Knowledge distillation remains the most effective method used in NAR models, but it requires pretraining an autoregressive model and performance on raw datasets remains an open question.Source: §10 · p. 15
  • Iteration-based models can achieve performance comparable to AR models, but their speedup diminishes with large batch sizes,.Source: §10 · p. 15
  • Maximum-likelihood training can approximate marginal distributions while dropping token dependencies, leaving suitable NAR training objectives an open question.Source: §10 · p. 15
  • Most NAR applications remain limited to bilingual settings, unlike the broader bilingual, multilingual, high-resource, and low-resource use of AR models.Source: §10 · p. 15
  • Only a few studies apply pretrained models to NAR, and NAR pretraining remains at a preliminary exploration stage, [130],,,.Source: §10 · p. 15

OVERVIEW OF IMPROVING METHODS FOR NAT

Figure 4 lays out the general NAT framework and data flow.

OVERVIEW OF IMPROVING METHODS FOR NAT

REPRESENTATIVE MODELING METHODS

Figure 5 presents representative modeling methods discussed in Section 4.

REPRESENTATIVE MODELING METHODS

CRITERION

The section illustrates how NAT criteria treat reordered predictions and contrasts decoding strategies. Its benchmark synthesis frames translation quality and speed as a trade-off, alongside curated resource collections.

COMPARISONS BETWEEN EXISTING METHODS

CRITERION

  • Traditional CE penalizes all tokens in a reordered prediction, while N-gram CE, AXE, and OAXE handle token order differently.N-gram CE flags the two-gram “night I”; AXE finds a monotonic alignment and penalizes unaligned tokens, whereas OAXE gives no penalty for the order errors.Source: CRITERION · p. 22, Figure 6 · p. 25

ILLUSTRATION OF DECODING STRATEGIES

  • Autoregressive and semi-autoregressive decoding generate left to right, while fully non-autoregressive and iterative strategies predict tokens in parallel or use insertion and deletion operations.Mask-and-predict methods predict all or part of the target in parallel; insert-and-delete methods add and remove tokens across iterations.Source: ILLUSTRATION OF DECODING STRATEGIES · p. 22, Figure 7 · p. 25

RESOURCES

  • The survey collects published papers and code for NAT, related general-purpose and task-specific text-generation tasks, and other NAR applications beyond text generation.The resource collections are summarized in Tables 3, 4, and 5.Source: Table 3 · p. 29, Table 4 · p. 30, Table 5 · p. 31

Related papers

Loading 2204.09269v2…