PaperlayerView exact revision on arXiv

Source-linked AI summary

AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation

Tong Wu, Zhihao Fan, Xiao Liu, Yeyun Gong, Yelong Shen, Jian Jiao, Hai-Tao Zheng, Juntao Li, Zhongyu Wei, Jian Guo, Nan Duan, Weizhu Chen

arXiv:2305.09515v3cs.CL

TL;DR

Existing diffusion language models decode tokens concurrently, but this weakens the sequential dependencies central to natural language generation. AR-Diffusion introduces position-dependent denoising so left tokens generate earlier and influence right tokens, achieving comparable results at 100×–600× higher decoding speed. The method still has a sampling-cost limitation because optimal performance may require many candidate samples.

  • Problem

    Diffusion language models generate tokens concurrently, but this sacrifices inter-token position dependency and can reduce generation performance compared with auto-regressive generation.

  • Method

    AR-Diffusion uses multi-level sentence- and token-level diffusion with position-dependent denoising, plus skipping for faster decoding.

  • Results

    AR-Diffusion surpasses existing diffusion models across text generation tasks in quality and diversity, achieving 100× faster machine translation and 600× faster summarization with comparable results.

  • Takeaways & Limitations

    AR-Diffusion combines AR-like generation behavior with efficient parallel decoding while maintaining diversity across multiple text generation tasks.

  • Takeaways & Limitations

    Optimal performance requires generating many candidate samples; on CNN/DAILYMAIL, k = 50 yields a 0.8 lower ROUGE-2 score than k = 500.

Abstract

from arXiv · show

Diffusion models have gained significant attention in the realm of image generation due to their exceptional performance. Their success has been recently expanded to text generation via generating all tokens within a sequence concurrently. However, natural language exhibits a far more pronounced sequential dependency in comparison to images, and the majority of existing language models are trained with a left-to-right auto-regressive approach. To account for the inherent sequential characteristic of natural language, we introduce Auto-Regressive Diffusion (AR-Diffusion). AR-Diffusion ensures that the generation of tokens on the right depends on the generated ones on the left, a mechanism achieved through employing a dynamic number of denoising steps that vary based on token position. This results in tokens on the left undergoing fewer denoising steps than those on the right, thereby enabling them to generate earlier and subsequently influence the generation of tokens on the right. In a series of experiments on various text generation tasks, including text summarization, machine translation, and common sense generation, AR-Diffusion clearly demonstrated its superiority over existing diffusion language models and that it can be $100\times\sim600\times$ faster when achieving comparable results. Our code is available at https://github.com/microsoft/ProphetNet/tree/master/AR-diffusion.

1 Introduction

Diffusion-based text generators decode all tokens simultaneously, gaining speed but weakening the sequential dependencies that support natural language quality. AR-Diffusion addresses this by assigning position-sensitive denoising dynamics and achieves strong quality with substantially faster decoding.

  • Auto-regressive language models generate tokens left to right, using position dependency to improve naturalness, coherence, and adherence to language conventions.
  • Diffusion text generators predict all tokens simultaneously, which accelerates decoding but sacrifices inter-token position dependency and generation performance.
  • AR-Diffusion assigns dynamic token-level movement speeds within a sentence-level diffusion process, allowing left tokens to denoise faster than right tokens.The strategy is designed so earlier-denoised tokens can provide information to later tokens.
  • 100× faster than SeqDiffSeq in machine translation and 600× faster than GENIE in text summarization, while delivering comparable results.The experiments cover text summarization, machine translation, and common sense generation.

2 Preliminary

Conditional language generation commonly contrasts auto-regressive models, which condition each token on preceding tokens, with faster non-auto-regressive models that generate tokens independently. Diffusion language models add iterative noising and denoising to this comparison, using latent representations and learned reverse transitions for text generation.

  • 2.1 Conditional Generative Language Models: Auto-regressive models factorize the conditional likelihood so each token is predicted from the visible tokens to its left.
  • 2.1 Conditional Generative Language Models: Non-auto-regressive models assume conditional independence among tokens and generate them uniformly, offering substantial speed advantages but lower quality than auto-regressive models.
  • 2.2 Diffusion Models for Text Generation: Diffusion language models use a forward noising process that gradually perturbs a latent variable until it becomes standard Gaussian noise.The forward process is described as a fixed linear Gaussian model with timestep-dependent coefficients.
  • 2.2 Diffusion Models for Text Generation: Their reverse process starts from standard Gaussian noise and repeatedly applies denoising transitions to generate text.The denoising transition is pθ(z_t−1|z_t; x), with model-learned mean and variance parameters.
  • 2.2 Diffusion Models for Text Generation: For discrete text, diffusion models map tokens into latent embeddings and use a trainable rounding step to recover token predictions.An encoder-decoder model approximates the clean latent state in a non-auto-regressive manner during denoising.

3 Methodology

AR-Diffusion uses multi-level diffusion with position-sensitive token timesteps so left tokens denoise faster than right tokens, preserving sequential dependencies. Its skipping mechanism reduces inference steps by traversing a decreasing timestep subsequence.

  • Multi-Level Diffusion: AR-Diffusion combines sentence-level and token-level diffusion, assigning each token a timestep based on its position and sentence-level timestep.The sentence-level timestep is sampled first, then positional information determines each token's diffusion timestep.
  • Token-Level Diffusion with Dynamic Movement Speed: Left-side tokens use higher movement speeds and denoise earlier, while right-side tokens move more slowly and use previously denoised information.This dynamic movement speed is implemented with a position-sensitive token-level timestep function.
  • Token-Level Diffusion with Dynamic Movement Speed: The forward process extends the sentence-level timestep range to [0, N + T] using a linear token-level diffusion strategy.The start point moves horizontally from (N, 0) to (0, 0), then vertically to (0, T).
  • Inference with Skipping: During inference, AR-Diffusion assigns token-level timesteps at each pair of sentence-level timesteps before reverse sampling.The model computes f(n, ti) and f(n, ti+1) for every token position and uses them in the reverse diffusion process.
  • Inference with Skipping: The implementation uses the anchor point (2 × N, T) and maps the final denoised representation to the nearest embedding.The inference algorithm initializes from Gaussian noise, performs M reverse-sampling iterations, and produces the predicted target embedding.
  • Inference with Skipping: Skipping reduces decoding work by selecting a decreasing subsequence of M timesteps from T + N to 0, where M is much smaller than T + N.For example, an interval of 500 over T + N = 2500 yields five decoding steps: [2500, 2000, 1500, 1000, 500, 0].

4 Experiments

Experiments evaluate AR-DIFFUSION across summarization, translation, and common-sense generation using multiple baseline groups and task-specific metrics. Across these tasks, it achieves strong quality, diversity, and decoding-efficiency results, with ablations supporting both proposed mechanisms.

  • Experimental Setup: AR-DIFFUSION achieves the best reported performance across the evaluated datasets and comparison tables.The experiments compare against NAR, Semi-NAR, AR, and diffusion baselines.
  • Text Summarization: It outperforms existing NAR and Semi-NAR summarization approaches across all metrics and consistently improves over GENIE.Against Transformer, it exceeds ROUGE-1 and ROUGE-L, matches ROUGE-2 comparably, and surpasses Transformer across all measures when k = 500.
  • Machine Translation: In machine translation, AR-DIFFUSION outperforms CNAT and GENIE, slightly exceeds AR Transformer, and shows strong results against DINOISER across evaluation settings.It is stronger than DINOISER when k = 500 and achieves further gains with larger candidate sampling.
  • Common Sense Generation: AR-DIFFUSION achieves superior performance across all COMMONGEN metrics compared with AR, NAR, and other diffusion methods.The evaluation uses ROUGE, BLEU, METEOR, and SPICE measures.
  • Inference Efficiency: 100× faster decoding than SeqDiffuSeq is achieved in machine translation while still outperforming it, and reduced-step inference remains more stable than GENIE.At two inference steps, the average score drop is 1.34 for AR-DIFFUSION versus 4.20 for GENIE; at three steps, GENIE’s decline is 2.81.
  • Diversity and Ablation: AR-DIFFUSION produces higher diversity than an autoregressive model and comparable diversity to GENIE with better performance.The ablation study also finds that both multi-level diffusion and the skipping mechanism are essential, while skipping outperforms DDIM at varied inference steps.

5 Related Work

Prior work contrasts auto-regressive and non-auto-regressive generation, continuous text diffusion, and related autoregressive diffusion approaches. AR-DIFFUSION instead targets left-to-right text generation in continuous embedding space, with anchor-point effects evaluated on IWSLT14.

  • Anchor-Point Evaluation: Table 9 evaluates anchor-point positions on the IWSLT14 DE→EN test set, using N for target length and T for total diffusion time.The table caption defines the quantities used in that anchor-point comparison.
  • AR and NAR Language Models: AR models generate text token by token, whereas NAR models generate the entire output sequence in parallel to improve speed and efficiency.The related-work discussion frames AR-DIFFUSION against this speed–sequentiality distinction.
  • Continuous Text Diffusion: Continuous text diffusion integrates continuous noise into word embeddings through embedding and rounding processes for sequence-to-sequence generation.DiffuSeq is cited as conditioning on one input part while perturbing the other.
  • Related Autoregressive Diffusion: ARDMs use order-agnostic masking and prediction, while TimeGrad combines RNNs with diffusion for conditional multivariate time-series forecasting.These methods share autoregression and diffusion terminology but address different generation settings.
  • Related Autoregressive Diffusion: AR-DIFFUSION focuses on continuous embedding-space diffusion for left-to-right text generation, unlike the related methods described above.The distinction is methodological and task-specific rather than merely terminological.

6 Conclusion

The paper introduces AR-DIFFUSION to combine AR-like sequential behavior with efficient parallel decoding. Its multi-level diffusion and skipping mechanism preserve quality and diversity while enabling substantially faster generation.

  • Contribution: AR-DIFFUSION combines AR-like generation behavior with efficient parallel decoding.The method uses multi-level diffusion to retain sequential structure while supporting parallel computation.
  • Contribution: Dynamic token movement speeds make left tokens denoise earlier, allowing them to influence later tokens.The sentence-level and token-level components assign fewer denoising steps to tokens on the left than to those on the right.
  • Contribution: A skipping mechanism facilitates parallel generation within the multi-level diffusion framework.The mechanism reduces the number of inference steps used during decoding.
  • Results: Across various tasks, AR-DIFFUSION surpasses existing diffusion models in quality while maintaining diversity.The conclusion reports this as an experimental result across multiple text-generation tasks.
  • Results: AR-DIFFUSION achieves comparable results while being 100×∼600× faster than existing diffusion language models.The reported speed range summarizes the paper’s efficiency comparison.

7 Limitation

The main limitation is that optimal performance requires generating many candidate samples, increasing sampling demands.

  • Limitation: AR-DIFFUSION requires a large number of candidate samples to achieve optimal performance.The authors identify more efficient sampling strategies as future work.
  • Limitation: 0.8 lower ROUGE-2 separates AR-DIFFUSION with k = 50 from AR-DIFFUSION with k = 500 on CNN/DAILYMAIL.The example illustrates the performance cost of reducing the candidate-sample count.

A Proof of Inference with Skipping

The skipping proof derives the reverse inference transition from the diffusion posterior and explains why dynamic token timesteps make earlier positions useful for inferring later ones. This supports accelerated generation through large denoising spans.

  • Skipping Inference: The skipping inference process uses the model gθ to infer a later state from the current state during reverse sampling.The supplied inference description identifies gθ as the model used in the skipped transition.
  • Dynamic Timesteps: When ti+1 ≪ ti, earlier-position tokens with smaller timesteps provide stronger auxiliary information for tokens farther to the right.Dynamic speed assigns positions n1 ≤ n2 different token-level timesteps, making earlier states closer to the target inference state.
  • Acceleration: This auxiliary information reduces the difficulty of inferring end-position states and makes the multi-level diffusion model suitable for acceleration.The proof connects position-sensitive denoising to the practical benefit of faster generation.
  • Diffusion Objective: The training objective maximizes the evidence lower bound of p(z0), equivalently minimizing divergence between the forward and reverse diffusion distributions.The reverse model approximates the posterior transition used in inference.
  • Posterior Derivation: The posterior q(z_ti+1|z_ti, z0) is rewritten by conditioning on the current noisy state and clean representation.This conditional form is then used to derive the reverse transition.
  • Reverse Transition: The derivation applies the same conditional reasoning to obtain the model transition pθ(z_ti+1|z_ti).The estimated clean representation is substituted into the posterior expression to form the parameterized reverse step.

B More Cases

The section presents multiple text passages, including variants about plaques, soldiers, China’s prime minister, and unrelated noisy strings. Several variants preserve partial wording while changing names, places, or surrounding tokens.

  • Other passages consist largely of corrupted or unrelated token sequences, including fragments mentioning tennis, organizational interception, and Barcelona.These strings contain special tokens, multilingual fragments, and incomplete wording.
  • Four passages describe a British soldier killed by an army in Afghanistan or corrupted variants, with wording about being named by a ministry or prosecutor.The clearest version names Afghanistan and the ministry of defence, while related passages contain altered wording.
  • Three related passages begin with China’s prime minister and mention emissions, the country’s crisis, and changing final phrases.The passages vary between “economic crisis,” “foundation,” and other corrupted continuations.
Loading 2305.09515v3…