PaperlayerView exact revision on arXiv

Source-linked AI summary

SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers

Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, Songfang Huang

arXiv:2212.10325v5cs.CL

TL;DR

Text diffusion has been less studied because text is discrete, and prior approaches cited here leave sequence-to-sequence generation as a distinct setting. SeqDiffuSeq applies an encoder-decoder Transformer with self-conditioning and token-level adaptive noise schedules. Across five tasks, it is competitive with AR and NAR baselines and improves generation performance and inference speed over DiffuSeq.

  • Problem

    Extending continuous diffusion to discrete text is nontrivial, and the cited text diffusion approaches focus on unconditional or controlled generation rather than sequence-to-sequence tasks.

    Source: §1 · p. 1

  • Method

    SeqDiffuSeq uses an encoder-decoder Transformer for sequence-to-sequence diffusion, combining self-conditioning with a token-level adaptive noise schedule.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    Across five generation tasks, SeqDiffuSeq is competitive with AR and NAR baselines and improves generation performance and inference speed over DiffuSeq.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments report that self-conditioning and adaptive noise schedules benefit sequence-to-sequence generation and complement each other.

    Source: §1 · p. 2

    Quality improvements from self-conditioning and adaptive schedules come at the cost of generation diversity across random seeds, leaving diversity improvement for future research.

    Source: A Limitation · p. 13

Abstract

from arXiv · show

Diffusion model, a new generative modelling paradigm, has achieved great success in image, audio, and video generation. However, considering the discrete categorical nature of text, it is not trivial to extend continuous diffusion models to natural language, and text diffusion models are less studied. Sequence-to-sequence text generation is one of the essential natural language processing topics. In this work, we apply diffusion models to approach sequence-to-sequence text generation, and explore whether the superiority generation performance of diffusion model can transfer to natural language domain. We propose SeqDiffuSeq, a text diffusion model for sequence-to-sequence generation. SeqDiffuSeq uses an encoder-decoder Transformers architecture to model denoising function. In order to improve generation quality, SeqDiffuSeq combines the self-conditioning technique and a newly proposed adaptive noise schedule technique. The adaptive noise schedule has the difficulty of denoising evenly distributed across time steps, and considers exclusive noise schedules for tokens at different positional order. Experiment results illustrate the good performance on sequence-to-sequence generation in terms of text quality and inference time.

1 Introduction

Text diffusion has been less explored, and earlier approaches described here do not address sequence-to-sequence generation. SeqDiffuSeq brings diffusion to this setting with an encoder-decoder Transformer, self-conditioning, and an adaptive noise schedule; experiments report competitive results against AR and NAR baselines and improvements over DiffuSeq.

1 Introduction

  • Diffusion models had shown strong synthesis performance in images, audio, and video, motivating investigation of their transfer to natural-language generation.The introduction contrasts these results with the challenge of applying continuous diffusion to discrete categorical text.Source: §1 · p. 1
  • Earlier text diffusion work described here focused on unconditional or controlled generation, while sequence-to-sequence generation covers tasks such as dialogue and machine translation.DiffuSeq is identified as a diffusion-based method for sequence-to-sequence generation, using encoder-only Transformers.Source: §1 · pp. 1–2
  • Across five generation tasks, SeqDiffuSeq is competitive with AR and NAR baselines and improves generation performance and inference speed over DiffuSeq.The reported ablations also find benefits from self-conditioning and the adaptive noise schedule, which are complementary in sequence-to-sequence settings.Source: §1 · p. 2
  • SeqDiffuSeq extends continuous text diffusion to sequence-to-sequence generation using an encoder-decoder Transformer, self-conditioning, and an adaptive noise schedule.Self-conditioning uses former-iteration information; the adaptive schedule controls noise and recovery at the token level.Source: §1 · p. 2

2 Related Work

Related work covers discrete categorical and continuous-feature approaches to text diffusion, including extensions to sequence-to-sequence generation. SeqDiffuSeq differs from earlier work through its architecture and token-level adaptive noise schedule.

2 Related Work

  • Text diffusion research includes discrete categorical transitions and continuous representations such as binary bits, latent spaces, and word embeddings.DiffusionBERT applies pre-trained models to language modeling, while DiffusionLM uses auxiliary losses for joint embedding and network learning.Source: §2 · p. 2
  • DiffuSeq extends DiffusionLM to sequence-to-sequence generation; SeqDiffuSeq instead changes the architecture and adds self-conditioning and an adaptive noise schedule.The passage also notes work on improving text generation quality after DiffusionLM.Source: §2 · p. 2
  • Unlike spindle and learned schedules for language modeling or machine translation, SeqDiffuSeq balances denoising difficulty across time steps with token-level schedules.Difformer is described as orthogonal to this work.Source: §2 · p. 2

3 Preliminary

The preliminary section describes diffusion as a forward process that adds noise and a learned reverse process that recovers samples. SeqDiffuSeq adopts the DDPM forward and reverse processes.

3 Preliminary

  • In the forward process, samples gradually mix with random noise; in the reverse process, noise is progressively denoised into synthetic samples.The paper states that its model adopts the forward and reverse processes from DDPM (Ho et al., 2020).Source: §3 · pp. 2–3
  • At each forward time step, a sample transitions to a noisier Gaussian distribution, eventually approaching standard Gaussian noise when the number of steps is large enough.The parameter α_t controls the noise added at time step t.Source: §3 · p. 3
  • The reverse process learns a denoising distribution to fit the forward process posterior, then generates a sample step by step from random noise.The denoising distribution is parameterized by θ.Source: §3 · p. 3

4 Approach

SeqDiffuSeq adapts continuous diffusion to sequence-to-sequence generation, denoising noisy output embeddings conditioned on the input. Its design adds former predictions as self-conditioning input and adapts noise schedules across time and token positions.

4.1 Diffusion Model

  • SeqDiffuSeq models noisy output sequences with an encoder-decoder Transformer, encoding the input once and using full decoder attention for sequence-level denoising.The reverse process conditions on the input sequence; the generated embeddings are rounded to words after denoising.Source: §4.1 · pp. 4–5

4.2 Self-Conditioning

  • Self-conditioning feeds former sequence predictions alongside noisier features, allowing the denoising function to refine predictions rather than start from scratch.Training uses an efficient scheme: half the time the former prediction input is zeroed, and otherwise it is estimated without backpropagating through that estimate.Source: §4.2 · p. 5

4.3 Adaptive Noise Schedule

  • The adaptive schedule fits token-position-specific noise schedules to training losses, aiming to make denoising difficulty increase linearly across time steps.The paper assumes different positions may need different schedules because their embedding features differ.Source: §4.3 · p. 5
  • To smooth the fitted mapping despite non-monotonic empirical losses, the method coarsens time-step sampling and periodically recalibrates schedules using evenly spaced values across recorded loss ranges.K is the stride for evenly downsampling time steps; schedule updates repeat as training progresses.Source: §4.3 · p. 6

5 Experiments

SeqDiffuSeq is evaluated across five generation tasks against autoregressive, non-autoregressive, and diffusion baselines. With MBR it surpasses all baselines on three generation tasks, while translation results trail autoregressive Transformers.

5.1 Datasets

  • Experiments cover six datasets across five tasks: paraphrase generation, text simplification, question generation, dialogue generation, and machine translation.The datasets include QQP, Wiki-Auto, Quasar-T, CCD, and German-English pairs from IWSLT14 and WMT14.Source: §5.1 · p. 6

5.2 Baselines

  • The comparisons include encoder-decoder Transformers and GPT-2 as AR baselines, LevT as an NAR baseline, and DiffuSeq as a diffusion baseline.CMLM is included for translation, and CDCD is compared on machine translation.Source: §5.2 · p. 6

5.3 Implementation Details

  • The model uses a six-layer encoder-decoder Transformer, a 2,000-step diffusion process, and a sqrt schedule to initialize its adaptive schedule.Training uses a 10^-4 learning rate, 10,000 warm-up steps, and schedule updates every 20,000 steps; MBR decoding is also explored.Source: §5.3 · pp. 6–7

5.4 Main Results

  • With MBR, SeqDiffuSeq surpasses all baselines on QQP, Wiki-Auto, and Quasar-T, but consistently trails AR Transformers on translation.Against one-iteration CMLM, it gains 6.32 and 6.75 averaged points across four datasets without and with MBR, respectively. Table 1 uses DiffuSeq-reported CCD results because reproducing them required more than 10 days of training on eight NVIDIA A100 80GB GPUs.Source: §5.4 · pp. 7–8, Table 1 · p. 7

6 Analysis and Discussion

Ablations show gains from both self-conditioning and adaptive noise scheduling, with the schedule contributing more in these tests. The learned schedules vary by token position, and SeqDiffuSeq is faster than DiffuSeq.

6.1 Ablation Study

  • Removing the adaptive schedule lowers BLEU by 2.29 on average; removing self-conditioning lowers it by 1.34, while removing both lowers it by 5.71 on average.The largest drop when removing both is 8.43 BLEU on Wiki-Auto; the ablation study covers QQP, Wiki-Auto, and IWSLT14.Source: §6.1 · p. 8

6.2 Time Schedule

  • Adaptive schedules vary by token position and are flatter than the sqrt schedule at time-step extremes; earlier positions have less noise and are recovered earlier.Losses increase linearly over time with the adaptive schedule, and earlier token positions have smaller losses.Source: §6.2 · pp. 8–9

6.3 Inference Speed

  • SeqDiffuSeq generates a batch of text samples 3.56 times faster than DiffuSeq.The encoder is run once rather than at each diffusion step, and each step models only the output sequence rather than concatenated input and output.Source: §6.3 · p. 9

6.4 MBR Inference

  • Self-conditioning and adaptive scheduling trade sequence-level diversity for higher single-sample quality; with both, MBR is unnecessary for high-quality SeqDiffuSeq samples.On QQP, MBR with 10 candidates improves SeqDiffuSeq by 1.06 BLEU, compared with more than 6 BLEU for DiffuSeq.Source: §6.4 · p. 9

7 Conclusion

SeqDiffuSeq uses an encoder-decoder Transformer with self-conditioning and an adaptive noise schedule for sequence-to-sequence generation. The conclusion reports strong generation quality and inference speed, while noting diversity and reverse-step costs as limitations.

7 Conclusion

  • SeqDiffuSeq models denoising with an encoder-decoder Transformer, self-conditioning, and token-specific adaptive noise schedules that distribute denoising difficulty across steps.Self-conditioning helps capture information from earlier generation iterations.Source: §7 · p. 9
  • Experiments report superior generation quality and inference speed for SeqDiffuSeq.The conclusion also says the experiments provide insights into the adaptive noise schedule.Source: §7 · p. 10

A Limitation

  • Thousands of reverse-process iterations remain computationally costly, despite nearly four times less computation per forward pass than DiffuSeq.Reducing reverse steps is identified as a promising direction for future research.Source: A Limitation · p. 13
  • Quality improvements from self-conditioning and adaptive schedules come at the cost of lower diversity across random seeds and only marginal MBR inference improvements.Improving SeqDiffuSeq's generation diversity is left to future research.Source: A Limitation · p. 13

B Ethic Statements and Boarder Impact

  • The authors frame SeqDiffuSeq as evidence of encoder-decoder diffusion's feasibility for sequence-to-sequence tasks and a starting point for further exploration.The datasets and baseline models used are publicly available.Source: B Ethic Statements and Boarder Impact · p. 13

D Derivation of Training Objective

E Datasets

The experiments cover several sequence-to-sequence tasks and describe dataset splits, licenses, and implementation settings. MBR decoding selects an output from sampled candidates by minimizing expected risk.

E Datasets

  • The datasets cover paraphrase generation, text simplification, question generation, dialogue, and machine translation.QQP uses positive pairs for paraphrase generation; IWSLT14 and WMT14 provide German-English translation pairs.Source: E Datasets · pp. 15–16

F.1 Details on Experiment Setting

  • Training uses one run per dataset on NVIDIA A100 GPUs, with batch size 1024 for WMT14 and 128 for other tasks.The maximum training step is 1,000,000; inference uses T = 2000, and the clamping trick is omitted because it does not consistently improve quality across datasets.Source: §F.1 · p. 16

F.2 Details on MBR

  • MBR decoding selects the candidate with minimum expected risk, using negative BLEU as the risk function.Candidates are generated from the diffusion model using different random seeds.Source: §F.2 · p. 16

G Sampling by Prior

The section explores replacing denoising-distribution sampling with prior-distribution sampling to trade off diversity and quality. It also presents adaptive-schedule and loss visualizations across three datasets.

G Sampling by Prior

  • Replacing pθ with the higher-variance prior distribution can increase sequence-level diversity; p1 is the replacement probability, and replacement can be limited to the first p2 percent of reverse-process steps.The variance difference is larger earlier in the reverse process, motivating the exploration of this time-step restriction.Source: G Sampling by Prior · p. 17
  • With replacing probability fixed at 0.05, generation diversity consistently and profoundly improves, while generation quality consistently degrades even at low replacement probability; MBR final sequences may improve with proper p2.With p2 = 0.5, final output sequences are consistently better across different p1 values.Source: G Sampling by Prior · pp. 17–18
  • The section characterizes prior sampling as a diversity-quality trade-off and reports that a suitable p1 and p2 combination can further improve SeqDiffuSeq's MBR generation quality.Source: G Sampling by Prior · p. 18

H More Results on Adaptive Noise Schedule

I Human Evaluation

A blind QQP evaluation compared SeqDiffuSeq and DiffuSeq outputs on 100 test examples. Both annotators preferred SeqDiffuSeq more often.

I Human Evaluation

  • The blind comparison sampled 100 QQP test inputs and asked annotators to choose which output was better, worse, or similar in quality.Source: I Human Evaluation · pp. 18–19
  • Annotators judged outputs by consistency with the input question and grammatical and syntactic quality.The annotators were graduate students proficient in English.Source: I Human Evaluation · p. 19
  • SeqDiffuSeq outputs won 36% and 44% of QQP comparisons for the two annotators, versus 24% and 30% for DiffuSeq.The authors conclude that SeqDiffuSeq generated higher-quality text in this human evaluation.Source: I Human Evaluation · p. 19

J Case Study

The case study describes how SeqDiffuSeq forms output sequences during sampling. It reports early sequence-length determination, left-to-right refinement, and a fixed [SEP] position despite token repetition.

J Case Study

  • The study examines three illustrative cases and says SeqDiffuSeq can generate reasonable sequences; Table 7 truncates samples to 15 tokens and notes that [PAD] tokens generally follow [SEP].Source: J Case Study · p. 19, Table 7 · p. 20
  • SeqDiffuSeq determines output length by generating [SEP] tokens early in sampling.Source: J Case Study · p. 19
  • The generation process appears to refine the sequence from left to right.Source: J Case Study · p. 20
  • The [SEP] position remains fixed during sampling, even when generated sequences contain token repetitions.Source: J Case Study · p. 20
Loading 2212.10325v5…