PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffuSIA: A Spiral Interaction Architecture for Encoder-Decoder Text Diffusion

Chao-Hong Tan, Jia-Chen Gu, Zhen-Hua Ling

arXiv:2305.11517v1cs.CLcs.AI

TL;DR

Conditional text diffusion has limited flexibility in single-encoder designs, and a separated conditional encoder does not perceive target text during diffusion. DiffuSIA uses spiral interactions between conditional and target information, achieving competitive performance across four text-generation tasks, though dialogue generation remains an area for improvement.

  • Problem

    Single-encoder conditional text diffusion offers limited flexibility, while encoder-decoder conditional encoders lack target-text understanding during diffusion.

    Source: §1 · p. 2

  • Method

    DiffuSIA uses multilayer spiral interactions so the diffusion decoder captures conditional information and the conditional encoder captures target information.

    Source: §1 · p. 2

  • Results

    DiffuSIA achieves competitive performance among previous methods on paraphrase, text simplification, question generation, and open-domain dialogue generation.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The results support the effectiveness and generalization of spiral interactions across the four evaluated text-generation tasks.

    Source: §1 · p. 2

    DiffuSIA falls short on dialogue response generation and incurs longer training times than pretrained-finetune workflows, which often need only 3–5 epochs.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Diffusion models have emerged as the new state-of-the-art family of deep generative models, and their promising potentials for text generation have recently attracted increasing attention. Existing studies mostly adopt a single encoder architecture with partially noising processes for conditional text generation, but its degree of flexibility for conditional modeling is limited. In fact, the encoder-decoder architecture is naturally more flexible for its detachable encoder and decoder modules, which is extensible to multilingual and multimodal generation tasks for conditions and target texts. However, the encoding process of conditional texts lacks the understanding of target texts. To this end, a spiral interaction architecture for encoder-decoder text diffusion (DiffuSIA) is proposed. Concretely, the conditional information from encoder is designed to be captured by the diffusion decoder, while the target information from decoder is designed to be captured by the conditional encoder. These two types of information flow run through multilayer interaction spirally for deep fusion and understanding. DiffuSIA is evaluated on four text generation tasks, including paraphrase, text simplification, question generation, and open-domain dialogue generation. Experimental results show that DiffuSIA achieves competitive performance among previous methods on all four tasks, demonstrating the effectiveness and generalization ability of the proposed method.

1 Introduction

DiffuSIA addresses limited flexibility in single-encoder conditional text diffusion and the inability of a separated conditional encoder to perceive target text. It uses spiral cross-attention interactions and achieves competitive results across four text-generation tasks.

1 Introduction

  • Text diffusion faces a challenge because natural language is discrete while diffusion processes typically operate in continuous space.Source: §1 · p. 1
  • A single-encoder design limits flexible conditional modeling, while a separate conditional encoder cannot perceive target text during diffusion.The authors note this target-awareness issue had not been studied in previous work.Source: §1 · p. 2
  • DiffuSIA spirally links target-aware conditional and condition-aware target information through two cross-attentions across multiple layers.The encoder first attends to target text; its target-aware conditional representations then interact with the decoder.Source: §1 · p. 2
  • Because diffusion is non-autoregressive, target information can assist conditional-text understanding without information leakage.Source: §1 · p. 2
  • DiffuSIA achieves competitive performance across paraphrase, text simplification, question generation, and open-domain dialogue generation.The authors attribute this to the effectiveness of spiral interactions and report generalization across the four tasks.Source: §1 · p. 2

2 Related Work

Prior text-diffusion work includes discrete diffusion and continuous diffusion on discrete data; this paper studies the latter. DiffuSIA addresses target-blind conditional encoding in encoder-decoder text diffusion through stronger interactions between conditions and targets.

2 Related Work

  • Text diffusion research includes discrete diffusion on discrete data and continuous diffusion on discrete data; this paper studies the latter.Source: §2 · p. 2
  • Diffusion-LM maps discrete tokens into continuous latent variables with embedding and rounding steps, enabling controllable text generation.The method learns the embedding as part of its training objective.Source: §2 · pp. 2–3
  • DiffuSeq uses partially noised input with one Transformer encoder, while SeqDiffuSeq extends continuous diffusion to sequence-to-sequence generation with an encoder-decoder.SeqDiffuSeq also adopts self-conditioned denoising and a token-level adaptive noise schedule.Source: §2 · p. 3
  • DiffuSIA targets the unaddressed problem of conditional text not perceiving target text in encoder-decoder diffusion and strengthens their interactions.The paper also analyzes encoder-decoder defects and the effects of different encoder and decoder layer counts.Source: §2 · p. 3

3 Preliminaries

The preliminaries describe diffusion as noise addition followed by denoising, then outline continuous diffusion on text embeddings and classifier-free guidance. For conditional sequence generation, the source information remains present throughout the training objective.

3 Preliminaries

  • The forward diffusion process adds Gaussian noise through Markov transitions, and the final state is approximately standard Gaussian.The reverse process learns transitions that reconstruct the original data from the noisy state.Source: §3 · p. 3
  • Continuous text diffusion maps discrete words to noisy embeddings and adds a trainable rounding step to recover token distributions.Diffusion-LM uses this embedding-space formulation.Source: §3 · p. 3
  • Classifier-free guidance trains conditional and unconditional generation in one network by randomly discarding conditioning information during training.Source: §3 · p. 4
  • For sequence-to-sequence generation, the target sequence is conditioned on the source, and conditional information is retained throughout the process.Source: §3 · p. 4

4 Approach

DiffuSIA combines encoder-decoder text diffusion with spiral cross-attention between conditional and target representations. It also uses self-conditioning to incorporate previous sampling predictions.

4.1 Encoder-Decoder Diffusion

  • The conditional encoder maps conditional tokens to hidden states, while the target decoder maps target tokens to continuous representations and uses cross-attention to incorporate conditional representations.Because diffusion is non-autoregressive, the decoder does not require a causal mask; encoding the condition once is computation-efficient, but its representation lacks target information and is degraded relative to full self-attention in DiffuSeq.Source: §4.1 · p. 4

4.2 Spiral Interaction Architecture

  • CACE layers attend to target information to form target-aware conditional representations, which then interact with decoder layers to form condition-aware target representations.The encoder and decoder layers are split and interleaved to create these spiral interactions.Source: §4.2 · p. 5
  • When encoder and decoder depths match, their layers are interleaved; with unequal depths, interleaving is limited to the shared range and remaining layers run individually.For Le < Ld, remaining decoder layers run individually; for Le > Ld, initial encoder layers run individually before interleaving.Source: §4.2 · p. 5

4.3 Self-Conditioning

  • Self-conditioning addresses discarded previous-step predictions by feeding a prior estimate into the denoising process, allowing the model to refine predictions rather than rebuild them from scratch.Following Chen et al. (2022), training uses self-conditioning with 50% probability and adds less than 25% training time.Source: §4.3 · p. 5
  • DiffuSIA combines the previous estimate with the current noisy target representation by concatenation and linear projection or by direct addition, and the concatenation approach is reported as more powerful.At inference, the model estimates x0 based on Eq. (11).Source: §4.3 · p. 5

5 Experiments

Across four test sets, DiffuSIA was competitive on Wiki-Auto and QQP, outperformed conventional baselines on Quasar-T, but lagged baselines on CCD. Ablations supported the spiral interactions and self-conditioning, while analysis found different layer-depth trends for DiffuSIA and DiffuED.

5.1 Datasets

  • Four sequence-to-sequence tasks were tested against autoregressive, partially autoregressive, and diffusion baselines using BLEU, ROUGE, and BERTScore.The six-layer encoder-decoder used 768-dimensional encoder and 128-dimensional decoder embeddings; experiments used 2,000 diffusion steps and MBR decoding with 10 candidates.Source: §5.1 · p. 6, §5.2 · p. 6, §5.4 · p. 6

5.5 Evaluation Results

  • Across four test sets, DiffuSIA was competitive on Wiki-Auto and QQP, beat conventional baselines on Quasar-T, but lagged baselines on CCD.The authors note that dialogue requires deeper language understanding and reasoning than the other tasks.Source: Table 2 · p. 7, §5.5 · p. 7
  • 0.81% BLEU, 1.14% ROUGEL, and 0.24% BERTScore: DiffuSIA exceeded the best baseline by these margins on Wiki-Auto.On QQP, it exceeded the best baseline by 0.86% BLEU and 0.69% ROUGEL, but trailed by 1.04% BERTScore.Source: §5.5 · p. 7

5.6 Ablation Study

  • On QQP, removing SI degraded all metrics, SC removal lowered both models’ performance, and concatenated SC outperformed direct-input SC for DiffuSIA.Direct-input SC caused no performance degradation for DiffuED.Source: Table 3 · p. 7, §5.6 · p. 7

5.7 Analysis

  • On QQP, DiffuSIA peaked at Ld = 6 and then declined as decoder layers increased, unlike DiffuED’s consistent improvement.The authors conclude that DiffuSIA performed best with a symmetrical encoder-decoder structure.Source: Figure 3 · p. 8, §5.7 · p. 8
  • Pretrained encoders played a negative role in DiffuED experiments, which the authors attribute to the gap between conditional encoding and diffusion decoding.They identify more effective use of pretrained language models as a direction for improvement.Source: §5.7 · p. 8
  • Generated samples generally followed conditions and varied across random seeds, but Quasar-T repeated targets and one CCD reply’s “no problem” was unsuitable.The case study used four randomly selected samples from each dataset.Source: Table 4 · p. 8, §5.7 · p. 8

6 Conclusion

DiffuSIA combines detachable encoder-decoder modules with spiral interaction that uses target information to improve understanding of conditional text. It achieves competitive performance across four text-generation tasks, though dialogue generation remains an area for improvement.

6 Conclusion

  • The encoder-decoder architecture's detachable modules offer greater flexibility and are described as extensible to multilingual and multimodal generation for conditions and targets.Source: §6 · p. 8
  • DiffuSIA's spiral interaction leverages target information to improve the understanding of conditional text.Source: §6 · pp. 8–9
  • DiffuSIA achieves competitive performance among previous methods on four tasks, while dialogue generation still has room for improvement.The evaluated tasks are paraphrase, text simplification, question generation, and open-domain dialogue generation.Source: §6 · p. 9

Limitations

The model falls short on tasks requiring stronger natural language understanding, such as dialogue response generation, and incurs longer training times than pretrained-finetune workflows.

Limitations

  • The model falls short on tasks demanding higher natural language understanding, such as dialogue response generation.Improving natural language understanding is identified as future research.Source: Limitations · p. 9
  • The model incurs longer training times for improved performance than pretrained-finetune workflows, which often achieve better downstream-task results in 3–5 epochs.Source: Limitations · p. 9
  • Effectively using pretrained language models is identified as a future research direction.Source: Limitations · p. 9

Related papers

Loading 2305.11517v1…