PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Diffusion for Language Generation

Justin Lovelace, Varsha Kishore, Chao Wan, Eliot Shekhtman, Kilian Q. Weinberger

arXiv:2212.09462v2cs.CLcs.LG

TL;DR

Language diffusion is less established than diffusion for continuous modalities because discrete text is difficult to model directly. LD4LG learns a compact latent space with pretrained encoder-decoder autoencoders and applies continuous diffusion there, producing strong results across generation settings while outperforming recent diffusion language models.

  • Problem

    Diffusion models have seen limited use in language because discrete states are less naturally connected to Gaussian noise, motivating methods that combine diffusion with pretrained language models.

  • Method

    LD4LG compresses pretrained encoder-decoder representations into compact fixed-size latents, learns continuous diffusion over those latents, and decodes generated latents into text.

  • Results

    LD4LG is effective for unconditional, class-conditional, and sequence-to-sequence generation and significantly outperforms recent diffusion language models across diverse datasets.

  • Takeaways & Limitations

    Latent diffusion provides an effective paradigm for language generation and offers some benefits over fine-tuned autoregressive language models.

  • Takeaways & Limitations

    LD4LG remains slow at inference because diffusion requires an iterative generative process, and faster sampling is left for future work.

Abstract

from arXiv · show

Diffusion models have achieved great success in modeling continuous data modalities such as images, audio, and video, but have seen limited use in discrete domains such as language. Recent attempts to adapt diffusion to language have presented diffusion as an alternative to existing pretrained language models. We view diffusion and existing language models as complementary. We demonstrate that encoder-decoder language models can be utilized to efficiently learn high-quality language autoencoders. We then demonstrate that continuous diffusion models can be learned in the latent space of the language autoencoder, enabling us to sample continuous latent representations that can be decoded into natural language with the pretrained decoder. We validate the effectiveness of our approach for unconditional, class-conditional, and sequence-to-sequence language generation. We demonstrate across multiple diverse data sets that our latent language diffusion models are significantly more effective than previous diffusion language models.

1 Introduction

Diffusion models are highly successful for continuous data but less natural for discrete language. LD4LG addresses this by combining pretrained encoder-decoder language models with latent continuous diffusion, achieving strong results across multiple generation settings.

  • Motivation: Diffusion models have seen limited use in language because transitioning discrete states to and from Gaussian noise is less natural than in continuous domains.Prior approaches using continuous word embeddings require careful regularization to avoid representation collapse.
  • Motivation: Latent diffusion shifts text generation into the continuous latent space of a pretrained autoencoder, leaving discrete distribution modeling to the autoencoder.This design lets diffusion focus on high-level semantic representations rather than directly modeling discrete text.
  • Approach: LD4LG uses a pretrained encoder-decoder network plus compression and reconstruction modules to create a compact, fixed-length language latent space.The compression module reduces high-dimensional, input-length-dependent encoder features, while reconstruction features guide the decoder through cross-attention.
  • Results: LD4LG supports unconditional, class-conditional, and sequence-to-sequence generation and significantly outperforms recent diffusion language models with fewer sampling steps.On ROCStories, LD4LG reaches MAUVE .716 with 250 steps versus Diffusion-LM’s .043 with 2000 steps; on XSum, it reaches ROUGE-L 31.9 versus DiffuSeq’s 14.1 with the same respective step counts.

2 Background

Diffusion models learn to transform Gaussian noise into samples from a data distribution through iterative noising and denoising. Training teaches a network to recover clean data from noisy inputs across randomly sampled timesteps.

  • Diffusion models: Diffusion models iteratively transform analytically sampled Gaussian noise into samples from an unknown data distribution.They define a forward process that adds noise and a generative process that reverses it.
  • Training: The denoising network is trained with a regression objective to recover clean data from a noisy latent at a sampled timestep.The noisy latent combines scaled clean data with Gaussian noise, using a time-dependent noise schedule and weighting term.
  • Sampling: Sampling starts from pure Gaussian noise and repeatedly denoises through decreasing noise levels until the final latent approximates the data distribution.The sampling sequence uses timesteps ordered from 1 toward 0.

3 Latent Diffusion For Language

LD4LG augments pretrained encoder-decoder language models with compression and reconstruction networks, then learns continuous diffusion in their compact latent space. The resulting latents can be reconstructed into natural language while accommodating fixed-length continuous diffusion and variable-length text.

  • Framework: LD4LG combines a pretrained encoder-decoder language model with two learnable networks and a continuous diffusion model over its latent distribution.The autoencoder creates a compact latent space, and diffusion generates samples that the decoder can convert into natural language.
  • Language Autoencoder: The language encoder maps variable-length token sequences to same-length continuous features, while the decoder cross-attends to features to generate text autoregressively.Pretrained models are frozen by default, and the autoencoding modules are trained to reconstruct the input with cross-entropy loss.
  • Compression Network: The Compression Network uses Perceiver Resampler attention to map variable-length encoder features into a fixed-length latent sequence.A linear projection further reduces the feature dimension, producing a latent space with length ℓ < L and dimension d_ae < d_LM.
  • Reconstruction Network: The Reconstruction Network projects compressed latents back to the decoder’s feature dimension, adds position embeddings, and processes them with a transformer.These reconstructed features guide the pretrained language decoder during generation.
  • Latent Diffusion: Diffusion training operates on autoencoder latents with a denoising regression objective, using v-parameterization and a cosine noise schedule.Generation begins from Gaussian latent noise, iteratively denoises it, and decodes the resulting latent through the reconstruction network and language decoder.
  • Conditioning: Self-conditioning supplies the denoising network with a previous data estimate, while class conditioning can be dropped during training to preserve unconditional generation.The class-conditioning procedure replaces labels with a null label with probability 0.1.

4 Datasets

LD4LG is evaluated on diverse datasets spanning stories, news, summarization, question pairs, and machine translation. The evaluation uses distributional, likelihood, diversity, memorization, and sequence-to-sequence similarity metrics.

  • Evaluation Metrics: MAUVE and perplexity measure generated-text distributional quality and likelihood under GPT-2-Large.MAUVE compares generated and reference distributions using divergence frontiers, while perplexity measures autoregressive-model likelihood.
  • Evaluation Metrics: Diversity is measured with distinct generated 4-grams, while memorization is the proportion of generated 4-grams found in the training set.These metrics distinguish varied generations from generations that reproduce training data.
  • Evaluation Metrics: ROUGE and BERTScore evaluate monolingual sequence-to-sequence generation by measuring reference overlap and contextual similarity.ROUGE includes unigram, bigram, and longest-common-sequence variants, while BERTScore uses pretrained contextual embeddings.
  • Evaluation Protocol: Unconditional and class-conditional experiments sample 1000 instances, repeat evaluation five times, and report means and standard deviations.Reference metric values are also computed from natural test-set samples.

5 Experiments

Experiments evaluate language autoencoders and LD4LG across unconditional, class-conditional, sequence-to-sequence, and multilingual generation. The results show strong reconstruction, improved efficiency from compression, and competitive or superior generation relative to diffusion and fine-tuning baselines.

  • 5.1 Language Autoencoder: BART-base compresses feature space by 24× while improving reconstruction fidelity, and both BART and FLAN-T5 achieve near-perfect reconstruction with low-dimensional latents.
  • 5.2 Unconditional Language Generation: Compact latent spaces improve absolute performance and reach BART-Diffusion’s peak MAUVE in one quarter of the time.Sequence compression also reduces self-attention overhead and speeds convergence.
  • 5.2 Unconditional Language Generation: Self-conditioning improves MAUVE and generated-text perplexity but reduces diversity.
  • 5.3 Class-Conditional Language Generation: Class labels guide generation effectively, with LD4LG more consistently effective for similar AG News classes such as business and sci/tech.
  • 5.4 Sequence-to-Sequence Language Generation: For sequence-to-sequence tasks, LD4LG outperforms recent diffusion models, is competitive with fine-tuning, and achieves the strongest results with oracle sampling across QQP and XSum.On machine translation, LD4LG surpasses Diffusion-LM and CDCD but trails DINOISER.

6 Future Work

Latent language diffusion models generate high-quality natural language across settings and may support applications such as language editing and controllable generation. However, diffusion sampling remains slow, and improved sampling or candidate-ranking methods are future directions.

  • LD4LG generates high-quality natural language across a variety of settings.
  • The authors identify language editing and controllable generation, including style transfer and toxicity mitigation, as potential applications.
  • LD4LG uses 250 sampling steps, improving on some prior continuous text diffusion models but remaining slower than desired.
  • Future work includes adapting faster image-diffusion techniques and improving candidate sampling or reranking for summarization and translation.

7 Related Work

Related work established diffusion models for continuous data and explored adaptations to language, including discrete-state processes and diffusion in token or pretrained embedding spaces.

  • Latent diffusion models learn image-generation distributions in the latent space of a pretrained autoencoder.
  • Prior language work modeled discrete state spaces or applied continuous diffusion to token and pretrained word embeddings.

8 Conclusion

The paper concludes that latent diffusion is effective for language generation by compressing language representations into compact fixed-size latents and applying continuous diffusion. The resulting models support multiple generation settings and outperform recent diffusion language models across datasets.

  • Latent diffusion compresses high-dimensional, variable-length language representations into compact fixed-size latents suitable for continuous diffusion.
  • The resulting latent language diffusion models support unconditional, class-conditional, and sequence-to-sequence generation and significantly outperform recent diffusion language models across datasets.
  • The forward diffusion process progressively adds Gaussian noise until the final latent is approximately independent Gaussian noise.
  • During training, a neural network estimates the original data from a noisy latent and timestep because the original data is unavailable during generation.
  • The denoising objective uses time-dependent weighting, and this work adopts the v-parameterization.
  • Generation starts from Gaussian noise and iteratively applies the learned generative process to produce samples.

B Additional Language Autoencoder Results

Additional experiments show that the language autoencoders are effective across XSum, QQP, and WMT14, while latent dimensionality and sampling steps affect reconstruction, quality, diversity, and runtime.

  • The proposed language autoencoders are similarly effective on XSum, QQP, and WMT14.
  • Reconstruction improves as latent dimensionality increases and degrades as the latent representation shrinks.
  • The default 32 × 64 latent dimensionality is generally effective for high-quality reconstructions across datasets.
  • Sampling configurations are evaluated across different timestep counts, with 250 steps used by default.
  • For BART-base, MAUVE is maximized at 100-250 steps, balancing text diversity and fluency.
  • Sampling time scales with the number of sampling steps, while reconstruction and autoregressive decoding add fixed costs.

D GPT-2 Sampling Ablation

The section reports sampling and implementation choices for LD4LG and its baselines, including nucleus-sampling sensitivity, latent normalization, noise schedules, and decoding procedures.

  • Sampling ablation: Memorization remains consistently higher than LD4LG across all tested nucleus-sampling settings, despite some sensitivity to p.The ablation varies the nucleus sampling parameter p.
  • Noise schedule: Machine translation uses a shifted cosine noise schedule with s = 0.1 to emphasize training at higher noise levels.Higher-noise training was reported to improve use of source conditioning in prior text-diffusion work.
  • Latent-space handling: Intermediate latent estimates are rescaled to enforce the latent norm constraint, improving robustness to large guidance weights without materially changing performance.The authors describe rescaling as non-critical because similar performance is obtained without it.
  • Decoding: Decoding uses beam search with beam size 4, repetition penalty 1.2, and duplicate-trigram prevention.These settings are applied when decoding sampled latent vectors.
  • Latent diffusion parameterization: The LD4LG baseline normalizes latent features to unit variance, while x-prediction is selected after v-prediction proved less effective and ϵ-prediction unstable.The x-prediction choice follows prior findings for high-dimensional data.

E.6 DiffuSeq

The DiffuSeq comparison discusses evaluation setup, computational differences, and sequence-length scaling. LD4LG uses fixed-length latent denoising and can incorporate longer source contexts more efficiently than DiffuSeq.

  • Evaluation setup: The reported comparison uses official or public implementations and standard metric implementations, with early stopping based on validation ROUGE-L for sequence-to-sequence models.The authors use the official DiffuSeq implementation for XSum and Hugging Face evaluate implementations for several metrics.
  • Training budget: The XSum DiffuSeq baseline was trained for over 3.8× more epochs than LD4LG.The comparison used 960k DiffuSeq iterations versus 250k LD4LG iterations.
  • Architecture: LD4LG denoises a fixed sequence of ℓ = 32 latents and cross-attends to source representations, whereas DiffuSeq concatenates source and target sequences.This architectural difference determines how each method scales with source and target length.
  • Complexity: LD4LG scales linearly with source-sequence length in its denoising network, enabling more efficient incorporation of long contexts than DiffuSeq.The frozen LD4LG encoder remains quadratic in source length, but its representations can be cached before training.
  • Sequence length: The official DiffuSeq implementation limits combined source-target length to 128 tokens, while LD4LG incorporates up to 256 source tokens.The authors also retrained DiffuSeq with a 256-token maximum, increasing training overhead.

F Dataset Statistics

The evaluation spans unconditional, class-conditional, paraphrase, summarization, and translation datasets with varied sizes and structures. Qualitative examples highlight vocabulary and long-context differences between LD4LG and baselines.

  • Summarization: XSum contains 204,045 training, 11,332 validation, and 11,334 test article-summary pairs.The articles span topics including news, politics, and sports.
  • Paraphrasing: QQP provides 144,715 training, 2,048 validation, and 2,500 test question pairs for paraphrase generation.The dataset labels whether paired questions have the same meaning.
  • Unconditional and class-conditional generation: Diffusion-LM often generates UNK tokens because it cannot model rare words, with the issue especially pronounced for AG News proper nouns.The qualitative samples compare random unconditional and class-conditional generations across ROCStories and AG News.
  • Sequence-to-sequence generation: DiffuSeq produces somewhat reasonable QQP paraphrases but completely fails to produce coherent XSum summaries despite longer training and 8× more sampling timesteps.The qualitative examples are presented in the QQP and XSum sample tables.
Loading 2212.09462v2…