Source-linked AI summary
Vector Quantized Diffusion Model for Text-to-Image Synthesis
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, Baining Guo
TL;DR
Existing text-to-image methods face unidirectional bias, accumulated prediction errors, and slow resolution-dependent autoregressive inference. VQ-Diffusion models VQ-VAE latents with conditional discrete diffusion and mask-and-replace updates; it improves image quality and, with reparameterization, is fifteen times faster than traditional AR methods.
Problem
Existing autoregressive text-to-image methods impose unidirectional context, accumulate prediction errors, and require inference time that increases with output resolution.
Method
VQ-Diffusion models VQ-VAE latent variables with a conditional discrete diffusion process and uses mask-and-replace updates to resample and correct image tokens.
Results
VQ-Diffusion achieves better text-to-image quality than similarly sized AR models, handles more complex scenes than prior GAN-based methods, and its reparameterized version is fifteen times faster than AR methods.
Takeaways & Limitations
The method provides a quality–speed trade-off while avoiding the unidirectional bias and error accumulation associated with autoregressive generation.
Abstract
from arXiv · showhide
We present the vector quantized diffusion (VQ-Diffusion) model for text-to-image generation. This method is based on a vector quantized variational autoencoder (VQ-VAE) whose latent space is modeled by a conditional variant of the recently developed Denoising Diffusion Probabilistic Model (DDPM). We find that this latent-space method is well-suited for text-to-image generation tasks because it not only eliminates the unidirectional bias with existing methods but also allows us to incorporate a mask-and-replace diffusion strategy to avoid the accumulation of errors, which is a serious problem with existing methods. Our experiments show that the VQ-Diffusion produces significantly better text-to-image generation results when compared with conventional autoregressive (AR) models with similar numbers of parameters. Compared with previous GAN-based text-to-image methods, our VQ-Diffusion can handle more complex scenes and improve the synthesized image quality by a large margin. Finally, we show that the image generation computation in our method can be made highly efficient by reparameterization. With traditional AR methods, the text-to-image generation time increases linearly with the output image resolution and hence is quite time consuming even for normal size images. The VQ-Diffusion allows us to achieve a better trade-off between quality and speed. Our experiments indicate that the VQ-Diffusion model with the reparameterization is fifteen times faster than traditional AR methods while achieving a better image quality.
1. Introduction
Existing text-to-image methods suffer from unidirectional context and accumulated prediction errors. VQ-Diffusion models discrete image latents with conditional diffusion, using mask-and-replace updates and reparameterization to improve quality and speed.
- Motivation: Autoregressive methods impose a fixed top-left-to-bottom-right order, which can ignore important context elsewhere in the image.Their inference errors also propagate because predicted tokens cannot be corrected.
- Method: VQ-Diffusion models a VQ-VAE latent space with a conditional DDPM for text-to-image generation.The model reverses a forward process that progressively corrupts latent variables with noise.
- Method: Mask-and-replace diffusion trains on masked and random tokens, then resamples all tokens so incorrect predictions can be modified.This avoids the error accumulation associated with teacher-forcing-based autoregressive inference.
- Results: Compared with similarly sized AR models, VQ-Diffusion achieves better image quality and is much faster across text-to-image experiments.The experiments cover CUB-200, Oxford-102, and MSCOCO, while comparisons with GAN-based methods show better handling of complex scenes.
- Results: 15 times faster than traditional AR methods, VQ-Diffusion with reparameterization achieves better image quality.Its global-context token prediction is independent of image resolution, unlike traditional AR inference.
2. Related Work
Prior text-to-image work includes GAN-based and autoregressive approaches, while diffusion models provide an alternative generative framework. These methods improve quality but retain important limitations or computational constraints.
- GAN-based Text-to-image generation: GAN-based text-to-image methods have produced promising generation results and have been repeatedly extended to improve quality.Conditional GAN formulations established a major line of text-to-image research.
- GAN-based Text-to-image generation: Some two-step GAN methods infer semantic layouts before generating objects but require fine-grained labels such as bounding boxes or segmentation maps.This adds annotation requirements beyond text-image pairs.
- Autoregressive Models: VQ-VAE, VQGAN, and ImageBART compress images into discrete low-dimensional latent spaces before modeling hidden-variable density.The compression improves image-generation performance.
- Autoregressive Models: DALL-E, CogView, and M6 use autoregressive frameworks to model joint text-image token distributions with large transformers and massive text-image datasets.These approaches substantially advance text-to-image quality.
- Denoising Diffusion Probabilistic Models: Autoregressive text-to-image models still exhibit unidirectional bias and accumulated prediction errors, while prior diffusion work mostly modeled continuous raw pixels.Discrete diffusion for categorical image tokens remained less explored.
3. Background: Learning Discrete Latent Space of Images Via VQ-VAE
VQ-VAE represents images with discrete latent tokens, reducing the sequence length needed for image modeling. An encoder, quantizer, decoder, and codebook are trained to reconstruct images from these tokens.
- Motivation: Transformers are computationally prohibitive on raw pixels because their computation cost is quadratic in sequence length.Discrete image tokens provide a shorter representation for transformer-based image synthesis.
- VQ-VAE representation: VQ-VAE uses an encoder, decoder, and finite codebook to map images into quantized latent representations.The quantizer assigns each spatial feature to its closest codebook entry.
- VQ-VAE representation: The encoded sequence length h × w is usually much smaller than the original image size H × W.The decoder reconstructs the image from the quantized latent representation.
- VQ-VAE representation: Image synthesis becomes sampling discrete image tokens from the latent distribution and decoding them into images.The encoder, decoder, and codebook can be trained end-to-end.
- Training: Exponential moving averages replace the codebook-loss update in practice because they work better than direct optimization through that loss.This is an implementation choice for updating codebook entries.
4. Vector Quantized Diffusion Model
VQ-Diffusion models VQ-VAE’s discrete image-token space with conditional discrete diffusion, replacing autoregressive sampling with non-autoregressive denoising. Its mask-and-replace process addresses error accumulation, while reparameterization accelerates reverse sampling.
- Motivation: Autoregressive image generation is limited by unidirectional context, exposure-bias-driven error accumulation, and one forward pass per predicted token.These limitations motivate a non-autoregressive model for the VQ-VAE latent space.
- Discrete diffusion process: VQ-Diffusion models discrete VQ-VAE image tokens with a conditional diffusion process instead of autoregressive factorization.The forward process corrupts tokens through a fixed Markov chain, and the learned reverse process restores them.
- Mask-and-replace diffusion: Mask-and-replace diffusion exposes corrupted locations and combines masking with random replacement to support reverse estimation and contextual token prediction.The cumulative transition matrix and q(x_t|x_0) can be computed in closed form, reducing their computation from O(tK^2) to O(K).
- Learning the reverse process: The denoising network predicts the noiseless token distribution pθ(x̃_0|x_t,y), which is used to construct reverse transition distributions conditioned on text and timestep.The model uses a text encoder and diffusion image decoder within an encoder-decoder transformer architecture.
- Fast inference strategy: Reparameterized inference skips diffusion steps using a time stride Δt, making sampling more efficient with little harm to quality.Inference samples x_T, x_T−Δt, x_T−2Δt, …, x_0 rather than every adjacent timestep.
5. Experiments
Experiments evaluate VQ-Diffusion across standard and large-scale text-to-image datasets, comparisons with prior methods, ablations, and broader image-synthesis tasks. Results show strong image quality, competitive scaling, efficient inference, and applicability beyond text conditioning.
- Experimental setup: Experiments cover CUB-200, Oxford-102, MSCOCO, Conceptual Captions, and LAION-400M, with additional class-conditional and unconditional synthesis tests.The standard datasets support text-to-image evaluation, while larger datasets assess scalability and ImageNet and FFHQ assess broader generation settings.
- Comparison with state-of-the-art methods: VQ-Diffusion-F achieves the best reported results and surpasses previous methods by a large margin, including DALL-E and CogView on MSCOCO despite having fewer parameters.VQ-Diffusion-S also performs strongly on CUB-200 and Oxford-102, while the base model further improves performance.
- Comparison with state-of-the-art methods: Visual comparisons show more realistic fine-grained details and better consistency with input text than DM-GAN and DF-GAN.
- Unified generation model: The model also supports global image generation and irregular mask inpainting without retraining, including unconditional and text-conditional variants.Images can be generated for broader in-the-wild settings and other synthesis tasks, including class-conditional and unconditional generation.
- Ablations: 15 times faster than VQ-AR with a better FID score, VQ-Diffusion achieves improved speed-quality trade-offs through fast inference.The comparison uses matched image and text encoders and evaluates throughput on a V100 GPU with batch size 32.
6. Conclusion
The conclusion presents VQ-Diffusion as a non-autoregressive text-to-image architecture with broad image-generation capabilities.
- VQ-Diffusion models the VQ-VAE latent space in a non-autoregressive manner.
- Its mask-and-replace diffusion strategy avoids the accumulated errors associated with autoregressive models.
- The model can generate more complex scenes than previous GAN-based text-to-image methods.
- The method also produces strong results for unconditional and conditional image generation.
A. Implementation details
The implementation uses VQGAN-derived VQ-VAE models for text-to-image and ImageNet generation, with different latent resolutions and transformer configurations.
- Text-to-image experiments use a VQ-VAE trained on OpenImages that downsamples 256 × 256 images to 32 × 32.
- The text encoder is pretrained CLIP ViT-B, which encodes each sentence into 77 tokens.
- The text-to-image diffusion decoder uses transformer blocks containing full attention, cross attention, and feed forward networks.
- The ImageNet model uses a VQ-VAE with 16 × 16 latent resolution and 24 transformer blocks.
- Figure 5 presents text-guided image editing by VQ-Diffusion.
B. Proof of Equation 8
The proof of Equation 8 uses mathematical induction, establishing the base case and then assuming the equation at one step before advancing to the next.
- Mathematical induction is used to prove Equation 8.
- At t = 1, the stated cases for x = x0, x ≠ x0, and x = K + 1 are asserted to hold.
- The induction step assumes Equation 8 holds at step t before considering t = t + 1.
C. Results
The results section presents visualizations of VQ-Diffusion across text-to-image synthesis, editing, comparisons, and conditional image-generation datasets.
- The paper compares VQ-Diffusion results with XMC-GAN and presents the comparison in Figure 7.
- Figure 5 shows irregular-mask inpainting results, while Figure 6 shows in-the-wild text-to-image results.
- Figures 8 and 9 report VQ-Diffusion results on FFHQ and class-conditional ImageNet synthesis, respectively.
- The visual examples include portraits, kitchens, landscapes, objects, people, and scenes described by text prompts.
- Additional examples include cartoon characters, movie posters, motorcycles, and unusual multi-object prompts.