Vector Quantized Diffusion Model for Text-to-Image Synthesis

1. Introduction

• VQ-Diffusion uses a VQ-VAE latent space modeled by a conditional DDPM, reversing a corruption process to denoise image tokens toward the desired distribution.The forward process gradually corrupts the latent variables through a fixed Markov chain until it produces pure noise.Source: §1 · pp. 1–2
• Its bidirectional image decoder uses context from all tokens at each step, eliminating the fixed top-left-to-bottom-right prediction bias of AR methods.An independent text encoder conditions diffusion over discrete image tokens, which start masked or random.Source: §1 · pp. 1–2
• Mask-and-replace diffusion trains on masked and random tokens and resamples all tokens at inference, allowing incorrect predictions to be revised instead of accumulating.Masked tokens focus attention on masked areas, reducing token combinations to examine and accelerating convergence.Source: §1 · p. 2
• Compared with GAN-based methods [67] [70] [73], VQ-Diffusion handles more complex scenes and improves image quality; it is comparable or better than DALL-E [48] and CogView [13] on image types seen in training.DALL-E and CogView have ten times more parameters than VQ-Diffusion.Source: §1 · p. 2
• Experiments also report strong results for unconditional and conditional image generation on FFHQ and ImageNet, showing the method’s broader applicability.The approach is described as general across both generation settings.Source: §1 · p. 2
• VQ-Diffusion is fifteen times faster than AR methods while achieving better image quality, and outperforms similarly sized AR models on CUB-200, Oxford-102, and MSCOCO.The reported AR comparison uses image-quality metrics and visual examination.Source: §1 · p. 2

GAN-based Text-to-image generation

• GAN-based methods produce high-fidelity images on single-domain datasets but struggle with complex, multi-object scenes because of convolutional locality bias.GAN-INT-CLS [50] introduced conditional GANs for text-to-image generation, and subsequent approaches [44] [67] [68] [69] [70] [73] sought to improve quality.Source: §2 · p. 2

Autoregressive Models

• Raw-pixel autoregressive models such as PixelRNN [53], Image Transformer, and ImageGPT are limited to low-resolution images such as 64 × 64 by computation.Other approaches [25] [33] first infer semantic layouts, but require fine-grained labels such as bounding boxes or segmentation maps.Source: §2 · p. 2
• VQ-VAE [41] [49], VQGAN [16], and ImageBART [15] compress images into low-dimensional discrete latent spaces and model their hidden-variable densities, improving image generation.Source: §2 · p. 2
• DALL-E [48], CogView [13], and M6 [35] advance text-to-image quality with large transformers and massive text-image pairs, but retain unidirectional bias and accumulated prediction errors.Source: §2 · pp. 2–3

Denoising Diffusion Probabilistic Models

• D3PMs [1] apply discrete diffusion to raw image pixels but generate only low-resolution images such as 32 × 32; Argmax Flow [26] applies discrete diffusion to text.Earlier diffusion work [59] and later image-generation studies [12] [23] [24] [39] considered continuous diffusion on raw pixels.Source: §2 · p. 3

• Discrete image tokens reduce transformer context length, addressing the quadratic cost that makes direct modeling of raw pixels computationally prohibitive [16] [41].Source: §3 · p. 3
• VQ-VAE encodes images into spatial features, quantizes each feature to its nearest codebook entry, and decodes the resulting tokens for faithful reconstruction [41].The encoded sequence has length h × w, usually much smaller than the pixel dimensions H × W; image synthesis is therefore sampling from the latent distribution.Source: §3 · p. 3
• Exponential moving averages update the codebook entries in place of the second loss term, which is reported to work better than direct loss-based updates [41].Source: §3 · p. 3

4. Vector Quantized Diffusion Model

• Unlike prior autoregressive models such as DALL-E [48] and CogView [13], VQ-Diffusion models VQ-VAE image tokens conditioned on BPE text using discrete diffusion, avoiding raster-order bias, error accumulation, and per-token inference.The framework maximizes the conditional transition distribution q(x|y), where image tokens come from a pretrained VQ-VAE and text tokens use BPE encoding [56].Source: §4 · p. 3

4.1. Discrete diffusion process

• A fixed Markov chain progressively corrupts discrete image tokens, while a text-conditioned transformer learns reverse transitions to restore the image tokens.For a token, the transition probabilities are represented by a matrix Qt; the categorical distribution at the next step is Qtv(xt−1).Source: §4.1 · pp. 3–4
• Mask-and-replace corruption makes altered locations explicit while retaining uniform noise, easing reverse estimation and encouraging the network to use context.The method adds a [MASK] state; ordinary tokens can be masked, uniformly replaced, or left unchanged, and the paper argues uniform noise avoids a trivial posterior and encourages contextual understanding.Source: §4.1 · pp. 4–5

4.2. Learning the reverse process

• The reverse model predicts noiseless token distributions, combining an auxiliary denoising loss with VLB; its encoder-decoder transformer uses text cross-attention and timestep-conditioned AdaLN.Skipping diffusion steps with stride Δt makes sampling more efficient with little harm to quality.Source: §4.2 · pp. 5–6

5. Experiments

• Text-to-image experiments cover CUB-200, Oxford-102, and MSCOCO, with additional large-scale training on Conceptual Captions and LAION-400M.The authors filter Conceptual Captions to a 7M subset and train on cartoon, icon, and human subsets of LAION-400M.Source: §5 · p. 6

5.1. Comparison with state-of-the-art methods

• VQ-Diffusion-F achieves the best results and surpasses prior methods by a large margin, including DALL-E and CogView on MSCOCO despite having ten times fewer parameters.VQ-Diffusion-S performs strongly on CUB-200 and Oxford-102, and VQ-Diffusion-B further improves performance; visual comparisons also show finer details and closer alignment with input text.Source: §5.1 · p. 7, Table 1 · p. 7

5.2. In the wild text-to-image synthesis

• VQ-Diffusion generates in-the-wild images from cartoon, icon, and human LAION-400M subsets, and supports unconditional or text-conditional irregular-mask inpainting without retraining.The LAION subsets contain 0.9M cartoon, 1.3M icon, and 42M human images; the model is also reported to perform strongly despite being smaller than DALL-E and CogView.Source: §5 · p. 6, §5.2 · p. 7

5.3. Ablations

• Training for 100 timesteps improves over 10 and appears saturated at 200, while inference retains good performance after dropping three-quarters of its steps.The authors therefore use 100 training timesteps by default; reducing inference steps may save about three-quarters of inference time.Source: §5.3 · p. 7, Table 2 · p. 8
• The method achieves its best performance at mask rate M = 0.9 and truncation rate r = 0.86.Mask rates above 0.9 may suffer error accumulation, while rates below 0.9 can make it difficult for the network to identify regions needing attention; truncation avoids sampling low-probability tokens.Source: §5.3 · pp. 7–8, Figure 4 · p. 8
• 15 times faster than VQ-AR with a better FID score, VQ-Diffusion retains its advantage on both small and base settings in CUB-200.The throughput comparison used a V100 GPU with batch size 32 and the fast inference strategy.Source: §5.3 · p. 8, Table 3 · p. 8

5.4. Unified generation model

• VQ-Diffusion also works for class-conditional ImageNet and unconditional FFHQ synthesis, providing a unified model even though some task-specialized GANs report better FID scores.For class conditioning, the method removes the text encoder and cross-attention and injects labels through AdaLN; the reported FID uses 50k generated images and all real images.Source: §5.4 · p. 8, Table 4 · p. 8

• VQ-Diffusion models the VQ-VAE latent space non-autoregressively for text-to-image generation.Source: §6 · p. 8
• Its mask-and-replace diffusion strategy avoids the accumulation of errors associated with AR models.Source: §6 · p. 8
• VQ-Diffusion generates more complex scenes than previous GAN-based text-to-image methods and produces strong unconditional and conditional image-generation results.Source: §6 · p. 8

A. Implementation details

• For text-to-image synthesis, the 370M-parameter base decoder has 19 transformer blocks with 1024 channels, while the small decoder has 18 blocks and 34M parameters.Both use a VQGAN-provided VQ-VAE [41] trained on OpenImages, downsampling 256 × 256 images to 32 × 32, and a CLIP [45] ViT-B text encoder producing 77 tokens. For class-conditional ImageNet generation, the VQGAN VQ-VAE downsamples images to 16 × 16, and the decoder uses 24 transformer blocks with 512 channels and convolutional FFNs with channel expansion rate 4.Source: A. Implementation details · p. 12

B. Proof of Equation 8

• The proof uses mathematical induction to establish Equation 8.It first presents the t = 1 case, then proceeds from the assumption that the equation holds at step t.Source: B. Proof of Equation 8 · p. 12
• The displayed t = 1 base case is stated to hold.The proof then assumes Equation 8 holds at step t and considers t + 1.Source: B. Proof of Equation 8 · p. 12

• The section shows VQ-Diffusion results for irregular-mask inpainting in Figure 5.Source: C. Results · p. 12
• Figure 6 shows VQ-Diffusion's in-the-wild text-to-image synthesis results.Source: C. Results · p. 12, Figure 6 · p. 13
• Figure 7 compares VQ-Diffusion results with XMC-GAN results taken from their paper.Source: C. Results · p. 12, Figure 7 · p. 14
• Figure 8 shows VQ-Diffusion results on the FFHQ1024 and FFHQ256 datasets.Source: C. Results · p. 12, Figure 8 · p. 14
• Figure 9 presents VQ-Diffusion results for class-conditional synthesis on ImageNet.Source: C. Results · p. 12, Figure 9 · p. 14