PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Text-Conditional Image Generation with CLIP Latents

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark Chen

arXiv:2204.06125v1cs.CV

TL;DR

Text-to-image generation needs to exploit CLIP’s semantic and stylistic image representations while supporting diverse outputs and related image manipulations. The paper combines a caption-conditioned prior with a CLIP-image-embedding decoder, finding samples comparable in quality to GLIDE but more diverse and diffusion priors more compute-efficient than autoregressive priors.

  • Problem

    The paper addresses text-conditional image generation and the challenge of leveraging CLIP image embeddings for generation and image manipulation.

  • Method

    The model uses a prior to generate CLIP image embeddings from captions and a decoder to generate images conditioned on those embeddings.

  • Results

    The unCLIP stack produces samples comparable in quality to GLIDE with greater diversity, while diffusion priors match autoregressive priors with greater compute efficiency.

  • Takeaways & Limitations

    CLIP’s joint embedding space supports image variations, interpolations, and zero-shot language-guided semantic manipulations.

  • Takeaways & Limitations

    unCLIP struggles more than GLIDE to bind attributes to objects and to produce coherent rendered text.

Abstract

from arXiv · show

Contrastive models like CLIP have been shown to learn robust representations of images that capture both semantics and style. To leverage these representations for image generation, we propose a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding. We show that explicitly generating image representations improves image diversity with minimal loss in photorealism and caption similarity. Our decoders conditioned on image representations can also produce variations of an image that preserve both its semantics and style, while varying the non-essential details absent from the image representation. Moreover, the joint embedding space of CLIP enables language-guided image manipulations in a zero-shot fashion. We use diffusion models for the decoder and experiment with both autoregressive and diffusion models for the prior, finding that the latter are computationally more efficient and produce higher-quality samples.

1 Introduction

The work combines a CLIP-image-embedding diffusion decoder with a text-conditioned prior to build a full text-to-image generative model. This framework supports diverse image generation, image variations, and analysis of which features CLIP recognizes or disregards.

  • CLIP image decoder: A diffusion decoder approximately inverts CLIP image embeddings non-deterministically, producing multiple semantically similar images from one embedding.The encoder-decoder pair also enables image encoding and decoding capabilities beyond text-to-image translation.
  • Image analysis and variation: Encoding and decoding images provides a tool for observing which image features CLIP recognizes or disregards.The non-deterministic decoder also produces image variations corresponding to a given image embedding.
  • Text-to-image generation: A prior generates possible CLIP image embeddings from text captions, which then condition the diffusion decoder to produce final images.The prior can be autoregressive or diffusion-based, while the CLIP model remains the representation basis for the generation process.
  • Generation quality and diversity: The system produces samples comparable in quality to GLIDE but with greater diversity in its generations.The comparison includes other systems such as DALL-E and GLIDE.

2 Method

The method stacks a caption-conditioned prior that generates CLIP image embeddings with a diffusion decoder that generates images from those embeddings. It supports autoregressive and diffusion priors, classifier-free guidance, and multistage diffusion upsampling for high-resolution synthesis.

  • 2 Method: The generative stack models captions-to-images by sampling a CLIP image embedding with a prior and then sampling an image with a decoder.The prior is P(zi|y), while the decoder is P(x|zi, y) and may optionally use the caption directly.
  • Decoder: The decoder uses diffusion models conditioned on CLIP image embeddings, projecting them into timestep embeddings and four additional context tokens while retaining GLIDE text conditioning.This architecture enables image generation from CLIP representations with optional text-caption conditioning.
  • Decoder: Two diffusion upsamplers increase resolution from 64×64 to 256×256 and then to 1024×1024, using Gaussian blur and BSR degradation to corrupt conditioning images during training.The staged upsampling design targets high-resolution image generation and improves robustness to conditioning-image variation.
  • Prior: Two prior classes are explored: an autoregressive model predicting discrete codes for zi and a diffusion model directly modeling the continuous embedding.Both priors are conditioned on the caption, and the diffusion prior uses a causal Transformer sequence containing text, CLIP text embeddings, timestep information, and noised image embeddings.
  • Prior: 319 principal components out of 1,024 preserve nearly all CLIP image-representation information for the autoregressive prior while reducing its dimensionality.The retained representation has less than 1% average mean-squared reconstruction error, and PCA yields a threefold reduction in predicted inference tokens.

3 Image Manipulations

The model supports image variations, image blending, and language-guided text-diff manipulations through CLIP-conditioned decoding. Its bipartite latent representation preserves CLIP-recognized content while retaining residual information needed for accurate reconstruction.

  • Image variations: A bipartite latent representation (z_i, x_T) separates CLIP-recognized image aspects from residual information needed for accurate reconstruction.z_i comes from the CLIP image encoder, while x_T is obtained through DDIM inversion.
  • Image variations: DDIM sampling with η > 0 produces images sharing essential content while varying details such as shape and orientation.Setting η = 0 deterministically reconstructs the input, whereas larger η values introduce stochasticity and perceptual variation.
  • Image blending: Spherical interpolation between two CLIP image embeddings generates intermediate representations that traverse concepts between the source images.Intermediate DDIM latents can be produced along the interpolation trajectory using the images’ inverted representations.
  • Language-guided manipulation: CLIP’s shared image-text latent space enables text diffs by interpolating between an image embedding and a normalized difference of two text embeddings.The target description is compared with a caption describing the current image, and outputs decode interpolated representations while fixing DDIM noise x_T.

4 Probing the CLIP Latent Space

The decoder enables direct visualization of what CLIP’s image encoder represents, including semantic errors caused by typographic attacks. PCA reconstructions further reveal that different latent dimensions encode coarse semantics and finer object details.

  • Decoder visualizations: The decoder makes CLIP image embeddings directly visualizable, enabling inspection of the image encoder’s representations.The paper presents this as a tool for probing CLIP’s latent space.
  • Decoder visualizations: In typographic attacks, overlaid text can cause CLIP to predict the text-described object instead of the object depicted in the image.These adversarial examples illustrate how CLIP can rely on overlaid text when making incorrect predictions.
  • PCA reconstructions: Higher PCA dimensions preserve finer-grained details about objects’ exact forms, whereas lower dimensions preserve coarse-grained semantic information.Figure 7 shows reconstructions at 20, 30, 40, 80, 120, 160, 200, and 320 dimensions.
  • PCA reconstructions: PCA reconstructions visualize CLIP embeddings across progressively more dimensions, exposing the semantic information encoded by different latent dimensions.The reconstructions use the decoder with DDIM on a fixed seed.

5 Text-to-Image Generation

The section shows that unCLIP’s CLIP-image-embedding pathway improves generation quality and diversity, with diffusion priors outperforming autoregressive priors. Against GLIDE, unCLIP achieves comparable photorealism while preserving greater diversity and reaches a zero-shot MS-COCO FID of 10.39.

  • 5.1 Conditioning Signals: The unCLIP stack achieves FIDs of 7.99, compared with 9.16 for the text-embedding decoder and 16.55 for zero-shot text-to-unCLIP decoding.These results suggest that conditioning through a generated CLIP image embedding is the strongest of the compared approaches.
  • 5.1 Conditioning Signals: Diffusion priors outperform autoregressive priors across comparable experiments while requiring less training compute.This finding holds in Sections 5.2, 5.4, and 5.5.
  • 5.2 Comparison to GLIDE: Humans slightly prefer GLIDE for photorealism, but strongly prefer unCLIP for diversity despite similar photorealism.The evaluations compare photorealism, caption similarity, and diversity using pairwise human judgments, including 4 × 4 sample grids for diversity.
  • 5.3 Guidance and Diversity: Increasing guidance improves unCLIP sample quality without collapsing scene semantics, because semantic information is fixed in the CLIP image embedding.For GLIDE, increasing guidance causes camera angle, color, and size semantics to converge.
  • 5.3 Guidance and Diversity: At guidance scale 2.0, GLIDE approaches unCLIP in photorealism and caption similarity but remains less diverse, while guidance harms unCLIP’s FID much less.The FID comparison fixes the unCLIP prior guidance and varies only decoder guidance.
  • 5.4 MS-COCO Evaluation: UnCLIP achieves a zero-shot MS-COCO FID of 10.39 with the diffusion prior, establishing a new state of the art among the compared zero-shot models.UnCLIP generalizes to the MS-COCO validation set without direct training on its training set.
  • 5.5 Aesthetic Quality: Decoder guidance improves unCLIP’s aesthetic quality without decreasing Recall5, whereas guiding its prior hurts results.The evaluation highlights a fidelity–diversity tradeoff that unCLIP guidance avoids according to Recall5.

6 Related Work

Prior work applies GANs, VQ-VAE-style autoregressive models, and hierarchical latent processes to text-conditional or high-quality image generation. Other approaches use diffusion or CLIP embeddings to guide or condition generation, including two-stage methods that sample representations before decoding images.

  • Text-Conditional Image Generation: GANs and VQ-VAE-derived autoregressive transformers have been applied to text-conditional image generation using image-captioning datasets and sequences of text tokens followed by image tokens.These approaches extend established unconditional image-generation techniques to text conditioning.
  • Hierarchical Generation: Hierarchical generative models progressively sample coarse-to-fine latent codes or use latent hierarchies whose resolution increases with the generated image.Examples include multi-layer discrete autoencoders and VAEs with progressively higher-resolution latent codes.
  • Latent Diffusion: Diffusion models have been used to model latent spaces in autoencoder, VAE, and VQGAN-based frameworks, motivated in part by computational benefits.These methods use diffusion or score-based models to generate or render latent variables.
  • CLIP-Guided Generation: CLIP has been used to steer generative models toward text prompts through gradients, while classifier-free guidance was reported to work more favorably than CLIP guidance for text-conditional generation.The cited methods include CLIP-gradient guidance for GANs and classifier or classifier-free guidance for diffusion models.
  • CLIP-Conditioned Generation: Several models directly condition image generation on CLIP embeddings, including GANs and diffusion models, while prior two-stage work sampled image representations with Kernel Density Estimation before diffusion decoding.These approaches include conditioning on CLIP image or text embeddings and generating images end-to-end from sampled representations.

7 Limitations and Risks

unCLIP improves diversity but has limitations in attribute-object binding, text rendering, and detail generation in complex scenes. Its performance also raises risks around deceptive content and training-data biases, which depend on deployment context.

  • Model limitations: unCLIP binds attributes to objects less reliably than GLIDE, especially when separate objects must receive separate attributes.Reconstructions can mix up object colors and fail to preserve relative sizes.
  • Model limitations: unCLIP struggles to produce coherent rendered text because CLIP embeddings may not precisely encode spelling information.BPE tokenization can obscure caption spelling, requiring the model to learn tokens independently from training images.
  • Model limitations: The decoder has difficulty generating details in complex scenes, potentially because it starts at a 64 × 64 base resolution before upsampling.A higher base resolution may alleviate this limitation but would require additional training and inference compute.
  • Risks: unCLIP’s performance improvements increase risks that generated images will be mistaken for authentic ones and vice versa.The technology leaves fewer traces and indicators that outputs are AI-generated, while its architectural effects on learned biases need more research.
  • Risks: Model risks should be assessed according to deployment context, including training data, guardrails, deployment space, and access.A preliminary analysis addressed these issues for the DALL·E 2 Preview platform, the first unCLIP deployment.

A Linear Probes for Evaluations

The evaluations use two linear probes built on CLIP ViT-L/14: a regression model for automated aesthetic quality ratings and a logistic model for pairwise image win probabilities.

  • Linear Probes for Evaluations: Two linear probes on CLIP ViT-L/14 support automated aesthetic evaluation and prediction of pairwise image win probabilities.The aesthetic probe is trained by linear regression on AVA images and mean ratings, while the pairwise probe uses logistic regression to reduce hyperparameter-sweep costs before human evaluations.

B Error Bars for Human Evaluation

Human-evaluation error bars use a normal-approximation interval at p = 0.95, justified by the large sample size of n = 1000.

  • Method: Error bars for human evaluations use the normal approximation interval with p = 0.95.This specifies the interval used to quantify uncertainty in human-evaluation results.
  • Method: The evaluation uses a sample size of n = 1000.The authors characterize this sample as large.
  • Method: The authors expect the normal approximation to be accurate for this large sample size.The expected accuracy is tied specifically to n = 1000.

C Training Details

The models use distinct CLIP and generative-stack datasets, with a GLIDE-based decoder, ADMNet upsamplers, and separate autoregressive and diffusion prior configurations. Training incorporates specific encoder, diffusion, and sampling hyperparameters across these components.

  • CLIP encoder: The CLIP model uses a ViT-H/16 image encoder for 256 × 256 images and a causal Transformer text encoder, trained with learning rate 3 × 10−4 and SAM ρ = 0.1.The image encoder has width 1280 and 32 Transformer blocks, while the text encoder has width 1024 and 24 blocks.
  • Datasets: The encoder samples equally from CLIP and DALL-E datasets totaling approximately 650M images, while the decoder, upsamplers, and prior use only approximately 250M DALL-E images.Including the noisier CLIP dataset in generative-stack training negatively affected sample quality in initial evaluations.
  • Decoder: The decoder is a 3.5-billion-parameter GLIDE model using Nichol et al.’s architecture and diffusion hyperparameters, with learned sigma and 250 strided sampling steps.The decoder follows the same architecture and diffusion hyperparameters as the cited GLIDE work.
  • Upsamplers: The two ADMNet upsamplers use different noising schedules and channel depths, with 320 channels and three resblocks in the first stage and 192 channels and two resblocks in the second.The first stage applies Gaussian blur, the second uses BSR degradation, and neither upsampler uses attention.
  • Priors: The AR prior uses 2048-wide and 1664-wide 24-block Transformer modules, whereas the diffusion prior uses a 2048-wide 24-block Transformer with 64 Analytic DPM sampling steps.The diffusion prior scales CLIP embedding inputs by 17.2 to match the empirical variance of RGB pixel values.

D Random samples

The paper presents random samples from the production unCLIP model for prompts introduced earlier, illustrating outputs across several text descriptions.

  • D Random samples: Random samples from the production model are shown for selected prompts from Figure 1.Examples include a vibrant Salvador Dali portrait with a robotic half face, a palm with leaves growing from it, and a teddy bear skateboarding in Times Square.
  • D Random samples: The examples cover a portrait, a close-up handpalm, and a teddy bear on a skateboard in Times Square.These samples correspond to Figures 18, 19, and 20, respectively.
Loading 2204.06125v1…