Source-linked AI summary
Text-to-Image Diffusion Models are Zero-Shot Classifiers
Kevin Clark, Priyank Jaini
TL;DR
Text-to-image diffusion models have informative representations, but their downstream knowledge and transfer abilities are not fully understood. The paper turns Imagen and Stable Diffusion into zero-shot classifiers using denoising-based likelihood proxies and compares them with CLIP. They match CLIP broadly, outperform it on shape-versus-texture robustness, and perform some attribute-binding tasks that CLIP cannot.
Problem
Diffusion models learn strong image-text representations, but their downstream discriminative abilities and captured knowledge have not been thoroughly explored.
Method
The paper uses text-conditioned denoising scores as likelihood proxies to evaluate Imagen and Stable Diffusion directly as zero-shot image classifiers.
Results
Imagen and Stable Diffusion perform comparably to CLIP across diverse zero-shot datasets, while Imagen achieves more than 50% error reduction over CLIP on Cue-Conflict and diffusion models bind attributes in some settings.
Takeaways & Limitations
The results support studying generative pre-training as an alternative to contrastive pre-training for vision-language tasks.
Takeaways & Limitations
Model comparisons are not direct because Imagen, Stable Diffusion, and CLIP differ in size, resolution, training data, and training duration.
Abstract
from arXiv · showhide
The excellent generative capabilities of text-to-image diffusion models suggest they learn informative representations of image-text data. However, what knowledge their representations capture is not fully understood, and they have not been thoroughly explored on downstream tasks. We investigate diffusion models by proposing a method for evaluating them as zero-shot classifiers. The key idea is using a diffusion model's ability to denoise a noised image given a text description of a label as a proxy for that label's likelihood. We apply our method to Stable Diffusion and Imagen, using it to probe fine-grained aspects of the models' knowledge and comparing them with CLIP's zero-shot abilities. They perform competitively with CLIP on a wide range of zero-shot image classification datasets. Additionally, they achieve state-of-the-art results on shape/texture bias tests and can successfully perform attribute binding while CLIP cannot. Although generative pre-training is prevalent in NLP, visual foundation models often use other methods such as contrastive learning. Based on our findings, we argue that generative pre-training should be explored as a compelling alternative for vision-language tasks.
1 Introduction
This section motivates evaluating text-to-image diffusion models beyond generation and introduces their direct use as zero-shot classifiers. The experiments compare them with CLIP across classification, texture robustness, and attribute binding.
- Diffusion models’ strong image generation suggests they learn effective image-text representations, but their downstream discriminative transfer remains insufficiently explored.
- The paper converts diffusion models into generative zero-shot classifiers by scoring how well class-conditioned text prompts denoise an input image.The method uses a re-weighted variational lower bound as a proxy for unavailable exact likelihoods.
- Up to 1000x lower compute enables the diffusion classifier to become usable on datasets with many classes, although it remains slow.The efficiency improvements include pruning obviously incorrect classes.
- Imagen and Stable Diffusion achieve zero-shot classification performance comparable to CLIP across diverse vision datasets.
- Diffusion models achieve state-of-the-art Cue-Conflict results, with Imagen reducing error by more than 50% versus CLIP and outperforming ViT-22B.
- Diffusion models can bind attributes in some settings, whereas CLIP cannot, suggesting generative representations capture capabilities beyond image generation.
2 Preliminaries
Diffusion models are latent-variable generative models that progressively noise data and learn to reverse that process. Their training objective uses a weighted reconstruction-style loss over sampled noise levels.
- Diffusion models define forward and reverse Markov chains over observations and progressively noisier latent variables.
- The forward process adds Gaussian noise according to a fixed schedule, while the learned reverse process denoises from a standard Gaussian initialization.
- Training optimizes a variational lower bound on marginal likelihood, including prior, reconstruction, and diffusion-loss terms.
- The re-weighted diffusion loss samples an observation, Gaussian noise, and a timestep, then predicts the clean observation from its noisy version.The timestep weight w_t controls each sampled loss contribution.
- Diffusion models can condition denoising predictions on text prompts, class labels, segmentation masks, or low-resolution images.
3 Zero-Shot Classification using Diffusion Models
The proposed classifier estimates each class’s diffusion loss under its text prompt and predicts the class with the lowest estimated loss. Shared noise, adaptive pruning, and weighting reduce the otherwise substantial computation.
- Diffusion Generative Classifier: The method maps each class label to a text prompt and treats its conditional diffusion loss as a proxy for class likelihood.Because exact likelihoods are unavailable, the method uses the diffusion component of the variational lower bound.
- Diffusion Generative Classifier: For each class, the model repeatedly noises and denoises the image under its prompt, estimates the diffusion loss, and selects the minimum-loss class.
- Weighting Function: The timestep weighting function aggregates per-class scores, with alternative weighting functions explored because the training weight may not maximize classification accuracy.
- Efficiency Improvements: Shared noise reduces variance by evaluating all class prompts on the same timestep and noised image.Differences between class estimates then primarily reflect text conditioning rather than different Monte Carlo samples.
- Efficiency Improvements: Candidate-class pruning discards classes unlikely to become the lowest-scoring class and allocates more computation to plausible candidates.The procedure uses statistical thresholds and minimum scoring counts to reduce the chance of pruning the correct class.
- Efficiency Improvements: Shared noise and pruning require up to 1000x less compute than naive scoring, but classification still typically takes tens of scores per class.
4 Empirical Analysis and Results
The empirical analysis evaluates diffusion models as zero-shot classifiers across diverse datasets, efficiency settings, shape–texture conflicts, and synthetic attribute-binding tasks. Results show competitive classification, strong robustness to misleading textures, and binding abilities that differ from CLIP.
- Evaluation scope: The evaluation covers diverse classification datasets, efficiency strategies, shape–texture conflicts, and synthetic attribute-binding tasks.The experiments compare Imagen, Stable Diffusion, and CLIP across these settings.
- Efficiency: Shared noise and candidate pruning reduce Imagen’s classification compute by up to 1000x versus naïve scoring.Shared noise improves sample efficiency, while pruning removes implausible classes early.
- Image classification: Imagen generally outperforms CLIP and Stable Diffusion on low-resolution datasets, while Stable Diffusion performs comparably to Imagen on higher-resolution datasets.Stable Diffusion’s ImageNet accuracy drops from 61.9% to 15.5% and 34.6% when images are down-sampled to 32×32 and 64×64, respectively.
- Shape–texture robustness: 84.4% shape accuracy lets Imagen outperform Stable Diffusion, CLIP, and ViT-22B by more than 12%, 30%, and 15%, respectively, on Cue-Conflict.Imagen also exceeds the supervised ResNet50 model’s top-5 accuracy by 5%.
- Shape–texture robustness: Diffusion models are much better than CLIP at disregarding misleading textural cues, indicating denoising-based classification is robust to texture bias.The authors connect this behavior to the denoising process, while presenting the causal explanation as a belief.
- Attribute binding: CLIP performs no better than chance on attribute-binding tasks, whereas Imagen binds shape and color in some tasks and Stable Diffusion binds shape and color but not position.Imagen’s frequent preference for the larger object’s color limits some Color|Shape predictions.
5 Conclusion and Future Work
The paper presents diffusion models as zero-shot classifiers and quantitative probes of their learned abilities, while identifying future research directions and practical limits.
- The framework enables diffusion models to act as zero-shot classifiers and supports quantitative evaluation of generative models through controlled classification tasks.It also improves efficiency, though the method remains computationally demanding.
- Imagen and Stable Diffusion achieve state-of-the-art robustness to misleading textures and sometimes bind attributes that CLIP does not appear to bind.The paper presents these findings as evidence that diffusion models capture useful properties beyond generation.
- Related Work: The paper distinguishes its analysis from concurrent work using Stable Diffusion as a zero-shot classifier, which studies different tasks and reports slightly worse results.The authors attribute the difference to class pruning and timestep weighting choices.
- Future Work: The authors propose studying diffusion models as foundation models for non-generative tasks, including downstream fine-tuning and comparisons of contrastive and generative scaling laws.They also suggest testing whether the findings reflect generative pre-training generally or diffusion pre-training specifically.
- The method is not a practical classifier because scoring many classes requires substantial compute, so its primary value is analysis of pre-trained diffusion models.The authors frame generative pre-training as a potentially useful alternative to contrastive pre-training for text-image self-supervised learning.
A Model Details.
The experiments use Imagen, Stable Diffusion, and CLIP as distinct pre-trained vision-language models with different architectures, sizes, resolutions, and training procedures.
- Imagen: Imagen combines a frozen T5 language encoder with a 64 ×64 pixel diffusion model and two cascaded super-resolution diffusion models.The experiments use the 64 × 64 model, which has 2B parameters.
- Stable Diffusion: Stable Diffusion v1.4 is a latent diffusion model using CLIP’s text encoder and a variational autoencoder, with 890M parameters and 512x512 inputs.It was trained on subsets of Laion-5B for 1.2M steps with batch size 2048.
- CLIP: CLIP encodes image and text features into a shared latent space and scores their similarity after contrastive pre-training.The comparison uses the largest CLIP model, with a ViT-L/14@224px image encoder and 400M parameters.
B Weighting Functions Details.
The weighting-function analysis compares heuristic and learned timestep weighting for diffusion-based classification, finding that the heuristic performs strongly while learned weighting adds little.
- Learned Weighting: Learned weighting bins timestep scores into 20 buckets and fits a 20-feature linear model over those bucketed scores.At test time, the weight is selected as wt = v⌊t/0.05⌋.
- Learned Weighting: Learned weights improve zero-shot classification accuracy by around 1% on average, but they are excluded from the main paper because learning them is not truly zero-shot.The learned functions are similar across datasets and transfer well between them.
- Comparison: The comparison evaluates VDM, Simple, Heuristic, and Learned weighting functions on Caltech101, whose table reports percent accuracy for different schemes.Caltech101 was chosen for its resolution, class diversity, and relatively small number of classes.
- Results: The heuristic weighting function outperforms Simple and VDM for both models, while Stable Diffusion is more robust than Imagen to the weighting choice.Imagen is less accurate at very small noise levels, which the Simple and VDM functions weight more heavily.
C Variances in Classification Accuracies.
Accuracy estimates vary across random splits, with diffusion models showing greater variance than CLIP because their image noising is random.
- Random-split accuracy variance is roughly ±0.4% for CLIP, ±0.7% for Imagen, and ±0.6% for Stable Diffusion.The diffusion models’ higher variance reflects randomness in noising images, whereas CLIP is deterministic.
D Details on Attribute Binding Tasks and Prompts
The attribute-binding experiments construct prompts and negative examples from paired objects with distinct attributes, using simple templates and a fixed ordering convention. Alternative prompts did not substantially improve these templates.
- Each image contains two objects with distinct shapes, colors, sizes, and positions for recognition and binding tasks.
- Task-specific descriptions select attributes such as position, size, color, or shape for each object.
- Positive prompts use one randomly selected object for recognition and binding, or combine both object descriptions for pair tasks.
- Negative binding prompts replace one positive attribute with the other object's corresponding attribute, creating mismatched attribute combinations.
- Pair prompts consistently state the leftmost position, shape, color, or size first, and alternative prompts did not work substantially better.
E Calibration
The calibration analysis addresses how to obtain reliable confidence estimates when diffusion classifiers do not directly produce class probabilities. It compares score-based scaling with a number-of-scores measure that is fairly well calibrated.
- Diffusion classifiers lack direct class probabilities, making calibration desirable for zero-shot predictions, including potentially out-of-domain examples.
- The proposed number-of-scores measure uses total diffusion calls as confidence evidence because harder examples require more scores to identify the predicted class reliably.
- Temperature scaling converts weighted average squared-error scores into estimated class probabilities but requires score estimates for every class.
- The alternative Platt-scaled confidence uses the total number of scores needed to eliminate competing classes during prediction.
- Number-of-scores confidence is fairly well calibrated, showing that diffusion classifiers can provide reasonable confidences without an explicit class-probability distribution.
F Imagen’s Super-resolution Models
Only Imagen's 64 × 64 low-resolution model worked well as a zero-shot classifier; its super-resolution models were less sensitive to text prompts and performed poorly alone or in combination.
- Imagen's 64 × 64 low-resolution model worked well as a zero-shot classifier, whereas its super-resolution models did not.
- Super-resolution models condition on low-resolution inputs, so they denoise effectively across prompts and become less sensitive to class labels.
- The 64 × 64-to-256 × 256 model achieved 16.1% accuracy on ImageNet.
- Combining low-resolution and super-resolution scores assigned almost no weight to the super-resolution model and did not significantly improve performance.