Intriguing properties of generative classifiers

1 INTRODUCTION

• The study examines whether generative models can classify challenging out-of-distribution images and whether their decisions favor shape or texture.This question arises because many discriminative classifiers struggle on out-of-distribution images, while humans tend to be shape-biased.Source: §1 · p. 2
• Generative classifiers are models trained to generate images from which the researchers extract zero-shot classification decisions.The comparison focuses on diffusion and autoregressive text-to-image models, discriminative models, and human psychophysical data.Source: §1 · p. 2
• The broader question is whether object recognition is better implemented through fast but potentially shortcut-prone discriminative inference or slower but potentially more robust generative modeling.The introduction notes that how the brain combines these approaches remains unresolved.Source: §1 · pp. 2–3
• The work evaluates three text-to-image models on 17 challenging out-of-distribution datasets and compares them with 52 discriminative models and human data.The generative models are Stable Diffusion, Imagen, and Parti.Source: §1 · p. 3
• Generative classifiers show human-like shape bias, near-human out-of-distribution accuracy, state-of-the-art alignment with human errors, and understanding of certain perceptual illusions.The study compares diffusion and autoregressive text-to-image models with discriminative models and human psychophysical data.Source: §1 · pp. 2–3

2 METHOD: GENERATIVE MODELS AS ZERO-SHOT CLASSIFIERS

• The generative models are Imagen, Stable Diffusion, and Parti, representing pixel-space diffusion, latent-space diffusion, and autoregressive sequence-to-sequence modeling.For Stable Diffusion, the evaluated input is a latent representation; for Imagen, it is raw image pixels.Source: §2 · pp. 3–4
• For each image, the method evaluates its conditional likelihood under every class and assigns the class with the highest likelihood.Parti provides an exact conditional likelihood in one forward pass; diffusion-model likelihoods are approximated using a diffusion variational lower bound.Source: §2 · p. 4
• The evaluation uses 17 challenging out-of-distribution datasets, including five single-manipulation datasets and twelve parametric distortion datasets.Human comparison data come from 90 observers and 85,120 trials collected in a calibrated psychophysical setting.Source: §2 · p. 4
• Images are resized to each model’s base resolution, and class labels are converted to prompts using “A bad photo of a yk.”Imagen uses 64 × 64 images, Parti 256 × 256, and Stable Diffusion 512 × 512.Source: §2 · p. 4
• The comparison includes 52 discriminative models, three ViT-22B variants, and CLIP alongside human observer data.The CLIP baseline uses the ViT-L/14@224px model and an ensemble of 80 prompts.Source: §2 · p. 5
• Models are compared using shape bias, out-of-distribution accuracy, and error consistency, which measures beyond-chance agreement in whether two decision makers make errors on the same images.Error consistency is measured with Cohen’s kappa; positive values indicate beyond-chance agreement and negative values indicate systematic disagreement.Source: §2 · p. 5

3.1 HUMAN-LIKE SHAPE BIAS

• 99% shape bias for Imagen; Stable Diffusion reaches 93% and Parti 92%, matching the reported human-level range.Imagen matches or exceeds human shape bias across nearly all categories and the three generative classifiers outperform ViT-22B on shape bias.Source: §3.1 · p. 6

3.2 NEAR HUMAN-LEVEL OOD ACCURACY

• Generative classifiers approximately match human responses on most OOD datasets, but all three struggle with rotation and high-pass results vary by model.Imagen and Stable Diffusion are close to human-level overall; some discriminative models, including ViT-22B, achieve super-human accuracy.Source: §3.2 · pp. 6–7

3.3 SOTA ERROR CONSISTENCY WITH HUMAN OBSERVERS

• Imagen has the most human-aligned error patterns and surpasses ViT-22B, though a substantial gap from human-to-human consistency remains.On cue-conflict images, the matrix sorts models by mean error consistency with humans, with higher consistency toward the right and bottom.Source: Figure 6 · p. 7, §3.3 · p. 7

3.4 UNDERSTANDING CERTAIN VISUAL ILLUSIONS

• Generative models reconstruct four images of bistable illusions or pareidolia in ways that align with human perception.For rabbit-duck images, the test prompts reconstruction as either animal; Muse replaces Parti in this analysis because Parti cannot directly edit images.Source: §3.4 · p. 8, Figure 7 · p. 8

4 ANALYSIS: WHERE DOES THE INCREASED SHAPE BIAS ORIGINATE FROM?

• A strong association links blurred-image performance with shape bias: Pearson’s r(58) = .59 and Spearman’s r(58) = .64, both p < 0.001.The authors explicitly note that these correlations establish a connection, not a causal link.Source: §4 · p. 9
• Shape bias rises from 21% for standard ResNet-50 to 78% with diffusion-style noise augmentation during training and evaluation.The reported 78% exceeds the 62% shape bias previously obtained by combining six techniques and augmentations.Source: §4 · p. 9
• The authors conclude diffusion-style training is a crucial factor in increased shape bias, while noting that other factors may also contribute.They report that diffusion-style training biases models toward low spatial frequency information, which is associated with increased shape bias.Source: §4 · p. 9

5 DISCUSSION

• Generative classifiers show a human-like shape bias (92–99%), near human-level OOD accuracy, Imagen’s most human-aligned error patterns to date, and qualitative capture of perceptual-illusion ambiguities.These models are zero-shot classifiers that were neither trained nor designed for classification.Source: §5 · p. 9
• Generative classifiers are leading behavioral models of human object recognition, but the findings do not resolve how brains might combine generative and discriminative models.The authors allow that human brains may use a different implementation yet arrive at a similar representation.Source: §5 · p. 9
• The experiments support generative pre-training as a compelling alternative for vision tasks and offer classification tasks as a way to probe generative-model behavior.All investigated generative models performed poorly on the rotation dataset and also struggled to generate rotated or upside-down objects.Source: §5 · p. 9
• The approach is computationally slow and does not yield a practical classifier; model comparisons are also imperfect because sizes, resolutions, training data, training duration, and language encoders differ.The current approach generates at least one image per class, which is described as expensive and slow.Source: §5 · p. 10
• The authors identify open questions about how language cross-attention shapes the findings and whether denoising diffusion training explains Imagen’s and Stable Diffusion’s performance.Source: §5 · p. 10

A BACKGROUND ON DIFFUSION MODELS

• Diffusion models add Gaussian noise through a forward process and learn a reverse denoising process; their training objective includes prior, reconstruction, and diffusion losses.For classification, the paper uses L2 loss and weights timestep scores with wt := exp(−7t).Source: A BACKGROUND ON DIFFUSION MODELS · pp. 14–15

• Muse achieved only approximately chance-level accuracy on clean, undistorted images, so the authors could not include it as a successful classifier.Source: B MUSE AS A CLASSIFIER · p. 15
• The authors caution that Muse’s weak classification result may reflect their extraction approach, and that other approaches might work better.Source: B MUSE AS A CLASSIFIER · p. 15

• The approach is too slow to yield a practical classifier, particularly because it generates at least one image per class.Source: C LIMITATIONS · p. 15
• Comparisons are imperfect because the models differ in training data, architecture, and training procedure, so differences cannot be attributed to a single factor.Source: C LIMITATIONS · p. 15

• For Muse, the closest reconstruction can belong to a category the model cannot realistically generate, making L2 distance to the input an infeasible classification rule.In the example, the closest outputs include airplane, car, and truck, while the correct bottle-shaped prompt is shown separately.Source: Figure 8 · p. 16

E DETAILS ON RESNET-50 TRAINING WITH DIFFUSION NOISE

• ResNet-50 training used diffusion noise as an added augmentation and ran for 300 rather than 90 epochs because the task was more challenging.The authors did not optimize the settings for shape bias or other observed findings.Source: E DETAILS ON RESNET-50 TRAINING WITH… · p. 17

F SHAPE BIAS OF IMAGE CAPTIONING MODELS

• CapPa B16 and L16 were more shape-biased than most discriminative models, but less shape-biased than the tested text-to-image classifiers.The authors suggest generative modeling may help shape bias, while the objective and data may determine its strength.Source: F SHAPE BIAS OF IMAGE CAPTIONING MODELS · p. 17
• CapPa evaluation used one “A bad photo of a class” prompt because 80 CLIP ImageNet templates reduced its ImageNet validation accuracy.The authors attribute this possibly to CapPa models not being designed to handle different prompts well.Source: F SHAPE BIAS OF IMAGE CAPTIONING MODELS · pp. 17–18

G ADDITIONAL PLOTS FOR MODEL-VS-HUMAN BENCHMARK

• The cited figure mapping assigns Figure 11 to out-of-distribution accuracy on parametric datasets and Figure 12 to out-of-distribution accuracy on nonparametric datasets.For nonparametric datasets, Figures 6 and 15 to 18 show human-to-human and model-to-model error consistency.Source: G ADDITIONAL PLOTS FOR MODEL-VS-HUMAN… · p. 18
• Figures 6 and 15 to 18 report human-to-human and model-to-model error consistency for nonparametric datasets.Figure 11 plots model-to-human error consistency.Source: G ADDITIONAL PLOTS FOR MODEL-VS-HUMAN… · p. 18
• Figure 10 plots benchmark results aggregated over datasets, while Figure 11 shows out-of-distribution accuracy and error consistency for twelve parametric datasets.The benchmark metrics are based on the model-vs-human toolbox.Source: G ADDITIONAL PLOTS FOR MODEL-VS-HUMAN… · p. 18, Figure 10 · p. 19

H QUANTITATIVE BENCHMARK SCORES AND RANKINGS

• ResNet-50 shape bias rose from 21% for the vanilla model to 78% after training and evaluation with diffusion noise.Figure 13 presents shape-bias results across models, and Figure 14 shows the before-and-after ResNet-50 comparison.Source: Figure 13 · p. 21, Figure 14 · p. 22
• Table 2 aggregates results across 17 datasets for accuracy difference, observed consistency, and error consistency, then ranks models by their mean rank across those metrics.The three metrics yield different model rankings.Source: H QUANTITATIVE BENCHMARK SCORES AND… · pp. 19, 25, Table 2 · p. 25
• Table 3 aggregates out-of-distribution robustness across 17 datasets; Figures 11 and 12 show accuracy across twelve parametric and four nonparametric datasets, respectively.Figure 11 also presents error consistency across the twelve parametric datasets.Source: H QUANTITATIVE BENCHMARK SCORES AND… · p. 19, Figure 11 · p. 20, Figure 12 · p. 21