PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Separation of Human and AI-Generated Images in CLIP Embedding Space

Andrea Asperti

arXiv:2608.25609v1cs.CV

TL;DR

The paper asks why human and AI-generated paintings separate spontaneously in CLIP embedding space and investigates the visual evidence behind that separation. Using progressive interpretable representations and gradient-based inversion, it finds distributed multiscale structure and strong sensitivity to nearly imperceptible image changes.

  • Problem

    The reasons why human and AI-generated paintings naturally separate in CLIP embeddings without supervised learning remain unexplained.

  • Method

    The paper progressively evaluates interpretable image statistics and uses gradient-based inversion to probe relationships between image structure and CLIP embedding directions.

  • Results

    Multiscale scattering is the most informative tested representation but explains only part of the geometry, while nearly imperceptible perturbations can substantially move images along dominant CLIP directions.

  • Takeaways & Limitations

    The findings indicate that CLIP reflects distributed visual evidence that differs substantially from what is readily accessible to human perception.

  • Takeaways & Limitations

    The analysis is not an exhaustive characterization of every factor contributing to the separation, and hidden generator-specific signatures or watermarking cannot be completely excluded.

Abstract

from arXiv · show

We identify a previously unreported phenomenon in CLIP representations: human and AI-generated paintings spontaneously separate along the dominant principal directions of their joint embedding distribution, without any supervised objective designed to distinguish the two classes. Rather than exploiting this phenomenon for detection, our objective is to interpret it: we seek to identify the visual information underlying the separation and to trace it back from the embedding space to the image domain. We pursue this objective through a progressive investigation combining interpretable image representations with gradient-based inversion, used systematically as an experimental probe of the relationships identified in feature space. Robustness experiments and increasingly expressive statistical descriptors progressively rule out several intuitive explanations based on global image properties and simple local statistics, and point instead to distributed multiscale image structure. Multiscale scattering provides the most informative interpretable representation considered, but offers only a partial account of the phenomenon. Direct inversion provides a complementary and striking observation: substantial displacements along the dominant CLIP directions can be induced by image perturbations that remain nearly imperceptible to human observers, showing that the directions involved in the separation are highly sensitive to image variations with very low perceptual salience for humans. Taken together, these results reveal a significant difference between the visual evidence reflected in CLIP representations and that readily accessible to human perception, raising broader questions about the relationship between artificial and human vision and, ultimately, between artificial and human aesthetic judgment.

1 Introduction

The paper studies why human and AI-generated paintings separate naturally along dominant CLIP embedding directions, rather than training a detector. Progressive interpretable analyses and inversion point to distributed multiscale structure and a mismatch between CLIP-sensitive evidence and human perception.

  • Robustness: The phenomenon remains robust across image generators, human-painting collections, datasets, and CLIP versions.Figure 1 uses AI-Pastiche, National Gallery of Art paintings, and CLIP ViT-L/14@336px.
  • Phenomenon: Human and AI-generated paintings separate along dominant CLIP principal directions without supervised discrimination objectives.The distinction is already present in the empirical distribution of mixed-image embeddings.
  • Progressive analysis: Grayscale conversion, aggressive downsampling, cropping, and resolution changes preserve the separation, while simple statistics and local gradients explain only limited variability.These tests rule out several intuitive explanations based on global appearance and basic local structure.
  • Interpretable representations: Multiscale scattering features over patches and spatial resolutions best explain the dominant directions, implicating distributed mesoscopic statistics across scales and locations.Scattering captures a significant but incomplete fraction of the observed embedding geometry.
  • Inversion: Gradient-based inversion induces substantial CLIP-direction displacements with image perturbations that remain almost imperceptible to human observers.This provides evidence that CLIP relies on visual information largely inaccessible to human perception.
  • Implications: The results provide quantitative evidence of a systematic mismatch between artificial and human visual perception.The paper contrasts CLIP-sensitive statistical regularities with human reliance on semantic, stylistic, and compositional properties.

2 Background and Related Work

Prior work established CLIP’s semantic organization, detection utility, interpretable descriptors, and inversion methods, but largely treated these areas separately. This paper instead investigates the unsupervised geometry of CLIP embeddings using interpretable statistics and inversion.

  • CLIP representations: CLIP provides a transferable shared image-text embedding space used across retrieval, recognition, segmentation, and multimodal reasoning.Its representations also organize high-level visual concepts and semantic attributes.
  • Open question: Existing analyses explain semantic organization within CLIP but provide limited insight into why particular image collections naturally separate in embedding space.This motivates studying empirical separation rather than only internal semantic structure.
  • AI-generated image detection: AI-image detection research increasingly uses CLIP features, but existing approaches formulate the task as supervised classification.Prior work emphasizes decision boundaries, robustness, and cross-generator generalization within that paradigm.
  • Position of this work: This work studies the geometry of the embedded dataset itself, asking why separation emerges before supervised learning.The paper addresses a gap linking unsupervised CLIP geometry with interpretable image statistics and inversion.
  • Interpretable descriptors: Handcrafted descriptors such as SIFT and HOG characterize local gradients, while wavelet-based models capture multiscale image statistics.Here, HOG and scattering serve as analytical tools for interpreting CLIP geometry rather than replacing deep representations.
  • Representation inversion: Inversion methods reconstruct or optimize inputs to probe learned representations, and this paper uses inversion as an analytical probe of CLIP embedding geometry.The stated role differs from ordinary visualization or adversarial applications.

3 Experimental Setting

The experiments use multiple human and AI-generated painting datasets to study their natural separation in CLIP embedding space, focusing on PCA geometry and direct image inversion. The setup tests how principal-coordinate movements can be induced while preserving visual similarity.

  • Datasets and CLIP models: The phenomenon is consistently observed across different datasets, image generators, and CLIP variants, including the AI-WikiArt dataset.AI-WikiArt contains authentic paintings and synthetic artworks generated with Stable Diffusion, Flux, and F-Lite.
  • Experimental goals: The study aims to explain what PC1 and PC2 represent in image space and identify image modifications that reduce the observed separation.Images are repeatedly projected into the original PC1–PC2 space to analyze transformation-induced displacements.
  • Direct CLIP inversion: Direct inversion searches for image perturbations that maximize or minimize selected principal coordinates while keeping the modified image visually close to the original.Optimization proceeds through the differentiable CLIP encoder and navigates the dataset-specific embedding geometry.
  • PCA representation: PCA coordinates are computed by projecting centered CLIP embeddings onto principal directions obtained from the combined embedding dataset.The empirical mean embedding is subtracted before projection.
  • Direct CLIP inversion: The perturbation parameterization bounds image changes through δ = ε tanh(u), while total variation regularization discourages high-frequency noise.The regularized objective favors spatially coherent transformations distributed across the image.
  • Direct CLIP inversion: Substantial movements along dominant CLIP directions can occur despite perturbations that remain visually almost indistinguishable from the original images.The perturbations remain numerically small and visually coherent, producing a perceptual dissociation between human observers and CLIP.

4 Methodological framework

The framework progressively tests whether interpretable image representations explain CLIP’s AI–human separation, moving from robustness and basic statistics toward spatial, multiscale, and inversion-based analyses.

  • The investigation treats robustness analysis, statistical descriptors, and inversion as successive steps that narrow plausible explanations for the separation.
  • Robustness analysis: Robustness experiments test file handling, color, resolution, cropping, datasets, generators, and CLIP variants before explanatory analyses.
  • Basic statistical descriptors: Global descriptors examine pixel and luminance histograms, color statistics, frequency spectra, and texture measures against CLIP principal directions.
  • Advanced image descriptors: HOG, scattering, frequency, and orientation-transport features extend the analysis toward richer local and multiscale representations.
  • Spatial organization: Feature aggregation and patch shuffling distinguish information in local statistics from information in their spatial arrangement.
  • Representation inversion: Regression accuracy alone is insufficient because descriptors can predict PCA coordinates well yet fail to produce meaningful inversion results.
  • Representation inversion: Inversion seeks visually negligible perturbations that maximize or minimize predicted PCA coordinates, testing whether descriptor–embedding relationships are actionable and visually interpretable.

5 Robustness Experiments

Robustness experiments show that CLIP’s AI–human separation survives re-encoding, grayscale conversion, substantial resolution loss, and sufficiently large crops across datasets and generators. These results rule out simple storage artifacts, isolated fine-scale cues, and purely global layout as complete explanations, pointing toward mesoscopic structure.

  • File-format robustness: Original and reprocessed embeddings have cosine similarity 1 up to numerical precision, ruling out re-encoding, compression, metadata, and color-pipeline artifacts.
  • Color robustness: Grayscale embeddings retain average cosine similarity of approximately 0.90±0.04 for AI-Pastiche and 0.89±0.04 for NGA.
  • Color robustness: Grayscale conversion slightly entangles the classes, but most PCA structure remains preserved, indicating that dominant differences are largely independent of color.
  • Resolution robustness: Resolution degradation increasingly displaces embeddings, yet the AI–human separation remains robust across substantial reductions and spatial scales.
  • Cropping robustness: Crops covering approximately 40% of the shortest image dimension recover almost full-image classification performance, whereas 10% crops contain essentially no discriminative information.
  • Cropping robustness: The dominant signal is neither purely local nor purely global: it requires sufficiently large regions while remaining largely independent of exact global layout.
  • Cross-model robustness: The separation persists across AI-Pastiche’s model generations and newer generators including Imagen4, FireflyImage4, Midjourney 7, Flux.2, and gpt-image-1.5.
  • Overall robustness: Across datasets, generators, preprocessing strategies, and transformations, the stable separation motivates increasingly expressive statistical descriptors.

6 Global Statistical Descriptors

Global color, intensity, and basic luminance statistics show only limited correspondence with CLIP’s dominant directions. Their weak explanatory power motivates descriptors that represent richer spatial organization.

  • Conclusion: Simple global statistics and elementary local contrast capture only a limited aspect of the phenomenon, motivating richer spatial descriptors.
  • Color and intensity: RGB statistics reveal modest dataset differences, particularly in the blue channel and intra-image variability, but do not explain the dominant CLIP directions.
  • Color and intensity: Global intensity normalization procedures do not substantially modify the embedding geometry.
  • Color and intensity: The datasets have broadly similar brown- and gray-dominated palettes, with only minor quantitative color differences.
  • Luminance and local contrast: Luminance variability and local contrast are the first descriptors to show non-negligible correlation with the principal components.
  • Luminance and local contrast: The luminance and contrast statistics are mutually correlated between 0.86 and 1, leaving little gain from linear combinations.

7 Histogram of Oriented Gradients

HOG analyses indicate that CLIP-relevant information is weak but spatially distributed, with preserved spatial organization outperforming spatial averaging. Pairwise neighboring-region descriptors remain insufficient, motivating richer multiscale representations.

  • HOG representation: Each 24 × 24 patch is encoded as a 36-dimensional HOG descriptor, producing a 23×23×2×2×9 field for a 336 × 336 image.
  • Aggregation: The full HOG field predicts PC1 and PC2 best, while spatial averaging nearly erases the signal.
  • Spatial distribution: Mean and maximum HOG maps are relatively uniform, whereas prevalent-orientation maps are chaotic.
  • Spatial distribution: The discriminative signal is low-amplitude, smooth, spatially coherent, and not concentrated in isolated artifact zones.
  • Nonlinear aggregation: A shallow MLP only modestly improves over Ridge, indicating that strong nonlinear interactions do not dominate the signal.
  • Interpretation: The overall pattern is consistent with weak, spatially distributed gradient statistics that are largely additive and receive limited nonlinear refinement.
  • Spatial organization: Preserving HOG spatial organization is essential: progressively larger shuffling blocks substantially reduce correlation with the principal components.
  • Local correlation descriptors: Pairwise neighboring-region descriptors have limited predictive power below full spatial HOG, so simple local correlations do not adequately describe the organization.

8 Scattering

The study evaluates global and patch-based scattering descriptors as interpretable predictors of CLIP principal components. Scattering captures substantial information, but spatially richer aggregation does not consistently improve explanatory or predictive performance.

  • Experimental design: Scattering descriptors are evaluated globally and as spatially organized patch features to predict the first two CLIP principal components.Global descriptors use Ridge regression, while patch descriptors are processed by lightweight attention and Transformer models.
  • Global scattering: Increasing the maximum scattering scale raises Ridge performance from R2 ≈0.33 to R2 ≈0.41 for PC1 and from R2 ≈0.44 to R2 ≈0.49 for PC2.The improvement is described as relatively modest despite the larger descriptor dimensionality.
  • Interpretation: Increasing receptive-field scale saturates, motivating regional descriptors rather than a single global representation.The saturation is interpreted as evidence that the limitation lies in global pooling rather than scattering scale alone.
  • Nonlinear prediction: A nonlinear MLP reaches approximately R2 = 0.60 for both principal components, demonstrating considerable information in the scattering coefficients.The paper cautions that predictive performance alone is not equivalent to explanatory power.
  • Patch scattering: Patch-based models generally fall between linear and MLP performance and do not improve inversion-related interpretability or controllability.Preserving spatial organization does not systematically recover the information missing from global scattering.

9 Inversion

Gradient-based inversion uses differentiable scattering surrogates to move images along predicted CLIP principal directions. These perturbations produce consistent but variable displacements, with scattering accounting for only part of the direct CLIP inversion effect.

  • Inversion procedure: The inversion procedure optimizes an image to maximize or minimize a differentiable scattering-based prediction of a selected CLIP principal component.Gradients pass through scattering descriptors and Transformer regressors back to the input image.
  • Perturbation constraints: The perturbation is parameterized from the original image, bounded pixelwise by ±ε, and concentrated around textured regions and object boundaries using a fixed mask.The mask is derived from a smoothed edge map while largely preserving homogeneous areas.
  • Displacement results: Scattering inversion produces displacements substantially smaller than direct CLIP inversion but with consistent magnitude, indicating that scattering captures part of the associated visual information.The scattering displacement is approximately one quarter of the direct CLIP displacement.
  • Model comparison: MLP inversion modestly improves mean displacement at J = 3 and J = 4, but this advantage disappears at J = 5 and J = 6.At larger scales, mean displacement decreases while variance remains approximately unchanged.
  • Trajectory variability: Inversion produces systematic average movement toward the opposite class region, yet individual trajectories show large variance and irregularity.The experiments visualize movements from AI-Pastiche toward the human region and from NGA paintings toward the AI region.
  • Ensemble inversion: An ensemble of four Ridge regressors for J = 3 to J = 6 reduces variance and discourages model-specific image modifications.The ensemble also reduces artifacts typical of inversion techniques.

10 Discussion

The discussion links the separation to distributed multiscale scattering statistics rather than a single global image attribute. These statistics remain only a partial explanation, and the paper identifies important limits concerning representation, image domain, and generator-specific effects.

  • Interpretable representation: Aggregating scattering coefficients reduces the J = 3 descriptor from 61 to 7 features while retaining scale-dependent information for visual comparison.The compact representation combines zeroth-order intensity, first-order scale responses, and second-order cross-scale interactions.
  • Scale robustness: The overall scattering differences are robust to the choice of maximum scattering scale, appearing similarly at J = 3 and J = 4.This robustness accompanies the scale-consistent first- and second-order shifts.
  • First-order statistics: The zeroth-order coefficient overlaps substantially across datasets, whereas AI-generated images show larger first-order responses across scales.The first-order shift remains consistent as scale increases, indicating a broad rather than frequency-localized difference.
  • Second-order statistics: AI-generated images exhibit larger second-order coefficients and longer right-hand tails, suggesting stronger dependencies between structures at different scales.The same systematic shift persists for additional coarse-scale coefficients at J = 4.
  • Spatial variability: Spatial coefficient of variation is approximately equal across image sets, so larger scattering coefficients are not attributed to regular repeated texture.The increased absolute variance is approximately proportional to the increased mean.
  • Scope: The study is not an exhaustive characterization of every factor contributing to human–AI separation; it investigates interpretable statistical representations progressively.Several important questions remain open.
  • Limitations: The analysis is restricted to CLIP representations, leaving unclear whether the identified mechanisms generalize to other artificial vision systems.The paper specifically proposes extending the methodology to self-supervised vision transformers and other foundation models.
  • Limitations: The experiments use only artistic images, so whether similar mechanisms distinguish human and AI-generated natural photographs remains open.The domain choice is motivated by the central role of human perception in artistic evaluation and the clear separation observed there.

11 Conclusion

The study finds that multiscale scattering descriptors explain meaningful but incomplete structure in CLIP’s human–AI separation. The results also expose differences between CLIP’s visual evidence and human perception, motivating broader scrutiny of vision models and feature-based metrics.

  • Scattering descriptors explain a meaningful component of the dominant CLIP directions, but much of their statistical structure remains unexplained.The authors identify incomplete interpretability as a central unresolved issue.
  • Attention-based descriptors may be needed because Vision Transformers encode long-range, content-dependent interactions that scattering does not naturally capture.The paper presents this as an important direction for future investigation.
  • Whether similar statistical mechanisms occur in self-supervised or purely visual foundation models remains an open question beyond CLIP.Comparisons across representation-learning paradigms could distinguish CLIP-specific properties from broader mechanisms.
  • The observed discrepancies challenge the assumption that modern vision foundation models extract information broadly aligned with human perception.The authors call for this assumption to receive closer experimental scrutiny.
  • Feature-based metrics such as FID may reflect biases that do not coincide with human visual judgment when model and human evidence differ systematically.This consequence follows because such metrics rely on representations extracted by deep vision models.
  • In artistic evaluation, semantic agreement between humans and artificial systems does not necessarily imply agreement in aesthetic judgment.The paper argues that future work should examine where artificial and human visual-quality judgments diverge.
Loading 2608.25609v1…