Source-linked AI summary
Measuring Style Similarity in Diffusion Models
Gowthami Somepalli, Anubhav Gupta, Kamal Gupta, Shramay Palta, Micah Goldblum, Jonas Geiping, Abhinav Shrivastava, Tom Goldstein
TL;DR
Semantic image retrieval leaves style attribution in generated images less understood, despite its relevance to artists and users assessing training-data relationships. The paper builds LAION-Styles and learns style descriptors with supervised and self-supervised objectives. CSD performs strongly across style-matching tasks, and the Stable Diffusion analysis reports greater style copying with more complex prompts.
Problem
Existing retrieval methods focus on semantic content, while the origin of style in generated images is less understood, limiting attribution relevant to artists and users.
Source: §1 · p. 2Method
The paper curates the artist-associated LAION-Styles dataset and trains CSD with supervised and self-supervised objectives to extract style descriptors.
Source: §1 · p. 2Results
CSD achieves state-of-the-art performance across style-matching tasks including DomainNet, WikiArt, and LAION-Styles, and the Stable Diffusion analysis finds greater style copying with more complex prompts.
Source: §8 · pp. 10–11Takeaways & Limitations
Style descriptors can help users assess potential conflicts and associations in generated images and help artists assess whether models use elements of their work.
Source: §1 · p. 2WikiArt artist-matching errors reflect nuanced style distinctions, and the authors suggest accuracy may have a ceiling below 100%, even for typical human evaluators.
Source: §6.1 · p. 7
Abstract
from arXiv · showhide
Generative models are now widely used by graphic designers and artists. Prior works have shown that these models remember and often replicate content from their training data during generation. Hence as their proliferation increases, it has become important to perform a database search to determine whether the properties of the image are attributable to specific training data, every time before a generated image is used for professional purposes. Existing tools for this purpose focus on retrieving images of similar semantic content. Meanwhile, many artists are concerned with style replication in text-to-image models. We present a framework for understanding and extracting style descriptors from images. Our framework comprises a new dataset curated using the insight that style is a subjective property of an image that captures complex yet meaningful interactions of factors including but not limited to colors, textures, shapes, etc. We also propose a method to extract style descriptors that can be used to attribute style of a generated image to the images used in the training dataset of a text-to-image model. We showcase promising results in various style retrieval tasks. We also quantitatively and qualitatively analyze style attribution and matching in the Stable Diffusion model. Code and artifacts are available at https://github.com/learn2phoenix/CSD.
1 Introduction
Style attribution matters because generated images may reproduce training data, while existing retrieval tools focus on semantic similarity. The paper combines style-preserving self-supervision with artist-labeled data to learn descriptors, then applies them to style retrieval and Stable Diffusion analysis.
1 Introduction
- Existing image retrieval primarily matches semantic content, leaving the origin of style in generated images less understood.The paper motivates attribution as a way for users to assess potential conflicts and associations and for artists to assess use of their work.Source: §1 · p. 2
- The authors evaluate style descriptors on public-domain datasets and report that CSD outperforms large-scale pretrained models and prior style retrieval methods.The contribution summary names WikiArt and DomainNet for zero-shot evaluation.Source: §1 · p. 2
- The paper uses CSD to study style replication in Stable Diffusion and proposes indicators of how likely an artist’s style is to be replicated.Source: §1 · p. 2
- The paper introduces LAION-Styles, a style-attribution dataset, and a multi-label contrastive method for extracting image style descriptors.The framework combines self-supervised and supervised objectives to represent style.Source: §1 · p. 2
2 Motivation
The case study uses CSD to compare generated images with artist prototypes under unconstrained and content-constrained prompts. It finds substantial artist-to-artist variation in similarity, with visual examples illustrating differences across artists and model versions.
2 Motivation
- Across 96 artists, the study computes General Style Similarity (GSS) by comparing generated-image embeddings with artist prototype vectors, and also evaluates content-constrained prompts.Artist prototypes average embeddings of their paintings; reported scores are means across repeated generations.Source: §2 · p. 2
- GSS scores are highly correlated with content-constrained scores, while Stable Diffusion 2.1 emulates some artists more strongly than others.Source: §2 · p. 2
- Leonid Afremov and Georges Seurat score highly and appear stylistically emulated, whereas Ruan Jia and Greg Rutkowski score low; Rutkowski was reported as excluded from SD 2.1 training.Source: §2 · p. 3
- The authors hypothesize that a single-image Style Similarity score below 0.5 indicates absence of an artist’s style, while a score above 0.8 strongly indicates its presence.This threshold proposal follows inspection of generations from 96 artists.Source: §2 · p. 3
- Figure 1 shows high similarity for MidJourney generations and cases where Greg Rutkowski, Ruan Jia, and Amano appear in SD 1.4 but not SD 2.1.The authors postulate that several artists may have been excluded from training or removed from SD 2.1; they note reports of removals.Source: §2 · p. 3
3 What is style?
The paper treats style as global image characteristics associated with an artist or movement, including color, brushwork, composition, and perspective. It situates its approach among feature-based, style-transfer, classification, and retrieval work.
3 What is style?
- Figure 2 plots artist-level similarity against prototypes for unconstrained prompts on the X-axis and woman-constrained prompts on the Y-axis.Source: Figure 2 · p. 3
- Style is defined as global image characteristics identified with an artist or artistic movement, including color use, brushstroke, composition, and perspective.Source: §3 · p. 3
- Earlier computational approaches modeled style with low-level visual features or rules for compositional elements, palettes, and brushstrokes [55].Source: §3 · p. 3
- Prior work includes style transfer and classification, while relatively few studies address in-the-wild style quantification, matching, and retrieval [23] [50] [66].Examples include Gram-matrix descriptors, photorealism regularization [34], and Markov random fields.Source: §3 · p. 3
- Unlike Wang et al.’s synthetic style-pair approach [63], this method uses real image pairs curated through captions and reports outperforming it on many zero-shot matching tasks despite using about 16% as much training data.Source: §3 · p. 4
4 Creating a dataset for style attribution
The paper addresses the lack of a large, style-diverse dataset with labels suitable for retrieval evaluation by constructing LAION-Styles from LAION-Aesthetics. Its filtering and deduplication pipeline yields a tagged subset for model training.
4 Creating a dataset for style attribution
- Existing datasets are limited for style retrieval: BAM is not public, while WikiArt is considered too small for training a strong style feature extractor.Source: §4 · p. 4
- LAION-Styles contains 511,921 images and 3,840 style tags, with each image assigned one or more tags.It is curated as a subset of LAION-Aesthetics images with predicted aesthetics scores of at least 6.Source: §4 · p. 4
- The curation starts with 5,600 tags from the CLIP Interrogator style bank, filters common tags and unavailable URLs, then deduplicates near-copies with SSCD at threshold 0.8 and merges their tags.The resulting dataset has 511,921 images and 3,840 tags.Source: §4 · p. 4
5 Contrastive Style Descriptors (CSD)
CSD learns style descriptors intended to separate style from semantic content and represent multiple styles. It combines style-preserving self-supervision with multi-label supervised learning on LAION-Styles.
5 Contrastive Style Descriptors (CSD)
- CSD seeks descriptors invariant to semantic content while capable of disentangling multiple styles.Its training pipeline combines contrastive self-supervision using style-preserving augmentations with alignment to human style perceptions through labeled LAION-Styles data.Source: §5 · p. 4
- The multi-label objective treats images as similar when they share any style label, using pairwise cosine similarity between projected ViT descriptors.The dataset has 511,921 images and 3,840 style tags; the approach considers ViT-B and ViT-L backbones.Source: §5 · pp. 4–5
- The self-supervised term uses spatial flips, resizing, and rotation, excluding color jitter and Gaussian blur because they alter image style.The two augmented views of each image are used in a contrastive SSL term, added because the supervised dataset is modest.Source: §5 · p. 5
- The final training loss sums the multi-label contrastive loss and SSL loss, weighted by λ; inference computes style similarity with the final-layer embedding dot product.The authors report that initializing with CLIP ViT-B and ViT-L weights improves performance.Source: §5 · p. 5
6 Results
CSD is evaluated for zero-shot style retrieval on DomainNet and WikiArt, against pretrained feature extractors and a prior style-attribution model. It outperforms the compared methods overall, while analyses show artist-level matching remains challenging and untrained humans trail many feature extractors.
6 Results
- Zero-shot retrieval tests whether a query’s style label appears among its nearest database neighbors, using Recall@k and mAP@k for k = 1, 10, and 100.The comparison includes GDA, VGG Gram Matrices, CLIP, DINO, MoCo, SSCD, and other pretrained feature extractors; WikiArt uses artist as a style proxy, while DomainNet excludes Quickdraw.Source: §6 · pp. 5–6
- CSD’s base model exceeds the next-best model in mAP@1 by 5 percentage points on WikiArt and 4.6 points on DomainNet; the large model reports similar margins.CSD performs best in all but one Table 1 comparison, and the prose reports consistent outperformance of pretrained extractors and GDA across both datasets.Source: Table 1 · p. 6, §6.1 · p. 6
6.1 Analyses and observations
- CSD’s improvements are attributed to multi-label style contrastive loss on curated LAION-Styles; authors hypothesize SSL can become style-invariant, so they use curated non-photometric augmentations to avoid stripping style information.WikiArt style-matching top-1 accuracy is 64.56; errors include confusion among impressionist painters and between Picasso and Braque, while most errors are concentrated along the movement-level heatmap diagonal.Source: §6.1 · p. 6
- WikiArt errors often preserve the art movement but miss the artist; some off-diagonal errors involve movements with related historical timelines or origins.The authors suggest exact artist-level accuracy may have a ceiling below 100%, even for a typical human evaluator, given the subtleties of the task.Source: §6.1 · pp. 6–7, Figure 3 · p. 7
6.2 Human Study
- In a survey of 30 untrained humans, participants performed worse than many feature extractors; SSCD alone did worse, and CSD beat all baselines.The survey used four choices per query, one from the same artist, making chance accuracy 25%; it collected three responses per item.Source: §6.2 · p. 7, Figure 4 · p. 7
7 Studying style in the wild: Analysis of Stable Diffusion
The section evaluates style retrieval on Stable Diffusion generations under different prompt settings and investigates which styles the model reproduces. CSD performs well across these settings, while qualitative and exploratory analyses examine style matches, artist combinations, and generalization to new subjects.
Creating synthetic style dataset
- The study compares synthetic collections from user-generated, simple artist-name, and content-constrained prompts against LAION-Styles matches.It uses Stable Diffusion 2.1 and generates 4,000 images for each prompt setting; user-generated prompts may carry multiple style labels.Source: §7 · pp. 7–8
Style retrieval on generated images
- CSD outperforms CLIP on simple prompts, is closer to CLIP on user-generated prompts, and beats both baselines in content-constrained scenarios.User-generated prompts are more complex and have a different label distribution; the authors note caveats for the quantitative study.Source: Table 2 · p. 7, Table 3 · p. 8, §7 · p. 8
Qualitative Results
- For content-constrained queries with different subjects, CSD matches style where CLIP often prioritizes image content.The reported examples include columns 1, 3, 4, 7, and 8.Source: Figure 6 · p. 9, §7 · p. 9
- CSD retrieves top-five style neighbors for generated images, and user-prompt matches often share colors, motifs, or textures with the query.Perceivable copying in user-generated prompts typically occurs when prompts are shorter and include elements characteristic of the artist’s typical content; green boxes mark a shared artist label.Source: Figure 5 · p. 8, §7 · p. 9
Does the model prioritize some artists over others in the prompt?
- In two-artist prompts, the higher-GSS artist’s style usually dominates, though Picasso prevailed over Klimt despite their small GSS difference.Reversing artist order generally preserved the result; the authors describe the study as preliminary rather than extensive.Source: Figure 7 · p. 9, §7 · p. 10
Which styles do diffusion models most easily emulate?
- For dog subjects, artist-level style diversity correlated 0.568 with the difference between general and content-constrained style similarity among artists with GSS above 0.7.The authors interpret this as suggesting that painters of diverse subjects are more likely to have their style replicated on out-of-distribution objects; dog was selected as an out-of-distribution subject.Source: §7 · p. 10, Figure 8 · p. 10
8 Conclusion
The paper presents a style-descriptor framework and reports strong style-matching performance across several datasets. Its Stable Diffusion analysis finds more style copying with complex prompts, while defining style through artist attribution as an operational choice rather than a definitive account.
8 Conclusion
- The framework learns style descriptors from labeled and unlabeled data, and its model achieves state-of-the-art performance on DomainNet, WikiArt, and LAION-Styles.Source: §8 · p. 10
- The Stable Diffusion analysis reports that style copying increases with prompt complexity, with complex prompts producing more copying than simple one-line prompts.Source: §8 · p. 11
- Style is defined through artist attribution for dataset construction, but the authors say this is not a definitive account and invite alternative or extended definitions.Source: §8 · p. 11
A Human Evaluation
The human evaluation recruited 30 participants, excluding the authors, to assess human-level style-matching performance.
A Human Evaluation
- Thirty volunteers with no prior task familiarity took part; the authors manually screened visuals and obtained exempt IRB status.Source: A Human Evaluation · p. 16
B List of artists in style analysis
The appendix lists the artists included in the style analysis.
B List of artists in style analysis
- The appendix names the artists used in the style analysis, including artists sourced from across different artistic traditions.Source: B List of artists in style analysis · p. 16
C Model Ablations
The appendix reports model and loss ablations, data and prompt construction, and observations about Stable Diffusion generations. It also cautions that noise in the dataset and model behavior limits interpretation.
C Model Ablations
- CLIP ViT-B initialization, combining SSL and multi-label contrastive losses, and τ = 0.1 each gave the best outcome in their respective WikiArt ablations.The loss comparison found the combined objectives best; the temperature comparison found τ = 0.1 best.Source: C Model Ablations · p. 16
D.1 LAION Deduplication
- Deduplication recursively removed connected images with SSCD inner-product similarity above 0.8, merging their labels into a master image and reducing the database from about 1.3 million to 0.5 million images.The procedure retained one image per connected graph.Source: §D.1 · pp. 16–17
E.0.1 Content-constrained templates.
- The content-constrained prompt templates vary subjects and contexts, including women, dogs, and houses.The templates include scenes such as a woman holding an umbrella, a dog playing in a field, and a house in a forest.Source: §E.0.1 · p. 17
E.0.2 Fig 5 prompts.
- The Figure 5 prompts pair four dog scenarios and four woman scenarios with named artists.Examples include a dog in Van Gogh's style and a woman holding an umbrella in Katsushika Hokusai's style.Source: §E.0.2 · pp. 17–18
E.0.3 First observations on synthetic datasets.
- For simple prompts, SD 2.1 borrowed artists’ contents along with styles: the Warhol-style painting prompt always produced a Marilyn Diptych version, while every Gustav Klimt prompt produced The Kiss, even across random seeds.The authors believed these associations reflected dataset memorization, as discussed in [57] [58].Source: §E.0.3 · p. 18
- When requested content diverged from an artist's conventional subject matter, Stable Diffusion sometimes ignored the content and generated artist-typical motifs instead.A woman prompt in Thomas Kinkade's style could produce cottages, streams, or gardens.Source: §E.0.3 · p. 18
E.0.4 Some Caveats.
- Interpretation is limited by noisy LAION captions and artist labels, as well as cases where Stable Diffusion does not follow the prompt's style or content.These factors can cause correct matches to be labeled incorrect and style matches to be classified as negatives.Source: §E.0.4 · p. 18
E.0.5 Qualitative results extended.
The extended qualitative results show Stable Diffusion generations alongside their nearest CSD matches. The authors caution that caption-derived ground-truth labels can misclassify stylistically similar matches.
E.0.5 Qualitative results extended.
- LAION-caption ground-truth labels may omit artist names, causing stylistically similar matches to appear as false positives.The authors note such mislabeling particularly in Figure 11.Source: §E.0.5 · p. 18
- The authors conclude that assessing style copying in Stable Diffusion remains challenging and that evaluation methods need further refinement.Source: §E.0.5 · p. 18