Source-linked AI summary
Unsupervised Learning of Artistic Styles with Archetypal Style Analysis
Daan Wynen, Cordelia Schmid, Julien Mairal
TL;DR
Style-transfer research has emphasized transforming images, while this paper addresses discovering and interpreting styles across painting collections. It learns archetypes from deep image representations and decomposes image styles into sparse convex combinations. The resulting coefficients support interpretation and style manipulation, though linking learned styles to art metadata remains future work.
Problem
The paper seeks to discover, summarize, and interpret artistic styles in large collections, rather than only improve style transfer.
Source: §1 · p. 2Method
Results
Image styles can be interpreted through their archetype proportions, and changing those coefficients produces style transfer, enhancement, and recombination effects.
Source: §1 · p. 2Takeaways & Limitations
The dual convex-combination view makes archetypes and image-style decompositions interpretable and allows coefficient-based style manipulation.
Source: §5 · p. 9Linking learned styles to metadata such as periods, schools, and art movements is identified as future work.
Source: §5 · p. 9
Abstract
from arXiv · showhide
In this paper, we introduce an unsupervised learning approach to automatically discover, summarize, and manipulate artistic styles from large collections of paintings. Our method is based on archetypal analysis, which is an unsupervised learning technique akin to sparse coding with a geometric interpretation. When applied to deep image representations from a collection of artworks, it learns a dictionary of archetypal styles, which can be easily visualized. After training the model, the style of a new image, which is characterized by local statistics of deep visual features, is approximated by a sparse convex combination of archetypes. This enables us to interpret which archetypal styles are present in the input image, and in which proportion. Finally, our approach allows us to manipulate the coefficients of the latent archetypal decomposition, and achieve various special effects such as style enhancement, transfer, and interpolation between multiple archetypes.
1 Introduction
The paper shifts from improving style transfer to unsupervised discovery and interpretation of styles across painting collections. It uses archetypal analysis to represent image styles and manipulate them through their latent coefficients.
1 Introduction
- Style transfer methods model deep-feature statistics or use feed-forward networks for single or multiple styles, while this work instead learns style representations from collections.The paper frames its goal as distinct from prior work focused on improving style transfer.Source: §1 · pp. 1–2
- Archetypal analysis, related to sparse coding [13] and non-negative matrix factorization, is chosen for its interpretability.The paper identifies interpretability as important for applied machine learning requiring model interpretation.Source: §1 · p. 2
- Each image style can be decomposed into proportions of archetypes, yielding interpretations richer than clustering and sometimes revealing interpolations between styles.Archetypes themselves are convex combinations of image-style representations and can be visualized.Source: §1 · p. 2
- Changing archetypal coefficients enables style transfer, enhancement of an existing style, and stylizations formed by recombining styles.The latent representation can create styles not necessarily seen in the dataset.Source: §1 · p. 2
- Unlike the latent space in [7], which supports generalization of a style-transfer network, this work seeks an interpretable dimension for each archetypal style.The paper describes [7] as the closest work in latent-space style description while distinguishing its tools and objective.Source: §1 · p. 2
2 Archetypal Style Analysis
The method represents painting styles with deep-feature statistics, reduces their dimensionality, and learns archetypes under convex-combination constraints. Sparse codes then describe both paintings and archetypes, supporting interpretation of new images.
2 Archetypal Style Analysis
- Figure 1 depicts an image as a convex combination of archetypes and shows archetypes as textures, artwork combinations, or stylized versions of the input.Free recombination of archetypal styles allows novel stylizations.Source: Figure 1 · p. 3
- The style descriptor concatenates feature-map statistics, then SVD reduces the vectors to 4,096 dimensions while retaining more than 99% of variance.The statistics are normalized by layer parameter count to prevent layers with more parameters from being over-represented.Source: §2 · p. 3
- Archetypal analysis approximates each sample as a convex combination of archetypes, while constraining each archetype to the convex hull of the data.The archetypes are themselves convex combinations of input samples.Source: §2 · p. 3
- Simplex constraints yield non-negative sparse codes, associating each sample with a few archetypes and each archetype with a few samples.The sample-code constraint has unit ℓ1-norm, which has a sparsity-inducing effect.Source: §2 · pp. 3–4
- The method typically learns 32 to 256 archetypes and represents each painting with a sparse low-dimensional code; a new image is encoded to quantify archetype presence.Archetypes are associated with a few input paintings, aiding interpretation.Source: §2 · p. 4
3 Archetypal Style Manipulation
The method uses convex combinations of archetypal style statistics to construct target styles for transfer, interpolation, and enhancement. Its layerwise transfer update also reuses the original content image to better preserve details when stylization is partial.
3 Archetypal Style Manipulation
- The procedure assumes paired encoders and decoders, with each decoder approximately inverting its encoder’s selected VGG feature map.The paper makes this assumption for this universal style transfer variant.Source: §3 · p. 4
- Universal style transfer matches content feature maps to a style image’s means and covariances through whitening and coloring, summarized as an affine transformation.The content features are processed coarse-to-fine, one network layer at a time.Source: §3 · p. 4
- The layerwise update combines a refinement of the preceding stylization with a separate stylization of the original content image.γ controls stylization amount, while δ controls how much to trust the preceding stylized image’s content information.Source: §3 · p. 4
- When γ < 1, the proposed update can preserve content details much better by explicitly reusing the original content image across layers.The update balances refinement of the current stylized image against stylizing the original content image at each layer.Source: §3 · pp. 4–5
- An input style is approximated by a convex combination of archetypes, whose combined statistics define a coloring operation for transfer, interpolation, or enhancement.A single selected archetype gives transfer; a convex combination gives interpolation; modifying the input decomposition enhances a style already present.Source: §3 · p. 5
4 Experiments
Experiments visualize learned archetypes and their image decompositions, then show how their coefficients support controllable style manipulation. The comparison with [11] reports improved preservation of content details, while visualizations also reveal interpretable patterns and some difficult-to-interpret decompositions.
4 Experiments
- GanGogh contains 95997 WikiArt artworks and uses 256 archetypes, while van Gogh’s 1154 paintings and drawings use 32; GanGogh’s style representation is biased.The GanGogh collection excludes artworks that are not paintings, and the van Gogh collection excludes works such as sketches and studies.Source: §4 · p. 5
4.1 Archetypal Visualization
- Strong archetype contributors usually share a stroke style or color choice, while one archetype consists only of portraits, highlighting content-style correlation.Figure 2 shows each archetype’s texture and its strongest contributing paintings in descending contribution order.Source: §4.1 · p. 6, Figure 2 · p. 6
- The 2D t-SNE embedding places Baroque and Renaissance styles in the middle and abstract and cubist styles on the right.Each of the 256 GanGogh archetypes is represented by a synthesized texture.Source: §4.1 · p. 6, Figure 3 · p. 7
- Image-style decompositions typically have only a few strong archetype contributions, but some are non-sparse and difficult to interpret.Some decompositions are trivial when a single archetype describes the image style well.Source: §4.1 · p. 6, Figure 4 · p. 7
4.2 Archetypal Style Manipulation
- Increasing archetype weights produces gentle style changes, with the middle panel near the original image and stronger emphasis toward the outer panels.The middle panel uses γ = δ = 0.5; the outer panels increase these parameters to 0.95 while emphasizing different prominent components.Source: §4.2 · p. 7, Figure 6 · p. 8
- Users can select archetypes unrelated to the input and interpolate their convex combinations to produce images akin to classical style-transfer results.The paper illustrates this free manipulation on “The Bitter Drunk” by Adriaen Brouwer.Source: §4.2 · p. 7, Figure 7 · p. 9
- The modified update improves content-detail preservation over [11]; Figure 5 shows [11] artifacts especially at γ = δ = 0, attributed to repeated lossy encoding and decoding.At γ = δ = 1, the approaches produce equal outputs.Source: §4.2 · p. 6, Figure 5 · p. 8
5 Discussion
Archetypal style analysis identifies styles in artwork collections without supervision and supports artwork and photo manipulation. Linking learned styles to WikiArt metadata remains future work, which the authors believe could extend their usefulness beyond style manipulation.
5 Discussion
- Archetypes are interpretable as convex combinations of input image styles, while image styles are represented as convex combinations of archetypes.Source: §5 · p. 9
- Archetypal coefficients can be used to manipulate the styles of artworks and photos.Source: §5 · p. 9
- Linking learned styles to WikiArt metadata such as period, schools, and art movement is identified as future work.The authors believe these links would make the styles more useful beyond style manipulation.Source: §5 · p. 9