Source-linked AI summary
Exploring the structure of a real-time, arbitrary neural artistic stylization network
Golnaz Ghiasi, Honglak Lee, Manjunath Kudlur, Vincent Dumoulin, Jonathon Shlens
TL;DR
The paper addresses the tension between flexible arbitrary style transfer and the speed of learned style-transfer networks. It predicts conditional normalization parameters directly from a style image, trains on a large corpus, and reports real-time transfer to unseen paintings alongside a structured semantic embedding space.
Problem
Existing fast style-transfer methods were computationally efficient but usually required a separate network for each style and generalized poorly to unseen styles.
Method
A style prediction network maps an arbitrary style image to the normalization-parameter embedding used by a style transfer network.
Results
The large-scale model generalizes real-time stylization to paintings never previously observed and produces a low-dimensional embedding capturing semantic properties of paintings.
Takeaways & Limitations
The learned embedding supports exploration of an artist’s artistic range and allows users to dial in stylization strength through interpolation.
Takeaways & Limitations
The painting embedding captures only a portion of the semantic information available, motivating future use of metadata to refine it.
Abstract
from arXiv · showhide
In this paper, we present a method which combines the flexibility of the neural algorithm of artistic style with the speed of fast style transfer networks to allow real-time stylization using any content/style image pair. We build upon recent work leveraging conditional instance normalization for multi-style transfer networks by learning to predict the conditional instance normalization parameters directly from a style image. The model is successfully trained on a corpus of roughly 80,000 paintings and is able to generalize to paintings previously unobserved. We demonstrate that the learned embedding space is smooth and contains a rich structure and organizes semantic information associated with paintings in an entirely unsupervised manner.
1 Introduction
Artistic style transfer progressed from computationally expensive optimization to faster learned networks, but early fast methods sacrificed flexibility and generalization. This work introduces a real-time arbitrary style-transfer network trained at scale, with an embedding space intended to capture painting semantics and styles unseen during training.
- Prior work: Traditional texture-synthesis methods transferred visual texture to photographs, while neural style transfer matched content and style constraints in classifier feature space.These approaches established the basis for conferring artistic appearance on arbitrary photographs.
- Prior work: Fast style-transfer networks improved computational speed but typically required a separate trained network for each painting style, limiting flexibility and shared representation.Multi-style conditional-normalization networks reduced this restriction but covered only 32 styles and generalized poorly to unseen styles.
- Contributions: The proposed network is trained on about 80,000 paintings and 6,000 visual textures to transfer arbitrary artistic styles in real time, including styles never previously observed.The paper presents this large-scale training corpus as enabling generalization to unobserved painting styles.
- Contributions: The model represents painting styles in a compact embedding space that captures features of painting semantics.The embedding is formed from style-dependent normalization parameters and is designed to support analysis of artistic structure.
- Contributions: The architecture combines a style prediction network that estimates normalization parameters from a style image with a style transfer network that stylizes the content photograph.Figure 2 describes the style prediction and transfer components and their VGG-based content and style losses.
2 Methods
The method formulates style transfer through content and style constraints, then uses conditional normalization and a learned style predictor to enable fast stylization of unseen styles.
- Artistic style transfer creates a stylized image x from content image c and style image s, typically a photograph and painting.
- Content similarity is measured using high-level classifier features, while style similarity is represented by spatial statistics of lower-level features.
- The optimization objective combines content loss Lc(x, c) and style loss Ls(x, s), weighted by the Lagrange multiplier λs.
- Style transfer networks replace expensive image optimization with an encoder/decoder that learns the transformation from content image c to rendered image x̂.
- Conditional instance normalization assigns style-specific normalization parameters, whose concatenation forms a roughly 3000-dimensional embedding for each painting style.
- The proposed style prediction network maps an arbitrary style image to its normalization embedding, allowing stylization of unseen styles at test time.
- Unlike fixed heuristic mappings, the method learns the mapping from style images to normalization parameters directly.
3 Results
The model performs real-time arbitrary stylization on observed and previously unseen paintings, with generalization improving as training diversity increases. Its embedding supports both flexible stylization and organization by visual and semantic similarity.
- Training setup: 79,433 labeled paintings from PBN and 5,640 textures from DTD were used as training style corpora.Content images came from ImageNet, and training images were augmented by flipping, rescaling, cropping, and hue or contrast changes.
- Training setup: A single style weighting produced reasonable stylizations across paintings and textures, without selecting a separate λ_s for each style.The networks were trained jointly on the DTD and PBN datasets.
- Generalization to unobserved paintings: Observed and unobserved PBN styles had similar losses: style 2.08e4 vs 1.95e4 and content 8.92e4 vs 8.94e4.The DTD-trained model also produced similar stylizations on unobserved PBN paintings, with style loss 2.67e4 ± 6.49e4 and content loss 8.76e4 ± 3.55e4.
- Generalization to unobserved paintings: On unobserved styles, the proposed method achieved losses comparable to direct optimization: style 1.95e4 vs 1.12e4 and content 8.94e4 vs 9.09e4.The comparison evaluates the proposed method against the optimization-based method on style and content objectives.
- Generalization to unobserved paintings: Compared with AdaIN, the proposed method achieved lower style loss 1.95e4 vs 2.56e4 and content loss 8.94e4 vs 12.3e4.Paired t-tests reported p-values of 1.9×10^-9 for style and 0.0 for content differences.
- Generalization to unobserved paintings: Style loss on unobserved paintings correlated positively with Gram-matrix proximity to training examples, with r^2 = 0.9.Unobserved images more similar to training examples tended to have lower style loss.
- Scaling to large numbers of paintings: For unobserved styles, the style-loss distribution did not asymptote until roughly 16,000 paintings, after which losses roughly matched those for trained styles.Content loss remained largely preserved across training-set sizes, while increasing painting count chiefly improved unseen-style generalization.
- Embedding space: The 2,758-dimensional style embedding is predicted from a 100-dimensional bottleneck representation.The embedding represents artistic and texture styles used by the style transfer model.
4 Conclusions
The method enables fast arbitrary artistic style transfer and generalizes to previously unobserved paintings. Its embedding supports semantic exploration of artistic range and controllable stylization, while leaving several refinements for future work.
- The model performs fast arbitrary artistic style transfer and generalizes to paintings never previously observed.
- Generalization is largely predictable from the proximity of an unobserved style to styles represented during training.
- The embedding captures semantic properties of paintings and represents an artist’s artistic range and vocabulary in a low-dimensional space.
- Interpolation through the embedding lets users arbitrarily control artistic stylization strength.
- Future work includes refining semantic information with metadata, improving visual quality, preserving photograph color or spatial regions, and enforcing temporal consistency in video.
- Improved stylization may help transfer models trained in simulation toward real-world domains where data is limited.
A.1 Variety of stylizations in trained model.
The trained model produces a broad mosaic of stylizations across 2592 paintings, applying those styles to an assortment of eight content images.
- 2592 paintings provide the styles in a mosaic of model-generated stylizations.
- Each painting is shown alongside stylizations applied across eight content images.
A.2 Structure of a low-dimensional representation of the embedding space
The paper visualizes low-dimensional t-SNE embeddings learned from paintings and visual textures, alongside versions in which inputs are replaced by metadata.
- Figure 2 shows a low-dimensional t-SNE style embedding for paintings from the PBN dataset across 20 painters.
- For both datasets, the bottom visualization replaces each painting or texture with metadata describing its artist or texture.
- Figure 3 shows a low-dimensional t-SNE style embedding for the DTD visual textures dataset across 10 categories.
A.3 Qualitative comparison with other methods
The paper compares its PBN-trained method with AdaIN, Gatys et al., and deepart.io across many style and content images.
- The comparison includes the proposed method trained on PBN images, AdaIN, Gatys et al., and deepart.io.
- The methods are evaluated across many combinations of style and content images.
A.4 Generalization of the model across training datasets
The model generalizes within each style-image domain and maintains similar stylization-loss distributions when trained on painting or texture datasets and evaluated across domains.
- The experiments test within-domain generalization and cross-domain similarity between models trained on painting and texture images.
- Within a domain, training-image and test-image style-loss distributions closely match, suggesting good generalization to unseen test images.
- Across datasets, style-loss distributions remain very similar when painting-trained and texture-trained models are evaluated on the same test images.
- Figure 4 summarizes style and content-loss distributions for training images, same-dataset tests, and cross-dataset tests.
A.4.1 Qualitative comparisons of generalization across datasets.
The appendix provides qualitative comparisons of models trained on PBN and DTD images using test style images from both datasets.
- Qualitative comparisons use models trained on PBN and DTD images with style images drawn from PBN and DTD test data.
- The figures present qualitative examples rather than reported numerical measurements.
- The comparison materials label the paired inputs as content image and style image.
A.5 Expanded analysis on the generalization to unobserved paintings
The expanded analysis relates generalization on unobserved paintings to their proximity to the training corpus in Gram-matrix space, then contrasts easier and harder examples qualitatively.
- Style loss correlates with the minimum L2 distance between an unobserved painting’s Gram matrix and the training Gram matrices.
- Figure 5 uses a log-log scatter plot, with each point representing a different test style image.
- Test styles near training examples tend to have relatively small style loss, whereas distant styles tend to have relatively large style loss.
- Yellow points denote easier examples, while red points denote difficult examples in the Figure 5 analysis.
- For difficult cases, test-style and nearest-training-style stylizations appear more perceptually different in color and texture.
- The authors hypothesize that difficult cases may lie in low-density embedding-space regions, leaving further analysis for future work.