Source-linked AI summary
A Learned Representation For Artistic Style
Vincent Dumoulin, Jonathon Shlens, Manjunath Kudlur
TL;DR
Existing style-transfer systems are flexible or fast but are limited by expensive optimization or models tied to individual styles. The paper introduces conditional instance normalization in a single scalable network to learn multiple styles and represent them in an embedding space. It reports comparable stylization quality to independently trained models and enables faster capture and novel combinations of styles.
Problem
Existing feedforward style-transfer networks require a separate network for each painting style, creating a memory problem for devices such as smartphones and discarding shared visual computation.
Method
The paper modifies feedforward style transfer with conditional instance normalization, conditioning normalized activations on style so one network can model multiple paintings.
Results
The multi-style network produces stylizations qualitatively similar to independently trained single-style models and represents styles as points in an embedding space that can be combined arbitrarily.
Takeaways & Limitations
A shared style representation can capture multiple artistic styles while supporting compact transmission of new styles to mobile devices and novel style combinations.
Takeaways & Limitations
The approach addresses the single-purpose limitation of feedforward networks, but the paper’s introduction identifies that limitation as remaining in prior work rather than reporting a limitation of its own method.
Abstract
from arXiv · showhide
The diversity of painting styles represents a rich visual vocabulary for the construction of an image. The degree to which one may learn and parsimoniously capture this visual vocabulary measures our understanding of the higher level features of paintings, if not images in general. In this work we investigate the construction of a single, scalable deep network that can parsimoniously capture the artistic style of a diversity of paintings. We demonstrate that such a network generalizes across a diversity of artistic styles by reducing a painting to a point in an embedding space. Importantly, this model permits a user to explore new painting styles by arbitrarily combining the styles learned from individual paintings. We hope that this work provides a useful step towards building rich models of paintings and offers a window on to the structure of the learned representation of artistic style.
1 INTRODUCTION
Style transfer aims to preserve an image’s content while imitating another image’s style, but existing feedforward networks require a separate model for each style. This work introduces a scalable network that learns multiple styles, represents them in an embedding space, and combines them in novel ways.
- Background: Style transfer renders an image in another image’s style while preserving some notion of content.It is closely related to texture synthesis, which models pixel statistics without the same content-preservation goal.
- Prior work: Optimization-based style transfer is flexible but expensive because it repeatedly carries out an optimization loop.Classifier features guide comparisons between synthesized, content, and style images.
- Prior work: Feedforward style transfer networks reduce test-time cost by producing a pastiche in one pass, but each network is tied to one style.Consequently, separate networks must be trained for every style being modeled.
- Motivation: Separate per-style networks waste shared visual regularities and create practical memory constraints for applications such as mobile processing.The paper also frames cross-style generalization as a measure of how parsimoniously systems capture higher-level image features.
- Contribution: Conditional instance normalization allows one style transfer network to learn multiple styles while remaining comparable to single-purpose networks in quality and convergence.The model reduces each style image to a point in an embedding space.
- Contribution: The embedding representation permits arbitrary combinations of learned artistic styles, producing novel pastiches.The approach is illustrated with a network trained on 32 varied styles.
2 STYLE TRANSFER WITH DEEP NETWORKS
Style transfer seeks a pastiche that preserves a content image while matching a style image, using classifier features to define content and style similarity. Feed-forward networks accelerate this process, and conditional instance normalization extends one network across multiple styles with most weights shared.
- Style transfer: Style transfer produces a pastiche whose content resembles a content image and whose style resembles a style image.The task is closely related to texture synthesis but additionally seeks to preserve content.
- Style transfer: Content similarity is measured by Euclidean distance between high-level classifier features, while style similarity uses Gram-matrix statistics of low-level features.These definitions operationalize content as higher-level abstraction and style as visual texture.
- Style transfer: The original neural style algorithm optimizes a weighted sum of style and content losses over the output image.The losses are computed from selected style and content layers of a trained classifier, with λc set to 1.0 in practice and λs left free.
- Feed-forward networks: Feed-forward style transfer networks learn a direct content-to-pastiche transformation, but conventional networks require a separate model for every style.That one-style-per-network design creates a substantial memory burden on devices such as smartphones.
- N-styles networks: Conditional instance normalization trains one conditional network using shared convolutional weights and style-specific normalization scales and shifts.The input activation is normalized across spatial dimensions, then transformed with γs and βs selected for style s.
- N-styles networks: The multi-style design requires O(N × L) style parameters, and adding an N + 1th style is cheap because only a small number of parameters must be trained.In a typical network, roughly 1.6M parameters are used, with about 3K specifying individual styles.
3 EXPERIMENTAL RESULTS
Experiments show that one conditional style-transfer network can model multiple artistic styles with mostly shared parameters, retain quality comparable to separate models, learn new styles efficiently, and interpolate between styles.
- Multiple styles in one network: A 10-styles network captured distinct color palettes and textures across Monet paintings while sharing 99.8% of its parameters.Only 0.2% of parameters were unique to each painting style.
- Comparison with independent models: The 10-styles network converged as quickly as single-style networks on style loss but lagged slightly on content loss.The comparison averaged losses over 32 random content-image batches.
- Comparison with independent models: The 10-styles network had 8.7 ± 3.9% higher content loss and 8.9 ± 16.5% lower, insignificant style loss than single-style counterparts.Training separate networks requires N times more parameter updates than training an N-styles network.
- Comparison with independent models: Pastiches from the 10-styles network were qualitatively similar to those produced by independently trained single-style networks.This supports comparable stylization quality despite consolidating styles into one model.
- Generalization across styles: The model represented 32 highly varied artistic styles despite substantial differences in color palette and spatial scale.The evaluation used artworks selected for stylistic diversity.
- Efficient learning and style representation: After training, learning only new γ and β parameters incorporated an additional style much faster than training a new network from scratch.The fine-tuned model produced comparable pastiches after eight times fewer parameter updates.
- Style interpolation: The learned γ and β style embeddings supported convex combinations that blended distinct styles smoothly and produced monotonically varying style losses.The N-styles network can combine multiple styles through interpolated normalization parameters.
4 DISCUSSION
The discussion interprets conditional style representations, notes complementary directions, and outlines possible extensions toward direct style-conditioned generation and generative models of artistic movements.
- 4 DISCUSSION: The learned style representation may reflect style elements whose expression is modulated by scaling and shifting factors, but this hypothesis remains unverified.Pruning produces qualitatively similar results, offering an alternative interpretation involving unused architectural capacity.
- 4 DISCUSSION: Loss revisions that control spatial scale, color information, and spatial localization are described as complementary avenues for studying artistic style representation.
- 4 DISCUSSION: A transformer could generate a pastiche directly from style and content images, avoiding an individual conditional embedding and test-time optimization.
- 4 DISCUSSION: Generative models could sample style embeddings from an artistic movement and use them with the style transfer network to produce random pastiches.The proposed example models enough paintings from a movement such as impressionism to learn a collection of style embeddings.
- 4 DISCUSSION: The work concludes that conditional instance normalization enables scalable multi-style modeling, compact style transmission, style combination, and flexible capture of diverse painted styles.The authors report little impact on training time and final performance relative to the network’s flexibility.
APPENDIX
The appendix includes a table of style transfer network hyperparameters and identifies a broad collection of paintings used or presented alongside the study.
- APPENDIX: Table 1 is titled “Style transfer network hyperparameters.”