Source-linked AI summary
Controlling Perceptual Factors in Neural Style Transfer
Leon A. Gatys, Alexander S. Ecker, Matthias Bethge, Aaron Hertzmann, Eli Shechtman
TL;DR
Neural Style Transfer lacked intuitive access to perceptual style factors such as spatial region, colour, and scale. This paper introduces independent controls for these factors, improving stylisation, enabling interpretable style combinations, and reducing high-resolution optimisation time by approximately 2.5×.
Problem
Neural Style Transfer needs intuitive access to perceptual factors in image-style statistics, including spatial region, colour, and scale.
Method
The paper controls Neural Style Transfer independently across spatial regions, colour and luminance, and spatial scales, including spatial guidance channels for regional control.
Results
The controls improve stylisation, alleviate common failure cases, combine styles from multiple images, apply to fast approximations, and reduce high-resolution optimisation time by approximately 2.5×.
Takeaways & Limitations
Factorising style into space, colour, and scale provides interpretable control for producing flexible, high-quality stylisations and new combinations of existing styles.
Takeaways & Limitations
Selecting inputs that combine into aesthetically pleasing new styles remains challenging, motivating prediction of suitable style combinations as an open research question.
Abstract
from arXiv · showhide
Neural Style Transfer has shown very exciting results enabling new forms of image manipulation. Here we extend the existing method to introduce control over spatial location, colour information and across spatial scale. We demonstrate how this enhances the method by allowing high-resolution controlled stylisation and helps to alleviate common failure cases such as applying ground textures to sky regions. Furthermore, by decomposing style into these perceptual factors we enable the combination of style information from multiple sources to generate new, perceptually appealing styles from existing ones. We also describe how these methods can be used to more efficiently produce large size, high-quality stylisation. Finally we show how the introduced control measures can be applied in recent methods for Fast Neural Style Transfer.
1. Introduction
The paper hypothesizes that image style can be factored into spatial, colour, and scale dimensions, then adds controls for these factors to Neural Style Transfer. These controls address failure cases, combine styles from multiple images, and improve high-resolution rendering efficiency.
- Image style is hypothesized to factor into spatial regions, colour and luminance information, and spatial scales.
- The paper introduces independent controls for spatial regions, colour and luminance, and spatial scales within Neural Style Transfer.
- The controls alleviate common stylisation failures and enable perceptually interesting styles by combining style information from multiple images.
- 2.5 approximate factor: a coarse-to-fine approach reduces optimisation time for high-resolution stylisations.
- The introduced controls also apply to recent fast approximations of Neural Style Transfer.
2. Related Work
Prior stylisation methods provide varying degrees of control, while Neural Style Transfer offers flexible new structures but lacks intuitive manipulation. This work targets that gap by making neural style control more fine-grained.
- Image Analogies introduced example-based stylisation, colour preservation, regional control, and coarse-to-fine texture synthesis.
- Procedural stylisation methods offer separate spatial, colour, and brush-stroke controls but cannot work from training data.
- Neural Style Transfer uses CNN summary statistics and can create image structures absent from the source images.
- Neural Style Transfer provides less intuitive control because complex interactions among deep texture parameters make perceptual effects difficult to predict.
- Concurrent work independently developed a similar approach for spatial control.
3. Neural Style Transfer
Neural Style Transfer optimizes an image to preserve content features while matching style statistics from a separate image. Its style representation uses Gram matrices computed from CNN feature responses across selected layers.
- Neural Style Transfer defines content and style images through CNN feature representations at layer ℓ.
- The generated image depicts the content image in the style of the style image by minimizing a loss with respect to the generated image.
- The content term compares feature maps at a single content layer ℓC.
- The style term compares a set of summary statistics across selected CNN layers.
- The Gram matrix Gℓ(x) = 1/Mℓ(x) Fℓ(x)^T Fℓ(x) summarizes feature-map correlations at layer ℓ.
4. Spatial Control
Spatial guidance aligns style transfer across corresponding image regions, helping avoid mismatched textures and combine styles from multiple sources. The section presents guided Gram-matrix and guided-sum approaches, with the latter trading texture quality for efficiency.
- Guidance setup: Spatial guidance channels specify which style-image regions should stylise corresponding content-image regions, supporting separate control over sky and ground.Channels may be binary or real-valued, and their layer-specific versions guide the style representation.
- Guided Gram Matrices: Guided Gram matrices compute a separate style target for each guidance region and weight its contribution with λr.Each target is used for the corresponding content region during optimisation.
- Guidance setup: Matching sky and ground regions between content and style images addresses cases where unconstrained transfer applies ground structures to the sky.The method is motivated by a content image with bright clouds and a style image with greyish clouds.
- Guidance propagation: Naively downsampling guidance masks can fail near region boundaries because neurons may have receptive fields spanning multiple regions.The paper instead describes eroded guidance channels plus a global channel for softer guidance.
- Guided Gram Matrices: Spatial guidance also combines styles from multiple example images, such as using one image’s sky style and another image’s ground style.The paper presents this as a way to generate new stylisations from multiple sources.
- Guided Sums: A guided-sum alternative uses one Gram matrix with additional guidance channels, making it more efficient than computing R separate Gram matrices.The efficiency gain comes at the expense of texture quality because the resulting model has limited capacity for complex textures.
5. Colour Control
The paper introduces two ways to control colour independently during Neural Style Transfer: luminance-only transfer and colour transformation of the style image. These methods preserve content colours while retaining stylisation, but involve different trade-offs.
- Motivation: Colour is treated as an independently controllable perceptual style factor because standard Neural Style Transfer copies the style image’s colour distribution.The authors motivate colour preservation for cases such as stylising a farmhouse while retaining the photograph’s colours.
- Luminance-only transfer: Luminance-only transfer applies Neural Style Transfer to the extracted luminance channels, then recombines the output luminance with the content image’s colour information.The method uses a colour space separating luminance from colour.
- Luminance-only transfer: Before luminance transfer, the style luminance histogram can be adjusted to match the content luminance by aligning their means and standard deviations.The adjustment is recommended when the style and content luminance histograms differ substantially.
- Colour histogram matching: Colour matching transforms the style image’s colours to match the content image, producing a new style image for otherwise unchanged Neural Style Transfer.The paper uses linear transformations whose parameters match RGB means and covariances.
- Colour histogram matching: Colour matching works reasonably well with Neural Style Transfer and can also preserve the style image’s colour distribution when that distribution is desirable.The latter use is illustrated for strong colour mismatches, including pencil drawings or line-art styles.
- Comparison: The colour-matching method can mismatch the content colours when the colour distributions cannot be matched perfectly.Luminance-only transfer preserves content colours perfectly but can lose dependencies between luminance and colour channels, sometimes affecting prominent brushstrokes.
6. Scale Control
The paper separates style across spatial scales to mix fine and coarse characteristics, then applies the same principle to high-resolution stylisation. This produces recombined styles and makes high-resolution optimisation more efficient.
- Scale control for style mixing: Scale control targets separate styles for fine- and coarse-scale image structures.The method defines style at a scale through image structures in neighbourhoods of a corresponding size.
- Scale control for style mixing: Gram Matrices from different CNN layers capture overlapping spatial-scale information, so directly combining them does not provide independent scale control.A Gram Matrix at one layer also captures information from smaller scales represented at lower layers.
- Scale control for style mixing: The method creates a new style image by combining fine-scale information from one image with coarse-scale information from another before applying Neural Style Transfer.It uses lower-layer Gram Matrices, initialises with the coarse-style image, and omits the content loss when constructing the combined style.
- Scale control for style mixing: Scale recombination generates new styles, including combinations of brush strokes with angular or round coarse structures.Figure 4 illustrates Style IV and Style V as recombinations of fine and coarse scales from existing styles.
- Scale control for efficient high resolution: 2.5 times fewer iterations were used for high-resolution optimisation with the coarse-to-fine procedure.The procedure first stylises at a target lower resolution, then upsamples and optimises at high resolution; repeated application supports very high-resolution outputs subject to input size and memory.
7. Controlling Fast Neural Style Transfer
The paper extends spatial and colour control to feed-forward Fast Neural Style Transfer, using luminance handling and guidance channels to control output appearance and regional styles.
- Controlling Fast Neural Style Transfer: Spatial and colour control can be applied to feed-forward Fast Neural Style Transfer methods.Scale control is applied by training on a style image that combines multiple scales.
- Colour control: Colour preservation combines the stylised luminance channel with the content image’s colour channels.An alternative trains the feed-forward network only on luminance channels, and both methods match output mean luminance to the content image.
- Spatial control: Regional style control adds two binary guidance channels to a style image formed by vertically concatenating the Candy and Feathers styles.The guidance channels identify the image regions used during spatially guided loss evaluation.
- Controlling Fast Neural Style Transfer: Figure 6 compares standard outputs, two colour-preservation methods, vertical and horizontal style separation, and person-background separation.The figure includes luminance-channel replacement, a luminance network, and region-specific stylisation configurations.
8. Discussion
The discussion presents space, colour, and scale as perceptual factors that make Neural Style Transfer more controllable. It also identifies style selection for aesthetically pleasing combinations as an open challenge.
- Discussion: The paper introduces intuitive controls over space, colour, and scale to improve Neural Style Transfer’s quality and flexibility.These controls are motivated by the hypothesis that image style includes these factors.
- Discussion: Factorising style into perceptual aspects enables interpretable combinations of styles from multiple images.The paper contrasts this with linearly interpolating in a style representation whose directions may not correspond to perceptual variables.
- Discussion: Choosing style inputs that combine into aesthetically pleasing new styles remains challenging.Predicting which style combinations will work well is identified as an open research question.
- Discussion: The perceptual effect of Neural Style Transfer’s underlying texture parameters is difficult to predict because their interplay is complex.The paper frames intuitive control as a way to combine Neural Style Transfer’s flexibility with finer-grained control.