Source-linked AI summary
Fast Patch-based Style Transfer of Arbitrary Style
Tian Qi Chen, Mark Schmidt
TL;DR
Artistic style transfer must reproduce one image’s content using another image’s style, while prior approaches are either slow optimizations or fast but style-limited feed-forward networks. The paper introduces local patch matching in a single CNN layer and trains an inverse network to approximate it, yielding efficient transfer for arbitrary content and styles. The method also supports intuitive tuning and frame-by-frame video application, although its simplifications can reduce quality and cause occasional flickering.
Problem
Existing style-transfer methods are either expensive optimization procedures that accept arbitrary styles or efficient feed-forward networks limited to trained styles.
Method
The method swaps content-image patches with style-image patches in one CNN activation layer and trains an inverse network to reconstruct the stylized image.
Results
The inverse method generalizes beyond its training styles and allows changing the style-image size without retraining the inverse network.
Takeaways & Limitations
The approach combines efficiency with arbitrary content and style images, while supporting intuitive patch-size tuning and frame-by-frame video stylization.
Takeaways & Limitations
The method lacks global style and neighboring-patch similarity measures, sacrificing quality for efficiency and occasionally causing local video flicker.
Abstract
from arXiv · showhide
Artistic style transfer is an image synthesis problem where the content of an image is reproduced with the style of another. Recent works show that a visually appealing style transfer can be achieved by using the hidden activations of a pretrained convolutional neural network. However, existing methods either apply (i) an optimization procedure that works for any style image but is very expensive, or (ii) an efficient feedforward network that only allows a limited number of trained styles. In this work we propose a simpler optimization objective based on local matching that combines the content structure and style textures in a single layer of the pretrained network. We show that our objective has desirable properties such as a simpler optimization landscape, intuitive parameter tuning, and consistent frame-by-frame performance on video. Furthermore, we use 80,000 natural images and 80,000 paintings to train an inverse network that approximates the result of the optimization. This results in a procedure for artistic style transfer that is efficient but also allows arbitrary content and style images.
1. Introduction
Artistic style transfer combines the structural content of one image with the style of another, but existing methods trade off computational efficiency against support for arbitrary styles. The paper proposes a one-layer objective and inverse network to make style transfer both efficient and adaptable.
- Motivation: Artistic style transfer transfers the desired style from one image onto the structural content of another.The problem is motivated by the time required to create artwork and by computational strategies for synthesizing artistic images.
- Background: CNN-based methods achieve visually appealing style transfer by using convolutional neural networks for feature extraction.These methods have also supported mobile applications that stylize user-provided images on demand.
- Problem: Existing methods are either expensive optimization procedures or fast feed-forward networks limited to their trained styles.Optimization can take minutes, while feed-forward networks cannot generalize beyond their trained set of style images.
- Contribution: The proposed method combines content and style information in a single CNN layer and uses an inverse network to produce stylized images.Restricting the objective to one layer enables deterministic inversion of stylized activations.
- Contribution: The paper develops a one-layer optimization objective and trains a neural network to invert its activations, followed by experiments on the method’s properties.The objective combines style and content statistics in one activation layer, while the inverse network yields an image from those activations.
2. Related Work
Prior work framed style transfer as multi-layer optimization, accelerated it with style-specific feed-forward networks, or added temporal losses for video. This paper instead constructs target activations directly in one layer and inverts them with a feed-forward network that supports arbitrary content and styles.
- Style Transfer as Optimization: Gatys et al. formulate style transfer as optimization combining texture synthesis and content reconstruction across multiple pretrained CNN layers.Gradients are computed by backpropagation and gradient-based optimization solves for the stylized image.
- Feed-forward Style Networks: Feed-forward style networks approximate the optimization result quickly but require retraining for each new style.Their speed comes at the cost of support for only one or more fixed styles.
- Style Transfer for Video: Video style transfer can use a temporal loss with optical flow, but this produces an order of magnitude slowdown compared with frame-by-frame processing.The temporal loss is added to Gatys et al.’s style and content losses to maintain consistency across nearby frames.
- Inverting Deep Representations: Existing inverse networks are trained on image datasets with losses defined in RGB space, often for visualization or autoencoding.The paper distinguishes its activation-based training setup from these prior inversion objectives.
- This Work: This paper directly constructs target activations for one CNN layer, then inverts them without restricting style transfer to trained styles.Its inverse network uses an activation-space loss and can invert activations outside the usual CNN activation range.
3. A New Objective for Style Transfer
The method constructs a stylized target activation in one CNN layer by replacing content patches with best-matching style patches, then reconstructs an image from those activations. The style-swap operation is implemented efficiently with convolution, channel-wise selection, and transposed convolution before optimization-based image reconstruction.
- Style Swap: Style swapping replaces each content activation patch with its closest-matching style patch, producing activations that retain content structure while adopting style textures.Patches are extracted from both content and style activations, matched using normalized cross-correlation, swapped, and recombined by averaging overlaps.
- Style Swap: The operation reconstructs complete target activations by averaging overlapping swapped patches, including overlaps created by multiple equally good matches.The resulting activations correspond to the content image’s structure with textures taken from the style image.
- Parallelizable Implementation: A 2D convolution computes cross-correlations between content patches and normalized style patches at every spatial location.The resulting tensor has one spatial position per content patch and one feature channel per candidate style patch.
- Parallelizable Implementation: A channel-wise argmax selects the best-matching style patch at each content location, represented as a one-hot vector for reconstruction.Only the selected style patch contributes at each location during the subsequent transposed convolution.
- Parallelizable Implementation: A 2D transposed convolution places the selected, unnormalized style patches into the target activation tensor, after which overlapping values are averaged.This reduces style swapping to standard efficient convolutional operations.
- Optimization Formulation: The stylized image is obtained by optimizing a squared activation-space reconstruction loss against the swapped target, with total variation regularization for spatial smoothness.The target is Φss(C, S), and subgradient-based optimization applies because the pretrained CNN mapping is subdifferentiable.
4. Inverse Network
The inverse network approximates the optimization-based stylization result while supporting arbitrary content and style images. Training addresses the CNN’s non-injective mapping and style-swapped activations that may fall outside the CNN’s activation range.
- Inverse-network objective: The inverse network approximates an optimum of the stylization loss for arbitrary target activations using a deterministic function.Its objective combines activation reconstruction with total-variation regularization.
- Inverse-network objective: Because the pretrained CNN is non-injective, the method learns an approximate inverse relation with a parametric neural network.Convolution, maxpooling, and ReLU layers are many-to-one and lack well-defined inverses.
- Training: The network is trained unsupervised on activation features because the optimization objective’s optimum need not be known.The parameters are trained using activation features from a dataset of size n.
- Training: Style-swapped activations are added to training because interpolation can produce targets outside the range of the pretrained CNN.Augmentation uses style-swapped activations formed from pairs of training images.
- Feedforward procedure: The feedforward procedure computes content and style activations, constructs style-swapped activations, and feeds them to the trained inverse network.This replaces the optimization procedure and uses one trained inverse network for new style images.
5. Experiments
Experiments show that the proposed style-swapping objective produces stable, tunable stylizations and supports an inverse network for arbitrary-sized inputs. Its computation is dominated by style-patch matching, while the single-layer formulation reduces optimization time and preserves arbitrary-style flexibility.
- Style Swap Results: The relu3 1 layer provides visually pleasing style textures while remaining structurally consistent with the content.The method is restricted to this layer for subsequent experiments and inverse-network training.
- Style Swap Results: Random initialization has almost no effect on the stylized result, indicating fewer local optima than competing style-transfer objectives.The formulation reaches the same optimum more consistently and in fewer iterations than existing formulations.
- Style Swap Results: Increasing patch size trades content structure for style texture, providing an intuitive control over stylization strength.Larger patches replace more content structure with textures from the style image.
- CNN Inversion: The inverse network achieves reasonable results on arbitrary full-sized images despite training only on 256 × 256 images.Training augmentation with style-swapped activations improves performance over an otherwise identical network without that augmentation.
- Computation Time: The style swap procedure consumes much more feedforward computation than the other neural networks because its filters correspond to style patches.The number of style patches grows linearly with image pixels, making style-image size the strongest computation-time factor.
- Computation Time: The procedure can handle large content images when the number of style patches remains manageable, although clustering or approximate nearest-neighbor search could reduce patch costs.Content-image computation may stop increasing at larger sizes, likely because of implementation parallelism.
6. Discussion
The method prioritizes speed, arbitrary-style adaptability, and simple video application, while accepting quality limitations and occasional local flickering.
- Patch size provides an intuitive tuning parameter, and frame-by-frame processing enables straightforward video stylization.
- The inverse network approximates style-transfer optimization much faster and generalizes beyond its training styles.It can also accommodate changes in style-image size without retraining.
- The method is slower than Johnson et al.’s feedforward approach, but it can stylize new style images without training a new network for each one.
- Missing global style and neighboring-patch similarity measures sacrifice quality for efficiency and can occasionally cause local flickering in videos.
Inverse Network Architecture
The experiments use a truncated VGG-19 encoder and an inverse network, with standard convolutional, normalization, pooling, and upsampling components; the architecture choice may limit results.
- Better results may be possible with other CNN architectures because the authors did not explore many alternatives.
- Convolutional layers use 3 × 3 filters, padding 1, and stride 1.
- ReLU applies the elementwise function ReLU(x) = max{x, 0}, and InstanceNorm standardizes each feature channel to mean 0 and standard deviation 1.
- Maxpooling downsamples by a factor of 2, whereas nearest-neighbor upsampling increases resolution by a factor of 2.
- The truncated VGG-19 network runs from the input layer through relu3_1, while the inverse network reconstructs images from its activations.These architectures are listed in Tables A1 and A2.