Multi-style Generative Network for Real-time Transfer

Hang Zhang; Kristin Dana

Source-linked AI summary

Multi-style Generative Network for Real-time Transfer

Hang Zhang, Kristin Dana

arXiv:1703.06953v2Mar 20, 2017cs.CV

TL;DR

Feed-forward multi-style and arbitrary-style transfer struggles to represent comprehensive styles with one-dimensional embeddings while retaining image quality and flexibility. The paper introduces CoMatch Layer and MSG-Net, which match second-order style statistics in a feed-forward architecture and use upsampled convolution. MSG-Net is reported to deliver superior image quality, real-time performance, and real-time brush-size control.

Problem
Existing feed-forward multi-style and arbitrary-style transfer methods compromise image quality and model flexibility, while one-dimensional style embeddings struggle to represent comprehensive style appearance.
Method
The paper introduces CoMatch Layer for matching target Gram-matrix statistics and builds MSG-Net with a Siamese style-statistics encoder, a transformation network, and upsampled convolution.
Results
MSG-Net achieves superior image quality and test speed compared with previous work, with MSG-Net-100 reaching more than 90 fps at 256×256 on an NVIDIA Titan Xp.
Takeaways & Limitations
MSG-Net supports multi-style transfer with real-time operation and enables real-time brush-size control in a purely feed-forward manner.

Abstract

from arXiv · show

Despite the rapid progress in style transfer, existing approaches using feed-forward generative network for multi-style or arbitrary-style transfer are usually compromised of image quality and model flexibility. We find it is fundamentally difficult to achieve comprehensive style modeling using 1-dimensional style embedding. Motivated by this, we introduce CoMatch Layer that learns to match the second order feature statistics with the target styles. With the CoMatch Layer, we build a Multi-style Generative Network (MSG-Net), which achieves real-time performance. We also employ an specific strategy of upsampled convolution which avoids checkerboard artifacts caused by fractionally-strided convolution. Our method has achieved superior image quality comparing to state-of-the-art approaches. The proposed MSG-Net as a general approach for real-time style transfer is compatible with most existing techniques including content-style interpolation, color-preserving, spatial control and brush stroke size control. MSG-Net is the first to achieve real-time brush-size control in a purely feed-forward manner for style transfer. Our implementations and pre-trained models for Torch, PyTorch and MXNet frameworks will be publicly available.

1. Introduction

The paper identifies limitations in feed-forward multi-style transfer: one-dimensional style embeddings restrict comprehensive style representation and existing models lack real-time brush-size control. It introduces CoMatch Layer and MSG-Net to improve style matching, image quality, flexibility, and speed.

Motivation: Existing feed-forward multi-style and arbitrary-style methods commonly encode styles through feature-map mean and variance in a one-dimensional space.The paper argues that comprehensive image-style appearance is fundamentally difficult to represent with this embedding.
Motivation: Current generative methods lack real-time brush-stroke-size adjustment, while resizing the style image changes the relative brush size.The paper links this limitation to one-dimensional style embeddings that cannot capture finer style behavior.
Contributions: CoMatch Layer embeds style in two dimensions and learns to match target Gram-matrix statistics during end-to-end training.The layer is differentiable, requires no additional supervision, and is compatible with existing generative architectures.
Contributions: CoMatch Layer enables multi-style generation from a single feed-forward network.The method explicitly matches second-order feature statistics rather than relying only on feature-map mean and variance.
Contributions: MSG-Net combines CoMatch Layer with upsampled convolution to support real-time feed-forward transfer while reducing checkerboard artifacts.Upsampled convolution applies an integer-stride convolution and outputs an upsampled feature map instead of using fractionally-strided convolution.
Contributions: MSG-Net is reported as the first purely feed-forward method to achieve real-time brush-size control for multistyle transfer.The paper also presents compatibility with style interpolation, color-preserving transfer, and spatial control.

2. Content and Style Representation

The paper represents content with CNN activations and style with Gram Matrices that capture feature distributions and second-order statistics.

CNN activations at scale i represent semantic content as feature maps Fi(x) with channel, height, and width dimensions.Fi(x) ∈ R^{Ci×Hi×Wi}, where Ci, Hi, and Wi describe feature-map channels, height, and width.
Gram Matrices G(Fi(x)) represent image texture or style through feature distributions at each CNN scale.The Gram Matrix has dimensions Ci×Ci.
Gram Matrices are orderless and, for zero-centered data, equal the covariance matrix scaled by Ci × Hi × Wi.This connects Gram-based style representation to second-order feature statistics.
The feature map is reshaped with Φ into a Ci×(HiWi) matrix before the Gram Matrix is computed.The reshaping operation enables efficient matrix-based calculation.

3. CoMatch Layer

The CoMatch Layer approximates style matching by transforming content features with a learnable weight matrix, while remaining differentiable and compatible with generative networks.

3. CoMatch Layer: The CoMatch Layer matches target style statistics by preserving content representations while aligning second-order feature statistics.It operates on content features Fi(xc) and style statistics G(Fi(xs)).
3. CoMatch Layer: The layer approximates an iterative content-style optimization and shifts its computational burden from inference to training.This approximation enables a real-time, differentiable implementation.
3. CoMatch Layer: A learnable matrix W ∈ R^{Ci×Ci} transforms reshaped feature maps to match the required dimensions and style statistics.The reshaping operation Φ supports the dimensional alignment.
3. CoMatch Layer: The weight matrix can minimize the content term through one choice and the style term through another, with learning balancing the trade-off.The paper relates these choices to inverse feature-map transformations and Cholesky decomposition of Gram statistics.
3. CoMatch Layer: The CoMatch Layer is differentiable, inserts into existing generative architectures, and requires no additional supervision.This supports end-to-end learning through the loss function.

4. Multi-style Generative Network

MSG-Net combines runtime style conditioning with multi-scale feature-statistics matching, while its decoder uses upsampled convolution to improve output quality and support brush-size control.

Network Architecture: MSG-Net takes both content and style images as inputs and explicitly matches the style image’s feature statistics at runtime.A Siamese encoder captures style statistics at multiple scales, which transformation-network CoMatch Layers match to the content representation.
Network Architecture: The style branch outputs Gram Matrices at multiple scales, providing the style representation used by the transformation network.The Siamese network shares weights with the transformation encoder and produces {G(F_i(x_s))} across K scales.
Network Architecture: Upsampled convolution replaces fractionally-strided convolution to avoid checkerboard artifacts during image reconstruction.The decoder recovers image details from downsampled feature maps, while the upsampling strategy addresses artifacts associated with fractional strides.
Network Architecture: Resizing the style image changes the relative brush size, enabling MSG-Net to control brush-stroke size through runtime style inputs.Prior generative models are limited because one-dimensional mean-and-variance embeddings cannot capture this finer style behavior.
Network Learning: The training objective combines content loss, multi-scale style loss, and total-variation regularization for smooth generated images.Content and style differences are computed through a pretrained loss network, with λ_c and λ_s balancing the two losses.

5. Experimental Results

Experiments compare MSG-Net with established style-transfer baselines on quality, speed, scalability, and control capabilities. The reported results show strong qualitative performance, practical inference speed, and no observed quality degradation when scaling to 1K styles.

Experimental Setup: Gatys et al. serves as the gold-standard baseline, alongside patch-based and one-dimensional style-embedding methods.The comparisons use implementations from the original authors for the multistyle and arbitrary-style baselines.
Model Size and Speed: MSG-Net-100 has 2.3M parameters and MSG-Net-1K has 8.9M parameters, with model size and inference speed evaluated at 256×256 and 512×512 resolutions.The table reports disk model size and frames-per-second inference speed on an NVIDIA Titan Xp, averaged over 50 samples.
Model Size and Speed: MSG-Net-100 has comparable model size and speed to single-style networks and is faster than arbitrary-style transfer using a pretrained VGG encoder.The reported speed advantage is attributed to a learned compact encoder.
Style Transfer: MSG-Net achieves superior performance compared with state-of-the-art generative-network approaches in qualitative comparisons.The paper notes a trade-off between style flexibility and image quality and focuses particularly on image quality.
Model Scalability: Increasing the style set to 1K images produced no observed quality degradation in MSG-Net-1K.MSG-Net-1K increases model capacity to 8.9M parameters and uses four times as many training iterations as MSG-Net-100.
Style Transfer: MSG-Net supports content-style interpolation, color-preserving transfer, and spatial manipulation in addition to multi-style transfer.The paper illustrates color-preserved results and spatial transfer using separate foreground and background styles.

6. Conclusion and Discussion

The paper concludes that MSG-Net improves the quality and flexibility of real-time multi-style transfer while supporting several existing control techniques. Its compact MSG-Net-100 model combines 2.3M parameters with high reported inference speed.

Conclusion and Discussion: MSG-Net uses CoMatch Layers to represent style through second-order statistics and achieves superior image quality compared with state-of-the-art approaches.The approach is presented as improving both the quality and flexibility of generative style-transfer models.
Conclusion and Discussion: MSG-Net-100 contains 2.3M parameters, runs at more than 90 fps on an NVIDIA Titan Xp, and reaches 15 fps on a GTX 750M-2GB laptop GPU.These speeds are reported for 256×256 input images.
Conclusion and Discussion: A single MSG-Net-100 generates diverse outputs from different style targets while preserving the same input content image.Figure 12 presents input content images in the first row and generated images for different style targets in subsequent rows.