Source-linked AI summary
Learning Linear Transformations for Fast Arbitrary Style Transfer
Xueting Li, Sifei Liu, Jan Kautz, Ming-Hsuan Yang
TL;DR
Arbitrary style transfer methods can require costly matrix computations or fail to model feature covariance. This paper derives a linear transformation framework and learns its matrix with feed-forward networks; it reports favorable state-of-the-art comparisons for image and video style transfer and demonstrations across four tasks.
Problem
Existing arbitrary style-transfer approaches either compute feature transformations expensively or fail to model feature covariance.
Source: §1 · p. 1Method
The paper derives general linear transformation solutions and learns the transformation matrix with lightweight feed-forward CNNs for flexible multi-level style transfer.
Source: §1 · p. 1Results
The method performs favorably against state-of-the-art methods for image and video style transfer and is demonstrated across artistic, photo-realistic, video, and domain-adaptation tasks.
Source: §1 · p. 1, §6 · p. 11Takeaways & Limitations
Optimizing the linear transformation can also minimize style reconstruction losses used by most existing approaches.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Given a random pair of images, an arbitrary style transfer method extracts the feel from the reference image to synthesize an output based on the look of the other content image. Recent arbitrary style transfer methods transfer second order statistics from reference image onto content image via a multiplication between content image features and a transformation matrix, which is computed from features with a pre-determined algorithm. These algorithms either require computationally expensive operations, or fail to model the feature covariance and produce artifacts in synthesized images. Generalized from these methods, in this work, we derive the form of transformation matrix theoretically and present an arbitrary style transfer approach that learns the transformation matrix with a feed-forward network. Our algorithm is highly efficient yet allows a flexible combination of multi-level styles while preserving content affinity during style transfer process. We demonstrate the effectiveness of our approach on four tasks: artistic style transfer, video and photo-realistic style transfer as well as domain adaptation, including comparisons with the state-of-the-art methods.
1. Introduction
Existing arbitrary style transfer methods either compute feature transformations expensively or fail to model feature covariance. This work derives a linear transformation framework and learns its matrix with a feed-forward network for efficient, flexible transfer across multiple tasks.
1. Introduction
- Existing arbitrary style-transfer methods compute second-order feature transformations directly, but can be expensive at high feature dimensions and do not explore general solutions.AdaIN matches means and variances, while WCT models covariance through whitening and coloring.Source: §1 · p. 1
- The paper derives general solutions for linear transformations and connects them to style reconstruction losses based on Gram-matrix differences.The authors state that optimizing the transformation can also minimize style reconstruction losses used by most existing approaches.Source: §1 · p. 1
- Two lightweight CNNs learn a transformation matrix that runs at about 140 fps, combines multi-level styles, and preserves content affinity.The transformation can be controlled by different levels of style losses.Source: §1 · p. 1
- The proposed method is demonstrated on artistic, photo-realistic, and video style transfer, as well as domain adaptation, with comparisons to state-of-the-art methods.Figure 1 depicts these four applications.Source: §1 · p. 1, Figure 1 · p. 2
2. Related Work
Earlier style-transfer work moved from iterative optimization to faster feed-forward models, but those models often lacked arbitrary-style support. Covariance-aware approaches address that gap with trade-offs in computation or flexibility, while this work learns the transformation matrix with feed-forward networks.
2. Related Work
- Gatys et al. use feature-statistics matching, while later work improves artistic quality, user control, diversity, or semantic handling.Earlier methods also used non-parametric sampling, non-photorealistic rendering, and image analogy, but often lacked high-level semantic information.Source: §2 · p. 2
- Feed-forward approaches reduce Gatys et al.’s iterative cost, but most models transfer only fixed styles; later models support 32 [6] or 1,000 styles, not arbitrary ones.The cited approaches use instance normalization or a binary selection unit, respectively.Source: §2 · p. 2
- Mean-and-variance methods support arbitrary styles but do not model feature covariance, while WCT models covariance at the cost of matrix decomposition and cascading modules for multi-level styles.A meta-network approach has extra memory costs for each content/style pair and does not explicitly model covariance shift.Source: §2 · p. 2
- This work replaces intermediate-layer matrix computation with feed-forward networks and demonstrates applications in artistic, video, photo-realistic, and domain-adaptation tasks.The authors describe the replacement as more flexible and efficient.Source: §2 · p. 2
3. Style Transfer by Linear Transformation
The method learns a linear style transformation from content and style features, with its objective tied to style reconstruction. Covariance-based inputs support flexible transformations, while affinity preservation and a shallow auto-encoder target less-distorted stylization.
3. Style Transfer by Linear Transformation
- A feed-forward model transforms content features with a learned matrix T, using a fixed encoder-decoder and two small CNNs to predict the transformation.The method runs at approximately 140 fps and computes style losses at multiple levels.Source: §3 · pp. 2–3
3.1. Objectives for Arbitrary Style Transfer
- The style objective minimizes the difference between centered covariances of transformed content features and a virtual style feature map.The resulting T depends on content and style feature covariances; adding the style mean aligns mean and covariance statistics.Source: §3.1 · p. 3
3.2. Model for learning transformation T.
- The model uses separate content and style CNNs on feature covariances to form T, allowing transformations independent of input dimensions.Covariance inputs also avoid the fixed-dimension constraint of image or feature-map inputs; the ablation reports better general style-transfer results with covariance inputs.Source: §3.2 · pp. 3–4
3.3. Efficient Model
- A feed-forward network replaces costly matrix decomposition, and one T can combine multiple style-loss levels without added inference cost.The paper contrasts this with WCT’s cascaded auto-encoders for multi-level styles, which add computational load.Source: §3.3 · p. 4
3.4. Un-distorted Style Transfer
- Linear transformations preserve content-feature affinity, and a shallower auto-encoder can use deeper style losses to produce less-distorted stylization.The method reports fewer distortions and more stable sequences than methods not based on the linear-transformation formulation; shallow features may not express abstract styles well.Source: §3.4 · pp. 4–5
4. Network Implementation
The implementation combines a fixed reconstruction encoder-decoder, a learned transformation module, and VGG-19 losses. The transformation module factorizes covariance-based inputs to keep the model compact.
4. Network Implementation
- A VGG-19 encoder and symmetric decoder are trained for faithful reconstruction, then fixed for the remaining training procedure.The encoder uses pre-trained ImageNet weights, while the decoder is trained from scratch on MS-COCO.Source: §4 · p. 5
- Two CNNs produce matrices that are multiplied into T; multiplying content features by T yields transformed features for decoding.The transformation module factorizes covariance inputs through convolutional units and a fully connected layer, then compresses and uncompresses content features.Source: §4 · p. 5
- A fixed pre-trained VGG-19 loss network measures style with Gram-matrix differences and content with feature differences.Content losses use relu4_1 features, and the final loss weights content and style terms as L = Lcontent + λLstyle.Source: §4 · p. 5
5. Experiment Results
The experiments evaluate learned linear style transfer across artistic, video, photo-realistic, and game-to-real adaptation tasks, alongside ablations of its inputs and style-loss settings.
5.2. Ablation Studies
- Using covariance as the transformation-module input yields more similar stylized details than the tested image- and feature-input alternatives; a single CNN model does not converge.The single-CNN failure is consistent with the authors’ architectural analysis.Source: §5.2 · p. 6, Figure 4 · p. 6
- Lower-layer content features and style losses favor more photo-realistic results, while higher-layer style losses produce more stylized images; relu3_1 bottlenecks are less distorted than relu4_1 bottlenecks.The model combines multiple style-loss levels in one transformation matrix; the experiments compare different encoder bottlenecks and loss-layer settings.Source: §5.2 · p. 6, Figure 5 · p. 7
5.3. Artistic Style Transfer
- On unseen content and style images, the proposed method performs favorably against artistic-transfer baselines, and users favored it among evaluated methods.The user study collected 540 votes from 36 users.Source: §5.3 · p. 7, Figure 6 · p. 8
- The slower relu4_1 variant runs at 100 FPS for 256×256 images and 27 FPS for 512×512 images, about three orders faster than optimization and two faster than WCT.Runtime was measured on a Titan XP GPU; Table 1 compares methods at three image scales.Source: §5.3 · pp. 7–8, Table 1 · p. 7
5.4. Video and Photo-realistic Style Transfer
- Video stylization is stable without optical-flow warping, and the style-branch output can be computed once at initialization and reused across frames.The reported heat-map difference between frames is closest to that of the original frames; the user study found the method performed well against other methods.Source: §5.4 · pp. 8–9, Figure 7 · p. 9, Figure 8 · p. 9
- With relu3_1 as bottleneck, the method produces undistorted photo-realistic stylizations comparable in visual quality to specialized methods, at 0.025 seconds for 512×256 images.A bilateral filter or guided upsampling can further improve object boundaries.Source: §5.4 · p. 9, Figure 10 · p. 10
5.3. Artistic Style Transfer; 5.4. Video and Photo-realistic Style Transfer; 5.5. Domain Adaptation: from Game to Real.
- Across tasks, results are favorable for artistic transfer, video stability, and photo-realistic quality, while game-to-real adaptation improves PSPNet segmentation.For 512×256 images, the photo-realistic method takes 0.025 seconds; the adaptation comparison uses PSPNet results before and after transfer.Source: §5.3 · p. 7, §5.4 · pp. 8–9, §5.5 · p. 10
5.5. Domain Adaptation: from Game to Real.
- After transferring GTA game images using Cityscapes styles and semantic masks, PSPNet segmentation results improve relative to results before adaptation.The authors describe narrowing the game-to-real domain gap in semantic space.Source: §5.5 · p. 10, Figure 11 · p. 11
6. Conclusions
The paper develops a theoretical foundation for arbitrary style transfer and presents an efficient, flexible learned-linear-transformation algorithm, with favorable results across image and video stylization tasks.
6. Conclusions
- The work develops a theoretical foundation for analyzing arbitrary style-transfer frameworks.Source: §6 · p. 11
- Its algorithm learns linear transformations and is described as computationally efficient and flexible across numerous tasks.Source: §6 · p. 11
- Experiments report favorable performance against state-of-the-art methods on image and video style transfer.Source: §6 · p. 11