Source-linked AI summary
A Closed-form Solution to Photorealistic Image Stylization
Yijun Li, Ming-Yu Liu, Xueting Li, Ming-Hsuan Yang, Jan Kautz
TL;DR
Photorealistic stylization must transfer reference style while preserving content and photographic realism, yet existing methods can yield inconsistent results and artifacts. The paper introduces a two-step closed-form pipeline combining stylization with photorealistic smoothing. It reports stronger user preference and photorealism judgments than competing methods, alongside substantially faster computation.
Problem
Existing photorealistic stylization methods tend to produce spatially inconsistent stylizations with noticeable artifacts.
Method
The method combines PhotoWCT stylization with pixel-affinity-based photorealistic smoothing, using efficient closed-form solutions.
Results
63.1% of users preferred the proposed stylization to Luan et al.’s, 73.5% judged it more photorealistic, and 1024×512 images ran in 13.16 seconds versus 650.45 seconds.
Takeaways & Limitations
The method delivers photorealistic stylization outputs preferred by human subjects over the state of the art while running much faster.
Takeaways & Limitations
The method can fail to transfer flower patterns and favors smooth color transitions on surfaces because smoothing uses content-pixel affinities.
Abstract
from arXiv · showhide
Photorealistic image stylization concerns transferring style of a reference photo to a content photo with the constraint that the stylized photo should remain photorealistic. While several photorealistic image stylization methods exist, they tend to generate spatially inconsistent stylizations with noticeable artifacts. In this paper, we propose a method to address these issues. The proposed method consists of a stylization step and a smoothing step. While the stylization step transfers the style of the reference photo to the content photo, the smoothing step ensures spatially consistent stylizations. Each of the steps has a closed-form solution and can be computed efficiently. We conduct extensive experimental validations. The results show that the proposed method generates photorealistic stylization outputs that are more preferred by human subjects as compared to those by the competing methods while running much faster. Source code and additional results are available at https://github.com/NVIDIA/FastPhotoStyle .
1 Introduction
Photorealistic image stylization transfers a reference photo’s style while preserving content and photographic realism. Existing approaches can produce artifacts or inconsistent stylization, motivating a faster two-step method with closed-form solutions.
- Motivation: Photorealistic stylization changes a photo’s style while preserving its content and making the result appear camera-captured.Examples include transferring summery imagery to snowy imagery and daytime imagery to nighttime imagery.
- Limitations of prior work: Neural style transfer performs well for artistic stylization but often introduces structural artifacts and distortions on photorealistic images.
- Results: The method produces more consistent outputs with fewer artifacts than existing methods while running an order of magnitude faster.
- Limitations of prior work: Existing photorealistic methods can produce inconsistent stylization in semantically uniform regions.
- Proposed approach: The proposed method combines a stylization step with a smoothing step, both of which have closed-form solutions and can be computed efficiently.The stylization step transfers reference style, while smoothing targets spatial consistency and artifacts.
2 Related Work
Prior photorealistic stylization methods use global matching, local correspondences, or optimization-based approaches, but often trade off generality, consistency, photorealism, or speed. The proposed method addresses these limitations with two closed-form processing steps.
- Existing approaches: Global methods stylize by matching pixel-color means and variances or color histograms, whereas local methods find dense content–style correspondences.
- Existing approaches: Local correspondence-based approaches are slow in practice and often designed for specific scenarios.
- Terminology: A closed-form solution is obtainable in a fixed finite number of operations, including convolutions, max-pooling, and whitening.
- Proposed approach: The proposed pipeline uses two closed-form steps: F1 transfers style to an intermediate image, and F2 removes noticeable artifacts to produce a photorealistic output.
- Related formulations: Neural style transfer methods improve artistic stylization but do not aim to preserve photorealism, while photorealistic stylization does not require a training dataset.
- Closest prior work: Luan et al. improve local-structure preservation but often produce inconsistent stylization with artifacts and require substantial computation.
3 Photorealistic Image Stylization
The method combines PhotoWCT stylization with graph-based photorealistic smoothing to transfer style while preserving local structure and reducing inconsistent stylizations. Both stages use closed-form formulations, with affinities derived from the content photo to encourage consistent results.
- 3.1 Stylization: The stylization transform F1 transfers style from IS to IC while minimizing structural artifacts, but can remain inconsistent across semantically similar regions.This motivates a separate smoothing function F2 after PhotoWCT.
- 3 Photorealistic Image Stylization: The complete algorithm maps the content and style photos through PhotoWCT and then smoothing, using closed-form solutions that can be computed efficiently.The smoothing inverse is efficient because the pixel graph is often sparsely connected.
- 3.1 Stylization: PhotoWCT replaces WCT upsampling layers with unpooling layers to preserve spatial information and suppress structural artifacts.The unpooling layers use pooling masks, while comparisons show straighter building boundaries and fewer artifacts than WCT.
- 3.2 Photorealistic Smoothing: The smoothing step represents pixels as graph nodes and balances local smoothness against fidelity to the PhotoWCT result.The formulation uses an affinity matrix W, a smoothness term, and a fitting term; its solution encourages similar stylization within semantically similar regions.
- 3.2 Photorealistic Smoothing: Matting affinity avoids choosing one global Gaussian-affinity scale and can smooth different regions simultaneously.Gaussian affinity may either over-smooth the photo or produce inconsistent stylization when σ is poorly chosen.
4 Experiments
Experiments compare the proposed method with photorealistic and artistic stylization algorithms through visual comparisons, user studies, design analyses, and runtime measurements. The method generally yields stronger stylization, greater photorealism, fewer artifacts, and substantially faster processing, while its guided-filter approximation trades some preference for speed.
- Visual comparison: Visual comparisons show stronger style transfer than classical color-matching methods, while avoiding the artifacts and inconsistent regional stylization seen in Luan et al..The proposed method synthesizes style patterns such as snow, whereas Luan et al. produces irregular brightness and inconsistent stylization in similar regions.
- Visual comparison: The proposed method produces more photorealistic results than artistic stylization algorithms, which transfer style effectively but introduce structural artifacts and inconsistent stylization.The distortions occur on object boundaries and detailed edges.
- User studies: 63.1% of users prefer the proposed method's stylization to Luan et al.'s, while 73.5% judge its outputs more photorealistic; against Pitié et al., 55.2% prefer its stylization.The proposed results are reported as equally photorealistic to those from Pitié et al.'s color-matching algorithm.
- User studies: 56.4% and 65.6% are the proposed method's user preference scores for stylization effect and photorealism against artistic algorithms.These scores are reported as significantly better than those of the other artistic stylization algorithms.
- Design analysis: 83.6% and 83.2% of comparisons favor the proposed method over the WCT variant for stylization and photorealism, respectively.The variant replaces the PhotoWCT step with WCT.
- Runtime and approximation: At 1024×512 resolution, the full algorithm takes 13.16 seconds versus 650.45 seconds for Luan et al., while the guided-filter approximation takes 0.64 seconds but is less preferred.The approximation remains preferred over other methods for both stylization effects and photorealism.
5 Conclusions
The paper presents a fast photorealistic image stylization method built from two efficient closed-form steps. Experiments report stronger human preference than state-of-the-art methods while running much faster.
- The method combines a stylization step with a photorealistic smoothing step.
- Both steps have efficient closed-form solutions.
- Experimental results show greater human preference for its stylization outputs than for state-of-the-art outputs, with faster runtime.
A Multi-level Stylization
The multi-level stylization scheme applies PhotoWCT across different VGG-19 feature levels. It uses an auto-encoder and feature transforms to reconstruct images while capturing style characteristics across levels.
- The PhotoWCT stylization step uses an auto-encoder with unpooling layers and two feature transforms, PC and PS.
- Four decoders are trained for image reconstruction in the multi-level stylization design.
- The method applies PhotoWCT to VGG features at different layers to capture style characteristics across levels.
B Network Architecture
The decoder network configurations are documented in Table 5. The implementation uses abbreviations for filter number, filter size, and stride.
- N denotes filter number in the decoder configurations.
- K and S denote filter size and stride, respectively.
- Table 5 presents the detailed configurations of the decoders.
C Semantic Label Map
Semantic label maps can improve content–style matching when available, but the method uses them only to locate matching areas. Precise alignment with object boundaries is unnecessary.
- Semantic label maps can support better content–style matching when available.
- The label map is used only to find matching areas between content and style images, not for specific class information.
- The label map need not be drawn precisely along object boundaries.
D Additional Results
Figures 13–17 present additional photorealistic stylization results using semantic label maps and comparisons with artistic and photorealistic methods. Across varied examples, the proposed method produces more photorealistic results with fewer structural artifacts and more consistent stylizations.
- Additional Results: Figures 13–17 show additional stylization results for content–style pairs accompanied by semantic label maps.Each color in the maps represents a different semantic label; maps come from Luan et al. or are roughly drawn by humans.
- Additional Results: The proposed method is compared with three artistic and three photorealistic stylization methods.
- Additional Results: Across a variety of examples, the proposed method generates more photorealistic results with fewer structural artifacts and more consistent stylizations.
- Additional Results: Figure 12 compares stylization results produced using coarse versus precise content label maps.
- Additional Results: Figures 13–17 provide comparisons among different stylization methods.