PaperlayerView exact revision on arXiv

Source-linked AI summary

Photorealistic Style Transfer via Wavelet Transforms

Jaejun Yoo, Youngjung Uh, Sanghyuk Chun, Byeongkyu Kang, Jung-Woo Ha

arXiv:1903.09760v2cs.CV

TL;DR

Photorealistic style transfer must avoid spatial distortions and artifacts while changing a photograph’s style. WCT2 uses wavelet-corrected whitening and coloring transforms with progressive single-pass stylization, achieving fast, photorealistic results without post-processing. The method also provides temporally stable video stylization without additional constraints.

  • Problem

    Photorealistic style transfer requires changing scene style while preserving fine structures, but existing methods suffer spatial distortions or unrealistic artifacts.

  • Method

    WCT2 replaces lossy VGG pooling and unpooling with wavelet operations and progressively applies whitening and coloring transforms within a single forward pass.

  • Results

    1024×1024 images are stylized in 4.7 seconds, with better reported photorealism and user preference, without post-processing.

  • Takeaways & Limitations

    Exact wavelet recovery preserves structural information, while progressive stylization supports scalable, lighter, faster, and temporally stable photorealistic transfer.

  • Takeaways & Limitations

    WCT2 depends on an accurate semantic map, and misaligned maps visibly affect its results; removing this dependency is left for future work.

Abstract

from arXiv · show

Recent style transfer models have provided promising artistic results. However, given a photograph as a reference style, existing methods are limited by spatial distortions or unrealistic artifacts, which should not happen in real photographs. We introduce a theoretically sound correction to the network architecture that remarkably enhances photorealism and faithfully transfers the style. The key ingredient of our method is wavelet transforms that naturally fits in deep networks. We propose a wavelet corrected transfer based on whitening and coloring transforms (WCT$^2$) that allows features to preserve their structural information and statistical properties of VGG feature space during stylization. This is the first and the only end-to-end model that can stylize a $1024\times1024$ resolution image in 4.7 seconds, giving a pleasing and photorealistic quality without any post-processing. Last but not least, our model provides a stable video stylization without temporal constraints. Our code, generated images, and pre-trained models are all available at https://github.com/ClovaAI/WCT2.

1. Introduction

Photorealistic style transfer must change a scene’s general style while preserving fine structures, but existing approaches suffer information loss, distortions, artifacts, or costly post-processing. WCT2 corrects the network’s downsampling and upsampling with wavelet operations and uses progressive stylization in one pass to produce efficient end-to-end transfer.

  • Motivation: Photorealistic transfer must alter color and tone while preserving fine scene structures, unlike artistic methods that can introduce severe distortions.The paper highlights changing the sky and sea while retaining ship and bridge details.
  • Limitations of prior work: PhotoWCT reduces spatial distortion with max-pooling masks but still loses information and requires costly post-processing that can blur outputs and add manually tuned hyperparameters.These steps use the original image to patch results and increase computation and time.
  • WCT2 correction: WCT2 replaces VGG pooling and unpooling with wavelet pooling and unpooling, enabling exact signal recovery with minimal information loss and no post-processing.The design is motivated by matching learned downsampling operations with inverse operations.
  • Progressive stylization: WCT2 progressively transforms features during a single pass rather than recursively stylizing from coarse to fine across multiple levels.This uses one decoder during training and inference and avoids repeated lossy encoding and decoding.
  • Results: 1024×1024 images are processed in 4.7 seconds, while the method is reported as 830 times faster than state-of-the-art models.PhotoWCT fails from out-of-memory, and DPST takes 3887.8 seconds in the reported comparison.

2. Related Work

Earlier photorealistic transfer methods either restrict style changes, require heavy optimization, or rely on post-processing to address distortions and artifacts. WCT2 instead preserves fine structures in an end-to-end process without additional post-processing, using wavelet decomposition and reconstruction to retain spatial information.

  • Style transfer: Artistic style-transfer techniques cannot be used directly for photorealistic transfer because their feature abstraction can distort image structure.Photorealistic transfer requires preserving content structure while applying style.
  • Photorealistic transfer: Classical methods mainly match color and tone, restricting their use, while DPST adds photorealism regularization but requires heavy optimization.DPST also uses a semantic segmentation mask.
  • PhotoWCT: PhotoWCT uses max-pooling masks to relax spatial distortion but needs smoothing and filtering because its raw outputs are unsatisfactory.The post-processing increases runtime with image resolution and can blur final outputs.
  • WCT2: WCT2 preserves fine image structures with little spatial distortion in an end-to-end manner, removing the need for additional post-processing.This is the paper’s stated distinction from existing methods.
  • Wavelet reconstruction: Unlike wavelet pooling in this work, a related wavelet method discards first-level sub-bands for dimensionality reduction rather than using all sub-bands for exact spatial recovery.WCT2 combines wavelet decomposition and reconstruction to minimize noise amplification.

3. WCT2

WCT2 combines wavelet pooling and unpooling with whitening and coloring transforms in a single encoder-decoder network. The wavelet representation preserves structural information through exact reconstruction, while progressive feature transformation applies stylization across scales in one forward pass.

  • Method overview: WCT2 uses wavelet pooling and unpooling to preserve content information and progressive stylization in one forward pass to apply style.These components address structural preservation and stylization respectively.
  • Wavelet pooling: Haar wavelet pooling produces LL, LH, HL, and HH channels that represent smooth content and vertical, horizontal, and diagonal edge information.The low-pass channel captures smooth surface and texture, while high-pass channels capture edge-like information.
  • Wavelet reconstruction: Wavelet unpooling exactly reconstructs the original signal through component-wise transposed convolution and summation, unlike max-pooling, which lacks an exact inverse.This supports stylization with minimal information loss and noise amplification.
  • Wavelet design: Haar wavelets are selected because their component channels support better stylization, although they are not the only operations capable of full signal recovery.The method uses wavelet decomposition and reconstruction together.
  • Model architecture: The encoder replaces max-pooling with wavelet pooling, passes only LL to the next encoding layer, and uses a mirrored decoder to aggregate wavelet components.The encoder is based on ImageNet-pretrained VGG-19 through conv4_1.
  • Whitening and coloring transforms: WCT transfers style by matching content and style correlations in VGG feature space through singular value decomposition before decoding the transferred features.WCT2 is built on this whitening and coloring transform.
  • Progressive stylization: Progressive stylization sequentially applies WCT at multiple feature scales within one encoder-decoder network while keeping the number of SVD computations unchanged.Additional WCT operations can strengthen stylization at increased cost.

4. Analysis

Wavelet pooling preserves spatial and frequency information for photorealistic stylization, while progressive and multi-level transformations trade stronger style effects against computation and photorealism. Ablations and comparisons show that WCT2 improves detail preservation and avoids post-processing artifacts.

  • Wavelet pooling: Wavelet pooling preserves fine details that PhotoWCT loses through max-pooling.Low-frequency LL components control surface and texture, while LH, HL, and HH components control edges.
  • Wavelet pooling: Stylizing only LL leaves edges unstylized, enabling separate control over surfaces and edge details.Applying WCT to LL changes overall texture, whereas applying it to high-frequency components stylizes edges.
  • Ablation study: Split pooling degrades fine details despite carrying whole information, which the authors attribute to limited representation power.The reported degradation is visible in details such as grass.
  • Ablation study: Learnable pooling fails to preserve content and faithfully transfer style because its flexibility alters VGG feature properties.The ablation attributes this behavior to excessive flexibility in the trainable stride-two convolution.
  • Unpooling options: Concatenation-based wavelet unpooling produces clearer images but increases parameters and reduces interpretability and theoretical correctness.It learns a weighted combination of four wavelet components and the pre-pooling feature output.
  • Progressive versus multi-level strategy: Additional whitening and coloring transforms strengthen stylization while increasing computation and potentially reducing photorealism through noise amplification.WCT2 maintains photorealism in the reported multi-level setting, whereas PhotoWCT produces spotty artifacts.
  • Progressive versus multi-level strategy: WCT2 produces photorealistic stylization without post-processing, unlike PhotoWCT results that require post-processing to remove spotty artifacts.The comparison includes DPST, PhotoWCT, PhotoWCT with post-processing, and WCT2.
  • Progressive versus multi-level strategy: WCT2 separately stylizes video frames using a shared style image without semantic segmentation maps or post-processing.The figure reports results for a day-to-sunset video sequence.

5. Experimental results

WCT2 improves photorealistic style transfer through wavelet-based reconstruction and progressive stylization, producing fewer artifacts, stable video results, strong quantitative performance, and efficient high-resolution processing without post-processing.

  • Qualitative evaluation: WCT2 shows fewer artifacts while faithfully transferring reference styles, without applying post-processing after network output.PhotoWCT requires post-processing to remove spotty artifacts, but smoothing can blur final images.
  • Video stylization: WCT2 performs stable video style transfer without temporal consistency regularization, whereas PhotoWCT produces spotty and varying artifacts across frames.Each video frame is stylized separately to the target style without semantic segmentation.
  • Statistics: WCT2 variants occupy the top-right of Figure 9, indicating superior SSIM and style-loss performance relative to PhotoWCT and DPST post-processed outputs.SSIM measures structural recovery, while VGG style loss measures covariance differences between style images and outputs.
  • Runtime & memory: 830 times faster than DPST, WCT2 processes 1024×1024 images while maintaining high-quality photorealistic transfer and avoiding PhotoWCT’s post-processing.WCT2 inherits WCT’s computational time and removes the cumbersome post-processing steps.
  • Runtime & memory: WCT2 uses only 51% GPU memory of PhotoWCT because progressive stylization uses a single encoder-decoder instead of four.PhotoWCT’s multi-level stylization requires four encoder-decoder models.
  • User study: Human subjects preferred WCT2 by a large margin in all evaluated aspects against PhotoWCT and other state-of-the-art methods.The study comprised 1640 responses and compared WCT2 without post-processing against PhotoWCT with two post-processing steps.
  • Failure cases: Accurate semantic maps remain important because misalignment is more visible in WCT2, which retains fine details instead of unintentionally smoothing them.Removing the dependency on semantic label maps is identified as future work.

6. Conclusion

The paper concludes that WCT2 combines wavelet-based exact recovery with progressive stylization to achieve scalable, efficient, and photorealistic transfer, while semantic labels remain necessary for flawless results.

  • 6. Conclusion: WCT2 satisfies the reconstruction condition through exact recovery of wavelet transforms, preserving structural information during stylization.The method is presented as an end-to-end photorealistic style transfer model.
  • 6. Conclusion: Progressive stylization produces better results with less noise amplification than the compared multi-level strategy.The conclusion reports quantitative and qualitative advantages over other state-of-the-art methods.
  • 6. Conclusion: WCT2 is described as scalable, lighter, faster, and more photorealistic than other state-of-the-art methods, with human preference in every evaluated aspect.The method’s outputs were preferred by human subjects by a significant margin.
  • 6. Conclusion: WCT2 still requires accurate semantic labels for flawless results, motivating future work to remove this dependency.The conclusion identifies semantic-label removal as a future research direction.

A. Frame-based signal reconstruction

WCT2 is motivated by frame-based signal reconstruction theory, including the properties of frames and tight Haar wavelets that support exact recovery.

  • A. Frame-based signal reconstruction: WCT2 is inspired by frame-based signal reconstruction approaches and introduces frame theory, tightness of Haar wavelets, and theoretical motivation.These topics are provided to make the paper’s theoretical basis self-contained.

A.1. Perfect reconstruction condition

Perfect reconstruction requires a synthesis operator that inverts the analysis transform; tight frames provide exact recovery without amplifying input power.

  • A.1. Perfect reconstruction condition: A family of functions in a Hilbert space is a frame when it satisfies bounds defined for input signals and positive frame constants.The frame bounds are denoted α and β.
  • A.1. Perfect reconstruction condition: The original signal f is exactly recovered from frame coefficients z = Φf when the synthesis operator satisfies ˜ΦΦ⊤ = I.Under this perfect reconstruction condition, f = ˜Φz = ˜ΦΦ⊤f = f.
  • A.1. Perfect reconstruction condition: A tight frame has equal frame bounds and can use the same operator for analysis and synthesis, equivalently satisfying ΦΦ⊤ = I.The passage states that tightness is equivalent to ˜Φ = Φ.
  • A.1. Perfect reconstruction condition: Tight frames do not amplify input power and therefore have the minimum noise amplification factor.This property motivates their use for reconstruction performance.

A.2. Wavelet frames

The method uses Haar wavelets as a tight filter-bank representation that preserves important signal information, including edges, during pooling and unpooling.

  • Wavelets compactly represent signals while maintaining important information such as edges, improving reconstruction with minimal noise amplification.
  • The proposed basis is a tight filter bank whose subband operators decompose the signal into complementary components.
  • Haar wavelets provide low- and high-pass subband decomposition as a simple tight filter-bank frame.
  • The Haar low-pass subband is equivalent to average pooling.

A.3. Theoretical motivation

The paper interprets encoder-decoder networks through convolution framelets and uses perfect reconstruction as the criterion for preserving input information. Max-pooling lacks an exact inverse, whereas wavelet decomposition and reconstruction enable exact recovery.

  • Encoder-decoder convolution networks learn local convolutional bases together with hand-crafted global bases supplied by pooling operations.
  • When the learned and global frames satisfy the perfect reconstruction condition, the input signal can be exactly recovered from its representations.
  • WCT and PhotoWCT cannot satisfy perfect reconstruction because max-pooling has no exact inverse.
  • WCT2 uses Haar wavelet pooling and unpooling to exploit encoder information while preserving spatial structure.

A.4. Proposed network architecture

WCT2 replaces VGG pooling and unpooling with wavelet operations and performs progressive stylization in a single pass. The architecture is presented as practical, post-processing-free, and compatible with multiple stylization methods.

  • Proposed network architecture: The proposed module uses Haar wavelet pooling and unpooling in paired encoder-decoder stages.
  • Proposed network architecture: Progressive stylization applies WCT during one pass, using encoder skip connections and optionally transforming decoder features.
  • Proposed network architecture: High-frequency wavelet components skip directly to corresponding decoding layers, while low-frequency components continue to deeper layers.
  • Proposed network architecture: WCT2 removes the need for post-processing and is compatible with alternative stylization methods, including AdaIN.
  • Qualitative results: Qualitative comparisons report that WCT2 preserves fine details while competing artistic methods produce distortions and artifacts.
  • Qualitative results: Additional qualitative comparisons include WCT2 variants and video stylization results.
Loading 1903.09760v2…