Source-linked AI summary
ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows
Jie An, Siyu Huang, Yibing Song, Dejing Dou, Wei Liu, Jiebo Luo
TL;DR
Existing universal style-transfer methods can corrupt image content after repeated stylization, creating the content leak problem. ArtFlow uses reversible neural flows and unbiased feature transfer in a projection-transfer-reversion pipeline, achieving comparable style-transfer results while avoiding content leak.
Problem
Universal style-transfer methods achieve strong results but can corrupt image content after several rounds of stylization, a phenomenon called content leak.
Method
ArtFlow combines a reversible neural-flow Projection Flow Network with an unbiased feature-transfer module for forward projection, style transfer, and reverse reconstruction.
Results
ArtFlow achieves comparable style-transfer results to state-of-the-art methods while avoiding content leak.
Takeaways & Limitations
Lossless forward and backward inference allows style-transfer modules to be evaluated without reconstruction error or image-recovery bias.
Abstract
from arXiv · showhide
Universal style transfer retains styles from reference images in content images. While existing methods have achieved state-of-the-art style transfer performance, they are not aware of the content leak phenomenon that the image content may corrupt after several rounds of stylization process. In this paper, we propose ArtFlow to prevent content leak during universal style transfer. ArtFlow consists of reversible neural flows and an unbiased feature transfer module. It supports both forward and backward inferences and operates in a projection-transfer-reversion scheme. The forward inference projects input images into deep features, while the backward inference remaps deep features back to input images in a lossless and unbiased way. Extensive experiments demonstrate that ArtFlow achieves comparable performance to state-of-the-art style transfer methods while avoiding content leak.
1. Introduction
Universal style transfer improves generalization across styles, but repeated stylization can corrupt image content. ArtFlow addresses this content leak with reversible, unbiased feature processing while maintaining comparable style-transfer results.
- Motivation: Universal style transfer methods improve generalization beyond methods specialized for particular style-transfer types.Representative methods include AdaIN, WCT, and Avatar-Net.
- Motivation: Content leak is the corruption of image content after several rounds of stylization, despite strong first-round performance.Existing methods preserve content less effectively when outputs and references are repeatedly processed.
- Motivation: Content leak arises from reconstruction errors or biased decoder and style-transfer-module training in encoder-based universal style-transfer systems.WCT variants primarily exhibit decoder reconstruction error, while AdaIN and Avatar-Net also involve decoder or transfer-module bias.
- Proposed Framework: ArtFlow replaces the encoder-transfer-decoder structure with a projection-transfer-reversion pipeline using forward and backward inference through a reversible neural-flow network.Its Projection Flow Network extracts latent features, applies unbiased transfer, and reconstructs stylized images through reverse inference.
- Proposed Framework: ArtFlow combines an unbiased, lossless, reversible Projection Flow Network with an unbiased style-transfer module.The framework is designed to avoid reconstruction error and image-recovery bias while enabling analysis of transfer-module bias.
- Results: ArtFlow achieves comparable style-transfer results to state-of-the-art methods while avoiding content leak.The contribution statement identifies this outcome as the framework’s principal objective.
2. Related Work
Style transfer research has progressed from handcrafted approaches to neural methods, but quality, generalization, and efficiency remain competing considerations. Neural flows provide reversible transformations that support ArtFlow’s lossless feature processing.
- Image Style Transfer: Early style-transfer methods used stroke rendering, image analogy, and image filtering before deep neural networks were applied.These approaches typically traded off style-transfer quality, generalization, and efficiency.
- Image Style Transfer: Neural style transfer introduced deep-feature representations and optimization or feed-forward networks to improve transfer quality or computational efficiency.The related-work passage places these developments before universal style transfer.
- Neural Flows: Neural flows learn exact likelihoods through chains of reversible transformations and include models such as NICE, RealNVP, GLOW, and Flow++.ArtFlow uses a reversible PFN and unbiased feature transfer to support lossless forward and backward inference.
3. Pre-analysis
The pre-analysis characterizes content leak as progressive loss of content information during repeated stylization and attributes it to reconstruction error and bias. ArtFlow’s projection-transfer-reversion scheme is presented as the corresponding unbiased alternative.
- What Content Leak Is: Content leak occurs when stylization results lose content information, which can be exposed by repeatedly feeding outputs back into style-transfer methods.The analysis uses repeated stylization as an indirect way to visualize corruption of content.
- Why Content Leak Happens: A decoder’s non-lossless reconstruction can accumulate errors across rounds and gradually disturb content details.This reconstruction-error mechanism is identified as a direct explanation for content leak in existing systems.
- What Content Leak Is: Repeated encoding and decoding with AdaIN produces increasingly visible artistic patterns, indicating that its auto-encoder can introduce style during reconstruction.The reported visualization propagates each reconstruction output into the next round for 50 rounds.
- Why Content Leak Happens: AdaIN’s combined content and style losses bias decoder training toward artistic rendering rather than perfect image reconstruction.Training with only content loss converges to a significantly smaller content-loss value than weighted content-and-style training.
- Why Content Leak Happens: Reconstruction error, biased image recovery, and biased style transfer are summarized as three causes of content leak.The latter two can shift outputs toward a biased style through decoding or feature stylization.
- ArtFlow Scheme: ArtFlow performs projection, feature transfer, and reversion through forward and reverse inference to transform content and style images into stylized outputs.The scheme extracts deep features, transfers style between them, and reconstructs the stylized image.
4. Method
ArtFlow replaces the common encoder-transfer-decoder scheme with a projection-transfer-reversion pipeline built around a reversible Projection Flow Network. Its lossless feature projection and inversion support unbiased analysis and selection of style-transfer modules, with AdaIN and WCT used for unbiased transfer.
- ArtFlow framework: ArtFlow performs style transfer through projection, feature transfer, and reversion using the reversible Projection Flow Network (PFN).PFN provides both forward and backward inference instead of the prevalent encoder-transfer-decoder structure.
- Projection Flow Network: PFN consists of reversible additive coupling, invertible 1×1 convolution, and Actnorm transformations, with squeezing used to reduce spatial feature-map size.The reversible components make PFN fully reversible; squeezing rearranges spatial patches into channels.
- Projection Flow Network: Additive coupling splits feature channels, applies a same-shaped neural network to one half, adds the result to the other half, and concatenates the halves.Its reverse computation is directly derived, and the paper reports additive coupling as more efficient and stable than affine coupling in training.
- Unbiased Content-Style Separation: AdaIN and WCT are identified as unbiased style-transfer modules, whereas Avatar-Net’s Style Decorator is empirically biased toward style.The paper defines unbiased transfer as preserving the content factor from the content image and the style factor from the style image.
- Unbiased feature transfer: PFN enables lossless and unbiased feature extraction and image recovery, avoiding reconstruction error and recovery bias in encoder-decoder frameworks.This makes comparisons among style-transfer modules less affected by the auto-encoder used for reconstruction.
- Unbiased Content-Style Separation: PFN-based analysis quantitatively verifies unbiased content and style reconstruction for AdaIN and WCT, unlike the VGG19-based auto-encoder.ArtFlow therefore adopts AdaIN and WCT as two transfer-module options for unbiased style transfer.
5. Experiments
Experiments evaluate ArtFlow’s stylization quality, efficiency, content-leak behavior, and content-style separation against universal style-transfer methods. ArtFlow preserves stylization while reducing content leak and producing less style-contaminated content features.
- Experimental Setup: ArtFlow is evaluated against StyleSwap, AdaIN, WCT, LinearWCT, OptimalWCT, and Avatar-Net for stylization effect, computing time, content leak, and content-factor visualization.The experiments also include portrait style transfer and reverse style transfer, an application enabled by ArtFlow.
- Stylization and Efficiency: ArtFlow with AdaIN achieves lower Gram loss than vanilla AdaIN, while ArtFlow with WCT has similar Gram loss to WCT.These results indicate that ArtFlow addresses content leak without hurting the stylization ability of AdaIN or WCT.
- Stylization and Efficiency: ArtFlow is slower than AdaIN but faster than vanilla WCT because PFN avoids pooling and does not require WCT’s multi-level stylization strategy.The computing-time comparison is evaluated on 512×512 images.
- Content Leak: After 20 rounds of stylization, ArtFlow substantially alleviates content leak compared with AdaIN, WCT, and Avatar-Net counterparts.The comparison visualizes results from the first and twentieth stylization rounds.
- Content-Style Separation: ArtFlow’s inverted content features contain fewer style effects and sharper structures than those of AdaIN and WCT.The visualization supports unbiased content-style separation for ArtFlow, whereas AdaIN and WCT do not achieve it.
6. User Study
A user study compares ArtFlow with established universal style-transfer methods across 1,161 content-style results per method. ArtFlow receives more votes than the other methods, supporting comparable style-transfer performance.
- Study Design: The study evaluates StyleSwap, AdaIN, WCT, LinearWCT, OptimalWCT, Avatar-Net, and ArtFlow on 43 content images and 27 style images.Each method produces 1,161 style-transfer results, one for every content-style pair.
- Results: ArtFlow receives more votes than the other style-transfer methods from 799 effective votes.Users select one preferred result for each listed content-style pair.
7. Conclusion
The conclusion identifies content leak as a common issue in universal style transfer and presents ArtFlow as a reversible, unbiased alternative. ArtFlow combines PFN with an unbiased transfer module to preserve content while retaining comparable stylization.
- Conclusion: ArtFlow addresses content leak, where image content may corrupt after repeated stylization in state-of-the-art style-transfer algorithms.The framework is motivated by analysis of the phenomenon’s main causes.
- Conclusion: PFN uses forward and reverse inferences to project images into feature space and invert features back into image space.The algorithm transfers style features and then applies reverse PFN propagation to obtain the stylized image.
- Conclusion: ArtFlow with AdaIN or WCT achieves comparable style-transfer results while avoiding content leak.PFN provides lossless and unbiased image projection and reversion, supporting style-transfer reversal.
B. Proof of the Theorem 1
The proof establishes that AdaIN is an unbiased style-transfer module by showing that its output standard deviation matches the style feature’s standard deviation under the stated centered-feature assumption.
- Theorem Statement: Theorem 3 states that adaptive instance normalization in AdaIN is an unbiased style-transfer module.The proof assumes both content and style features are centered without loss of generality.
- Proof: AdaIN’s transformed feature has standard deviation σ(fs), matching the style feature’s standard deviation.The derivation uses σ(fcs) = σ(fc / σ(fc))σ(fs) = 1 · σ(fs) = σ(fs).
- Proof: Therefore, the adaptive instance normalization transfer module is unbiased.This is the proof’s stated conclusion.
C. Proof of the Theorem 2
Theorem 4 establishes that WCT’s whitening and coloring transforms are unbiased, under the proof’s centered-feature assumption. The proof concludes that the style-transferred features preserve this unbiasedness.
- Theorem 4 states that WCT’s whitening and coloring transforms form an unbiased style transfer module.
- The proof assumes that both content and style features are centered before deriving the transformed feature expression.
- The derivation concludes that WCT’s whitening and coloring transforms are unbiased.
E. Network Structure Analysis of PFN
The analysis compares alternative neural-flow architectures for PFN and examines their stylization quality, details, reverse transfer, and broader applications. PFN provides the reference architecture for these comparisons.
- Architecture variants: Three alternative architectures vary the number of cascaded flow modules and flow blocks: Flow8-Block1, Flow16-Block1, and Flow4-Block4.Each flow stacks Actnorm, invertible 1×1 convolution, and additive coupling layers.
- Architecture variants: Flow16-Block1 variants are less colorful, while Flow4-Block4 achieves comparable transfer but weaker details than the proposed PFN.The PFN produces more colorful details around the Itsukushima Shrine example.
- Reverse style transfer: Reverse style transfer uses the original content image as a new style image to recover the original content image losslessly.The comparison includes AdaIN, WCT, and ArtFlow.
- Comparative results: Figures 15–19 provide additional comparisons between ArtFlow and state-of-the-art style transfer algorithms.
- Portrait transfer: Portrait style transfer applies ArtFlow using FFHQ content images and Metfaces style images.
- Content leak: Figures 22 and 23 show further comparisons of the Content Leak phenomenon.
I. More Content Factor Reconstruction Results
This section presents additional visual comparisons covering ArtFlow’s style transfer, portrait transfer, Content Leak, and reconstructed content factors. The supplied passages identify the comparisons but do not state their outcomes.
- Content factor reconstruction: Figure 24 presents additional comparisons of content factor reconstruction results.
- Style transfer comparisons: Figures 15–19 compare ArtFlow’s style transfer results with state-of-the-art algorithms.
- Portrait style transfer: Figures 20 and 21 present portrait style transfer results produced by ArtFlow.
- Content Leak: Figures 22 and 23 provide further comparisons of the Content Leak phenomenon.