Source-linked AI summary
Arbitrary Style Transfer with Deep Feature Reshuffle
Shuyang Gu, Congliang Chen, Jing Liao, Lu Yuan
TL;DR
Existing neural style transfer methods struggle to combine global style similarity with faithful local and semantic matching. This paper reshuffles style-image deep features, connects global and local style losses, and reports effective transfer for arbitrary styles with global similarity and local plausibility.
Problem
Neural parametric and non-parametric style transfer methods separately struggle with local distortions or semantic mismatches and with global inconsistency or wash-out artifacts.
Method
The method spatially rearranges style-image deep features, defines a reshuffle-based style loss, and progressively optimizes it in feature space with a level-wise decoder.
Results
Experiments show effective transfer for arbitrary styles, yielding global similarity to the style and local plausibility.
Takeaways & Limitations
Deep feature reshuffle provides a simple way to unify global and local style losses while combining advantages of parametric and non-parametric transfer.
Takeaways & Limitations
Constraining neural-patch usage for style can reduce matching accuracy and damage content structure, while the optimal usage parameter λ must be determined for each input.
Abstract
from arXiv · showhide
This paper introduces a novel method by reshuffling deep features (i.e., permuting the spacial locations of a feature map) of the style image for arbitrary style transfer. We theoretically prove that our new style loss based on reshuffle connects both global and local style losses respectively used by most parametric and non-parametric neural style transfer methods. This simple idea can effectively address the challenging issues in existing style transfer methods. On one hand, it can avoid distortions in local style patterns, and allow semantic-level transfer, compared with neural parametric methods. On the other hand, it can preserve globally similar appearance to the style image, and avoid wash-out artifacts, compared with neural non-parametric methods. Based on the proposed loss, we also present a progressive feature-domain optimization approach. The experiments show that our method is widely applicable to various styles, and produces better quality than existing methods.
1. Introduction
Neural style transfer methods trade off global style statistics against faithful local and semantic matching. Deep feature reshuffle connects these approaches through a new loss and supports progressive feature-domain optimization.
- Neural style transfer uses CNN representations to change image style while preserving content, extending earlier texture-synthesis and transfer methods.
- Parametric methods preserve content and overall artwork appearance but distort local style patterns and miss locally semantic correspondences.Examples include distorted circles and weak spatial constraints in face regions.
- Non-parametric methods improve local texture and semantic matching, but greedy patch selection reduces style richness and introduces wash-out artifacts.
- Deep feature reshuffle spatially rearranges style-image neural activations, combining global style consistency with locally semantic matching.The paper theoretically relates reshuffling to Gram-matrix optimization and non-parametric matching.
- Experiments report effective transfer for arbitrary styles, with global similarity to the style and local plausibility.
- The proposed reshuffle energy function is simple and flexible, while a level-wise decoder enables efficient progressive optimization from high-level to low-level features.
2. Related Work
Style synthesis methods use either global statistics or local example patches, each with characteristic weaknesses. Neural feature-based approaches improve semantic structure, while the proposed direction adds global constraints to local matching.
- Style synthesis methods are broadly categorized as parametric, which match global statistics, and non-parametric, which directly search for similar neural patches.
- Early parametric texture methods iteratively match desired statistics but struggle to provide proper statistical models for representation and fine matching.
- Gatys et al. use pretrained VGG features and Gram matrices to statistically match content and style, producing results across varied artistic styles.
- Neural parametric transfer can distort local patterns or miss semantic correspondences, while non-parametric transfer improves local fidelity but may lose style richness and create wash-out artifacts.
- Non-parametric neural methods search local patches using Markov Random Fields, and Deep Analogy adds bidirectional constraints and pyramid refinement for semantic-level matching.
- The proposed method retains local similarity and semantic transfer while adding global constraints intended to avoid wash-out artifacts and improve global consistency.
3. Understanding Neural Style Transfer
Neural parametric and non-parametric methods use different style losses: one matches global feature statistics, while the other matches local neural patches. Deep feature reshuffle provides a theoretical bridge, simultaneously achieving zero global and local style loss under the stated condition.
- Existing neural style losses: Parametric methods measure global style statistics with Gram matrices, whereas non-parametric methods measure neural patch similarity without a global constraint.The global loss ignores spatial layout; the local loss optimizes each patch's layout independently.
- Feature reshuffle: Feature reshuffle permutes style-feature spatial locations to reconstruct the output feature map through a bijective mapping.Each output location corresponds to exactly one style-feature location.
- Global-style connection: Reshuffling style features leaves their Gram matrices unchanged, making the global style loss zero.The result follows because spatial permutation preserves channel correlations.
- Local-style connection: When patch size is 1 × 1, the reshuffled feature at each location is its own nearest-neighbor style patch.This identifies the reshuffle construction with the local patch-matching formulation.
- Combined objective: The derivation concludes that feature reshuffling simultaneously minimizes global and local style losses when the patch size is 1 × 1.This establishes reshuffle as a complementary solution connecting the two neural style-transfer formulations.
4. Method
The method integrates global and local style losses through deep feature reshuffling, then optimizes features progressively across layers and decodes them into an image. It uses constrained patch matching and a shared, level-wise decoder to improve efficiency.
- Reshuffle Loss Function: The reshuffle loss integrates global and local style losses in a single neural style-transfer objective.It modifies the local style-loss term and connects parametric and non-parametric approaches.
- Reshuffle Loss Function: Constrained nearest-neighbor search limits patch reuse, while relaxed usage softly encourages uniform patch usage for greater robustness.Strict reshuffling permits each source patch only once; the relaxed formulation accommodates cases such as multiple content faces and one style face.
- Single Layer Optimization: Image-domain optimization repeatedly backpropagates through VGG19, but hundreds of forward-backward passes and nearest-neighbor searches make it prohibitively slow.The feature-domain alternative avoids updating the image after every iteration.
- Single Layer Optimization: An EM-like feature-domain algorithm alternates constrained patch matching with average-voting and feature blending until the optimal feature map is reached, typically in fewer than 10 iterations.The optimized feature map is decoded back into the output image.
- Multi-layer Progressive Optimization: Progressive optimization proceeds from layer 4 to layer 2, decoding coarse features downward and blending them with finer-layer content features.Multi-layer processing enriches textures, reduces poor local minima, and narrows later search ranges through coarse-layer guidance.
- Decoder Training: A single decoder is trained bottom-up by fixing and sharing earlier decoder parts, unlike Li et al.’s strategy of retraining all parts for each layer.The decoder combines image-reconstruction and feature-reconstruction losses while using a VGG-like architecture.
5. Ablation Study
The ablations show that reshuffle loss combines global style similarity with local texture plausibility, while patch usage and size expose a trade-off between style statistics, content, and spatial coherence.
- Style Loss Analysis: Reshuffle loss achieves lower global loss than the non-parametric method and lower local loss than the parametric method in all tested cases.The comparison uses 60 image pairs optimized from random noise for 500 iterations in layer 4.
- Patch Usage Count: Global style loss increases while content loss decreases as the maximum patch usage count increases.Relaxing usage constraints preserves local loss at 0 but changes the balance between global style matching and content fidelity.
- Patch Size Selection: Increasing patch size sacrifices global style loss but can preserve better local structure in style patterns.Large patches are useful when spatial coherence matters, despite the global-style trade-off.
- Patch Size Selection: The method uses 3 × 3 patches in layer 4 and 5 × 5 patches in layers 2 and 3.These sizes are selected empirically because coarser-layer patches cover larger receptive fields and can be harder to match.
6. Results
Across comparisons with neural style-transfer methods, the approach preserves local and semantic style structure while retaining global style properties; it is preferred by users and has intermediate runtime.
- Comparisons: The method preserves local textures and semantic-level matches better than parametric methods while retaining more globally faithful style than non-parametric methods.The comparison includes more than 100 content-style pairs and uses author-released code with default settings for baselines.
- Comparisons: The constrained nearest-neighbor field search is the computational bottleneck.The reported timings were measured on an Intel E5 2.6GHz CPU with an NVIDIA Tesla K40C GPU.
- Perceptual Study: Subjects preferred the method overall in a 150-group study averaging rankings from 15 participants.Participants ranked six randomized outputs from 1, best, to 6, worst.
- Comparisons: 114 seconds was the method's average running time on 512×512 image pairs.It was slower than [20] [30], comparable to [32], and faster than [16] [28].
7. Discussion and Conclusion
The paper presents deep feature reshuffle as a unifying view of global and local style losses, paired with progressive feature-domain optimization, while acknowledging a usage-constraint limitation.
- Discussion and Conclusion: Deep feature reshuffle unifies the commonly used global and local style losses.The paper frames this as a new perspective connecting parametric and non-parametric neural style-transfer methods.
- Discussion and Conclusion: Progressive feature-domain optimization provides an efficient algorithm based on the reshuffle loss.Features are progressively optimized from high-level to low-level layers and decoded back into an image.
- Discussion and Conclusion: Constraining neural-patch usage can reduce matching accuracy and damage content structure.Finetuning λ can address the issue, but automatically choosing the optimal value for each input remains future work.