PaperlayerView exact revision on arXiv

Source-linked AI summary

Stable and Controllable Neural Texture Synthesis and Style Transfer Using Histogram Losses

Eric Risser, Pierre Wilmot, Connelly Barnes

arXiv:1701.08893v2cs.GRcs.CVcs.NE

TL;DR

CNN-based texture synthesis and style transfer can produce strong results but face instability, artifacts, manual parameter tuning, and limited user controls. This paper introduces a multiscale CNN pipeline with histogram and localized losses, reporting improved quality, faster convergence, greater stability, and artistic controls.

  • Problem

    CNN-based methods for texture synthesis and style transfer have limitations in stability, ghosting artifacts, per-image parameter tuning, large-scale feature reproduction, and artistic controls.

  • Method

    The paper combines multiscale CNN synthesis with histogram losses, localized style losses, and automated parameter selection.

  • Results

    The approach improves quality and stability, converges in fewer iterations, reduces ghosting, and supports artistic controls such as paint by numbers.

  • Takeaways & Limitations

    Histogram losses better constrain texture statistics, while localized losses and automated tuning improve control for texture synthesis and style transfer.

  • Takeaways & Limitations

    The discussion leaves additional statistics, including translational co-occurrence statistics, beyond the paper’s scope.

Abstract

from arXiv · show

Recently, methods have been proposed that perform texture synthesis and style transfer by using convolutional neural networks (e.g. Gatys et al. [2015,2016]). These methods are exciting because they can in some cases create results with state-of-the-art quality. However, in this paper, we show these methods also have limitations in texture quality, stability, requisite parameter tuning, and lack of user controls. This paper presents a multiscale synthesis pipeline based on convolutional neural networks that ameliorates these issues. We first give a mathematical explanation of the source of instabilities in many previous approaches. We then improve these instabilities by using histogram losses to synthesize textures that better statistically match the exemplar. We also show how to integrate localized style losses in our multiscale framework. These losses can improve the quality of large features, improve the separation of content and style, and offer artistic controls such as paint by numbers. We demonstrate that our approach offers improved quality, convergence in fewer iterations, and more stability over the optimization.

1 Introduction

CNN-based texture synthesis and style transfer can produce high-quality results but suffer from instability, artifacts, parameter-tuning demands, limited large-scale reproduction, and missing artistic controls. The paper addresses these issues with histogram losses, localized losses in a multiscale framework, and automatic parameter selection.

  • Motivation: CNN-based methods can achieve state-of-the-art results for some style-transfer inputs but have limitations in stability, artifacts, parameter tuning, and large-feature reproduction.They also lack artistic controls such as painting by numbers.
  • Motivation: Larger synthesized textures become more unstable, with brightness and contrast varying across the image and ghosting remaining after manual tuning.Figure 2 illustrates both instability and residual artifacts.
  • Contributions: Histogram losses match the full statistical distribution of CNN activations, improving quality, reducing ghosting, and accelerating convergence.The losses are first formulated for texture synthesis and then extended to style transfer.
  • Contributions: Localized losses in a multiscale synthesis framework improve quality and control, including large-feature reproduction, content-style separation, and paint-by-numbers controls.The framework also includes automatic parameter selection to remove manual per-image tuning.
  • Results: The combined contributions achieve state-of-the-art quality for neural style transfer and parametric neural texture synthesis.The reported scope covers both applications.

2 Related work

Prior work spans parametric, neural, feed-forward, and non-parametric texture synthesis and style-transfer methods. The paper builds on these approaches while targeting their statistical, speed, flexibility, and control trade-offs.

  • Parametric texture synthesis: Early parametric methods synthesized textures by matching histograms and modeling multiscale wavelet statistics, including cross-correlations between filter responses.These methods motivate the paper’s interest in histogram matching.
  • Scope: The paper’s stated quality claim excludes regular textures because it does not include Berger et al.’s regularity-improving loss.This is an explicit scope limitation on the state-of-the-art claim.
  • Neural texture synthesis and style transfer: Gatys et al. used CNN feature co-occurrence statistics represented by Gram matrices for texture synthesis, later combining Gram-matrix style loss with content loss for style transfer.The neural approach generally improved on earlier parametric texture synthesis results.
  • Feedforward neural texture synthesis: Feed-forward neural methods trade faster runtime and lower memory use for training separately on each style or texture.Their fixed network weights are specialized to the training exemplar.
  • Non-parametric texture synthesis: Non-parametric methods synthesize textures or transfer style by copying exemplar neighborhoods or patches according to local similarity.This provides a contrasting local-similarity strategy to parametric neural models.

3 A brief introduction to neural texture synthesis and style transfer

The introduced neural synthesis framework optimizes CNN-based statistics from exemplar images, using Gram-matrix losses for texture and combined style-content losses for style transfer. Outputs are initialized with noise and refined by gradient descent.

  • Texture synthesis: Texture synthesis passes source and output images through a CNN such as VGG and optimizes layerwise Gram-matrix losses over their feature activations.The source texture supplies the statistics that the synthesized output seeks to preserve.
  • Texture synthesis: The Gram matrix contains inner products between pairs of features within a CNN layer, while user parameters weight the loss terms.The matrix is defined over feature maps and has dimensions N_l × N_l.
  • Optimization: The output image is initialized with white noise and optimized by gradient descent, with gradients computed by backpropagation.This optimization applies to the texture-synthesis loss.
  • Style transfer: Style transfer passes content, style, and output images through a CNN and combines style loss with content loss.The style loss uses Gram-matrix statistics, while the content term compares content and output features.
  • Style transfer: The content loss is a feature distance intended to make the output image resemble the content image.Style-transfer output is likewise initialized with white noise and optimized using gradient descent.

4 Our basic method using histogram losses

The section diagnoses why Gram-matrix synthesis is unstable and introduces histogram losses to better preserve activation distributions. It also extends the approach to style transfer while retaining Gram and content constraints.

  • 4.1 Prelude: useful statistics for parametric synthesis: The baseline method combines mean, Gram-matrix, full-histogram, and Gram-plus-histogram statistics to compare alternative synthesis constraints.The paper notes that additional statistics, including translational co-occurrences, are beyond its scope.
  • 4.2 The problem: instabilities: Gram matrices capture non-central second moments rather than feature means or covariances, allowing different activation distributions to share the same statistic.For one feature, the normalized Gram value equals σ^2 + µ^2, so mean and variance can trade off while the Gram matrix remains fixed.
  • 4.2 The problem: instabilities: Equal Gram matrices can therefore permit arbitrary variance changes, producing instability in synthesized textures.In multiple dimensions without feature correlations, each feature can exhibit the same mean–variance ambiguity independently.
  • 4.2 The problem: instabilities: The instability artifact becomes more pronounced as the output texture is enlarged, while smaller outputs may require manual tuning of learning rates, gradient sizes, and contributing layers.The cited prior methods are commonly demonstrated at the input resolution or only slightly larger.
  • 4.3 Solution: Histogram Losses: Histogram losses match the full marginal distributions of feature activations, addressing the missing mean and variance guarantees of Gram-based synthesis.The texture objective also includes Gram and total-variation losses; total variation slightly improves output smoothness.
  • 4.4 Extension to style transfer: For style transfer, the method adds histogram and per-pixel content losses to the parametric synthesis objective.This preserves the content constraint while applying the histogram-based stabilization strategy to style transfer.

5 Localized losses for control and stability

The paper combines multiresolution synthesis with localized style losses to improve coarse-scale transfer, content–style separation, and user control. Indexed masks let one CNN match distinct regional textures while blending adjacent regions through overlapping receptive fields.

  • 5.1 Multiresolution synthesis: Multiresolution pyramids blend coarse-scale style features better, transfer coarse features more effectively, and reduce CNN noise artifacts.The pyramid uses coarse-to-fine synthesis across image scales.
  • 5.2 Localized style loss for artistic controls: Localized losses treat an image or style as multiple distinct textures rather than one combined parametric model.This framing supports painting-by-numbers or painting-by-texture synthesis.
  • 5.2 Localized style loss for artistic controls: Painting by numbers uses indexed masks on source and output images to identify M corresponding regions.The method tracks these indices through the image pyramid during coarse-to-fine synthesis.
  • 5.2 Localized style loss for artistic controls: A single CNN matches Gram matrices and histograms separately within each indexed region, while overlapping receptive fields automatically blend adjacent regions.This avoids using multiple CNNs and separately blending regional outputs.
  • 5.2 Localized style loss for artistic controls: For style transfer, semantically corresponding textures in style and content images enable transfer of higher-level features such as eyes and lips.The approach requires manually painted masks but is not specialized for faces and can apply across domains.

6 Automatic tuning of parameters

The method automatically determines loss parameters during optimization instead of relying on tedious per-image manual tuning. Its dynamic process adjusts the coefficients governing Gram, content, histogram, and total-variation losses using gradient information.

  • 6 Automatic tuning of parameters: Previous methods often manually tuned parameters per image, whereas this paper automatically determines parameters for all reported results.The authors also automatically determine parameters for previous approaches to ensure fair comparisons.
  • 6 Automatic tuning of parameters: The tuned parameters are α_l for Gram loss, β_l for content loss, γ_l for histogram loss, and ω for total variation loss.These coefficients control the respective loss terms in the optimization.
  • 6 Automatic tuning of parameters: Dynamic tuning adjusts α_l, β_l, γ_l, and ω during optimization using gradient information.The authors note that tuning with a more mathematically well-founded approach may be possible.

7 Implementation details

The implementation uses a VGG-19 network, multiresolution image pyramids, and circular convolution for texture synthesis and style transfer. Figure 7 illustrates indexed-mask controls and portrait style-transfer examples.

  • 7 Implementation details: The implementation is based on the ImageNet-pretrained VGG-19 network and uses selected convolutional layers for synthesis losses.The referenced implementation uses Johnson’s code for comparisons with Gatys et al. [2016].
  • 7 Implementation details: Figure 7 shows painting-by-numbers texture synthesis, portrait style transfer, and comparison with Selim et al. [2016].The authors report better preservation of fine-scale artistic texture, alongside somewhat greater transfer of identity through hair and eye-color changes.
  • 7 Implementation details: Gram losses use relu1_1 through relu4_1, histogram losses use relu4_1 and relu1_1, content loss uses relu4_1, and total variation uses the first convolutional layer.The total variation term smooths optimization noise.
  • 7 Implementation details: Synthesis starts with white noise at the bottom of an image pyramid and bilinearly upsamples each completed level to the next.This implements the paper’s multiresolution coarse-to-fine process.
  • 7 Implementation details: Circular convolution replaces padding so synthesized textures tile with themselves without apparent strange effects during style transfer.

8 Results and discussion

The results report improved stability and convergence from histogram losses, state-of-the-art quality, and substantially shorter CPU iteration time than the Gatys baseline. The evaluation uses 700 iterations for the proposed method versus 1000 for Gatys et al. [2016].

  • 8 Results and discussion: Histogram losses preserve the full feature-activation distribution, improving quality, stability, and convergence over Gram-matrix losses.The authors report a mean of 700 iterations for their results versus 1000 for Gatys et al. [2016], which sometimes remains unstable.
  • 8 Results and discussion: 45.7% of the Gatys method’s CPU running time is required per iteration for a 512x512 output.The proposed method takes 7 minutes 8 seconds per iteration, compared with 15 minutes 35 seconds for Gatys et al. [2016], using CPU-only measurements over 50 iterations.

9 Conclusion

The paper improves neural texture synthesis and style transfer by addressing stability, quality, parameter tuning, and artistic control. Its key insight is that histogram losses better constrain texture-statistic dispersion.

  • Histogram losses improve the stability and quality of CNN-based texture synthesis and style transfer by better constraining texture-statistic dispersion.
  • Automated parameter tuning removes the requirement for human manual tuning in the synthesis pipeline.
  • The approach adds artistic controls while improving both texture synthesis and style transfer applications.
Loading 1701.08893v2…