PaperlayerView exact revision on arXiv

Source-linked AI summary

Style-Transfer via Texture-Synthesis

Michael Elad, Peyman Milanfar

arXiv:1609.03057v3cs.CV

TL;DR

Style-transfer has no well-defined target, including whether content edges and other elements should shift. This paper adapts texture-synthesis for style-transfer while tying selected regions to the content and allowing departures elsewhere. Its results are reported as competitive with recent CNN- and texture-synthesis-based methods, and the authors conclude effective, satisfactory style-transfer is within reach.

  • Problem

    Style-transfer lacks a well-defined target, including whether content edges and other elements should shift.

    Source: §1 · pp. 1, 3

  • Method

    The paper adapts Kwatra's texture-synthesis approach, preserving content in selected regions while allowing departures elsewhere.

    Source: §1 · p. 4

  • Results

    The results are reported as competitive with recent CNN [39] and texture-synthesis-based style-transfer methods, with visually pleasing, rich images.

    Source: §1 · p. 6

  • Takeaways & Limitations

    The authors conclude that effective and satisfactory style-transfer is within reach.

    Source: §5 · p. 36

    The algorithm relies on segmentation; removing that requirement needs more work, although it may operate without segmentation under some circumstances.

    Source: §5 · p. 36

Abstract

from arXiv · show

Style-transfer is a process of migrating a style from a given image to the content of another, synthesizing a new image which is an artistic mixture of the two. Recent work on this problem adopting Convolutional Neural-networks (CNN) ignited a renewed interest in this field, due to the very impressive results obtained. There exists an alternative path towards handling the style-transfer task, via generalization of texture-synthesis algorithms. This approach has been proposed over the years, but its results are typically less impressive compared to the CNN ones. In this work we propose a novel style-transfer algorithm that extends the texture-synthesis work of Kwatra et. al. (2005), while aiming to get stylized images that get closer in quality to the CNN ones. We modify Kwatra's algorithm in several key ways in order to achieve the desired transfer, with emphasis on a consistent way for keeping the content intact in selected regions, while producing hallucinated and rich style in others. The results obtained are visually pleasing and diverse, shown to be competitive with the recent CNN style-transfer algorithms. The proposed algorithm is fast and flexible, being able to process any pair of content + style images.

1 Introduction

This section frames style transfer as combining image content and style despite an ill-defined balance, and presents texture synthesis as an approach that retains content in selected regions while allowing stylistic hallucination elsewhere. It also outlines the paper’s progression from background and method through results, conclusions, and supplementary examples.

1 Introduction

  • The algorithm extends Kwatra et al.’s texture-synthesis method through noisy content initialization and iterative style-to-content color transfer.The initialization ties the output to content in selected areas while allowing it to depart elsewhere; color transfer helps retain style richness and avoid repetitive patterns.Source: §1 · p. 4
  • At each iteration, patch aggregation merges the intermediate result with content in segmented regions, keeping their edges and objects mostly in place despite slight shifts from patch matching.The segmentation pre-process defines which regions to preserve and the extent of preservation; local contrast changes are allowed during content pre-processing.Source: §1 · p. 4
  • The method reduces gracefully to Kwatra’s classic texture synthesis algorithm, applied on the style image.The whole process takes 25–50 seconds for a 400 × 400-pixel image in a direct Matlab implementation without Mex.Source: §1 · pp. 4, 6
  • The method allows outputs to depart from content and hallucinate random combinations of style elements, while merging style patches and varying scales and patch sizes to discourage large copied regions.The paper identifies this freedom to hallucinate as a source of its high-quality artistic results.Source: §1 · p. 6
  • Every style-image detail is a candidate for transfer, with randomized patch matches influenced by content regions allowed to drift; an additional mask can restrict style regions, such as faces.The method treats the appearance of style-image faces in the result as acceptable unless they are masked out.Source: §1 · p. 6
  • Output quality depends chiefly on segmentation preserving important content, as well as palette compatibility and the match between content detail and style-object scale.The authors caution that a robust, consistent algorithm does not make every output successful.Source: §1 · p. 6
  • The proposed algorithm’s results are competitive with recent style-transfer algorithms, including CNN [39] and texture-synthesis-based methods, producing visually pleasing images with rich, diverse style-derived hallucinated parts while preserving content essence.Source: §1 · p. 6
  • Section 2 reviews related work, and Section 3 presents the proposed algorithm in relation to Kwatra’s work.The related-work discussion provides background for the method.Source: §1 · p. 6
  • Section 4 presents results, discusses the method’s key features, and includes both successes and failures.The results demonstrate various aspects of the proposed algorithm.Source: §1 · pp. 6–7
  • Section 5 concludes the work and highlights possible extensions; an appendix provides additional examples.Source: §1 · p. 7

2 Past Work on Style-Transfer

Earlier texture-synthesis work developed patch-matching and patch-fusion methods, with Kwatra et al. adding global optimization and multiscale processing. Style-transfer approaches later included image analogies, texture-synthesis methods, and CNN-based optimization; this work extends patch-based texture synthesis while aiming for results closer to CNN style-transfer.

Past Work on Style-Transfer

  • Earlier patch-based synthesis methods used seam curves or belief propagation to merge patches, while analogy-based stylization requires an input image and its stylized counterpart.The analogy-based branch is distinguished from the problem addressed here.Source: §2 · p. 7
  • Kwatra et al. formulate texture synthesis as global minimization of accumulated local patch distances, alternating between patch matching and updating the synthesized image.Their method processes multiple patch sizes from large to small and can operate on a Gaussian pyramid.Source: §2 · p. 7
  • The proposed method adopts and extends Kwatra et al.’s texture-synthesis ideas, aiming for CNN-like style-transfer while retaining a classic patch-matching approach.The adopted ideas include sequential processing from large to small patches and operation on a Gaussian pyramid.Source: §2 · pp. 7–8
  • Frigo et al. adapt texture synthesis to content using adaptive quad-tree patches, but each patch must remain close to its content patch, limiting hallucination.Their subdivision depends on style-image nearest-neighbor distance and content-patch variance.Source: §2 · p. 8
  • Gatys et al. use a pretrained CNN feature domain to seek an image close to the content while matching style-image feature correlations.Their approach frames transfer as energy minimization and requires numerical optimization.Source: §2 · p. 8

3 The proposed Algorithm

The algorithm performs style transfer through style-patch matching and image updates across resolutions and patch sizes, incorporating content-preservation penalties weighted by a segmentation mask. It uses palette matching and reduced-dimensional nearest-neighbor search to guide and accelerate a coarse-to-fine process that retains delicate textures.

3 The proposed Algorithm

  • The proposed algorithm adapts ingredients borrowed from Kwatra et al. with changes aimed at handling style transfer.Source: §3 · pp. 8–9

3.1 Energy Minimization Point-of-View

  • The energy matches output patches to style patches while a segmentation-weighted content term preserves marked regions and a prior encourages spatial smoothness across pyramid layers.The algorithm evaluates the penalty at multiple patch sizes and image scales.Source: §3.1 · pp. 9–10

3.2 The EM Optimization Structure

  • It uses block-coordinate descent: find each output patch’s nearest style patch, then update the output with matches fixed through Plug-and-Play denoising using Domain-Transform filtering.The Domain-Transform filter is described as a fast approximation to the bilateral filter.Source: §3.2 · pp. 10–12

3.3 Multi-Scale and Multi-Patch Sizes

  • Rather than combine scales and patch sizes into one objective, the algorithm sweeps through them sequentially to improve the steady-state result and reduce local-minimum risk.Source: §3.3 · p. 13
  • Coarse-to-fine resolutions and progressively smaller patches refine large style elements locally, while mask weights keep important regions near the content and allow low-weight regions to hallucinate.The content image influences the temporary result more strongly where W is high and more delicately where W is low.Source: §3.3 · p. 13

3.4 Segmentation and its Role

  • Segmentation is outside the work’s scope, and no tested option is clearly preferable; options include edge-based region filling and face/body detection followed by GrabCut [47].Affinity-based segmentation was also explored but excluded from the reported experiments.Source: §3.4 · pp. 13–14
  • With constant W, removing segmentation has inconclusive results: α = 0 yields style-only texture synthesis, α →∞ prevents style transfer, and intermediate values often look unpleasing.The authors also describe the method’s clear building blocks as enabling meaningful user control, but note it tends to influence important content regions less than CNN-based methods, a shortcoming they plan to address.Source: §3.4 · p. 14

3.5 Color Transfer

  • Palette matching before optimization and after each refinement round helps retain the style palette’s diversity; matching to a richer style palette often gives better visual results.The reported results use Matlab histogram matching; without palette re-projection, the algorithm may omit style colors such as yellow.Source: §3.5 · p. 15
  • The algorithm treats every style-image element as style, so distinctive objects such as faces may appear in unrelated output locations; masking style regions can manage this.Style images with unnatural dominant colors or large dark areas can also produce unpleasant results, so the authors advise careful selection and preprocessing.Source: §3.5 · pp. 15–16

3.6 Fast Nearest Neighbor (NN) Used

  • For faster nearest-neighbor patch matching, PCA reduces patch-vector dimensionality and linear comparisons across style patches, retaining 95% of the original data energy.Style patches are organized by resolution and patch size; the reduced dimension is chosen to preserve that energy.Source: §3.6 · p. 16
  • Style patches and query vectors are projected into a reduced PCA space for nearest-neighbor search, after which the corresponding original style patch is retrieved.Source: §3.6 · p. 16
  • Following Kwatra et al., the search uses a clustering tree whose stages allow 50% of examples to overlap between clusters to avoid inaccuracies.Source: §3.6 · p. 16

3.7 An Overall Description of the Proposed Algorithm

  • The algorithm proceeds from coarsest to refined resolutions, updating and scaling up X at each layer, and omits the final weighted average of C to preserve delicate textured regions.Source: §3.7 · pp. 16–17

4 Results

The results section presents style-transfer examples using face-based segmentation and reports that foreground content is largely preserved while background regions can hallucinate style. It also gives baseline settings for the experiments.

4 Results

  • Unless stated otherwise, experiments use 400 × 400-pixel images, patch sizes [33, 21, 13, and 9], subsampling gaps [28, 18, 8, and 5], Lmax = 3 resolution layers, and robust fusion r = 0.8.Source: §4 · p. 17

4.1 Style-Transfer Examples

  • The first and third results include parts of the face from the style image, which the method permits as part of the style definition.The effect is more pronounced in auxiliary results.Source: §4.1 · p. 17
  • A darker-than-desired bottom region in one example is attributed to the palette transfer.Source: §4.1 · p. 17
  • In some face-segmented examples, segmentation marks a hand or cell phone as background, and those regions disappear from the result.The text reports a vanished hand in another example as well.Source: §4.1 · p. 20
  • Stopping while larger patches are being matched would change the appearance of the segmented content, which otherwise appears nearly unchanged with slight modifications.Source: §4.1 · p. 20
  • The face-segmented examples produce an acceptable transfer competitive with state of the art, allowing background hallucination while largely preserving foreground content.The face-segmentation figure shows content, style, and resulting images; foreground regions are those where content is preserved.Source: §4.1 · pp. 17, 20, Figure 4 · p. 19

4.2 No-Segmentation Results

The paper shows some visually successful transfers without segmentation, but reports that their success depends on the content and style images.

4.2 No-Segmentation Results

  • The no-segmentation variant is produced by setting W to a constant value.Source: §4.2 · p. 20
  • Without segmentation, some examples yield relatively successful style transfer, but these are the better-looking cases rather than a consistent outcome.Other no-segmentation results are reported as not visually pleasing.Source: §4.2 · p. 20
  • Successful no-segmentation transfer depends on content images without too many details and style images based on localized rather than global brush elements.Source: §4.2 · p. 20

4.3 Tendency to Randomness

Repeated runs with identical inputs and parameters produce slightly different style-transfer results because the algorithm contains inner randomness. The authors note that randomness could be increased through rougher approximate nearest-neighbor search and random patch initialization, but they did not explore these options.

4.3 Tendency to Randomness

  • Repeated runs on the same content–style pair with identical parameters produce slightly different results because of inner algorithmic randomness.Figure 7 shows four runs under the same settings.Source: §4.3 · p. 20
  • The authors suggest that rougher approximate nearest-neighbor search and random patch assignments at each round could strengthen the randomness.Random patch initialization was proposed in.Source: §4.3 · p. 20
  • The captions label Figure 5 as style transfer via edge-segmentation and Figure 6 as no-segmentation, while both also state that all images were segmented using the Edge-Based method.Source: Figure 5 · p. 21, Figure 6 · p. 22

4.4 Comparison to [36] and [3]

Against Gatys et al., the method is described as competitive but perhaps slightly inferior, while being much faster and requiring no special pre-training. Against Frigo et al., its results tend to be more vivid and varied, with differences in palette transfer and foreground handling.

Comparison to Gatys et al. [36]

  • The method's style-transfer is competitive with Gatys et al., perhaps slightly inferior, while being far faster and requiring no special pre-training.Palette transfer plays a key role in the images compared with Gatys' work.Source: §4.4 · p. 24
  • In the compared images, the content-image skies are forced to remain bluish, limiting which style-image elements are adopted there.The supplied passage also contrasts this with long brush-strokes in Gatys' results.Source: §4.4 · p. 24
  • Compared with Gatys' results, the method lacks very long brush-strokes and is more conservative in handling segmented foreground content.The authors identify these differences as directions for future research.Source: §4.4 · pp. 24, 26

Comparison to Frigo et al. [3]

  • Compared with Frigo et al., the method's images tend to be more vivid and use larger, more diverse style ingredients.The authors attribute this mostly to freedom for some regions to hallucinate their final appearance.Source: §4.4 · pp. 24, 26

4.5 Failure Cases

The failure cases expose sensitivities to palette transfer, segmentation, content–style scale mismatch, and style-image choice. In particular, poorly chosen palettes or absent segmentation can wash out or erase recognizable content, including faces.

Failure Cases

  • The authors report unsatisfactory results on some faces despite describing the algorithm as stable and robust.In these cases, face regions were overridden too aggressively by style.Source: §4.5 · p. 26
  • Poor initial palette transfer can wash out important object details when brightness and contrast are set badly.The described matching of the content's brightest parts to the style's brightest colors can be overdone.Source: §4.5 · p. 26
  • When content heads are much smaller than the style's core elements, faces can become unrecognizable despite a rich, interesting outcome.The authors say most users would consider this a failure.Source: §4.5 · p. 26
  • A style image with large bright or dark regions, or too few expressive colors, can ruin transfer results.The paper frames a rich palette as a partial criterion for an appropriate style image.Source: §4.5 · pp. 26, 28
  • Without segmentation, the method can lose critical details that make content photos recognizable, especially in images containing faces.The described unsegmented examples had unpleasant outcomes.Source: §4.5 · p. 26, Figure 16 · p. 29
  • Poor edge-based segmentation can fail to recover critical face regions and thereby produce poor-quality results.The reported examples used the Edge-Based segmentation method.Source: §4.5 · p. 26, Figure 13 · p. 27

4.6 A Closer view of the Optimization Process

The optimization proceeds over a resolution pyramid and decreasing patch sizes, alternating patch matching and aggregation with content enforcement and palette transfer. These steps affect the objective differently, and the authors identify some iteration choices and stages as adjustable.

4.6 A Closer view of the Optimization Process

  • The algorithm processes three resolution levels, from 100 × 100 through 200 × 200 to the native 400 × 400 image size.The penalty-function graph separates into one part for each resolution layer.Source: §4.6 · p. 29
  • Within each layer, the algorithm sweeps through several patch sizes from largest to smallest and targets an objective for each layer–patch-size pair.These objectives are denoted E_L,n, where L is the resolution layer and n is the patch size.Source: §4.6 · p. 29
  • Patch matching finds nearest neighbors for sampled-grid patches, but approximate nearest-neighbor search does not necessarily decrease the objective.The method then aggregates the found neighbors using IRLS.Source: §4.6 · pp. 29–30
  • Patch aggregation sharply lowers the objective, whereas content enforcement and palette transfer each raise it.Content enforcement merges content into selected regions according to the segmentation map; palette transfer then enforces the style palette.Source: §4.6 · p. 30
  • The authors consider 110 iterations excessive, suggesting that 2–3 would have been sufficient.Their discussion concerns the patch-aggregation step.Source: §4.6 · p. 30
  • At the final E_1,7 stage, the method omits content enforcement and palette transfer to process content regions delicately.This omission applies to the very last stage only.Source: §4.6 · p. 30
  • Across the pyramid and decreasing patch sizes, the image is refined, and some steps might be omitted without much harm to the final image.Figure 19 displays initialization, patch-aggregation steps, content enforcement, and palette-transfer results for each resolution layer.Source: §4.6 · pp. 30–31

4.7 Parameters and their Influence

The experiments examine how pyramid depth, patch sizes, and patch overlap affect style-transfer results. Deeper pyramids adopt more style, while denser patch grids add little visual benefit and increase runtime.

4.7 Parameters and their Influence

  • Higher pyramid depth adopts more style; L = 4 is richer but risks copying whole style-image pieces, while L = 1 is conservative and resembles results by.Figure 20 compares pyramid depths from L = 4 to L = 1.Source: §4.7 · p. 31, Figure 20 · p. 33
  • Avoiding smaller patches yields a more textured content region; finer patches without content enforcement may refine the result while retaining the textured foreground.The tested patch-size sets range from [33, 21, 13, 9, 5] to [33, 21].Source: §4.7 · pp. 31, 34, Figure 21 · p. 34
  • Coarser sampling produces nearly the same results as denser overlap grids, while density increases runtime linearly, so the authors use coarse sampling.The reported changes are attributed to algorithm randomness.Source: §4.7 · pp. 34–35

4.8 Computational Complexity

The implementation runs in tens of seconds on 400 × 400 images, with patch matching—especially at full resolution—the most demanding part. Runtime comparisons with CNN methods are difficult because implementations differ, but the algorithm needs no pretraining for new content-style pairs.

4.8 Computational Complexity

  • 25–50 seconds are reported for 400 × 400 images, with fast-nearest-neighbor tree files preloaded, a step taking 20–30 seconds.The implementation used a desktop Windows machine with a 3.5GHz Intel Xeon E5-1650 v3 CPU and 32 GByte RAM.Source: §4.8 · p. 35
  • Patch matching is the most demanding part of the algorithm, particularly at the final full-resolution stage.The reported runtime varies with the patch sizes used.Source: §4.8 · p. 35
  • The method is described as faster and simpler than Gatys’ algorithm [36] and roughly as complex as Frigo et al., but runtime comparisons with CNNs are difficult because implementations differ.Unlike CNN methods that train a feed-forward network, this algorithm works on any content-style pair without pretraining.Source: §4.8 · p. 35

5 Conclusions

The paper presents a texture-synthesis-based style-transfer algorithm built on Kwatra et al.’s work, with modifications for transferring style while preserving content. Its conclusions identify image-scale matching and segmentation as important to outcomes, while outlining limitations and directions for improvement.

5 Conclusions

  • The algorithm builds on Kwatra et al.’s texture-synthesis method, augmented with changes for the style-transfer task.Source: §5 · pp. 35–36
  • Effective and satisfactory style transfer is described as within reach, with pleasing results strongly tied to matching the scale of style objects to content details.The authors also report that the palette-transfer stage has both good and bad effects.Source: §5 · p. 36
  • The authors say palette transfer has both beneficial and adverse effects, and that its design needs further work to improve transfer quality.Source: §5 · p. 36
  • Tests used 400 × 400 images, and the authors expect transfer results to change as the relative scale of content and style images changes.They call for more study to understand the effect and choose scales automatically.Source: §5 · p. 36
  • The method relies on segmentation to protect important content, although it can operate without segmentation under some circumstances.Further work is needed to remove segmentation while preserving result quality.Source: §5 · p. 36
  • Texture-synthesis-based transfer may copy pieces from the style image; enriching the patch dataset, for example with rotated or scaled patches, is proposed as a way to prevent this.Source: §5 · p. 36
  • The algorithm remains conservative on segmented content, and the authors envision changing content while preserving its essence.They note that this requires defining what is important in a content image.Source: §5 · p. 36

A Auxiliary Results

The auxiliary results add examples for empty-content, face-segmented, edge-based-segmented, and non-segmented inputs. They also refer to differing outcomes across runs and scales.

A Auxiliary Results

Loading 1609.03057v3…