PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Style Transfer: A Review

Yongcheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Yizhou Yu, Mingli Song

arXiv:1705.04058v7cs.CVcs.NEeess.IVstat.ML

TL;DR

Recent Neural Style Transfer advances require an overview of algorithms and more reliable ways to evaluate their benefits and suitability. This review classifies current methods, presents evaluation approaches, compares algorithms, and summarizes challenges and future directions.

  • Problem

    Neural Style Transfer has rapidly advanced, but reliable criteria for comparing methods and assessing their suitability for particular scenarios remain limited.

  • Method

    The review proposes a taxonomy of Neural Style Transfer algorithms, presents evaluation methods, compares algorithms, and summarizes challenges and possible future directions.

  • Results

    The review experimentally compares different Neural Style Transfer algorithms using several qualitative and quantitative evaluation methods.

  • Takeaways & Limitations

    The review identifies interpretability, aesthetic evaluation, style diversity, perceptual quality, speed, and hyperparameter tuning as challenges for future Neural Style Transfer research.

  • Takeaways & Limitations

    Data-driven arbitrary-style methods depend on training-style diversity, while learning-free image-transformation methods remain slower than other approaches.

Abstract

from arXiv · show

The seminal work of Gatys et al. demonstrated the power of Convolutional Neural Networks (CNNs) in creating artistic imagery by separating and recombining image content and style. This process of using CNNs to render a content image in different styles is referred to as Neural Style Transfer (NST). Since then, NST has become a trending topic both in academic literature and industrial applications. It is receiving increasing attention and a variety of approaches are proposed to either improve or extend the original NST algorithm. In this paper, we aim to provide a comprehensive overview of the current progress towards NST. We first propose a taxonomy of current algorithms in the field of NST. Then, we present several evaluation methods and compare different NST algorithms both qualitatively and quantitatively. The review concludes with a discussion of various applications of NST and open problems for future research. A list of papers discussed in this review, corresponding codes, pre-trained models and more comparison results are publicly available at https://github.com/ycjing/Neural-Style-Transfer-Papers.

1 INTRODUCTION

Neural Style Transfer (NST) emerged as a CNN-based approach for rendering a content image in different artistic styles without requiring paired ground-truth training data. This review surveys NST advances, evaluates algorithms, and discusses applications and future challenges.

  • Motivation: Earlier artistic-rendering methods were often designed for particular styles or relied on low-level features that poorly captured image structures.These limitations motivated more flexible approaches to artistic stylisation.
  • Neural Style Transfer: Gatys et al. modelled photographic content with CNN feature responses and artistic style with summary feature statistics.Their experiments showed that CNNs can extract content from photographs and style from artworks.
  • Neural Style Transfer: NST renders a content image in different styles using a Convolutional Neural Network.Gatys et al.'s method imposed no explicit restrictions on style images and required no ground-truth results for training.
  • Motivation: The seminal NST work prompted extensive academic follow-up and successful industrial applications, including Prisma, Ostagram, and Deep Forger.The review identifies a lack of a comprehensive survey of recent NST advances and challenges as its motivation.
  • Review Scope: The review classifies recent NST advances, compares algorithms with several evaluation methods, and summarizes challenges and possible future directions.Its stated coverage extends through March 2018 and includes the organization of prior rendering methods, NST foundations, algorithms, improvements, evaluation, applications, and open problems.

2 STYLE TRANSFER WITHOUT NEURAL NET-

Before CNN-based NST, image-based artistic rendering used stroke, region, example-based, and filtering techniques. These methods could depict prescribed styles or exploit local image structure, but commonly lacked flexibility, style diversity, or effective structure extraction.

  • Overview: Image-based artistic rendering is a long-standing area focused on artistic stylisation of 2D images without CNNs.The review follows an established taxonomy and discusses the strengths and weaknesses of each category.
  • Stroke-Based Rendering: Stroke-based rendering incrementally composites virtual strokes onto a source photograph to produce a non-photorealistic image in a prescribed style.Such algorithms can simulate styles such as oil paintings, watercolours, and sketches.
  • Region-Based Techniques: Stroke-based and region-based rendering methods generally cannot simulate arbitrary styles, limiting their flexibility despite local control over details in region-based methods.Region segmentation enables different stroke patterns and local detail levels across semantic regions.
  • Example-Based Rendering: Example-based rendering learns transformations from pairs of unstylised and stylised images, but such training pairs are usually unavailable and low-level features can fail to capture content and style.These limitations can reduce the performance of image analogies.
  • Image Processing and Filtering: Image-filtering methods combine filters to create simplified or abstract artistic effects and are generally straightforward and efficient, but have limited style diversity.Bilateral and difference-of-Gaussians filters were used to produce cartoon-like effects.
  • Overview: The limitations of pre-CNN artistic-rendering methods in flexibility, style diversity, and image-structure extraction motivated the emergence of NST.The review presents NST as a response to these shortcomings.

3 DERIVATIONS OF NEURAL STYLE TRANSFER

The review derives NST from visual texture modelling and CNN representation reconstruction. It explains how style can be represented through summary statistics, while reconstruction methods determine how those representations are converted back into images.

  • Foundations: The review frames style modelling as a central NST problem and relates it to visual texture modelling, while noting that style also includes simplification and shape abstraction.After style representation is obtained, image reconstruction is needed to produce an image containing the desired style information.
  • Visual Texture Modelling: Visual texture modelling uses parametric summary statistics or non-parametric Markov Random Fields (MRFs).These constitute two distinct approaches developed in texture synthesis.
  • Parametric Texture Modelling: Gatys et al. model CNN-domain textures with Gram-based representations encoding second-order correlations between filter responses in pre-trained VGG layers.The representation captures summary statistics rather than directly modelling pixels.
  • Parametric Texture Modelling: A Gram matrix is computed from a reshaped CNN feature map to represent correlations among its channels.For a feature map F^l(I_s) ∈ R^(C×H×W), the reshaped representation has dimensions C×(HW), and the Gram matrix has dimensions C×C.
  • Parametric Texture Modelling: Gram-based representations model many natural and non-natural textures effectively, but discard spatial arrangements and can perform poorly on regular textures with long-range symmetric structure.Spatially shifted feature correlations were proposed to incorporate arrangement information for symmetric textures.
  • Non-Parametric Texture Modelling: Non-parametric MRF methods synthesize textures by assuming each pixel is characterized by its spatial neighbourhood and resampling similar neighbourhoods from a source texture.The approach synthesizes pixels sequentially by searching for matching neighbourhoods.
  • Image Reconstruction: CNN representation reconstruction reverses feature extraction by producing an image with a desired representation, either through iterative image optimisation or a feed-forward network.Online optimisation is time-consuming, whereas offline model optimisation shifts computation to training and enables a forward pass at test time.

4 A TAXONOMY OF NEURAL STYLE TRANSFER ALGORITHMS

The review organizes NST methods into image-optimisation-based online methods and model-optimisation-based offline methods, then examines their representations, improvements, and trade-offs. Across these categories, methods address issues such as speed, style flexibility, detail coherence, depth preservation, and computational cost.

  • Taxonomy: NST methods are divided into IOB-NST, which iteratively optimizes an image, and MOB-NST, which trains a model offline for single-pass stylization.This taxonomy extends earlier IB-AR categories to neural methods.
  • Image-Optimisation-Based Online Neural Methods: Gram-based style representations enable NST but can cause optimization instability, manual parameter tuning, and weak preservation of fine details and depth.Gatys et al.'s approach also generally fails for photorealistic synthesis and does not model brush-stroke variation or content semantics explicitly.
  • Image-Optimisation-Based Online Neural Methods: Matching feature-activation histograms produces more stable style transfer with fewer iterations and less parameter tuning, but increases computational complexity.Depth and detail-coherence weaknesses remain.
  • Image-Optimisation-Based Online Neural Methods: Adding a Laplacian loss constrains pixel-space filter responses to improve coherence of fine structures that CNN feature matching can distort.The Laplacian loss is the squared Euclidean distance between content and stylized-image Laplacian responses.
  • Model-Optimisation-Based Offline Neural Methods: MOB-NST reconstructs stylized images with a feed-forward network, addressing the efficiency limitations of iterative image optimization.Per-style-per-model methods can achieve real-time transfer, while multi-style approaches trade model size or style scalability against detail and depth preservation.
  • Model-Optimisation-Based Offline Neural Methods: Instance normalization improves stylization quality and convergence over batch normalization by normalizing each image's style statistics.The method is interpreted as directly normalizing each content image toward the desired style, leaving the network to learn the content loss.
  • Model-Optimisation-Based Offline Neural Methods: AdaIN enables the first real-time arbitrary-style-per-model stylization, but its data-driven training limits generalization to unseen styles and rich local patterns.It transfers channel-wise mean and variance statistics between content and style activations.
  • Model-Optimisation-Based Offline Neural Methods: Learning-free artistic style transfer removes the generalization limitation but remains ineffective at sharp details, fine strokes, depth information, and brush-stroke variation.Li et al.'s method is identified as the first ASPM approach with a learning-free style-transfer procedure.

5 IMPROVEMENTS AND EXTENSIONS

The review surveys improvements that control perceptual factors in NST and extensions to specialized image types, videos, and photorealistic outputs. These methods address issues including stroke-size flexibility, semantic correspondence, object boundaries, facial structure, temporal consistency, and structural artefacts.

  • Controlling Perceptual Factors: Spatial style control uses guidance channels to specify which style transfers to each content region.
  • Controlling Perceptual Factors: Stroke-size control is complicated by scale-sensitive style statistics and differs across image resolution and optimization strategy.For high-resolution images, coarse-to-fine procedures address the limited effect of resizing style images.
  • Controlling Perceptual Factors: A StrokePyramid module enables one model to provide continuous stroke-size and spatial stroke-size control while preserving stroke consistency.The review states that this avoids trading off quality and speed, whereas ASPM is less effective for fine strokes and details.
  • Extensions: Semantic style transfer establishes correspondences between semantically similar regions before transferring style region by region.Patch matching supports this region-based correspondence, while segmentation accuracy remains important for the resulting transfer.
  • Extensions: Specialized extensions target individual objects, doodles, portraits, stereoscopic images, and video sequences with task-specific constraints.Examples include boundary smoothing, segmentation-based transfer, disparity loss, gain maps, and smooth transitions between adjacent frames.
  • Extensions: Photorealistic style transfer combines stylisation with smoothing to reduce structural artefacts, while unpooling layers produce results with fewer distortions.

6 EVALUATION METHODOLOGY

The review evaluates NST qualitatively through observer judgements and saliency, and quantitatively through speed, training, loss, convergence, and scalability measures. Its comparisons reveal trade-offs among visual quality, flexibility, and computational efficiency across NST categories.

  • Qualitative Evaluation: Qualitative evaluation relies on aesthetic judgements, but ratings vary across observers even when participants share occupation and age.The authors therefore use saliency comparison as an additional objective-oriented evaluation approach.
  • Evaluation Datasets: The evaluation uses ten diverse style images and twenty content images from the NPRgeneral benchmark, with content images excluded from offline-model training.The style set spans artistic styles, media, and image characteristics such as detail, contrast, complexity, and color distribution.
  • Qualitative Evaluation: IOB-NST produces appealing visual results but is computationally expensive, whereas PSPM-MOB-NST produces real-time stylisation while requiring a separate model for each style.PSPM-MOB-NST methods based on Ulyanov et al. and Johnson et al. appear visually similar, while Li and Wand’s GAN-based results are somewhat less impressive and training is less stable.
  • Qualitative Evaluation: MSPM-MOB-NST incorporates multiple styles into one model, but parameter-tied methods grow in model size while shared-weight methods experience interference among styles.These design choices create a flexibility–model-size or flexibility–quality trade-off.
  • Qualitative Evaluation: ASPM-MOB-NST transfers arbitrary styles, but its results are generally less impressive because speed, flexibility, and quality must be traded off.Patch-based methods may omit style elements, data-driven methods depend on training-style diversity, and global-statistics matching struggles with complex patterns.

7 APPLICATIONS

NST has moved from artistic stylisation research into social communication and industrial applications. The review also identifies prospective uses in creative tools, entertainment, film, animation, games, and 3D rendering.

  • Social Communication: NST applications such as Prisma, Ostagram, and Deep Forger provide artistic stylisation services and have begun delivering commercial benefits.Users can create and share stylised artwork through social platforms including Twitter and Facebook.
  • Social Communication: Social sharing of NST artwork can reinforce connections between people and provide feedback that helps researchers improve algorithms.The review also connects social-communication demands, such as real-time video stylisation, with advances in related techniques.
  • Creation Tools: NST is proposed as a user-assisted creation tool for painters, fashion designers, and architects, although the review notes that popular creation-tool applications were not yet established.Potential outputs include computer-made artworks, stylised fashion elements, and stylised CAD drawings.
  • Entertainment: Applying NST to movies, animations, and games could reduce the time and cost of producing stylised visual content.Animation production may require 8 to 24 painted frames per second, motivating automatic video stylisation.
  • Entertainment: Production studies apply NST to movie-scene redrawing and 3D-rendering tasks such as previews, shading autocomplete, and style transfer without a reference 3D model.These examples indicate potential uses beyond 2D image stylisation.

8 FUTURE CHALLENGES

The review highlights unresolved challenges in evaluating NST, interpreting and controlling its models, and balancing speed, flexibility, and quality. It discusses benchmark gaps, subjective aesthetics, disentangled representations, normalization, and adversarial vulnerability.

  • Aesthetic Evaluation: Aesthetic evaluation lacks reliable criteria and a gold-standard method because observers can assign substantially different ratings to the same stylised result.The authors report ratings from eight observers and note that evaluation becomes increasingly important as NPR and NST mature.
  • Aesthetic Evaluation: NST lacks a standard benchmark image set, so authors typically evaluate algorithms on independently selected images.This review uses the NPRgeneral content benchmark, while acknowledging limitations in its own image selection.
  • Interpretability: NST remains difficult to interpret and control because its CNN-based processing behaves like a black box.The review focuses on disentangled representations, normalization methods, and adversarial examples as three interpretability issues.
  • Representation Disentangling: Disentangled representations could allow individual style factors such as color, shape, stroke size, orientation, and composition to be controlled independently during stylisation.The proposed benefit depends on learning representations whose dimensions correspond precisely to separate factors of variation.
  • Normalisation Methods: Instance normalization can normalize image style through feature statistics, while conditional instance normalization scales and shifts parameters for different styles.These normalization developments also influence other vision tasks, including image recolourisation and video colour propagation.
  • Adversarial Examples: Adversarial examples also occur in generative style-transfer networks, where a perturbed input can produce a stylised output whose original content is hardly recognizable.This extends adversarial-example concerns beyond the classification networks emphasized in earlier studies.
  • Three-way Trade-off in Neural Style Transfer: NST exhibits a three-way trade-off: IOB-NST favors quality, PSPM-MOB-NST favors speed, MSPM-MOB-NST improves flexibility within style sets, and ASPM-MOB-NST supports arbitrary styles with weaker quality or speed.ASPM data-driven methods also depend on training-style diversity, while image-transformation methods are slower despite being learning-free.

9 DISCUSSIONS AND CONCLUSIONS

NST remains a fast-paced research area whose techniques have inspired work beyond style transfer, while important challenges remain in robustness and broader stylisation. Future directions include 3D surface stylisation and creating new AI-generated styles by combining existing ones.

  • NST research is fast-paced, with major advances summarised in Figure 2 and style-transfer loss functions surveyed in Table 5.The review describes continued activity driven by scientific challenges and industrial demands.
  • NST has inspired methods in image reconstruction and image transformation, including deep image priors and deep feature interpolation.Deep feature interpolation provides a new baseline for tasks such as face aging and smiling.
  • NST remains immature because stylisation quality and failure cases vary across different style and content images.The review notes that NST performs particularly well for some styles but remains weak for others.
  • A promising direction is directly optimising 3D objects for photorealistic and non-photorealistic surface stylisation.This extends stylisation beyond 2D image outputs to 3D surfaces.
  • Another direction is AI-created art that combines different styles under underlying aesthetic principles.One approach fuses coarse texture distortions from one style with fine brush strokes from another to produce a new style.
Loading 1705.04058v7…