A Neural Algorithm of Artistic Style

Leon A. Gatys; Alexander S. Ecker; Matthias Bethge

Source-linked AI summary

A Neural Algorithm of Artistic Style

Leon A. Gatys, Alexander S. Ecker, Matthias Bethge

arXiv:1508.06576v2Aug 26, 2015cs.CVcs.NEq-bio.NC

TL;DR

The algorithmic basis for combining image content and style was unknown, and no artificial system had similar capabilities. This paper uses neural representations from a deep network to separate and recombine content and style, producing perceptually meaningful artistic images.

Problem
A neural representation separating content from style in whole natural images had not been demonstrated, a gap relevant to creating artistic images.
Source: p. 6
Method
The system derives content and style representations from feature responses of an object-recognition network and synthesises images matching both representations.
Source: pp. 4, 6
Results
The network’s content and style representations are separable, enabling images that preserve a photograph’s global arrangement while adopting an artwork’s colours and local structures.
Source: p. 4
Takeaways & Limitations
The method provides a tool for studying art perception and neural representations using stimuli that vary image appearance and content independently.
Source: p. 8
Content and style cannot be completely disentangled because an image that perfectly matches both constraints usually does not exist.
Source: p. 6

Abstract

from arXiv · show

In fine art, especially painting, humans have mastered the skill to create unique visual experiences through composing a complex interplay between the content and style of an image. Thus far the algorithmic basis of this process is unknown and there exists no artificial system with similar capabilities. However, in other key areas of visual perception such as object and face recognition near-human performance was recently demonstrated by a class of biologically inspired vision models called Deep Neural Networks. Here we introduce an artificial system based on a Deep Neural Network that creates artistic images of high perceptual quality. The system uses neural representations to separate and recombine content and style of arbitrary images, providing a neural algorithm for the creation of artistic images. Moreover, in light of the striking similarities between performance-optimised artificial neural networks and biological vision, our work offers a path forward to an algorithmic understanding of how humans create and perceive artistic imagery.

Methods

The method uses VGG feature activations to represent image content and correlations between filter responses to represent style. It synthesizes images by gradient descent to match these representations, either separately or through a joint content-and-style loss.

Network

The method uses VGG’s 16 convolutional and 5 pooling layers, replacing max pooling with average pooling for improved gradient flow and slightly more appealing images.The fully connected layers are not used; VGG is cited as rivaling human performance on a visual object recognition benchmark 23, and the model is available in the caffe framework 24.Source: Methods · p. 9

Content reconstruction

Content reconstruction starts from white noise and uses gradient descent to produce an image whose responses in a chosen CNN layer match those of the original image.Each layer encodes an image through its filter responses; illustrated reconstructions use layers ‘conv1 1’ through ‘conv5 1’.Source: Methods · p. 10

Style reconstruction

Style is represented by spatially averaged correlations between filter responses, and texture reconstruction minimizes the mean-squared difference between the original and generated images’ Gram matrices.Gradient descent begins from a white-noise image; style reconstructions progressively match representations from ‘conv1 1’ through ‘conv5 1’.Source: Methods · pp. 10–11

Combined synthesis

For combined synthesis, the method jointly matches photograph content in ‘conv4 2’ and painting style across ‘conv1 1’ through ‘conv5 1’.For Fig. 2, style-layer weights were 1/5; the content-to-style loss ratio was 1×10^-3 for panels B–D and 1×10^-4 for E–F. Fig. 3 varies relative loss weights and the number of style-matching layers.Source: Methods · pp. 11–12