Source-linked AI summary
A Style-Aware Content Loss for Real-time HD Style Transfer
Artsiom Sanakoyeu, Dmytro Kotovenko, Sabine Lang, Björn Ommer
TL;DR
Prior style-transfer methods were constrained by single-image style representations, RGB or ImageNet-based content comparisons, and related data or supervision assumptions. The paper jointly trains a style-aware content loss and encoder-decoder network for real-time, high-resolution stylization, reporting improved capture of how styles affect content across qualitative, quantitative, and expert evaluations.
Problem
Prior methods represented style with a single image or used RGB comparisons and ImageNet-pretrained features, limiting an art-historically grounded treatment of how style preserves content.
Method
The paper jointly trains a style-aware content loss from scratch with an encoder-decoder network, using encoded input and stylized output representations to learn style-specific content retention.
Results
The approach enables real-time, high-resolution stylization of images and videos and significantly improves stylization by capturing how style affects content.
Takeaways & Limitations
Style-aware content loss provides a content representation that adapts to the specific details each artistic style retains or disregards.
Takeaways & Limitations
The backward projection of abstract art to the original image is highly underdetermined, causing the approach to fail as styles become abstract.
Abstract
from arXiv · showhide
Recently, style transfer has received a lot of attention. While much of this research has aimed at speeding up processing, the approaches are still lacking from a principled, art historical standpoint: a style is more than just a single image or an artist, but previous work is limited to only a single instance of a style or shows no benefit from more images. Moreover, previous work has relied on a direct comparison of art in the domain of RGB images or on CNNs pre-trained on ImageNet, which requires millions of labeled object bounding boxes and can introduce an extra bias, since it has been assembled without artistic consideration. To circumvent these issues, we propose a style-aware content loss, which is trained jointly with a deep encoder-decoder network for real-time, high-resolution stylization of images and videos. We propose a quantitative measure for evaluating the quality of a stylized image and also have art historians rank patches from our approach against those from previous work. These and our qualitative results ranging from small image patches to megapixel stylistic images and videos show that our approach better captures the subtle nature in which a style affects content.
1 Introduction
Style transfer must separate subject matter from artistic style while preserving content, yet prior approaches were limited by single-style references, RGB comparisons, ImageNet pretraining, or slow optimization. This paper proposes a style-aware content loss with an encoder-decoder network and evaluates its high-resolution, real-time stylization using qualitative, quantitative, and art-historical assessments.
- Motivation: Style transfer exchanges an image’s style while retaining its content, requiring a vision system to represent the two aspects separately.Here, content means the subject matter or depicted scene and objects.
- Related Work: Previous feed-forward methods commonly used VGG networks pre-trained on ImageNet, introducing dependence on millions of labeled object bounding boxes and possible bias without artistic consideration.Earlier approaches also included methods requiring similar content-style image pairs, which do not scale to arbitrary content images.
- Method: The proposed style-aware content loss is learned from scratch with an encoder-decoder, comparing encoded input and stylized output so retained content adapts to each style.The encoder measures reconstruction loss while the adversarial discriminator and network stylize the input content image.
- Motivation: Artistic style comprises diverse elements such as form, color, brushstroke, and light, so one artwork may not represent its full scope.The paper contrasts this with art history’s grouping of works by shared qualities and notes that datasets such as WikiArt contain more than 100K images.
- Evaluation: The approach targets real-time, high-resolution stylization of images and videos and is evaluated against numerous baselines using qualitative results, a deception-rate metric, and art-history experts.The paper reports more visually detailed stylizations than current state-of-the-art approaches while retaining real-time inference speed.
2 Approach
The approach uses a feed-forward encoder-decoder with adversarial training and a jointly learned style-aware content loss. It groups related artworks automatically and adapts the encoded content representation to the target style.
- Architecture: A feed-forward encoder-decoder maps content images to latent representations and generates stylized outputs in a single forward pass.The encoder E produces z = E(x), and the decoder G produces y = G(z).
- Training objective: The model uses an adversarial discriminator to distinguish generated stylizations from real examples drawn from a set of related style images.The adversarial term is combined with content and transformed-image losses in the full objective.
- Style-aware content loss: The style-aware content loss compares input and stylized images in a jointly learned latent space, retaining details relevant to the target style.The loss is a normalized squared Euclidean distance between E(x_i) and E(y_i), where y_i = G(E(x_i)).
- Style-aware content loss: The latent space produces style-specific sketchy content representations, with reconstructions showing that different styles retain different structures and details.Pollock representations ignore much object structure, while Gauguin representations emphasize same-color surfaces and omit fine details.
- Style-aware content loss: A transformed image loss applies a learned one-layer convolutional transformer before measuring differences, avoiding the structural rigidity of direct RGB comparison.The transformer T produces an image of the same size and is initialized with uniform weights during training.
- Style image grouping: Related style images are automatically retrieved from WikiArt using a VGG16 artist-classification network trained from scratch and nearest-neighbor distances.The retrieval uses fc6 activations and a 10% quantile threshold over pairwise dataset distances; no style labels are used.
3 Experiments
The experiments compare the proposed method with multiple style-transfer baselines, ablate its components, and assess qualitative detail, high-resolution output, video speed, deception rate, and expert judgments.
- Evaluation setup: The evaluation combines qualitative comparisons, deception-rate measurements, art-history expert scores, and ablations against numerous style-transfer baselines.The experiments also measure inference time and GPU memory consumption.
- Qualitative results: The proposed method produces more style-dependent content changes and fine-grained details than competing approaches across varied styles and content images.Competitors are described as oversmoothing, producing repetitive artifacts, misplacing colors, or mainly shifting color histograms.
- Qualitative results: 1280x1280-pixel outputs from the approach contain visible fine details and brushstrokes.The figure presents a style example alongside the generated high-resolution image.
- Ablation studies: Removing the style-aware content loss significantly degrades results and can make training unstable, while a trained-from-scratch encoder outperforms a pre-trained VGG16 encoder.Removing the transformed image loss is reported to cause mode collapse after 5000 iterations.
4 Conclusion
The conclusion presents the style-aware content loss as addressing limitations involving single-style-image training, pixel-wise comparisons, and ImageNet-pretrained models. It enables real-time, high-resolution stylization of images and videos while capturing how styles affect content.
- Conceptual contributions: The method addresses the limitation of relying on only a single style image or requiring style and content training images with similar content.The conclusion contrasts this approach with prior requirements for style and content data.
- Conceptual contributions: The style-aware content loss avoids merely pixel-wise comparisons and models pre-trained on millions of ImageNet bounding boxes.The conclusion frames these as conceptual issues in state-of-the-art style-transfer approaches.
- Supported outcome: The proposed encoder-decoder enables real-time, high-resolution stylization of images and videos while capturing how style affects content.The conclusion states that this significantly improves stylization within that setting.
A.1 Network Architecture
The method uses an encoder-decoder generator with residual and upscaling blocks, a transformer block, and a patch-based discriminator. Auxiliary multi-scale classifiers are added to the discriminator to capture details and alleviate artifacts.
- Overall architecture: The architecture is instantiated as an encoder-decoder network with a discriminator, following a naming convention similar to prior work.Layer notation specifies convolutional, downsampling, residual, and upscaling components.
- Encoder-decoder architecture: The encoder-decoder uses five encoder convolutional layers, nine residual blocks, four upscaling blocks, and a final 7 × 7 sigmoid convolution.The encoder begins with InstanceNorm and uses one stride-1 and four stride-2 convolutional layers.
- Transformer block: The transformer block combines a convolutional layer with weight normalization constrained by fixed norm ||W|| = 1.Its convolution uses three uniformly initialized 10 × 10 kernels with stride 1.
- Discriminator: The discriminator is fully convolutional and classifies input image patches as real or fake using seven convolutional layers and Leaky ReLU activations.The Leaky ReLU slope is 0.2, followed by a one-filter output convolution without activation.
- Auxiliary classifiers: Four auxiliary one-filter classifiers are inserted after discriminator layers 1, 2, 4, and 6 to capture details at multiple scales and alleviate artifacts.Their losses are summed across scales.
A.2 Style Image Grouping Details
The style-image set varies with the query neighborhood and style frequency in WikiArt, and experiments favor using many examples from the same style rather than too few.
- Style image grouping details: Style-image sets contain 55 to 1391 related images, depending on the query neighborhood size and the style’s frequency in WikiArt.The number of style images is not fixed across styles.
- Style image grouping details: Using more style examples improves results when the examples belong to the same style.The passage reports this as an experimental observation.
- Style image grouping details: Using too few style examples, such as one, leads to mode collapse in the reported ablation studies.
A.3 Training Details
Training uses randomly cropped 768 × 768-pixel patches and alternates encoder-decoder and discriminator updates according to discriminator accuracy.
- Training setup: All networks are trained from scratch on randomly cropped 768 × 768-pixel patches.Training runs for 300000 iterations with batch size 1 using Adam and learning rate 2 × 10−4.
- Training setup: The learning rate is reduced by a factor of 10 after 200000 iterations.The initial learning rate is 2 × 10−4.
- Optimization schedule: Encoder-decoder and discriminator updates alternate, with only the discriminator updated below accuracy 0.8 and only the encoder-decoder updated otherwise.Discriminator accuracy is calculated using an exponential moving average.
B Extra Qualitative Results
Additional results demonstrate stylization of artistic content, real-time high-definition videos, and high-resolution images, alongside an ablation of the independent transformer block.
- Artistic content: The method changes the style of existing artworks and handles artistic content images, a capability the authors report as absent from previous work.The passage directs readers to project-page results.
- Real-time HD video stylization: The approach stylizes 1920 × 1280-pixel videos, including Muybridge’s horse-in-motion video in Picasso’s style.Another 1920 × 1280 video is shown with Cezanne, Kandinsky, and Picasso stylizations.
- Transformer-block ablation: Replacing the proposed transformer-block loss with Lconv1 produces worse stylization results.The alternative loss is directly tied to the encoder’s conv1 layer, whereas the transformer block can be learned independently from early encoder layers.
C Extra ablation studies
The ablation replaces the transformed image loss with a conv1-feature loss, revealing that independently learning the transformer block is important for better results.
- Replacing the transformed image loss L_T with a conv1-feature loss L_conv1 produced worse results.The ablation used conv1 features from the encoder.
- L_conv1 performs worse because it is directly tied to the encoder convolutional layer used for the style-aware content representation.
- The proposed transformer block can instead be learned independently from the encoder’s early layers.