Source-linked AI summary
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Justin Johnson, Alexandre Alahi, Li Fei-Fei
TL;DR
Per-pixel losses do not adequately capture perceptual differences in image transformation tasks. The paper trains feed-forward networks with perceptual losses from pretrained features, achieving comparable style-transfer quality with three-orders-of-magnitude faster generation and visually pleasing super-resolution.
Problem
Per-pixel losses do not capture perceptual differences between transformed outputs and ground-truth images.
Method
The paper trains feed-forward transformation networks with perceptual losses defined from high-level features in a fixed pretrained classification network.
Results
Style transfer achieves comparable qualitative and objective performance while generating outputs three orders of magnitude faster, and perceptual-loss super-resolution produces visually pleasing results.
Takeaways & Limitations
Perceptual-loss training combines feed-forward image transformation with optimization-based image-generation benefits across style transfer and single-image super-resolution.
Takeaways & Limitations
Evidence is limited to style transfer and single-image super-resolution, while broader applications such as colorization and semantic segmentation remain future work.
Abstract
from arXiv · showhide
We consider image transformation problems, where an input image is transformed into an output image. Recent methods for such problems typically train feed-forward convolutional neural networks using a \emph{per-pixel} loss between the output and ground-truth images. Parallel work has shown that high-quality images can be generated by defining and optimizing \emph{perceptual} loss functions based on high-level features extracted from pretrained networks. We combine the benefits of both approaches, and propose the use of perceptual loss functions for training feed-forward networks for image transformation tasks. We show results on image style transfer, where a feed-forward network is trained to solve the optimization problem proposed by Gatys et al in real-time. Compared to the optimization-based method, our network gives similar qualitative results but is three orders of magnitude faster. We also experiment with single-image super-resolution, where replacing a per-pixel loss with a perceptual loss gives visually pleasing results.
1 Introduction
The paper combines feed-forward transformation networks with perceptual losses based on pretrained-network features, addressing perceptual limitations of per-pixel losses. It evaluates this approach on style transfer and single-image super-resolution, achieving similar style-transfer results three orders of magnitude faster and visually pleasing super-resolution results.
- Introduction: Per-pixel losses can poorly capture perceptual differences between output and ground-truth images, even for perceptually similar images shifted by one pixel.The one-pixel offset example illustrates why pixel-level differences may disagree with perceptual similarity.
- Introduction: The proposed method trains feed-forward transformation networks with perceptual losses based on high-level features extracted from a pretrained loss network.This combines efficient feed-forward inference with feature-based image similarity during training.
- Introduction: Perceptual losses transfer semantic knowledge from the loss network to transformation networks for inherently ill-posed style-transfer and super-resolution tasks.Style transfer has no single correct output, while multiple high-resolution images can correspond to the same low-resolution input.
- Introduction: Three orders of magnitude faster: feed-forward style-transfer networks produce results similar to the optimization-based method both qualitatively and by objective function value.The networks are trained to solve the optimization problem from Gatys et al.
- Introduction: Visually pleasing results: replacing per-pixel loss with perceptual loss works for ×4 and ×8 single-image super-resolution.The paper evaluates both style transfer and single-image super-resolution as image transformation tasks.
2 Related Work
Prior image-transformation methods commonly train feed-forward convolutional networks with per-pixel losses, while related work explores perceptual optimization using high-level convolutional features. This paper builds on feature inversion and style-transfer approaches by training feed-forward networks to approximate perceptual optimization more efficiently.
- Feed-forward image transformation: Feed-forward image-transformation methods commonly use deep convolutional networks trained with per-pixel losses.
- Feed-forward image transformation: Semantic segmentation, depth estimation, and surface-normal estimation transform images with feed-forward networks trained using per-pixel classification or regression losses.Some methods additionally penalize image gradients or use CRF loss layers to enforce local consistency.
- Perceptual optimization: Perceptual optimization generates or reconstructs images using objectives based on high-level features extracted from pretrained convolutional networks.Applications include maximizing class scores or individual features, generating fooling images, and inverting convolutional features to study retained image information.
- Perceptual optimization: Unlike feature-inversion work trained with per-pixel reconstruction loss, this paper trains feed-forward networks to directly optimize feature reconstruction loss.Dosovitskiy and Brox trained a feed-forward network to quickly approximate feature-inversion optimization, whereas the present networks directly optimize the feature reconstruction objective.
- Style Transfer: Gatys et al. perform artistic style transfer by jointly minimizing content feature-reconstruction and style-reconstruction losses based on a pretrained convolutional network.Their optimization produces high-quality results but is computationally expensive because each optimization step requires forward and backward passes through the pretrained network.
3 Method
The method trains a deep residual image-transformation network using perceptual losses defined by a pretrained classification network. Feature reconstruction measures content similarity, while style reconstruction captures stylistic statistics without necessarily preserving spatial structure.
- Method overview: The system combines an image transformation network f_W with a loss network φ that defines multiple scalar losses between outputs and target images.The transformation network maps input x to output ŷ = f_W(x), while each loss compares ŷ with a target.
- Method overview: The loss network φ is a pretrained convolutional classification network whose representations encode perceptual and semantic information.All experiments use the 16-layer VGG network pretrained on ImageNet.
- Perceptual losses: Feature reconstruction replaces exact pixel matching with similarity between output and target feature representations computed by φ.The loss uses activations from selected layers, represented as feature maps with dimensions C_j × H_j × W_j.
- Perceptual losses: Style reconstruction penalizes differences in colors, textures, and common patterns using statistics of feature activations, while preserving stylistic features rather than spatial structure.Higher-layer reconstruction transfers larger-scale structure from the style target.
- Pixel loss: The method also defines a pixel loss as the normalized Euclidean distance between the output image ŷ and target image y.This loss depends only on low-level pixel information.
4 Experiments
Experiments evaluate perceptual-loss image transformation networks on style transfer and single-image super-resolution. The networks produce qualitatively strong results, support larger inputs, and achieve real-time style transfer while perceptual super-resolution improves fine-detail reconstruction.
- Overview: Experiments cover style transfer and single-image super-resolution, comparing perceptual-loss feed-forward networks with optimization-based or per-pixel-loss baselines.The style-transfer baseline is Gatys et al., while super-resolution comparisons examine per-pixel and feature-reconstruction losses.
- Style Transfer: Style-transfer outputs preserve semantically salient objects selectively, with recognizable people and cat faces while less salient backgrounds or bodies become distorted.The authors suggest this reflects VGG-16 loss features selective for people and animals.
- Style Transfer: At 512 × 512 and 1024 × 1024, the style-transfer model achieves a loss comparable to 50 to 100 baseline iterations despite training on 256 × 256 images.The model applies fully convolutionally at test time and generalizes to larger images.
- Style Transfer: Three orders of magnitude faster than 500 baseline iterations, the style-transfer network produces qualitatively similar results and processes 512 × 512 images at 20 FPS.Across image sizes, its runtime is approximately twice the speed of one baseline optimization iteration.
- Single-Image Super-Resolution: Compared with the per-pixel-loss model, the feature-loss super-resolution model reconstructs fine details and avoids indiscriminate sharpening, producing pleasing visual results.The per-pixel model has fewer visual artifacts and higher PSNR values, whereas the feature-loss model better reconstructs fine details.
5 Conclusion
The paper combines feed-forward image transformation networks with optimization-based image generation through perceptual loss functions. It applies this approach to style transfer and single-image super-resolution, while identifying broader image-transformation tasks and alternative loss networks for future work.
- Conclusion: The method trains feed-forward transformation networks with perceptual loss functions, combining feed-forward image transformation with optimization-based image generation.This framework is applied to both style transfer and single-image super-resolution.
- Conclusion: Style transfer achieves comparable performance with drastically improved speed compared to existing methods.
- Conclusion: For single-image super-resolution, perceptual-loss training allows the model to better reconstruct fine details.
- Future work: Future work will explore perceptual losses for colorization and semantic segmentation, and investigate different loss networks trained on other tasks or datasets.The authors aim to determine whether alternative loss networks impart different types of semantic knowledge to image transformation networks.