PaperlayerView exact revision on arXiv

Source-linked AI summary

Characterizing and Improving Stability in Neural Style Transfer

Agrim Gupta, Justin Johnson, Alexandre Alahi, Li Fei-Fei

arXiv:1705.02092v1cs.CV

TL;DR

Real-time image style-transfer methods can flicker when applied to video, while optimization-based video stylization is too slow for real-time use. The paper analyzes instability through the style Gram-matrix solution set and introduces a recurrent network with temporal consistency loss. It reports stable, high-quality video style transfer with a 1000× speed improvement.

  • Problem

    Real-time feedforward style-transfer methods can produce visible flicker in videos, while optimization-based video stylization is too slow for real-time use.

    Source: §1 · p. 1

  • Method

    The paper examines Gram-matrix matching solution sets and uses a recurrent convolutional network trained with a temporal consistency loss for video stylization.

    Source: §1 · pp. 1–2

  • Results

    The method reports a 1000× speed improvement for stable video style transfer without sacrificing quality.

    Source: §1 · p. 2

  • Takeaways & Limitations

    For Gram-matrix-matching neural style-transfer methods, larger style Gram-matrix traces are associated with greater instability.

    Source: §1 · p. 2, §6 · p. 8

    The method sometimes produces slight object transparency during occlusion, typically only in the first one or two frames and not very perceptibly when viewing videos.

    Source: §5.4 · p. 7

Abstract

from arXiv · show

Recent progress in style transfer on images has focused on improving the quality of stylized images and speed of methods. However, real-time methods are highly unstable resulting in visible flickering when applied to videos. In this work we characterize the instability of these methods by examining the solution set of the style transfer objective. We show that the trace of the Gram matrix representing style is inversely related to the stability of the method. Then, we present a recurrent convolutional network for real-time video style transfer which incorporates a temporal consistency loss and overcomes the instability of prior methods. Our networks can be applied at any resolution, do not re- quire optical flow at test time, and produce high quality, temporally consistent stylized videos in real-time.

1. Introduction

Real-time feedforward style-transfer methods can flicker across video frames, while optimization-based video stylization is too slow for real-time use. The paper links instability to the style Gram-matrix trace and proposes recurrent video stylization with a temporal consistency loss.

1. Introduction

  • Optimization-based image stylization is high quality but takes minutes per image, whereas feedforward methods run in real time but flicker on video.Optimization-based video stylization also produces high-quality results but is too slow for real-time application.Source: §1 · p. 1
  • Figure 1 contrasts visually indistinguishable input frames that yield perceptibly different stylized images with the proposed temporally consistent output.The figure presents the instability of real-time methods and the recurrent network’s intended outcome.Source: Figure 1 · p. 1, §1 · p. 1
  • The trace of a style image’s Gram matrix determines the radius of a spherical solution set, so large traces are associated with greater instability.With a nonconvex objective, small content changes can lead to distant solutions and visibly different stylized images.Source: §1 · p. 1
  • The paper characterizes instability by examining the solution set and reports an inverse relation between Gram-matrix trace and stability.The characterization applies to neural style-transfer methods based on Gram-matrix matching.Source: §1 · p. 2
  • A recurrent convolutional network with a temporal consistency loss encourages similar Gram-matching solutions across time and produces stable, high-quality stylized video.The paper reports a 1000× speed improvement for stable video style transfer without sacrificing quality.Source: §1 · pp. 1–2

2. Related Work

Related work spans texture synthesis, neural style transfer, optical-flow methods, and video stylization. The paper positions its approach against prior methods that trade off speed, perceptual quality, and temporal consistency.

Texture Synthesis

  • Texture synthesis includes parametric methods that model global feature-space statistics and non-parametric methods that synthesize from local conditional distributions.Non-parametric approaches resample pixels or patches from the original texture.Source: §2 · p. 2
  • Neural texture and style methods use Gram matrices as summary statistics, while style swap replaces content activations with the closest-matching style patches.The style-swap optimization is described as more stable than earlier feedforward methods but still too slow for real-time video.Source: §2 · p. 2

Style Transfer

  • Feedforward methods made neural style transfer real-time, while conditional instance normalization enabled one model to learn multiple styles without sacrificing speed or perceptual quality relative to earlier feedforward methods.Earlier feedforward methods required a separate network for each style, and their perceptual quality was inferior to optimization-based stylization.Source: §2 · p. 2

Optical Flow

  • Optical-flow research includes variational methods and CNN estimators, with FlowNet reported to match variational-method performance.The passage identifies Horn and Schunck’s work as a classical variational approach.Source: §2 · p. 2

Style Transfer on Videos

  • Earlier video stylization tracked motion or added optical and spatial constraints to address flicker, while later painting methods extended brush-based stylization to video.The passage describes work on temporal coherence, flicker, and scintillation in non-photorealistic rendering.Source: §2 · p. 2
  • Optimization-based video stylization uses optical-flow constraints and multi-pass processing for temporal consistency, but takes a few minutes per frame.Ruder et al. extend optimization-based image stylization with adjacent-frame constraints and a multi-pass algorithm for long-term consistency.Source: §2 · p. 2

3. Stability in Style Transfer

The section reviews Gram-matrix style transfer and examines why small content changes can produce different stylized outputs. It links style-specific instability to the spread of solutions and reports a correlation between Gram-matrix trace and instability.

3.1. Style Transfer on Images

  • Style transfer combines content and style reconstruction losses, defined using a pretrained VGG-19 network and its layer activations.The style loss uses Gram matrices of layer activations; the losses encourage matching content features and style feature correlations.Source: §3.1 · p. 3

3.2. Gram Matrix and Style Stability

  • Imperceptible content-image changes can yield drastically different stylized images, with Composition XIV more unstable than The Great Wave in the examples.The section examines only the style loss for a single layer to analyze how instability depends on the style image.Source: §3.2 · p. 3
  • Gram-matrix matching has a solution set on a sphere whose radius is determined by the style Gram matrix trace, so larger traces place solutions farther apart in feature space.The section illustrates the geometry with two minima in 1-D and a circle of minima in 2-D.Source: §3.2 · p. 3, Figure 2 · p. 3
  • On static-camera, no-motion videos, styles with larger Gram-matrix traces show greater instability, measured as adjacent-frame stylized-image MSE.The comparison covers twelve feedforward models and traces at the relu1_1 and relu2_1 layers of the VGG-16 loss network.Source: §3.2 · p. 3, Figure 3 · p. 4

4. Method: Towards Stable Style Transfer

The method uses a recurrent network and a temporal consistency loss to produce stable stylized video while retaining style and content similarity. It compares stability with real-time and optimization-based baselines.

4. Method: Towards Stable Style Transfer

  • The method targets the speed of feedforward image stylization and the stability of optimization-based video stylization.The authors report a 1000× speed improvement for stable video style transfer without sacrificing quality.Source: §4 · p. 4

4.1. Overall Architecture

  • Each output is synthesized from the current content frame and previous stylized frame, with content, style, and temporal losses combined during training.The network is trained on video sequences to minimize the weighted sum of the three losses.Source: §4.1 · p. 4

4.2. Style Transfer Network

  • The style transfer network feeds each output into the next time step and concatenates the previous stylization with current content along the channel dimension.Its convolutional architecture uses downsampling, residual blocks, and nearest-neighbor upsampling.Source: §4.2 · p. 4, Figure 4 · p. 4

4.3. Temporal Consistency Loss

  • Temporal consistency warps the current stylized frame using input optical flow and compares it with the previous stylization, masking occluded and disoccluded regions.The mask excludes occlusions and motion boundaries; flow and occlusion masks are needed for training but not at test time.Source: §4.3 · p. 5, Figure 5 · p. 5

4.4. Implementation Details

  • Training first uses content and style losses on COCO, then fine-tunes with all three losses on Sintel, whose rendered images provide pixel-perfect flow and occlusion masks.Frames are resized to 512 × 218; training runs for 10 epochs with four-step BPTT.Source: §4.4 · p. 5
  • On DAVIS background patches, the method is generally more stable than the Real-Time baseline and has comparable stability to the Optim baseline.Stability is measured by mean PSNR and SSIM between corresponding 100 × 100 stylized patches across videos.Source: Table 1 · p. 5

5. Experiments

The experiments compare the proposed method with real-time and optimization-based baselines across controlled distortions, video stability, speed, and qualitative evaluations. They report robustness to input distortions, stability comparable to the Optim baseline, and substantially faster processing than that baseline.

5. Experiments

  • The method achieves image quality and stability comparable to optimization-based methods while retaining the speed advantage of feedforward methods.The experiments report comparable image quality and stability to the Optim baseline, while the method matches the real-time baseline's speed and processes 512 × 512 images at 20 FPS.Source: §5 · p. 5, §5.3 · p. 6

5.2. Datasets

  • The experiments use Sintel for training and DAVIS for qualitative and quantitative evaluation; DAVIS includes occlusion, motion blur, appearance change, and camera motion.Sintel provides pixel-perfect optical flow and occlusion masks; its test set is used for qualitative results.Source: §5.2 · p. 6

5.3. Quantitative Evaluation

  • The method is significantly more robust than the Real-Time baseline to translation and blur or sharpening distortions across two styles.As distortion magnitude increases, its output similarity decays comparably to the input similarity; the baseline's output similarity decays sharply.Source: §5.3 · p. 6, Figure 6 · p. 6
  • The method produces stylized videos with stability comparable to the Optim baseline, measured using corresponding background patches in adjacent DAVIS frames.The patch pairs are selected from the background using foreground masks, then compared using PSNR and SSIM.Source: §5.3 · p. 6
  • At test time, the method requires no explicit optical flow, matches the Real-Time baseline's speed, and runs 512 × 512 images at 20 FPS.Both it and the Real-Time baseline are roughly three orders of magnitude faster than the Optim baseline.Source: §5.3 · p. 6

5.4. Qualitative Evaluation

  • The Real-Time baseline flickers across adjacent frames, while the proposed method and Optim baseline maintain consistent stylization.A user study found significantly less qualitative flickering than the Real-Time baseline, with stability almost on par with the slower Optim baseline.Source: §5.4 · pp. 7–8
  • After occlusion and reappearance, the method sometimes maintains an object's style better than Optim, although long-term consistency is not explicitly enforced.In the shown sequence, the man looks similar before and after occlusion with the proposed method, while Optim smears the girl's red color onto him.Source: §5.4 · p. 7, Figure 7 · p. 7
  • The method sometimes produces slight object transparency during occlusion, typically only in the first one or two frames and not very perceptibly in video.The reported example is in the supplementary material.Source: §5.4 · p. 7

6. Conclusion

The conclusion characterizes Gram-matrix-matching instability and proposes a recurrent network for real-time video style transfer. It identifies long-term consistency through occlusion and reappearance as future work.

6. Conclusion

  • The paper characterizes instability in Gram-matrix-matching methods and argues that it is exacerbated when the style Gram matrix has a large trace.The characterization applies to neural style transfer methods based on Gram matrix matching.Source: §6 · p. 8
  • The proposed recurrent convolutional network for real-time video style transfer overcomes the instability of prior methods.Source: §6 · p. 8
  • Future work will investigate real-time methods that encode long-term consistency for objects that are occluded and later reappear.Source: §6 · p. 8
Loading 1705.02092v1…