PaperlayerView exact revision on arXiv

Source-linked AI summary

CCPL: Contrastive Coherence Preserving Loss for Versatile Style Transfer

Zijie Wu, Zhen Zhu, Junping Du, Xiang Bai

arXiv:2207.04808v4cs.CV

TL;DR

Single-frame style transfer methods can struggle to maintain temporal consistency in videos, motivating a versatile approach that does not require video training. The paper combines local-patch CCPL with SCT, reporting improved video consistency alongside satisfying stylization and results across style-transfer settings.

  • Problem

    Single-frame style transfer methods face the challenge of maintaining temporal consistency in video without video inputs during training.

    Source: §1 · p. 1

  • Method

    CCPL uses contrastive learning on local patch differences to preserve content coherence, while SCT aligns second-order statistics of content and style features.

    Source: §1 · pp. 2–3

  • Results

    The authors report that CCPL improves video temporal consistency while stylization remains satisfying, and that their full model produces satisfying image and video results.

    Source: §1 · p. 2, §5 · p. 14

  • Takeaways & Limitations

    CCPL transfers to other style-transfer networks and image-to-image translation, and the resulting SCTNet is reported to be light and fast.

    Source: §1 · p. 3, §5 · p. 14

    The temporal-coherence assumption relates content and generated-image differences when the frame interval is small.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

In this paper, we aim to devise a universally versatile style transfer method capable of performing artistic, photo-realistic, and video style transfer jointly, without seeing videos during training. Previous single-frame methods assume a strong constraint on the whole image to maintain temporal consistency, which could be violated in many cases. Instead, we make a mild and reasonable assumption that global inconsistency is dominated by local inconsistencies and devise a generic Contrastive Coherence Preserving Loss (CCPL) applied to local patches. CCPL can preserve the coherence of the content source during style transfer without degrading stylization. Moreover, it owns a neighbor-regulating mechanism, resulting in a vast reduction of local distortions and considerable visual quality improvement. Aside from its superior performance on versatile style transfer, it can be easily extended to other tasks, such as image-to-image translation. Besides, to better fuse content and style features, we propose Simple Covariance Transformation (SCT) to effectively align second-order statistics of the content feature with the style feature. Experiments demonstrate the effectiveness of the resulting model for versatile style transfer, when armed with CCPL.

1 Introduction

The paper targets artistic, photorealistic, and video style transfer without video training, addressing local inconsistency that can make stylized videos appear incoherent. It introduces CCPL to preserve content coherence through local patch differences and SCT to fuse content and style features.

1 Introduction

  • The paper presents a versatile approach for artistic, photorealistic, and video style transfer without explicitly training with videos.The introduction illustrates artistic and photorealistic image and video style transfer.Source: §1 · p. 1, Figure 1 · p. 2
  • Independent frame stylization can produce flicker, while optical-flow methods are computationally costly and single-frame global constraints can fail to preserve temporal consistency.The paper motivates a local-patch constraint because local noise can make videos appear inconsistent despite good structural rigidity.Source: §1 · pp. 1–2
  • CCPL encourages corresponding patch differences in content and generated images to be similar, using contrastive learning to avoid forcing the stylized output to match the content.Nearby patches in one image stand in for patches from consecutive frames, allowing the constraint to be applied with single-frame training images.Source: §1 · p. 2
  • The authors report substantially improved temporal consistency while stylization remains satisfying after applying CCPL.The introduction points to Fig. 5 for this result.Source: §1 · p. 2
  • SCT fuses style and content features by multiplying style-feature covariance with normalized content features, aligning their second-order statistics.The authors describe SCT as simpler than AdaIN and Linear while capturing precise style information.Source: §1 · p. 3
  • CCPL is transferable across style-transfer networks and to image-to-image translation, where the authors report improved temporal consistency and visual quality without further modifications.The combined SCTNet is reported to run at about 25 frames per second at 512 × 512 resolution.Source: §1 · p. 3

2 Related Works

Prior work spans image style transfer methods and video methods that either use optical flow or preserve content coherence without it. CCPL is presented as architecture-independent, while its neighbor-regulating contrastive design supports image and video generation and transfers to other tasks.

Image Style Transfer

  • Image style transfer ranges from optimization-based and feed-forward methods to multi-style, patch-swapping, feature-statistics, attention-based, and photorealistic approaches.The section names NST, AdaIN, WCT [41], SANet, and other methods across these approaches.Source: §2 · p. 4

Video Style Transfer

  • Optical-flow video methods improve temporal consistency but can introduce artifacts from inaccurate motion estimates and incur high computational cost, especially at larger image sizes.Other video methods avoid optical flow by seeking to maintain coherence in content inputs.Source: §2 · p. 4
  • CCPL requires no particular network architecture, and the authors report that SCTNet surpasses state-of-the-art methods in video temporal consistency while retaining satisfying stylization.They also report similar video-stability improvements when applying CCPL to other networks.Source: §2 · p. 5

Contrastive Learning

  • Unlike CUT, which matches patches at the same locations for image-to-image translation, CCPL regulates neighboring patches for image and video generation.The authors also report CCPL's effectiveness on top of CUT for image-to-image translation.Source: §2 · p. 5

3 Methods

The method pairs patch-level contrastive coherence preservation with covariance-based content/style fusion, then trains the network using content, style, and CCPL losses.

3.1 Contrastive Coherence Preserving Loss

  • CCPL assumes local patch differences preserve short-interval frame coherence and contrasts same-location generated/content difference pairs rather than forcing them equal.This avoids direct conflict with style losses while improving video temporal consistency without using information from other input frames.Source: §3.1 · p. 6
  • Training samples N feature vectors and their eight nearest neighbors, forming generated/content difference vectors at matching locations.The resulting loss has no influence on inference speed and can extend image-generation methods to video outputs.Source: §3.1 · pp. 6–7

3.2 Simple Covariance Transformation

  • SCT normalizes content and style features, computes style-feature channel correlations, and multiplies the covariance with normalized content features before decoding.SCTNet is reported comparable to Linear in stylization effects while lighter and faster.Source: §3.2 · pp. 7–8

3.3 Loss Function

  • Training combines content, style, and CCPL losses as a weighted sum, using content-feature distance and style-feature channel means and standard deviations.Feature layers and default weights vary by artistic versus photo-realistic style transfer.Source: §3.3 · p. 8

4 Experiments

The experiments compare artistic, photo-realistic, and video style transfer, then test CCPL’s settings and applications. The results report improved temporal consistency with stylization retained, alongside strong image quality and efficiency.

4.2 Comparison with Former Methods

  • SCTNet with CCPL exceeds the single-frame methods and matches ReReVST in short- and long-term temporal consistency; it also ranks best in human preference.SCTNet without CCPL scores best in SIFID, and the full model’s SIFID exceeds ReReVST’s.Source: Table 1 · p. 9, §4.2 · pp. 10–11
  • With CCPL, artistic results better preserve content structure and vivid color while reducing irregular textures and local color distortions.The paper also reports that CCPL can improve stylization and preserve semantic information.Source: §4.2 · p. 11, Figure 6 · p. 11
  • For photo-realistic transfer, the full model produces results comparable to or better than the compared methods, with high visual quality and appropriate stylization.The setup uses a shallower encoder, ending at relu3_1, and feature maps from all three layers for CCPL.Source: §4.2 · p. 12, Figure 7 · p. 12
  • SCTNet surpasses the compared methods in execution speed across three input scales, with tests run on one 12GB Titan XP GPU.The authors describe this efficiency as indicating feasibility for real-time use.Source: §4.2 · p. 12, Table 3 · p. 13

4.3 Ablation Studies

  • Increasing CCPL layers improves short- and long-term consistency but weakens stylization and increases computation; layer four adds little consistency over layer three.The authors therefore choose three layers by default.Source: §4.3 · pp. 12–13, Figure 8 · p. 13
  • Sampling 64 difference vectors per layer gives temporal-consistency results close to 128 with less computation and style degeneration; a 0.5 loss-weight ratio is chosen as a trade-off.The ablation varies layer count, sampled-vector count, and the CCPL-to-style-loss weight ratio.Source: §4.3 · pp. 12–13

4.4 Applications

  • CCPL improves temporal consistency when added to AdaIN, SANet, and Linear, and Fig. 9 presents its application to these methods.The paper describes CCPL as applicable to existing style-transfer networks with minor modifications.Source: §4.4 · p. 13, Figure 9 · p. 14
  • Applied to CUT and trained on the same horse2zebra dataset, CCPL improves visual quality and temporal consistency.Fig. 10 compares CUT with and without CCPL.Source: §4.4 · p. 14, Figure 10 · p. 14

5 Conclusions

The conclusion presents CCPL as a way to preserve content coherence while balancing temporal consistency and stylization. Together with SCT, it yields a light, fast model for image and video results, and CCPL also shows potential beyond style transfer.

5 Conclusions

  • CCPL preserves content coherence by encouraging patch-difference similarity between content and generated images, and offers a trade-off between temporal consistency and style effects.The loss is also demonstrated on other models and tasks, including image-to-image style transfer.Source: §5 · p. 14
  • SCT aligns second-order statistics of content and style features; combined with CCPL, it forms a light and fast model producing satisfying image and video results.Source: §5 · p. 14
  • The conclusion presents CCPL’s use on other models and tasks, including image-to-image style transfer, as evidence of potential for broader applications.Source: §5 · p. 14

Related papers

Loading 2207.04808v4…