PaperlayerView exact revision on arXiv

Source-linked AI summary

Image-to-Image Translation via Group-wise Deep Whitening-and-Coloring Transformation

Wonwoong Cho, Sungha Choi, David Keetae Park, Inkyu Shin, Jaegul Choo

arXiv:1812.09912v2cs.CV

TL;DR

The paper addresses the limited style representation and computational difficulty of covariance-based exemplar transfer in unsupervised image translation. It proposes end-to-end deep whitening-and-coloring and a group-wise extension, which experiments report as fast and effective at reflecting exemplar style.

  • Problem

    Channel-wise statistics may inadequately reflect target style, while direct whitening-and-coloring is computationally expensive and difficult to backpropagate.

  • Method

    The paper approximates whitening-and-coloring end to end with regularization and extends it to a group-wise transformation for efficiency and image quality.

  • Results

    Experiments report that the method is fast in training and inference and effective at reflecting an exemplar’s style, with competitive image-translation and style-transfer outputs.

  • Takeaways & Limitations

    Group-wise deep whitening-and-coloring offers a practical framework for exemplar-conditioned translation with competitive qualitative performance.

  • Takeaways & Limitations

    The approach assumes images decompose into domain-invariant content and domain-specific style spaces, and full-channel whitening can discard content features.

Abstract

from arXiv · show

Recently, unsupervised exemplar-based image-to-image translation, conditioned on a given exemplar without the paired data, has accomplished substantial advancements. In order to transfer the information from an exemplar to an input image, existing methods often use a normalization technique, e.g., adaptive instance normalization, that controls the channel-wise statistics of an input activation map at a particular layer, such as the mean and the variance. Meanwhile, style transfer approaches similar task to image translation by nature, demonstrated superior performance by using the higher-order statistics such as covariance among channels in representing a style. In detail, it works via whitening (given a zero-mean input feature, transforming its covariance matrix into the identity). followed by coloring (changing the covariance matrix of the whitened feature to those of the style feature). However, applying this approach in image translation is computationally intensive and error-prone due to the expensive time complexity and its non-trivial backpropagation. In response, this paper proposes an end-to-end approach tailored for image translation that efficiently approximates this transformation with our novel regularization methods. We further extend our approach to a group-wise form for memory and time efficiency as well as image quality. Extensive qualitative and quantitative experiments demonstrate that our proposed method is fast, both in training and inference, and highly effective in reflecting the style of an exemplar. Finally, our code is available at https://github.com/WonwoongCho/GDWCT.

1. Introduction

Unsupervised exemplar-based image translation aims to transfer style while preserving content, but common channel-wise statistics can miss richer style information. This paper introduces efficient deep and group-wise whitening-and-coloring transformations, supported by qualitative and quantitative comparisons.

  • 1. Introduction: AdaIN transfers style by matching channel-wise means and variances, whereas feature covariances can capture interaction effects among channels.The paper motivates covariance-aware transformation because matching only two statistics may produce sub-optimal outputs.
  • 1. Introduction: Applying WCT directly to image translation is costly at O(n^3) and difficult to backpropagate through its singular value decomposition.Here, n denotes the number of channels in an activation map.
  • 1. Introduction: The proposed deep whitening-and-coloring approach enables end-to-end training for transferring richer style semantics.The method is presented as a novel approach for conveying profound style semantics in image translation.
  • 1. Introduction: The paper proposes group-wise deep whitening-and-coloring transformation to efficiently convey exemplar style in image translation.The group-wise design is intended to improve computational efficiency and image quality through simple forward propagation.
  • 1. Introduction: Experiments evaluate the method extensively against state-of-the-art methods using both qualitative and quantitative comparisons.The introduction identifies these experiments as evidence of the method’s effectiveness.

2. Related Work

Related work develops unsupervised, multimodal image translation and arbitrary neural style transfer. These methods differ in how they represent content and style, from latent separation and feature concatenation to channel statistics and covariance matrices.

  • 2. Related Work: Earlier unsupervised translation methods addressed domain conversion, but some produced only one output or required paired images.The literature identifies limited diversity and pairing requirements as limitations of earlier approaches.
  • 2. Related Work: StarGAN provides a unified model for unsupervised image translation across multiple domains.It is presented as one approach within the broader unsupervised translation literature.
  • 2. Related Work: DRIT and MUNIT generate multimodal outputs without paired data by separating domain-specific style from domain-invariant content.Their approaches assume that latent image space can be decomposed into style and content spaces.
  • 2. Related Work: Gram-matrix and covariance-based feature interactions capture image style, but iterative style-transfer optimization requires multiple forward and backward passes at inference.Later methods sought to avoid this time-consuming inference procedure.
  • 2. Related Work: AdaIN aligns content and style feature means and variances, while WCT represents style through the feature covariance matrix.These approaches support arbitrary neural style transfer from an exemplar image.

3. Proposed Method

The proposed model translates between unpaired image domains by combining separate content and style features, then applying group-wise deep whitening-and-coloring inside the generator. Regularized whitening, learned coloring, multi-hop style injection, and consistency-based losses make the transformation efficient while preserving content and reflecting exemplar style.

  • 3.1. Model Overview: GDWCT combines content and exemplar-style features inside the target-domain generator to produce translated and reconstructed images.The model extracts domain-specific content and style representations, applies GDWCT, and trains generators in both translation directions.
  • 3.1. Model Overview: The model applies style through multiple generator residual blocks, allowing low-level fine patterns and high-level coarse patterns to be reflected simultaneously.A learnable parameter controls the amount of applied style at each transformation.
  • 3.2. Loss Functions: Style, content, cycle, identity, and adversarial losses jointly encourage exemplar-style transfer, content preservation, reconstruction, and realistic outputs.The full objective trains both translation directions simultaneously.
  • 3.3. Group-wise Deep Whitening-and-Coloring Transformation: Deep whitening replaces eigendecomposition-based whitening by regularizing content-feature covariance toward the identity, reducing whitening to mean subtraction.The encoder is trained to naturally produce a whitened feature after centering.
  • 3.3. Group-wise Deep Whitening-and-Coloring Transformation: Group-wise deep whitening addresses small-batch covariance inaccuracies and excessive content loss by applying whitening separately to channel groups.The forward operation matches DWT, while channel rearrangement supports group-wise regularization.

4. Experiments

Experiments evaluate GDWCT across multiple datasets and baselines using user preference, classification accuracy, inference time, and qualitative analyses. The results indicate effective style transfer, competitive speed, and benefits from multi-hop transformation and regularization.

  • Experimental setup: The evaluation covers CelebA, Artworks, cat2dog, BAM, and Yosemite datasets, comparing GDWCT with MUNIT, DRIT, and WCT.The study uses user preference and classification accuracy as evaluation measures alongside qualitative and inference-time comparisons.
  • User study: Users preferred GDWCT over other baselines in five of six CelebA class pairs, with the Female →Male case favoring DRIT because it consistently generated facial hair.The study measured preferences for output quality and exemplar-style rendering across 60 comparisons per participant.
  • Classification accuracy: GDWCT achieves higher classification accuracy than the compared models in all reported face-attribute translation cases.
  • Inference time: GDWCT matches existing image-translation methods in inference speed while rendering style information comparable to WCT.Inference time was measured per 256×256 image on a NVIDIA Titan XP GPU over 1,000 trials.
  • Stylization comparisons: On Artworks, GDWCT transfers object colors and overall style mood while preserving more detail than the compared methods.MUNIT preserves boundaries but limits style transfer, DRIT produces high contrast, and WCT can substantially lose content.
  • Number of hops on style: Multi-hop GDWCT transfers finer and coarser style details more effectively than the single-hop variant.The multi-hop result agrees with the exemplar’s detailed style, whereas the single-hop variant mainly follows its overall color pattern.
  • Effects of regularization: Increasing the whitening regularization strengthens style erasure, producing more visible texture removal in cloth and hair.The experiment varies whitening and coloring coefficients on Watercolor-to-Pen-Ink translation.
  • Whitening visualization: Visualizing the whitened feature shows that whitening removes detailed color and texture information from the content image.

5. Conclusion

GDWCT is proposed as a group-wise framework for exemplar-based image translation, with experiments showing competitive outputs and improved stylization. The paper reports effectiveness across image translation and style transfer, including user agreement that it reflects exemplar styles.

  • GDWCT is proposed as a framework for exemplar-based image translation with improved stylization capability.
  • Experiments report competitive outputs in both image translation and style transfer domains.
  • Real-user evaluations found majority agreement that GDWCT successfully reflects the given exemplar style.

6. Appendix

The appendix details implementation choices, group-size effects, and additional comparisons on CelebA and other datasets. Results favor GDWCT over ELEGANT in qualitative attribute translation and classification accuracy, while group-size effects remain difficult to generalize.

  • 6.2. Effects of Different Number of Groups: Increasing the number of groups reduces model parameters according to C^2/|G|, while larger group counts shift outputs toward stronger object-level style contrast.With |G| = 8, outputs mainly reflect exemplar colors; with |G| = 64, color tone is not necessarily maintained.
  • 6.3. Additional Comparison Results: GDWCT achieves better intended attribute translation than ELEGANT in qualitative CelebA comparisons, including bangs, smile, and eye-related outputs.The comparison uses 216×216 images and reports outputs closer to the given style.
  • 6.3. Additional Comparison Results: Higher classification accuracy by a large margin is reported for GDWCT across all cases in Table 4.The table reports classification accuracy in percentages.
  • 6.4. Extra Results on CelebA Dataset: The appendix reports additional unseen-data outputs on CelebA and verifies superior GDWCT performance across the extra results.
Loading 1812.09912v2…