PaperlayerView exact revision on arXiv

Source-linked AI summary

The Spatially-Correlative Loss for Various Image Translation Tasks

Chuanxia Zheng, Tat-Jen Cham, Jianfei Cai

arXiv:2104.00854v1cs.CV

TL;DR

Unpaired image translation must change an image’s appearance while preserving its scene structure, but pixel- and feature-based losses can conflate structure with domain-specific appearance. The paper learns spatially-correlative maps from within-image self-similarity and reports improvements across translation tasks.

  • Problem

    Pixel-level and feature-level content losses can conflate scene structure with appearance, limiting their fit for translation across domains.

    Source: §1 · p. 1

  • Method

    F/LSeSim compares within-image self-similarity patterns and uses contrastive learning to learn task-specific spatially-correlative maps that separate structure from appearance.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    The authors report solid performance improvements across various unpaired I2I tasks after replacing existing content losses with F/LSeSim.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    F/LSeSim is presented as suitable for structure-consistency tasks and readily applicable to existing architectures.

    Source: §5 · p. 8

    The evaluation focuses on image translation rather than translations incorporating shape modification.

    Source: §4 · p. 5

Abstract

from arXiv · show

We propose a novel spatially-correlative loss that is simple, efficient and yet effective for preserving scene structure consistency while supporting large appearance changes during unpaired image-to-image (I2I) translation. Previous methods attempt this by using pixel-level cycle-consistency or feature-level matching losses, but the domain-specific nature of these losses hinder translation across large domain gaps. To address this, we exploit the spatial patterns of self-similarity as a means of defining scene structure. Our spatially-correlative loss is geared towards only capturing spatial relationships within an image rather than domain appearance. We also introduce a new self-supervised learning method to explicitly learn spatially-correlative maps for each specific translation task. We show distinct improvement over baseline models in all three modes of unpaired I2I translation: single-modal, multi-modal, and even single-image translation. This new loss can easily be integrated into existing network architectures and thus allows wide applicability.

1. Introduction

The paper proposes self-similarity patterns as a domain-invariant way to represent scene structure during unpaired image translation. Its learned spatially-correlative loss improves on existing one- and two-sided methods and can be integrated into varied architectures.

1. Introduction

  • Pixel-level losses do not explicitly separate structure from appearance, while feature-level losses can retain domain-specific attributes and rely on fixed ImageNet features.This limits their ability to adapt to arbitrary domains.Source: §1 · p. 1
  • The method compares within-image self-similarity patterns between source and translated images, rather than directly comparing their pixels or features.It treats matching patterns as evidence of preserved structure even when appearances change substantially.Source: §1 · pp. 1–2
  • FSeSim computes a spatially-correlative map from features extracted by a pre-trained network, but those features can be biased toward photorealistic imagery.The paper reports comparative or better results than state-of-the-art methods on some tasks for this basic version.Source: §1 · p. 2
  • LSeSim uses contrastive learning to bring homologous structures closer and dissimilar structures apart, producing task-specific maps that are less tied to appearance.F/LSeSim also explicitly encodes structure, avoiding the multi-cycle looping used by cycle-consistency architectures.Source: §1 · p. 2
  • F/LSeSim outperformed existing one-sided and two-sided translation methods across the reported tasks.The authors also report that it can be integrated into architectures including CycleGAN, MUNIT, and StyleGAN.Source: §1 · p. 2

2. Related Work

Prior unpaired translation methods preserve content through cycle-consistency, pixel or feature losses, or indirect relationships between input and output. The paper instead learns structure maps by contrasting within-image self-similarity patterns.

2. Related Work

  • Two-sided methods commonly use cycle-consistency, but translated content can still distort without explicit constraints, and cycle-based methods require reverse-mapping generators and discriminators.Cycle-consistency may operate in image space or latent space.Source: §2 · p. 2
  • One-sided methods use predefined implicit distances or feature-space content losses, but fixed pretrained features and domain-specific peculiarities can limit their assumptions.CUT [39] learns a task-specific PatchNCE loss, yet its distance is still computed from extracted features and can be affected by domain-specific peculiarities.Source: §2 · p. 2
  • The proposed approach uses patch-wise contrastive learning within an image to learn a structure map that excludes appearance attributes.Self-supervised contrastive methods associate positive pairs and dissociate negative pairs.Source: §2 · p. 3

3. Methods

The method preserves scene structure by comparing within-image self-similarity patterns rather than matching pixels or deep features directly. It extends fixed self-similarity with task-specific contrastive learning and evaluates the resulting losses through qualitative comparisons.

3. Methods

  • The model translates an input into the target domain while retaining scene structure, comparing within-image deep-feature self-similarity patterns across source and translated images.The paper considers fixed self-similarity from a pretrained network and learned self-similarity trained without label supervision.Source: §3 · p. 3

3.1. Fixed Self-Similarity (FSeSim)

  • FSeSim forms maps from query-point features and nearby patch features, then compares collections of these maps between input and translated images.The maps represent spatial correlations rather than original feature attributes; the sampled collection is semi-sparse and more computationally efficient.Source: §3.1 · p. 3

3.2. Learned Self-Similarity (LSeSim)

  • LSeSim learns task-specific structure representations with contrastive learning, pairing a query patch with the same-position patch in an augmented image and contrasting other patches.The augmented image preserves structure while varying appearance; the generator still uses the spatially-correlative loss.Source: §3.2 · pp. 3–4, Figure 4 · p. 4

3.3. Full Objective

  • Training jointly minimizes losses for the structure-representation network and translation generator, whose objective combines style and structure terms.A hyperparameter trades off style and content.Source: §3.3 · p. 4

3.4. Analysis

  • LSeSim assigns low errors to aligned images across domains and high errors to unaligned images, while qualitative comparisons report better visual results than alternative content losses.The comparisons use the same translation-network setting, changing only the content loss.Source: §3.4 · pp. 4–5, Figure 5 · p. 4, Figure 6 · p. 5
  • F/LSeSim represents structure with multiple spatially-correlative maps intended to transcend domain specificity, unlike attention maps used as saliency guidance alongside cycle-consistency.The paper also distinguishes its local patch-level structure representation from self-similarity computed at random positions.Source: §3.4 · p. 5

4. Experiments

Experiments replace baseline content losses with F/LSeSim across single-modal, multi-modal, and single-image unpaired translation, focusing on preserving structure rather than modifying shape.

4. Experiments

  • The experiments target translations that preserve scene structure, rather than translations that incorporate shape modification.The authors apply F/LSeSim by replacing baseline content losses in suitable architectures across the three settings.Source: §4 · p. 5
  • Across the three settings, the authors report improved visual results, including gains over single-modal state-of-the-art methods and higher-quality multi-modal and single-image translations.The reported comparisons include single-modal methods, MUNIT for multi-modal translation, and existing methods for painting-to-photo translation.Source: §4.1 · p. 6, §4.2 · p. 7, §4.3 · p. 8

4.1. Single-Modal Unpaired Image Translation

  • With the same generator and discriminator settings, F/LSeSim significantly improved results over CycleGAN and CUT, using one training pass without CUT’s identity pass.The authors also describe the one-sided model as memory-efficient.Source: §4.1 · p. 6

4.2. Multi-Modal Unpaired Image Translation

  • F/LSeSim outperformed MUNIT and DRIT++ with the same network architecture; compared with MUNIT, it produced higher-quality and more diverse winter-to-summer and night-to-day results.BicycleGAN achieved greater diversity on all tasks, but with worse visual results.Source: §4.2 · pp. 6–7
  • In the ablation, local attention improved performance over global attention, and multiple layers with a fixed patch size improved it further; cosine distance substantially increased diversity.The full LSeSim model improved many metrics and visual results; the ablation compares single- and multi-modal translation.Source: Table 3 · p. 7, §4.2 · pp. 7–8

4.3. Single-Image Unpaired Image Translation

  • For high-resolution painting-to-photo translation trained on one unpaired source and target image, FSeSim produced results reported as higher quality and closer to the target style than CUT and other methods.This setting uses FSeSim rather than contrastively learned LSeSim.Source: Figure 9 · p. 8, §4.3 · p. 8

5. Conclusion

F/LSeSim is a structure-consistency loss based on spatial correlations rather than visual appearance, and the authors report performance improvements across unpaired image-to-image translation tasks.

5. Conclusion

  • F/LSeSim focuses on spatially-correlative relationships without regard to visual appearance, and replacing existing content losses with it led to reported improvements across varied unpaired translation tasks.The authors state that it can be applied to existing architectures.Source: §5 · p. 8

A.1. Additional comparisons

The appendix extends visual comparisons across single- and multi-modal translation, reporting improved single-modal results and varied multi-modal outputs.

A.1. Additional comparisons

  • Under the same training setting, results look better than CUT and CycleGAN, including challenging examples; the model also improves visual results in most cases.Source: §A.1 · p. 12, Figure A.1 · p. 12
  • Multi-modal samples extend Fig. 8: winter→summer outputs vary in color and lighting within summer appearances, while night→day shows obvious appearance modification.Source: §A.1 · p. 12, Figure A.2 · p. 13

B.1. Architecture details

The experiments reuse established generator and discriminator architectures, while the structure representation uses VGG16 features with fixed or task-selected weights.

B.1. Architecture details

  • The losses are evaluated with CycleGAN for single-modal, MUNIT for multi-modal, and CUT for single-image translation, using their generator and discriminator architectures.Source: §B.1 · p. 14
  • The structure representation network uses VGG16 without batch normalization; global FSeSim uses fixed ImageNet-pretrained relu3_1 features at 64×64.Source: §B.1 · p. 14
  • Local FSeSim uses relu3_1 and relu4_1 features with 32×32 patches; LSeSim uses those layers plus two task-specific 1×1 convolution layers.Source: §B.1 · p. 14

B.2. Evaluation details

Evaluation uses distributional, semantic-segmentation, perceptual-diversity, and manifold-based measures, with task-specific sampling and comparison procedures.

B.2. Evaluation details

  • FID compares generated and real image means and variances in deep feature space; multi-modal scores average FID across 19 sampled sets per test image.Source: §B.2 · p. 14
  • Cityscapes semantic-segmentation accuracy is measured by comparing DRN predictions on generated images with ground-truth paired semantic maps.Source: §B.2 · p. 14
  • Average LPIPS measures feature-domain perceptual distance, with larger distances indicating greater appearance diversity; scores average 19 generated-image pairs across 100 samples.Source: §B.2 · p. 14
  • Density and Coverage (D&C) jointly assesses diversity and fidelity through feature-manifold precision and recall, using 1900 generated samples and 2048-dimensional features.Source: §B.2 · p. 14

Related papers

Loading 2104.00854v1…