PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Patch-wise Semantic Relation for Contrastive Learning in Image-to-Image Translation Tasks

Chanyong Jung, Gihyun Kwon, Jong Chul Ye

arXiv:2203.01532v1cs.CV

TL;DR

Image-translation contrastive methods may overlook diverse semantic relations among patches. The paper combines decoupled contrastive learning with semantic-relation consistency and semantic hard-negative mining, reporting state-of-the-art results across translation and GAN compression tasks.

  • Problem

    Some patch-wise contrastive methods ignore semantic relations among patches, while cycle consistency can require an inverse generator and impose a constraint reported to cause distortion.

    Source: §1 · p. 1

  • Method

    The method combines decoupled contrastive learning with consistency regularization of patch-wise semantic relations and semantic-relation-based hard-negative mining.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    The method reports state-of-the-art performance across single- and multi-modal image translation and GAN compression tasks.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The reported results support using heterogeneous patch semantics and relational knowledge for image translation and its GAN compression.

    Source: §5 · p. 8

Abstract

from arXiv · show

Recently, contrastive learning-based image translation methods have been proposed, which contrasts different spatial locations to enhance the spatial correspondence. However, the methods often ignore the diverse semantic relation within the images. To address this, here we propose a novel semantic relation consistency (SRC) regularization along with the decoupled contrastive learning, which utilize the diverse semantics by focusing on the heterogeneous semantics between the image patches of a single image. To further improve the performance, we present a hard negative mining by exploiting the semantic relation. We verified our method for three tasks: single-modal and multi-modal image translations, and GAN compression task for image translation. Experimental results confirmed the state-of-art performance of our method in all the three tasks.

1. Introduction

The paper targets image translation methods that either impose restrictive cycle consistency or neglect semantic relations among patches. It proposes decoupled contrastive learning with semantic-relation consistency and semantic hard-negative mining, reporting state-of-the-art results across translation and GAN compression tasks.

1. Introduction

  • Cycle consistency requires an inverse-mapping generator and discriminator, and prior work warns its restrictive constraint may cause distortion.One-sided translation methods instead seek to enhance input-output correspondence.Source: §1 · p. 1
  • Prior patch-wise similarity methods ignored semantic relations between patches, while NEGCUT's random-vector negative generator may not match real negatives and can destabilize training.The proposed mining strategy avoids easy negatives with unrelated semantic information.Source: §1 · pp. 1–2
  • Semantic relation consistency preserves patch-wise semantic relationships between input and output, while hard-negative mining prioritizes semantically related negatives.The regularization is described as preventing artifacts that violate semantic relationships.Source: §1 · p. 2
  • The method achieves state-of-the-art performance in single- and multi-modal image translation and GAN compression.The conclusion also reports noticeable gains over baseline models.Source: §1 · p. 2

2. Related works

Related work covers one-sided translation correspondence, relational knowledge transfer, and contrastive learning. This paper applies semantic relational knowledge to input-output correspondence and focuses on heterogeneous patch semantics and controllable hard negatives.

2.1. One-sided image translation

  • One-sided translation methods preserve different relations: geometric consistency in GcGAN, mutual information in DistanceGAN, and embedding arithmetic in TraVeLGAN.Contrastive approaches also maximize mutual information between corresponding input and output locations.Source: §2.1 · p. 2

2.2. Relational knowledge transfer

  • Relational knowledge transfer uses inter-sample relations in distillation; here, semantic relations match input and output features rather than teacher and student features.The approach focuses on heterogeneous patch semantics overlooked in prior works, with real-data negatives and curriculum-controlled hardness unlike NEGCUT.Source: §2.2 · p. 2

2.3. Contrastive learning

  • Contrastive learning relates positive and negative pairs; prior work explores informative negatives, loss changes, and encoding semantic structures.CO2 regularizes semantic relations of positive pairs with negatives, while PCL encodes data structures using EM.Source: §2.3 · p. 2

3. Main Contribution

The method focuses on heterogeneous semantic relations among image patches, which may represent related object parts or unrelated background. It combines semantic-relation consistency with hard-negative mining and decoupled contrastive learning.

3. Main Contribution

  • Patch relations vary across image content, from related horse patches to unrelated background patches, and can differ even between parts of the same horse.The paper argues that these diverse relations should be preserved for accurate image translation.Source: §3 · p. 2, Figure 2 · p. 3

3.1. Consistency of semantic relation distribution

  • The method preserves patch-wise semantic relations by matching input and output similarity distributions with Jensen-Shannon divergence.The distributions encode semantic closeness between patches; the regularization is applied across sampled vectors.Source: §3.1 · p. 3

3.2. Hard negative mining by semantic relation

  • Hard negatives are sampled with higher probability when they are more similar to the positive, and curriculum learning gradually increases hardness through γ.γ starts small because semantic relations are unstable early in training.Source: §3.2 · pp. 3–4
  • Explicitly controlling negative hardness improved performance compared with NEGCUT, which controls hardness implicitly through a negative generator.The authors attribute the improvement to explicit hardness control.Source: §3.2 · p. 4

3.3. Effect of DCE for heterogeneous semantics

  • Decoupled InfoNCE removes the positive-pair term from the denominator, preventing easy unrelated negatives from diminishing gradients through the negative-positive coupling effect.The paper identifies unrelated randomly sampled patches as easy negatives that can cause this effect in image translation.Source: §3.3 · p. 4

4. Experiments

The experiments cover single-modal and multi-modal translation and image-translation GAN compression, then ablate the method’s components.

4. Experiments

  • The experiments apply the method to single-modal and multi-modal translation, GAN compression, and supplementary painting-to-photo translation.The experiment overview is schematized in Figure 5.Source: §4 · p. 4, Figure 5 · p. 5

4.1. Single modal image translation

  • The method outperforms baseline models across all reported single-modal datasets and metrics.The comparison includes one-sided, two-sided, and recent contrastive-learning methods.Source: Table 1 · p. 5, §4.1 · p. 6
  • Qualitative comparisons report preserved source structure and target-domain appearance across translation tasks, with clear boundaries and textures in compressed outputs.For Cityscapes, generated images correspond correctly to input labels; for horse-to-zebra, outputs reflect target texture while preserving source structure.Source: Figure 6 · p. 5, §4.1 · p. 6, Figure 7 · p. 6

4.2. Multi-modal image translation

  • The method improves image quality and diversity on both multi-modal datasets, while SeSim improves quality but lowers diversity.The comparison uses FID for image quality and LPIPS for generation diversity.Source: Table 2 · p. 6, §4.2 · p. 7

4.3. Compression of image translation model

  • Compressed models achieve better FID and mIoU with similar or fewer MACs and parameters than baseline students.On Cityscapes, the method also achieves better mIoU than the teacher model.Source: Table 3 · p. 7, §4.3 · p. 8

4.4. Ablations studies

  • Similarity maps become more consistent between input and output with SRC, while hard negative mining concentrates high similarity on query-related points.InfoNCE and basic DCE show high similarity at unrelated points and different input/output patterns.Source: Figure 9 · p. 8, §4.4 · p. 8
  • Adding SRC and hard negative mining to DCE improves both datasets, and the full component set gives the best performance.SRC and hard negative mining also improve results with InfoNCE, though InfoNCE remains worse overall than the best DCE model.Source: Table 4 · p. 8, §4.4 · p. 8

5. Conclusion

The paper concludes that its image-translation method uses relational knowledge from heterogeneous image-patch semantics, combining hDCE with SRC. Across image translation and GAN compression tasks, it reports state-of-the-art scores compared with baseline models.

5. Conclusion

  • The method combines decoupled infoNCE with hard negatives (hDCE) and semantic relation consistency (SRC) to improve performance.It uses relational knowledge formed by heterogeneous image-patch semantics.Source: §5 · p. 8
  • Across image translation and GAN compression tasks, the method achieves state-of-the-art scores compared with baseline models.The paper reports a noticeable performance increase.Source: §5 · p. 8
  • The paper reports that discussions of limitations and negative social impacts are in the Supplementary Materials.Source: §5 · p. 8

Related papers

Loading 2203.01532v1…