Source-linked AI summary
Cross-domain Correspondence Learning for Exemplar-based Image Translation
Pan Zhang, Bo Zhang, Dong Chen, Lu Yuan, Fang Wen
TL;DR
The paper addresses the challenge of establishing meaningful correspondence between distinct input domains and exemplar images for general image translation. CoCosNet jointly learns correspondence and translation through an intermediate feature domain, achieving strong quantitative and qualitative performance across applications.
Problem
Exemplar-based translation requires meaningful correspondence from masks, edges, or keypoints to photos, while pretrained features may fail and region-specific methods are task specific.
Method
CoCosNet jointly learns cross-domain correspondence and image translation, aligning inputs in an intermediate feature domain and synthesizing from a semantically warped exemplar.
Results
CoCosNet achieves preferable quantitative and qualitative performance, including a large image-quality margin over state-of-the-art methods across application tasks.
Takeaways & Limitations
The learned dense cross-domain correspondence supports exemplar-faithful image synthesis and several applications.
Takeaways & Limitations
Correspondence learning has no direct supervision and is jointly trained with image translation.
Abstract
from arXiv · showhide
We present a general framework for exemplar-based image translation, which synthesizes a photo-realistic image from the input in a distinct domain (e.g., semantic segmentation mask, or edge map, or pose keypoints), given an exemplar image. The output has the style (e.g., color, texture) in consistency with the semantically corresponding objects in the exemplar. We propose to jointly learn the crossdomain correspondence and the image translation, where both tasks facilitate each other and thus can be learned with weak supervision. The images from distinct domains are first aligned to an intermediate domain where dense correspondence is established. Then, the network synthesizes images based on the appearance of semantically corresponding patches in the exemplar. We demonstrate the effectiveness of our approach in several image translation tasks. Our method is superior to state-of-the-art methods in terms of image quality significantly, with the image style faithful to the exemplar with semantic consistency. Moreover, we show the utility of our method for several applications
1. Introduction
Exemplar-based image translation converts masks, edges, or pose keypoints into photorealistic images while matching style from semantically corresponding exemplar objects. CoCosNet addresses weakly supervised cross-domain correspondence and translation jointly, targeting instance-level exemplar fidelity and broad applicability.
- Exemplar-based image translation converts segmentation masks, edge maps, or pose keypoints into photorealistic images guided by a user-given exemplar.The formulation enables flexible multimodal control through exemplar selection.
- Existing latent style codes capture global exemplar style but can wash away spatially relevant local style.
- Pretrained deep-feature matching may fail across masks, edges, keypoints, and photos, while semantic-region methods are task-specific rather than general.
- Learning dense cross-domain correspondence is weakly supervised because correspondence annotations and synthesis ground truth for random exemplars are unavailable.
- CoCosNet jointly learns correspondence and translation by mapping distinct domains into an intermediate feature domain, aligning exemplar appearance, and synthesizing with spatially varying style injection.The two subnetworks facilitate each other and are trained end-to-end with novel losses.
- The method aims to reproduce exemplar fine structures at instance level and reports large-margin image-quality gains across varied application tasks.
2. Related Work
Image-to-image translation learns mappings between domains, but existing approaches do not provide delicate, explicit control over exemplar style. CoCosNet instead establishes cross-domain correspondence before synthesizing from aligned exemplar appearance.
- Image-to-image translation learns mappings between different image domains, commonly using conditional generative adversarial networks with paired or unpaired data.
- Latent-space sampling increases multimodal diversity, but complex latent representations lack explicit correspondence to image style and limit delicate output control.
- CoCosNet first establishes correspondence between source and exemplar images in different domains, warps the exemplar, and synthesizes an output preserving source content while resembling aligned exemplar style.
3. Approach
CoCosNet jointly learns cross-domain correspondence and image translation by aligning heterogeneous inputs in a shared feature domain, warping exemplar appearance, and synthesizing the output with spatially varying style injection.
- Domain alignment: The method first maps the source input and exemplar into a shared domain where their semantic features can be matched.Feature pyramid networks extract multi-scale features, which are transformed into representations xS and yS with matching dimensions.
- Correspondence within shared domain: A correspondence matrix matches shared-domain features across spatial positions, enabling the exemplar to be warped toward the source semantics.The warped exemplar selects highly correlated exemplar pixels and combines them through a weighted average.
- Joint learning: Correspondence and translation are jointly trained because generating high-quality outputs encourages the network to refer to semantically corresponding exemplar regions.The method uses indirect supervision from image translation rather than direct correspondence annotations.
- Translation network: The translation network uses the warped exemplar to condition synthesis through spatially adaptive denormalization and progressively inject exemplar style across layers.The denormalization parameters encode exemplar style and are produced by projecting the warped exemplar.
- Losses for pseudo exemplar pairs: Pseudo exemplar pairs are created by applying geometric distortions such as warping or random flips to paired target images, with feature matching penalizing output differences.The distorted target serves as the exemplar while the original target remains the expected translation counterpart.
- Exemplar translation losses: Additional losses enforce shared-domain embedding alignment, input-semantic consistency, exemplar appearance transfer, and cycle-consistent correspondence.The cycle-consistency regularizer is crucial because endpoint losses alone may produce plausible images without meaningful correspondence.
4. Experiments
Experiments evaluate CoCosNet across multiple image-translation tasks and datasets, using qualitative, quantitative, style-relevance, user-study, correspondence, and ablation analyses. The method shows strong image quality and style fidelity, while the ablations identify correspondence-related losses as important.
- Experimental setup: Experiments cover mask-to-image synthesis on ADE20k, CelebA-HQ, and Flickr, edge-to-face synthesis on CelebA-HQ, and pose-to-body synthesis on DeepFashion.The datasets include diverse segmentation, face-edge, and human-pose representations.
- Experimental setup: The comparison includes Pix2pixHD, SPADE, MUNIT, SIMS, and EGSC-IT, with released codes used where available for exemplar-based generation.SIMS results are taken from reported figures because preparing its database is computationally prohibitive.
- Quantitative evaluation: 7.5 FID points: CoCosNet improves over previous leading methods on the challenging ADE20k dataset.Image quality is assessed with FID for semantic realism and SWD for low-level patch-distribution distance.
- Qualitative and subjective evaluation: CoCosNet achieves the best reported style relevance and visual quality, with fewer artifacts and fine structures matching semantically corresponding exemplar regions.Style relevance measures color and texture distance using low-level VGG features.
- Qualitative and subjective evaluation: CoCosNet ranks first in 84.2% of user-study cases across image-quality and style-relevance comparisons.The study compares 30 images across the evaluated tasks.
- Ablation and correspondence analysis: Removing domain alignment causes oversmooth dense warping, while removing correspondence regularization produces incorrect matches; the full model performs best across all Table 4 metrics.The regularization loss helps make correspondence meaningful and facilitates image synthesis.
5. Applications
CoCosNet supports image editing and makeup transfer by using semantic correspondence to reuse appearance from an original image or transfer edits between portraits.
- Image editing: Image editing modifies a segmentation layout and synthesizes the changed image using the original image as a self-exemplar.The application supports moving, adding, and deleting instances.
- Makeup transfer: Makeup transfer applies makeup strokes from one portrait to other portraits by matching semantic correspondence.The method transfers artistic edits to semantically corresponding facial regions.
6. Conclusion
The conclusion presents CoCosNet as a correspondence-based image-translation method with strong quantitative and qualitative performance and applications enabled by dense cross-domain correspondence.
- Conclusion: CoCosNet translates images by relying on cross-domain correspondence and achieves preferable quantitative and qualitative performance over leading approaches.The conclusion also describes dense correspondence as supporting several applications.
Appendix A. Additional Qualitative Results
The appendix provides additional qualitative results across mask-to-image, edge-to-face, and pose-to-body synthesis tasks on the reported datasets.
- Mask-to-image synthesis: Additional mask-to-image results are shown for ADE20k, CelebA-HQ, and Flickr.Figures 12–14 place exemplars in the first row and segmentation masks with results in the second row.
- Edge-to-face synthesis: Additional edge-to-face results are shown on CelebA-HQ, with exemplars in the first row and edge maps with outputs in the second.The appendix presents this task in Figure 15.
- Pose-to-body synthesis: Additional pose-to-body results are shown on DeepFashion, with exemplars in the first row and pose images with outputs in the second.The appendix presents this task in Figure 16.
Appendix B. Additional Results of Dense Correspondence
CoCosNet establishes dense correspondence across domains and uses it to warp the exemplar into semantic alignment with the input domain.
- Dense correspondence produces a warped exemplar that is semantically aligned with the image in domain A.The warping is denoted r_y→x in Equation 4.
Appendix C. Additional Ablation Studies
The ablations examine normalization, semantic editing, makeup transfer, and the architecture supporting correspondence and style-conditioned synthesis.
- Positional Normalization: Positional normalization preserves structure information from prior layers and improves the translation quality lower bound.It helps produce vibrant images even when correspondence is difficult in complex ADE20k scenes.
- Feature normalization for correspondence: Channel-wise feature normalization helps transfer fine exemplar structures, including window patterns, into the final output.The normalization computes statistics along channels while retaining the spatial feature-map dimensions.
- Image editing: Changing segmentation masks enables semantic image editing by manipulating instances in the image.The original image and segmentation mask are used to alter the semantic layout.
- Makeup transfer: Dense semantic correspondence enables makeup transfer from edited portraits to a batch of other portraits.The transfer operates by matching semantically corresponding regions.
- Architecture: CoCosNet uses separate domain adaptors, a shared intermediate domain, correlation-based correspondence, and a translation network conditioned on warped-exemplar style.The translation network progressively synthesizes outputs through style-modulated normalization layers.
Appendix F. Detailed User Study Results
User studies compare image quality and style relevance across ADE20k, CelebA-HQ, and DeepFashion, while additional figures show multimodal landscape synthesis.
- ADE20k: 67.3% of ADE20k users preferred image quality and 91.9% preferred style relevance for CoCosNet.
- CelebA-HQ: 91.3% of CelebA-HQ users preferred image quality and 90.6% judged CoCosNet most similar to the exemplar.
- DeepFashion: 90.6% of DeepFashion users preferred image quality and 98.8% preferred style resemblance for CoCosNet.
- User-study presentation: The detailed user-study results are presented for ADE20k, CelebA-HQ, and DeepFashion.
- Flickr: Using different exemplars, CoCosNet supports multimodal landscape synthesis from Flickr segmentation masks.The reported landscape outputs are described as highly realistic.
Appendix H. Limitation
The method has limitations when correspondence mappings are ambiguous and when high-resolution processing requires substantial GPU memory.
- One-to-many mappings may produce mixed-color artifacts, while many-to-one mappings may give multiple instances the same style.
- Computing the correlation matrix requires tremendous GPU memory, limiting scalability to high-resolution images.The paper leaves resolving this issue for future work.