Source-linked AI summary
Image-to-Image Translation: Methods and Applications
Yingxue Pang, Jianxin Lin, Tao Qin, Zhibo Chen
TL;DR
Image-to-image translation seeks to transfer images between domains while preserving source content and target style, but its methods and applications span a broad and fragmented literature. This paper surveys I2I generative models, task categories, applications, and evaluation practices, finding that supervised methods generally produce better results than comparable unsupervised methods while identifying continuing challenges in quality, efficiency, and model complexity.
Problem
I2I requires controlled transfer between image domains, including fine-grained attribute or object changes, while preserving other image content.
Method
The paper surveys I2I generative models, evaluation metrics, two-domain and multi-domain tasks, and applications using a structured taxonomy.
Results
Supervised methods usually produce better translated results than unsupervised methods on similar network structures, while BicycleGAN achieves 0.047 more LPIPS scores than MUNIT.
Takeaways & Limitations
I2I methods support applications including synthesis, segmentation, style transfer, inpainting, pose estimation, colorization, super-resolution, and domain adaptation.
Abstract
from arXiv · showhide
Image-to-image translation (I2I) aims to transfer images from a source domain to a target domain while preserving the content representations. I2I has drawn increasing attention and made tremendous progress in recent years because of its wide range of applications in many computer vision and image processing problems, such as image synthesis, segmentation, style transfer, restoration, and pose estimation. In this paper, we provide an overview of the I2I works developed in recent years. We will analyze the key techniques of the existing I2I works and clarify the main progress the community has made. Additionally, we will elaborate on the effect of I2I on the research and industry community and point out remaining challenges in related fields.
I. INTRODUCTION
Image-to-image translation converts images from a source domain to a target domain while preserving source content and transferring target style. This survey reviews I2I foundations, task taxonomies, generative models, evaluation, and applications.
- Problem setting: I2I maps a source-domain image to a target domain while preserving intrinsic content and transferring extrinsic style.The paper illustrates this with selfies translated into artistic images using cartoons as the target reference.
- Applications: I2I encompasses applications including semantic synthesis, segmentation, style transfer, inpainting, pose estimation, colorization, super-resolution, domain adaptation, cartoon generation, and registration.These applications span image processing, computer graphics, and computer vision.
- Survey scope: The survey reviews recent I2I progress and presents analysis, methodology, and related applications as an overview of the field.The authors describe it as the first overview to cover these three aspects of I2I.
- Foundations: It introduces representative generative models and evaluation metrics, then analyzes how these models learn and produce translation results.The survey also discusses subjective and objective metrics for assessing translated-image quality.
- Organization: The paper categorizes I2I into two-domain and multi-domain tasks and provides a matching taxonomy of I2I applications.Its organization also includes a review of generative models, applications, and a concluding outlook.
A. Variational AutoEncoder
VAEs use an encoder to approximate the latent posterior and a decoder to generate data, optimizing a variational lower bound instead of the data log-likelihood. They offer stable training but face a sample-quality versus reconstruction-quality trade-off.
- VAEs use an encoder qφ(z|x) to approximate p(z|x) and a decoder pθ(x|z) to map latent variables z to data x.
- A VAE introduces a diagonal-covariance multivariate Gaussian recognition model because naive Monte Carlo gradient estimation can have very high variance.
- The VAE optimizes the variational lower bound L(xi, θ, φ) rather than directly differentiating logpθ(xi).
- VAEs provide more stable training than GANs but often trade off sample quality against reconstruction quality because of weak or overly simplistic approximate posteriors.
1) Unconditional GANs:
Unconditional GANs generate images from random noise through an adversarial generator–discriminator game, while conditional GANs add information such as labels or text to control outputs. GAN training can suffer from mode collapse and convergence difficulties, motivating alternative objectives and normalization techniques.
- Unconditional GANs: An unconditional GAN feeds random noise z to generator G, while discriminator D distinguishes generated samples from real data.
- Conditional GANs: Conditional GANs concatenate additional information y, such as labels, text, or image attributes, with z to steer generated images toward desirable results.
- The Way to Train GANs: GAN optimization seeks a Nash equilibrium where discriminator D cannot distinguish generated samples from real samples.
- The Way to Train GANs: GAN training is often trapped in mode collapse and can be difficult to converge.
- The Way to Train GANs: WGAN uses Wasserstein distance, WGAN-GP uses gradient penalties, and SNGAN uses spectral normalization to improve GAN training stability.
- I2I evaluation combines subjective and objective metrics to assess visual quality more comprehensively.
1) Subjective image quality assessment: •
I2I image quality is assessed through human perceptual studies and objective measures covering fidelity, similarity, diversity, coverage, content preservation, and target-domain realism. Different metrics emphasize different aspects of translation quality.
- Subjective image quality assessment: AMT perceptual studies ask participants to choose which image in a real–fake pair appears real, producing scores from human judgments.
- Objective image quality assessment: PSNR and SSIM compare translated images with ground truth, with higher scores indicating closer intensity or perceptual similarity.
- Objective image quality assessment: IS and CIS measure diversity across outputs, with CIS conditioning diversity on a single input image.
- Objective image quality assessment: Mask-SSIM and Mask-IS reduce background influence and are designed for evaluating person image generation.
- Objective image quality assessment: Perceptual distance measures content similarity between translated and source images, with lower scores indicating more similar content.
- Objective image quality assessment: FID and KID measure distributional differences between real and generated images, where lower scores indicate better performance or more shared visual similarity.
- Objective image quality assessment: LPIPS evaluates output diversity and perceptual similarity, while DC simultaneously assesses generative diversity and fidelity through manifold estimation.
- Objective image quality assessment: FCN scores assess semantic-map-to-photo translation by comparing predicted label maps with ground-truth labels using segmentation metrics.
III. TWO-DOMAIN IMAGE-TO-IMAGE TRANSLATION
Two-domain I2I translates between source and target domains across diverse applications, with methods distinguished by supervision, output modality, and architectural design. The section reviews supervised, unsupervised, semi-supervised, and few-shot settings alongside representative improvements and limitations.
- Two-domain I2I supports applications including style transfer, semantic segmentation, colorization, and super-resolution.
- Supervised methods learn mappings from aligned source-target image pairs, whereas unsupervised methods use large unpaired image sets because paired data are difficult and costly to collect.
- The section further distinguishes semi-supervised methods using limited paired data and abundant unlabeled data from few-shot methods using extremely limited examples.
- Pix2pix established a conditional-GAN framework for supervised I2I, combining pixelwise L1 regression with adversarial training and inspiring later variants.
- Pix2pix and related single-network variants remain limited for drastically different views, severe deformations, high-resolution computation, and dense semantic correspondence.SelectionGAN addresses cross-view translation, ASAPNet targets computational cost, and hierarchical refinement reduces high-resolution correspondence demands.
2) Multimodal Outputs:
Multimodal two-domain I2I models generate multiple target-domain outputs while preserving source content, addressing the ambiguity that single-output mappings cannot represent. Unsupervised approaches organize around cycle consistency, alternatives for large domain gaps, fine-grained object translation, and knowledge from other fields.
- Multimodal Outputs: Multimodal I2I maps one input image to a distribution of possible target-domain outputs while remaining faithful to the input.
- Multimodal Outputs: Diverse outputs address mode collapse, with methods such as BicycleGAN, PixelNN, and disentangled representations supporting multimodal or controllable translation.
- Single-modal Output: Cycle consistency enables unpaired translation by reconstructing a source image through two opposing translators, but can fail when domains require substantial clutter, heterogeneity, or shape deformation.Cycle-based methods include DualGAN, DiscoGAN, and CycleGAN; later work targets translation beyond cycle consistency for large domain gaps.
- Single-modal Output: Fine-grained object translation separates local instance changes from global style changes, supporting scenarios such as changing clothing while leaving other image regions unchanged.DAGAN, attention GAN, attention-guided I2I, and InstaGAN are representative approaches.
- Single-modal Output: Other unsupervised methods combine I2I with geometry distances, semantic transformations, or related techniques to improve translation beyond strict cycle-consistency constraints.
2) Multi-modal Outputs:
Multimodal unsupervised I2I models address the limitation of one-to-one mappings by learning domain-invariant content alongside domain-specific attributes. The section also highlights unresolved difficulties with content-rich target images containing multiple discrepant objects.
- Multi-modal Outputs: Shared-latent-space assumptions capture domain-invariant information but can miss domain-specific information, motivating one-to-many or many-to-many mappings.
- Multi-modal Outputs: Disentangled representations separate domain-invariant content from domain-specific attributes to support multimodal unsupervised translation.DRIT, MUNIT, cd-GAN, and EGSC-IT use related content and attribute representations.
- Multi-modal Outputs: Existing multimodal methods still struggle with target-domain images containing multiple discrepant objects and with content relationships omitted by shared content spaces.INIT separately translates instance-level objects and background or global areas using different style codes.
- Multi-modal Outputs: Mode-seeking regularization and latent filter scaling provide additional approaches for increasing diversity in multimodal unsupervised translation.
C. Semi-Supervised Image-to-Image Translation
Semi-supervised and few-shot I2I reduce the amount of paired or total training data needed for translation. The reviewed methods use unlabeled-data consistency, transfer learning, shared representations, or coarse-to-fine training to operate under limited supervision.
- Semi-Supervised Image-to-Image Translation: Semi-supervised I2I combines a small amount of paired data with abundant unlabeled data, which can considerably improve translation performance in specialized applications.
- Semi-Supervised Image-to-Image Translation: Transformation consistency regularization aligns predictions for transformed unlabeled inputs and transformed predictions, encouraging learning from source and target distribution structure.
- Semi-Supervised Image-to-Image Translation: Less than 1% of labeled data can support image colorization, denoising, and super-resolution when unlabeled data are incorporated.
- Few-Shot Image-to-Image Translation: Few-shot I2I seeks transfer or generalization to new categories from very few samples, using pretrained networks, shared autoencoder layers, or selective backpropagation.
- Few-Shot Image-to-Image Translation: TuiGAN performs one-shot unpaired I2I with two images by progressively translating through coarse-to-fine generator and discriminator pyramids.The approach is presented as a step toward unsupervised learning with extremely limited data.
IV. MULTI-DOMAIN IMAGE-TO-IMAGE TRANSLATION
Multi-domain I2I seeks to translate among multiple domains with one unified model rather than separate pairwise models. The surveyed methods use multimodules, a shared generator–discriminator pair, auxiliary labels, consistency losses, and related techniques.
- With n domains, pairwise two-domain methods require n×(n−1) generators, creating efficiency and network burdens.
- Multi-domain I2I uses a single unified model to produce outputs with different semantic contents or style textures.
- Training with multimodules: Multimodule methods train all domains simultaneously by composing several two-domain translators, reducing training time and model parameters.
- Training with one generator and discriminator pair: A shared generator and discriminator pair can translate to a randomly selected target domain using an auxiliary target-domain label.
- Relative attributes support finer-grained editing by representing the desired change between input and target attributes rather than requiring complete target attributes.
- Multipath consistency regularization encourages direct brown-to-blonde translation to match the indirect brown-to-black-to-blonde route.
2) Multimodal Outputs:
Multimodal multi-domain I2I addresses the limitation that earlier approaches learn deterministic mappings between arbitrary domains. The surveyed methods combine domain supervision, disentangled representations, and diversity-oriented losses to generate varied outputs.
- Earlier multi-domain methods learn deterministic mappings, motivating research on multimodal outputs.
- DosGAN uses explicit domain supervision by pretraining a CNN to predict an image’s domain and combining it with a domain classifier.
- GANimation continuously synthesizes anatomically aware facial movements by controlling the activation magnitude of each action unit.
- Disentanglement-based models separate content and style representations and use consistency, adversarial, regression, reconstruction, and mode-seeking losses.
B. Semi-Supervised multi-domain Image-to-Image Translation
Semi-supervised and few-shot multi-domain I2I reduce dependence on fully labeled data and address rapid adaptation to new domains. The reviewed methods use mixed labeled and unlabeled data, pseudo-labeling, few-shot translation, or meta-learning.
- AGUIT is described as the first method handling multimodal and multi-domain I2I with semi-supervised learning.
- AGUIT decomposes content and style, reconstructs and translates with AdaIN, and enforces cycle and feature consistency using labeled and unlabeled images.
- Few-shot multi-domain I2I targets rapid generalization from a few examples, contrasting with methods that struggle to adapt quickly.
- FUNIT translates source images toward target classes using many source-class images and few target-class images after multiclass training.
- FUNIT can lose domain-invariant content appearance because target-image style codes exert severe influence; COCO-FUNIT redesigns the style encoder to address this.
- ZstGAN uses meta-learning to transfer translation knowledge from seen domains to unseen classes using annotated attributes.
A. Datasets
The survey evaluates I2I methods on paired shoe-edge translation and multi-domain facial-attribute manipulation using common quality and diversity metrics. Results favor supervised methods in comparable settings, while newer architectures and training strategies can outperform older methods.
- A. Datasets: UT-Zap50K provides 49,826 training pairs and 200 test images for two-domain edge-to-shoes evaluation.
- FID, IS, and LPIPS evaluate generated-image quality and diversity; lower FID and higher IS or LPIPS indicate better outcomes.
- C. Results: Supervised pix2pix and BicycleGAN achieve better FID, IS, and LPIPS scores than CycleGAN, U-GAT-IT, and GDWCT on single-modal edge-to-shoes translation.
- C. Results: CUT obtains the best FID and IS scores among the compared edge-to-shoes methods without supervision, associated with its StyleGAN backbone and contrastive content constraint.
- C. Results: BicycleGAN and MUNIT generate multimodal realistic outputs, with BicycleGAN achieving 0.047 more LPIPS than MUNIT.
- C. Results: StarGANv2 achieves the best multi-domain FID and IS scores and generates realistic vivid translations, while other methods show artifacts or blurring.
- C. Results: StarGANv2 produces the best LPIPS in multimodal multi-domain translation, whereas StarGAN fails to generate diverse outputs despite injected randomness.
VII. SUMMARY AND OUTLOOK
This review synthesizes recent progress in image-to-image translation by analyzing its methods, generative-model foundations, tasks, and applications. It also identifies unresolved challenges involving network complexity, output quality, efficiency, and extending translation beyond images.
- The paper reviews recent I2I progress, analyzing methodology and related applications to clarify advances in the community.
- Methodology: It introduces representative generative models and evaluation metrics, then examines I2I methods for two-domain and multi-domain tasks.
- Applications: The review provides a thorough taxonomy of I2I applications organized according to the categorizations used for I2I methods.
- Challenges: A central outlook challenge is balancing network complexity against result quality at higher resolution while maintaining efficient generation of diverse, high-fidelity outputs.
- Outlook: The paper highlights lightweight I2I networks as a practical direction and identifies cross-modality translation beyond images as an additional research trend.