Source-linked AI summary
Pretraining is All You Need for Image-to-Image Translation
Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, Fang Wen
TL;DR
Existing image-to-image translation methods often require task-specific designs and training from scratch, limiting high-quality synthesis for complex scenes and scarce paired data. PITI adapts a pretrained diffusion model to multiple translation tasks, adding adversarial diffusion training and normalized guidance; experiments report strong realism and faithfulness across challenging benchmarks, with promising few-shot potential. Its main limitation is imperfect alignment with inputs, including missed small objects.
Problem
Existing image-to-image translation methods require dedicated designs and struggle with high-fidelity generation for complex scenes, especially when paired training data are limited.
Method
PITI adapts a pretrained diffusion model as a generative prior for downstream translation tasks, using task-specific input projection plus adversarial training and normalized guidance.
Results
PITI achieves high-quality synthesis across mask-to-image, sketch-to-image, and geometry-to-image tasks on ADE20K, COCO-Stuff, and DIODE, with promising few-shot potential.
Takeaways & Limitations
Pretraining can support a generic image-to-image translation framework and reduce data requirements, with comparable FID to full-data prior methods using 25% of ADE20K training data.
Takeaways & Limitations
Sampled images can have difficulty faithfully aligning with inputs and may miss small objects, possibly because the pretrained model’s intermediate space lacks accurate spatial information.
Abstract
from arXiv · showhide
We propose to use pretraining to boost general image-to-image translation. Prior image-to-image translation methods usually need dedicated architectural design and train individual translation models from scratch, struggling for high-quality generation of complex scenes, especially when paired training data are not abundant. In this paper, we regard each image-to-image translation problem as a downstream task and introduce a simple and generic framework that adapts a pretrained diffusion model to accommodate various kinds of image-to-image translation. We also propose adversarial training to enhance the texture synthesis in the diffusion model training, in conjunction with normalized guidance sampling to improve the generation quality. We present extensive empirical comparison across various tasks on challenging benchmarks such as ADE20K, COCO-Stuff, and DIODE, showing the proposed pretraining-based image-to-image translation (PITI) is capable of synthesizing images of unprecedented realism and faithfulness.
1 Introduction
PITI uses pretraining to address difficult image-to-image translation by adapting a pretrained diffusion model as a generative prior. Across diverse downstream tasks, it reports high-quality, diverse synthesis and improved performance, including with limited data.
- Existing image-to-image translation methods use task-specific customization yet still struggle to produce high-fidelity outputs for practical use.
- PITI pretrains a synthesis network to capture the natural image manifold, then adapts downstream inputs into the pretrained model’s latent representation.
- GLIDE provides PITI’s pretrained diffusion prior because it can model diverse images and offers a semantic latent space through text conditioning.
- Adversarial training improves texture synthesis during diffusion upsampling, while normalized guidance sampling is introduced to improve generation quality.
- PITI achieves high-quality and diverse results across mask-to-image, sketch-to-image, and geometry-to-image translation on ADE20K, COCO-Stuff, and DIODE.
2 Related Work
Prior work includes cGANs, autoregressive models, GAN latent-space priors, and diffusion-based translation, but these approaches have limitations in generality, speed, or robustness. PITI instead presents a generic pretrained diffusion framework for multiple translation tasks without task-specific customization or hyper-parameter tuning.
- cGANs are widely used for image-to-image translation, while autoregressive models offer expressivity but are slow at inference and prone to overfitting.
- GAN latent spaces support semantic manipulation but are limited by class-specific modeling, mode dropping, and stability issues for general-image synthesis.
- Unlike prior task-specific or stroke-focused approaches, PITI uses pretraining to support various translation tasks without task-specific customization or hyper-parameter tuning.
- Diffusion models achieve competitive quality and diversity across benchmarks and show strong capacity when trained on large-scale text-image pairs.
3 Approach
The framework pretrains a diffusion model to learn a semantic latent space, then adapts it to downstream image-to-image translation through conditional finetuning. It further improves synthesis with adversarial upsampling and normalized classifier-free guidance.
- Preliminary: Diffusion models reverse a gradual Gaussian noising process, starting from pure noise and iteratively predicting the added noise to generate images.Conditional diffusion additionally injects inputs such as class labels, text prompts, or degraded images.
- Pretraining: Pretraining on huge, diverse text-image data provides a semantic latent space that models general image statistics for downstream synthesis.The framework builds on GLIDE, whose text embeddings supply the semantic conditioning space.
- Downstream adaptation: Downstream adaptation trains a task-specific head to project conditions into the pretrained embedding space, followed by staged finetuning of the encoder and decoder.The decoder is first fixed while the task-specific encoder learns the mapping, after which both are jointly finetuned for improved spatial alignment.
- Adversarial diffusion upsampler: Adversarial and perceptual losses are added to diffusion upsampling because Gaussian denoising otherwise produces oversmoothed local structures.The upsampler is finetuned with degraded inputs to reduce the gap between training images and base-model samples.
- Normalized classifier-free guidance: Classifier-free guidance improves individual-sample quality at the cost of diversity, but its mean and variance shifts can accumulate into saturated images with oversmoothed textures.Normalized guidance explicitly matches the guided noise statistics to the original conditional estimate, especially for large guidance strength w.
- Evaluation: On multiple image-translation tasks, Table 1 compares FID scores across methods, while Table 2 reports user preference rates on COCO.The captions identify the best FID score and preference rate over baselines as the table outputs.
4 Experiments
Experiments evaluate PITI across multiple image-to-image translation tasks and datasets, comparing it with strong baselines, from-scratch diffusion, and ablated training strategies. Results show improved quantitative and qualitative performance, with benefits from two-stage finetuning, adversarial upsampling, normalized guidance, and pretraining under reduced data.
- Quantitative results: PITI consistently outperforms the model without pretraining, improving FID over OASIS by 5.9 on ADE20K, 3.6 on COCO, and 4.4 on Flickr.The evaluation uses CLIP-based FID and also reports promising performance on sketch-to-image and geometry-to-image synthesis.
- Qualitative results: Compared with from-scratch methods, PITI produces higher-quality and more diverse images with vivid details and correct semantics in complex scenes.The comparison covers COCO, ADE20K, and other datasets and input modalities.
- Human evaluation: A user study with 3,000 votes from 20 participants finds that PITI outperforms from-scratch models and other baselines in mask-to-image synthesis.Participants selected the more realistic image from paired comparisons on COCO-Stuff.
- Ablation study: Two-stage finetuning achieves the best ablation performance, whereas fixing the decoder preserves visual quality but fails to align outputs with the semantic map.The staged procedure first trains the task-specific encoder with the decoder fixed, then jointly finetunes the full model.
- Ablation study: Adversarial and perceptual losses improve diffusion upsampling quality, while normalized classifier-free guidance produces more vivid details than classifier-free guidance at large guidance strengths.Input degradations also lead to better FID scores, whereas reconstruction loss alone yields fewer high-frequency details.
- Smaller training dataset: With 25% of ADE20K training data, PITI achieves a comparable FID to previous methods trained on the full dataset.The reduced-data experiments use 5k and 10k training images.
5 Conclusion and Limitation
PITI provides a simple, universal framework for image-to-image translation, but its sampled images can struggle to align faithfully with inputs and preserve small objects.
- PITI brings pretrained generative modeling to varied image-to-image translation tasks through a simple and universal framework.
- Sampled images may have difficulty faithfully aligning with given inputs and may miss small objects.The authors suggest that the pretrained model's intermediate space may lack accurate spatial information.
- The authors plan to explore alternative pretraining methods to address these limitations.
Broader Impact
The work examines whether large-scale pretraining can benefit conditional image synthesis and reports advantages for complex objects and general scenes. It also discusses the energy and carbon costs of pretraining alongside reuse across downstream tasks and reduced data requirements.
- The work explores whether large-scale image pretraining can improve conditional image synthesis for complex objects and general scenes.
- Pretraining requires energy and produces carbon emissions, but one pretrained model can be shared across downstream tasks through conditional finetuning.
- Pretraining can enable generative-model training with fewer data when privacy constraints or annotation costs limit available data.
A Implementation details
The implementation adapts a pretrained diffusion decoder with a condition encoder and trains the system in two stages. The setup uses specified diffusion schedules for training and sampling.
- The pretrained GLIDE diffusion decoder is paired with a convolutional-and-ViT condition encoder that maps inputs to semantic tokens.Both the base and upsampling models use 1,000 diffusion steps during training.
- The two-stage finetuning scheme first freezes the decoder while training the encoder, then jointly finetunes the full model.The first stage runs for 200K iterations; the second stage uses joint optimization.
- Sampling uses 250 diffusion steps for the base model and 27 steps for the upsampling model.
B Additional results
Additional experiments report high-quality, diverse visual outputs, image-manipulation applications, and quantitative evaluation using CLIP-based and Inception-based FID measures across diverse translation tasks.
- Visual comparisons and sampled outputs show high quality and diversity across additional results.
- The model supports image composition, object removal, semantic-class changes, and shape changes while preserving unedited content with DDIM sampling.
- Quantitative evaluation reports CLIP-based FID-C and InceptionNet-based FID-I for comparing generative models.The paper motivates CLIP-based evaluation because InceptionNet FID may not align with human judgments of perceptual quality.
- Table 5 presents quantitative comparisons across diverse image translation tasks.
C Limitation
The proposed approach has two reported limitations: correlated styles among similar objects within one sample and imperfect alignment with input conditions, especially for small objects.
- C Limitation: Similar objects within a single generated sample can exhibit highly correlated styles, termed intra-image mode collapse.This differs from conventional mode collapse, which concerns limited diversity across samples.
- C Limitation: Generated images may show minor misalignment with input conditions, with some small objects missed.Examples include improperly synthesized trees and missed lighting.
- C Limitation: The authors conjecture that inaccurate spatial information in the latent space contributes to these alignment errors because pretraining used text-to-image synthesis.They leave alternative image-pretraining strategies for future work.