Source-linked AI summary
One-Step Image Translation with Text-to-Image Models
Gaurav Parmar, Taesung Park, Srinivasa Narasimhan, Jun-Yan Zhu
TL;DR
Conditional diffusion models face slow iterative inference and often require paired data, while image translation can also lose important input details. This paper adapts pretrained one-step diffusion models with a compact end-to-end generator and adversarial objectives, achieving strong paired and unpaired translation results with one-step inference. The authors conclude that such models can serve as versatile backbones for downstream image synthesis tasks.
Problem
Conditional diffusion models are limited by slow iterative inference, reliance on paired training data, and detail loss during image translation.
Method
The method adapts a pretrained one-step diffusion model through adversarial learning and consolidates the Encoder, U-Net, and Decoder with LoRA, direct conditioning, and skip connections.
Results
CycleGAN-Turbo significantly outperforms existing GAN-based and diffusion-based methods on unpaired translations, while pix2pix-Turbo is visually comparable to recent conditional diffusion models on paired tasks.
Takeaways & Limitations
Pretrained one-step text-to-image models can serve as strong, versatile backbones for paired and unpaired image-synthesis objectives without multi-step diffusion training.
Takeaways & Limitations
The method cannot specify guidance strength or use negative prompts because its SD-Turbo backbone lacks classifier-free guidance.
Abstract
from arXiv · showhide
In this work, we address two limitations of existing conditional diffusion models: their slow inference speed due to the iterative denoising process and their reliance on paired data for model fine-tuning. To tackle these issues, we introduce a general method for adapting a single-step diffusion model to new tasks and domains through adversarial learning objectives. Specifically, we consolidate various modules of the vanilla latent diffusion model into a single end-to-end generator network with small trainable weights, enhancing its ability to preserve the input image structure while reducing overfitting. We demonstrate that, for unpaired settings, our model CycleGAN-Turbo outperforms existing GAN-based and diffusion-based methods for various scene translation tasks, such as day-to-night conversion and adding/removing weather effects like fog, snow, and rain. We extend our method to paired settings, where our model pix2pix-Turbo is on par with recent works like Control-Net for Sketch2Photo and Edge2Image, but with a single-step inference. This work suggests that single-step diffusion models can serve as strong backbones for a range of GAN learning objectives. Our code and models are available at https://github.com/GaParmar/img2img-turbo.
1 Introduction
The paper adapts pretrained single-step diffusion models for paired and unpaired image translation, addressing slow iterative inference, paired-data requirements, and input-detail loss. Its CycleGAN-Turbo and pix2pix-Turbo models deliver efficient, realistic translations across multiple tasks.
- Motivation: Conditional diffusion models are slow because inference uses iterative denoising and often require costly paired datasets.These limitations restrict real-time applications and make some training settings infeasible.
- Approach: The method adapts a pretrained one-step diffusion model to new tasks and domains through adversarial learning in paired and unpaired settings.It reduces inference to 1 step while retaining the internal knowledge of the pretrained model.
- Approach: The generator consolidates the Encoder, U-Net, and Decoder into one end-to-end network using LoRA, direct conditioning, and zero-conv skip connections.These choices target conditioning conflicts, reduce overfitting and fine-tuning time, and preserve high-frequency input details.
- Results: CycleGAN-Turbo significantly outperforms existing GAN-based and diffusion-based methods on unpaired day-night and weather-effect translation tasks.The comparison covers distribution matching, input structure preservation, and efficiency relative to diffusion-based methods.
- Results: pix2pix-Turbo achieves visually comparable results to recent conditional diffusion models for Edge2Image and Sketch2Photo with single-step inference.The model also supports diverse outputs by interpolating between pretrained noise maps and encoder outputs.
2 Related Work
Prior image-translation methods include paired and unpaired GAN approaches, conditional diffusion models, and accelerated one-step or few-step generative models. The paper positions its architecture as a pretrained diffusion backbone adapted to new domains without multi-step diffusion training.
- Image-to-Image Translation: Paired image-translation methods combine reconstruction and adversarial losses to map source images to target domains.
- Image-to-Image Translation: Conditional diffusion methods extend pretrained text-to-image models with spatial or textual controls for image translation.
- Image-to-Image Translation: Unpaired methods use cycle consistency, shared latent spaces, content-preservation losses, or contrastive learning when corresponding image pairs are unavailable.Existing GAN-based and diffusion-based approaches typically train from scratch on new domains.
- Positioning: This work adapts pretrained diffusion models to unpaired translation instead of training new GAN or diffusion models from scratch.
- One-Step Generative Models: One-step generative models accelerate inference by using fast ODE solvers or distilling multistep teacher models into faster students.
3 Method
The method adapts a one-step diffusion model for image translation by directly incorporating conditioning, preserving details with skip connections, and training a consolidated generator with adversarial objectives. It supports unpaired and paired translation while targeting structural fidelity, domain matching, and diverse outputs.
- 3.1 Adding Conditioning Input: The method starts from a one-step text-to-image model and incorporates the input image through conditioning mechanisms for image translation.The approach uses Stable Diffusion Turbo v2.1 with one-step inference as its base network.
- 3.1 Adding Conditioning Input: Separate condition-encoder branches can conflict with the noise-derived layout, deteriorating downstream decoder features and output quality.The method therefore feeds conditioning information directly into the noise encoder branch rather than through an additional adapter branch.
- 3.2 Preserving Input Details: Skip connections preserve fine-grained input details, including text, street signs, and distant cars, during day-to-night translation.Without skip connections, these details are not preserved in the translated image.
- 3.2 Preserving Input Details: The generator consolidates the Encoder, UNet, and Decoder into one end-to-end trainable architecture and uses LoRA to adapt pretrained weights with reduced overfitting.The architecture combines the modules while retaining SD-Turbo weights and adding skip connections between encoder and decoder stages.
- 3.3 Unpaired Training: Unpaired training uses shared translation functions with domain-specific captions and combines cycle consistency, adversarial, and identity losses.The adversarial loss uses discriminators for both domains, while cycle consistency reconstructs each source image after translation in both directions.
- 3.4 Extensions: The architecture extends to paired translation and stochastic outputs by using reconstruction, GAN, and CLIP alignment losses and interpolating encoder features with noise.The interpolation coefficient γ controls the transition between pretrained stochastic behavior and deterministic translation; γ = 0 disables reconstruction enforcement, whereas γ = 1 gives deterministic translation.
- 3.4 Extensions: On 256 × 256 datasets, the unpaired method is compared with CUT and Instruct-pix2pix, whose outputs respectively show artifacts and weak input-structure preservation.These comparisons motivate evaluating both visual quality and structural fidelity in unpaired translation.
4 Experiments
Experiments evaluate the method across unpaired and paired image-translation tasks using distribution alignment, structure preservation, realism, speed, and ablations. The method generally outperforms or competes with prior baselines while preserving structure and enabling one-step inference.
- Unpaired comparisons: The method achieves the lowest DINO-Struct. across standard CycleGAN tasks and the lowest FID on all except Horse →Zebra, while being magnitudes faster than diffusion models.Cycle-Diffusion obtains a slightly better FID on one task but has a large increase in DINO-Struct.
- Unpaired comparisons: On 512 × 512 driving datasets, the method outperforms all GAN-based and diffusion-based baselines across the evaluated datasets.InstructPix2pix can obtain a slightly lower DINO-Struct for Day →Night but has much higher FID, indicating poorer target-distribution matching.
- Human evaluation: Human users prefer the method across all datasets except Clear to Foggy, where InstructPix2Pix produces more artistic fog images.For Clear to Foggy, InstructPix2Pix still has worse structure and distribution metrics than the proposed method.
- Ablation study: The full method outperforms ablation variants in distribution matching and structure preservation, while jointly training encoder and decoder improves structure preservation at a small FID cost.Random initialization produces unnatural outputs and increases FID; alternative conditioning branches can hallucinate objects or introduce artifacts.
5 Discussion and Limitations
The method is positioned as a versatile one-step backbone for downstream image synthesis, but its capabilities remain bounded by guidance, prompting, memory, and resolution constraints.
- Discussion: One-step pretrained models can support downstream image synthesis through GAN objectives with relatively few additional trainable parameters.The paper presents this as a broader implication of the proposed adaptation approach.
- Limitations: The method cannot specify guidance strength because SD-Turbo does not use classifier-free guidance.Guided distillation is identified as a possible way to enable guidance control.
- Limitations: The method does not support negative prompts, and cycle-consistency training with high-capacity generators is memory-intensive.The paper also identifies higher-resolution synthesis as an area for future work.
Appendix
The appendix organizes supplementary experiments, baseline comparisons, analyses, and implementation details supporting the main study.
- Appendix organization: Appendix A provides additional ablations, Appendix B adds GAN-based and diffusion-based baseline comparisons, and Appendix C analyzes conditioning conflicts and training choices.Appendix D reports hyperparameters and training details.
A Additional Ablation Study
Additional ablations on Horse →Zebra and Day →Night confirm that the complete method provides the best combined distribution matching and structure preservation among the tested variants.
- Additional datasets: The appendix repeats ablations on Horse →Zebra and qualitatively extends them to Day →Night using variants with random initialization, ControlNet encoding, and T2I-Adapter conditioning.These experiments supplement the main-paper ablation comparisons.
- Additional datasets: The full method outperforms all other variants in both FID-based distribution matching and DINO Structure Distance.The result supports the combined design choices evaluated in the ablation study.
B Additional Baseline Comparisons
Additional comparisons show that the method produces realistic translations while preserving input structure across GAN and diffusion baselines, with ablations supporting its component choices.
- The method consistently produces more realistic outputs while retaining the structure of input images across additional GAN and diffusion comparisons.
- Ablations on Horse ↔Zebra, Day →Night, and Night →Day show the final method achieving the best translation results, detail preservation, or convincing outputs.
- The additional comparisons include CycleGAN and CUT for Horse ↔Zebra and Day →Night translation tasks.
C Additional Analysis
The additional analysis examines conditioning conflicts, training-data requirements, and skip connections, emphasizing structure preservation and fine-detail retention.
- Using a separate Condition Encoder can cause the original network to be ignored, with different noise maps producing perceptually similar outputs for the same condition image.
- The additional diffusion-baseline comparison evaluates the method on the Horse ↔Zebra translation task.
- Structure preservation remains largely unchanged across different settings despite a slight increase in FID, suggesting training on small datasets is possible.
- Without skip connections, the VAE encoder-decoder baseline fails to preserve fine details such as text and street signs.
D Training Details
The experiments use specified datasets, optimization settings, and paired or unpaired objectives, alongside analyses of noise, skip connections, and training-data scale.
- Unpaired evaluations cover Day ↔Night, Clear ↔Foggy, Horse ↔Zebra, and Yosemite Winter ↔Summer datasets.
- The unpaired objective uses Adam with learning rate 1e-6, batch size 8, λidt = 1, and λGAN = 0.5.
- The paired objective combines reconstruction, GAN, and CLIP text-image alignment losses with λGAN = 0.4 and λCLIP = 4.
- The paired pix2pix-Turbo method is trained on Edge2Image and Sketch2Image using a community-collected artistic-image dataset and ControlNet preprocessing.
- Edge2Image uses Canny edges with random thresholds, while Sketch2Image uses HED-based synthetic sketches and augmentations.
- Different noise maps do not alter image structure, while removing skip connections degrades preservation of text and street-sign details.