Source-linked AI summary
Diffusion-based Image Translation using Disentangled Style and Content Representation
Gihyun Kwon, Jong Chul Ye
TL;DR
Diffusion-based image translation struggles to preserve input content while changing semantics under text or target-image guidance, particularly without impractical matched training pairs. The paper uses ViT-derived content and style representations with CLIP, semantic divergence, and resampling losses, and reports state-of-the-art performance across both guidance modes. The method nonetheless has difficulty with large source–target domain gaps and depends on the pretrained score model’s quality.
Problem
Diffusion image translation must change semantic style while preserving input structure, but disentangling these components is difficult and matched training pairs are often impractical.
Method
The method uses intermediate DINO ViT attention keys for content preservation, [CLS] tokens and CLIP for style guidance, plus semantic divergence and resampling for accelerated diffusion.
Results
The framework achieves state-of-the-art performance in both text-guided and image-guided semantic image translation tasks.
Takeaways & Limitations
The framework supports unconditional diffusion-based translation guided by either text or a target image while preserving input content.
Takeaways & Limitations
Translation is often unsuccessful when the source and target domains have a large semantic gap, and generation quality depends heavily on the pretrained score model.
Abstract
from arXiv · showhide
Diffusion-based image translation guided by semantic texts or a single target image has enabled flexible style transfer which is not limited to the specific domains. Unfortunately, due to the stochastic nature of diffusion models, it is often difficult to maintain the original content of the image during the reverse diffusion. To address this, here we present a novel diffusion-based unsupervised image translation method using disentangled style and content representation. Specifically, inspired by the splicing Vision Transformer, we extract intermediate keys of multihead self attention layer from ViT model and used them as the content preservation loss. Then, an image guided style transfer is performed by matching the [CLS] classification token from the denoised samples and target image, whereas additional CLIP loss is used for the text-driven style transfer. To further accelerate the semantic change during the reverse diffusion, we also propose a novel semantic divergence loss and resampling strategy. Our experimental results show that the proposed method outperforms state-of-the-art baseline models in both text-guided and image-guided translation tasks.
1 INTRODUCTION
Diffusion-based image translation must disentangle semantic style changes from input content and structure, especially without matched training pairs. The paper introduces ViT-guided losses and accelerated sampling for text- and image-guided translation.
- Motivation: Diffusion-based image translation remains challenged by separately controlling input-image and text conditions.The paper identifies disentanglement and separate control as an open problem when multiple conditions are provided.
- Motivation: Maintaining input structure while changing only semantic style is difficult for unconditional diffusion models because content and semantics can change together.Matched conditional training can address this issue, but it is impractical for many translation tasks.
- Approach: The method uses intermediate DINO ViT attention keys for content regularization and [CLS] tokens for style regularization.Similarity and contrastive losses preserve structure, while [CLS] matching supports image-guided style transfer.
- Approach: CLIP loss enables text-driven style transfer, while semantic divergence loss and resampling accelerate semantic changes during diffusion.These components are designed to improve control and sampling speed.
- Results: The framework achieves state-of-the-art performance in both text-guided and image-guided style transfer tasks.The reported evaluation is quantitative and qualitative.
2 RELATED WORK
Prior image-translation methods span text-conditioned generation, single-shot style transfer, and domain-specific or unseen-image translation. Their limitations motivate a more general diffusion-based approach.
- Text-guided image synthesis: CLIP-conditioned approaches generate images from text prompts, but GAN-based methods often poorly control semantic changes for out-of-distribution images.The related work describes this limitation in text-driven image generation.
- Text-guided image synthesis: Diffusion models achieve strong text-conditioned image generation, yet input images and text conditions are not typically disentangled for image translation.DiffusionCLIP partially addresses this with DDIM sampling and pixelwise regularization.
- Single-shot image translation: Single-target-image translation methods include early style transfer, StyleGAN adaptation, and unseen-image semantic conversion.These approaches broaden translation settings beyond conventional paired or domain-specific models.
- Single-shot image translation: StyleGAN adaptation methods can perform well but remain domain-specific, while unseen-image conversion methods often suffer degraded image quality.The cited examples include human-face-focused models and methods for unseen images.
3 PROPOSED METHOD
The method guides unconditional reverse diffusion to preserve input structure while changing semantic style toward a target image or text. It combines ViT-based structure and style losses with CLIP guidance and semantic divergence.
- 3.1 DDPM Sampling with Manifold Constraint: DDPM reverse sampling starts from Gaussian noise and iteratively denoises samples through learned Gaussian transitions.The forward process adds Gaussian noise over time, while the reverse process uses a learned mean and fixed variance.
- 3.1 DDPM Sampling with Manifold Constraint: Matched-target conditional diffusion tightly controls generation but requires paired targets, limiting applicability to many translation tasks.Unconditional alternatives are also described as insufficient for reliably preserving content.
- 3.2 Structure Loss: The structure loss uses DINO ViT self-attention keys and contrastive learning to preserve spatial content between input and output.Keys encode structure and spatial patch information; same-position keys are brought closer while different positions are separated.
- 3.3 Style Loss: Text-guided translation uses an input-aware directional CLIP loss that combines target text, source image, and source text embeddings.The formulation removes source-domain information, reflects source-image information, and uses augmentation to prevent adversarial artifacts.
- 3.3 Style Loss: Image-guided translation matches ViT [CLS] tokens between denoised samples and the target image to guide semantic style.The method also addresses color-statistic misalignment associated with using [CLS] tokens alone.
- 3.4 Acceleration Strategy: Semantic divergence accelerates translation by maximizing [CLS]-token distance between successive denoised outputs while preserving structure.The authors report that pixel-wise or perceptual divergence harms content structure, whereas semantic divergence controls appearance.
4 EXPERIMENT
Experiments evaluate text-guided and image-guided translation across natural-image datasets, including quantitative, qualitative, user-study, and ablation comparisons. The proposed model generally improves perceptual quality while balancing target semantics with source-content preservation.
- Qualitative comparisons: Qualitative figures report better perceptual quality than baselines for Animals, Landscapes, and image-guided translation, while the full ablation setting performs best.
- Text-guided semantic image translation: The text-guided evaluation uses Animals and Landscapes datasets, containing 14 and 7 image classes, respectively.
- Text-guided semantic image translation: The model achieves the best SFID and CSFID image-quality scores among baselines, while ranking second in LPIPS content preservation.FlexIT obtains the best LPIPS score but is overly confined to source images and fails to produce adequate semantic change.
- Text-guided semantic image translation: User studies show the proposed model has the best overall performance for text-guided translation across text-match, realism, and content criteria.
- Image-guided semantic image translation: For image-guided translation, the model obtains the best style-matching score and second-best realism and content-preservation scores.WCT2 leads realism and content preservation but performs worst in style matching because its outputs change little beyond overall colors.
- Ablation study: Ablations show that the full configuration best combines target-semantic conversion with source-content preservation.Removing semantic divergence, content loss, or resampling damages semantic change or content retention; replacing the ViT-based losses also causes artifacts or structural degradation.
5 CONCLUSION
The conclusion presents a ViT-guided DDPM framework for content preservation and semantic change, supplemented by resampling for diffusion initialization. Experiments report superior performance over baselines in text- and image-guided translation, but large source–target domain gaps remain challenging.
- The method uses a pre-trained ViT to guide DDPM generation toward content preservation and semantic changes.
- A resampling strategy is proposed to improve diffusion-process initialization.
- Extensive experiments show superior performance compared with baselines in both text-guided and image-guided semantic image translation.
- The method often fails to translate image styles when the source and target domains have a large gap.
A.1 IMPLEMENTATION DETAILS
Implementation uses fixed diffusion and resampling schedules, DINO ViT and ensemble CLIP guidance, and several quality-improvement procedures. Text-guided and image-guided settings share the diffusion framework but use distinct sampling and guidance choices.
- Text-guided manipulation uses T = 100 initial samples, skips 40 steps, and performs 10 resampling steps for 70 total iterations.
- The implementation uses a pre-trained DINO ViT with layer l = 11 keys and the final-layer [CLS] token, while images are resized to 224×224 for ViT and CLIP.
- A restart procedure reruns generation when the initial regularization loss exceeds 0.01, but this trick is excluded from quantitative comparisons.
- Geometric and noise augmentations are used, with noisy-image mixing helping remove artifacts.
- For landscape image-guided translation, output colors are matched to the target image distribution to improve perceptual quality.
- Baseline experiments use official implementations and suggested settings, with diffusion baselines using unconditional score models trained at 256×256 resolution.
A.2 DATASET DETAILS
The appendix describes filtered and manually grouped datasets, selected source images, and user-study construction for text- and image-guided translation. Evaluation uses mammal-face and landscape categories plus randomly selected single-image conditions.
- Dataset construction: The Animals dataset is filtered to 14 mammal classes, with 100–300 images per class and four testing source images per class.
- Dataset construction: The Landscapes dataset is manually divided into seven classes, generally with 300 images per class, except for the desert class with 100 images.
- Image-guided translation: Single-image-guided experiments use random AFHQ animal faces and random images from the Landscapes datasets.
- User studies: The text-guided user study generates 130 images from 13 text conditions, randomly selects 65, and evaluates text-match, realism, and content.
- User studies: The image-guided user study generates 40 images from eight image conditions and evaluates style-match, realism, and content.
- Procedure: Algorithm 1 provides a detailed explanation of the proposed image-translation procedure.
B QUANTITATIVE ABLATION STUDY
The ablation study evaluates the proposed losses and acceleration strategies across quantitative metrics and related translation settings. Results favor the complete ViT-based configuration while showing that alternatives can weaken content preservation or generation quality.
- Removing acceleration strategies degrades FID because outputs are not properly changed from the original source images.
- Removing content regularization lowers SFID and CSFID while severely increasing LPIPS, indicating poor reflection of source-image content.
- Using VGG features instead of ViT produces decent SFID and CSFID but very high LPIPS, damaging structural consistency.
- The best setting obtains the best overall output across the reported ablation scores.
- The method supports artistic style transfer with either text or image conditions and face translation with target-domain semantic changes.
- Semantic segmentation comparisons show high source-output structural similarity while the proposed model also applies the target semantic change.
G ADDITIONAL COMPARISON ON IMAGE-GUIDED TRANSLATION
Additional image-guided comparisons examine baseline conditioning and pixel-wise l2 regularization. ViT-based losses do not make SANet training work properly, while weak l2 loss improves target-color matching.
- Training SANet with only the proposed ViT-based style and content losses does not work properly, so existing VGG losses are retained for fair comparison.
- SANet trained with combined VGG- and ViT-based losses still produces incomplete semantic transfer results.
- Removing pixel-wise l2 preserves target semantics but slightly misaligns output colors with the target image.
- Weak l2 loss helps accurately apply the target image’s color to translated outputs.
H LIMITATION AND FUTURE WORK
The method has two stated limitations: translation can fail when source and target semantics are very distant, and image quality depends heavily on the pretrained score model.
- Extremely large semantic distances, such as building →Tiger, can prevent proper translation.
- Image generation quality depends heavily on the performance of the pretrained score model.
- The authors propose better text-to-image embedding and diffusion backbones as future directions.
I ADDITIONAL RESULTS
Additional qualitative results show semantic changes in natural images under both text-guided and image-guided translation conditions.
- Figure 13 presents additional image translation outputs generated using text conditions.
- Figure 14 presents additional results from image-guided image translation.
- The method successfully changes the semantics of various natural images with text and image conditions.