Source-linked AI summary
Decoupled Residual Denoising Diffusion Models for Unified and Data Efficient Image-to-Image Translation
Ziyue Lin, Jiahe Hou, Hongyu Xia, Xinrui Xie, Feifei Wang, Yuyin Zhou, Wei Wang, Jiawei Liu, Liangqiong Qu
TL;DR
Unified I2I translation is challenged by domain gaps and limited paired data, while coupled diffusion erodes the harmonization benefit of injected noise. DRDD decouples noise diffusion from residual diffusion, preserving harmonization during mapping and using unpaired target images for denoising. The paper reports robust unified translation across diverse tasks, including under limited paired data, with compatibility across mainstream diffusion paradigms.
Problem
Unified I2I translation faces substantial domain gaps and difficulty collecting large-scale paired source-to-target images, while coupled diffusion progressively erodes noise-induced harmonization.
Method
DRDD uses sequential stochastic noise diffusion for domain harmonization and manifold lifting, deterministic residual diffusion in a fixed-noise domain, and subsequent denoising refinement.
Results
DRDD delivers robust performance across diverse unified I2I tasks, especially with limited paired data, and is compatible with DDPM, DDIM, and SDE-based diffusion models.
Takeaways & Limitations
Gaussian noise functions as a domain harmonizer, while decoupling residual and noise removal supports simpler unified mappings and denoising trained on abundant unpaired target-domain images.
Takeaways & Limitations
The described decoupled forward process assumes paired input and target images for the residual definition.
Abstract
from arXiv · showhide
We propose Decoupled Residual Denoising Diffusion models (DRDD) for unified and data-efficient image-to-image (I2I) translation. While diffusion models have advanced I2I translation in terms of quality and diversity, we uncover a previously under-explored property in diffusion models. Crucially, beyond its conventional role of manifold lifting (i.e., moving data off low-dimensional manifolds), injecting Gaussian noise facilitates domain harmonization by implicitly aligning feature distributions across domains, a property particularly advantageous for unified I2I translation. However, existing diffusion models prematurely erode this harmonization effect, as noise and residuals are simultaneously removed in a single coupled diffusion process. To address this, DRDD decouples the diffusion process into two sequential and independent diffusion stages: (1) a stochastic noise diffusion for domain harmonization and manifold lifting, and (2) a deterministic residual diffusion that learns the core semantic mapping entirely within the fixed-noise domain. This decoupling preserves harmonization and manifold lifting effects throughout the transformation, substantially simplifying the learning of unified mappings across diverse tasks and domains. Notably, the noise diffusion stage is trained exclusively on abundant, unpaired target-domain images, greatly improving data efficiency. Comprehensive theoretical and empirical analysis demonstrates that DRDD is broadly compatible with mainstream diffusion models and consistently delivers robust, unified I2I translation, even under limited paired data. Our code is available at https://github.com/HKU-HealthAI/DRDD.
1. Introduction
The paper identifies Gaussian noise as a domain harmonizer that can reduce representation gaps, then proposes DRDD to preserve this effect while learning unified, data-efficient I2I mappings.
- Motivation: Unified I2I translation remains difficult because tasks have substantial domain gaps and large-scale paired source-to-target data are difficult to collect.These challenges motivate re-examining diffusion models for unified and data-efficient translation.
- Noise as a domain harmonizer: Gaussian noise can bridge representation gaps by implicitly aligning feature distributions across disparate domains, beyond moving data off low-dimensional manifolds.The paper formalizes this under-explored function as noise acting as a “domain harmonizer.”
- Limitation of coupled diffusion: Coupled diffusion models progressively remove injected noise during reverse diffusion, eroding harmonization before source-to-target mapping is complete.They simultaneously remove residuals and noise in one reverse process.
- DRDD: DRDD separates diffusion into stochastic noise diffusion for harmonization and manifold lifting, followed by deterministic residual diffusion within a fixed-noise domain.A final denoising stage refines fidelity after the core source-to-target transformation.
- Advantages: The decoupled design preserves harmonization during mapping, simplifies unified learning, and trains denoising exclusively on unpaired target-domain images for improved data efficiency.The framework is reported as compatible with DDPM, DDIM, and SDE-based diffusion models and robust under limited paired data.
2. Related Works
Diffusion models have become prominent for I2I translation after earlier GAN-based approaches, offering strong output quality and diversity while addressing prior limitations in stability and mode coverage.
- I2I translation: I2I translation transfers images between source and target domains while preserving content representations, supporting applications including restoration, super-resolution, and style translation.The field spans a wide range of computer-vision tasks.
- Diffusion-based methods: Diffusion models learn reversible processes that iteratively add and remove noise, achieving strong output quality and diversity in I2I tasks.SR3 pioneered conditioning diffusion sampling on the input image for super-resolution.
3. Method
DRDD reinterprets Gaussian noise as a domain harmonizer and decouples diffusion into noise and residual stages so semantic mapping occurs before denoising. The resulting reverse process preserves harmonization while separating residual removal from fidelity refinement.
- Role of Noise: Gaussian noise can reduce feature-representation distances across domains, extending diffusion noise beyond manifold lifting and score-estimation signal enrichment.The paper characterizes fixed-level Gaussian noise as a domain harmonizer for unified I2I tasks.
- Role of Noise: Injecting nonzero Gaussian noise into two distributions yields a lower KL divergence between their noisy versions than between the original distributions.This is stated as Proposition 3.1: D_KL(P_σ || Q_σ) < D_KL(P || Q).
- Decoupled Forward Process: DRDD performs stochastic noise diffusion first, then deterministic residual diffusion within the resulting fixed-noise domain.The forward process transforms a target image into a noise-carrying state before injecting the source-target residual.
- Motivation: Conventional coupled reverse diffusion removes injected noise together with residuals, eroding harmonization before source-to-target mapping is complete.The paper identifies this simultaneous removal as a difficulty for unified, data-efficient I2I translation.
- Decoupled Reverse Process: The reverse process first removes residuals in the noise-carrying domain and then denoises to recover a clean target image.This ordering preserves domain harmonization and manifold lifting during core mapping and is intended to simplify unified image-mapping learning.
9 end
The training procedure separates denoising from paired source-domain supervision. This design supports greater data efficiency and compatibility with several diffusion formulations.
- Training: DRDD trains the denoising network solely on clean images without corresponding source-domain images, significantly enhancing data efficiency.The denoising network can also be initialized from weights pretrained on large-scale natural-image datasets.
4. Experiments
Experiments evaluate DRDD across unified multi-task restoration, multi-domain single-task translation, standard single-task translation, limited paired data, and diffusion-framework extensions. Across these settings, DRDD generally achieves strong or state-of-the-art performance while retaining quality with less training data and extending to SDE-based diffusion.
- Evaluation scope: Experiments cover multi-task restoration, multi-domain denoising, single-domain translation, data pruning, and compatibility with SDE-based diffusion.The evaluation uses All-in-One-5, CDD-11, MNMD, All-in-One-3, Low-Light, and task-specific datasets.
- Unified restoration: DRDD achieves state-of-the-art performance on most All-in-One-5 restoration tasks and consistently outperforms diffusion and non-diffusion baselines across all three reported metrics.The reported metrics are SSIM, LPIPS, and FID; visual comparisons show richer details and fewer artifacts.
- Unified restoration: On CDD-11’s 11 degradation types, DRDD achieves the highest average SSIM and remains advantageous in challenging composite degradations.The cited examples include L+H+S and L+H+R, where several comparison methods experience notable performance drops.
- Multi-domain translation: Across natural, medical, and remote-sensing domains in MNMD, DRDD achieves the highest SSIM and lowest LPIPS for a single multi-domain denoising task.The benchmark adds different noise types to images from WED+BSD400, UC-Merced, and BrainWeb.
- Single-task translation: DRDD also achieves superior performance on single-task translations including inpainting, while visual evaluations include low-light enhancement, blur restoration, face inpainting, and super-resolution.Inpainting comparisons use CelebA-HQ under center and irregular masks; the visual results span LoLV1, GoPro, CelebA-HQ, and FFHQ.
- Data efficiency and extensions: With training data reduced to 75%, 50%, and 25%, DRDD’s relative performance drop remains substantially smaller than other methods, and decoupling also improves or preserves performance in SDE-based diffusion.The limited-data study uses LowLight and All-in-One-3; the SDE extension outperforms the baseline on deraining and inpainting and achieves comparable denoising with better FID.
5. Conclusion
DRDD is a diffusion model that separates noise diffusion from residual diffusion for unified image translation. The framework treats Gaussian noise as a domain harmonizer and enables denoising to use unpaired images.
- DRDD decouples standard diffusion into sequential noise diffusion and residual diffusion stages.
- Gaussian noise acts as a domain harmonizer by reducing feature-distribution distances across domains.
- The decoupled mechanism simplifies learning a unified mapping across tasks and permits denoising training exclusively on unpaired images.
- Theoretical and empirical analyses demonstrate DRDD’s effectiveness for unified image translation systems.
A.1. Proofs of Proposition 3.1
The proof establishes that adding nonzero Gaussian noise reduces the KL divergence between two distinct distributions. The appendix also derives reverse-sampling and training objectives for DRDD’s stages and extends the decoupling paradigm to SDE-based diffusion.
- Proof of Proposition 3.1: For distinct distributions P and Q, injecting Gaussian noise with σ ≠ 0 yields D_KL(P_σ ∥ Q_σ) < D_KL(P ∥ Q).
- Proof of Proposition 3.1: The proof uses a Gaussian kernel and joint-distribution KL decomposition to relate noisy and original divergences.
- Reverse sampling: The residual-removal reverse step subtracts a predicted residual scaled by α_t from the current residual-stage image.
- Reverse sampling: The denoising reverse process predicts noise, while deterministic generation uses η = 0.
- Training objectives: Training minimizes L1 error between predicted and target residuals or between predicted and target noise for the respective stages.
- Decoupled SDE: The decoupled forward process can be formulated in SDEs by adding noise in a first stage before degradation-specific information.
B. Experiment Settings and Dataset
The experiments use standardized image-restoration training settings, multiple unified and single-task benchmarks, and both distortion-based and perceptual evaluation metrics. The datasets cover noise, haze, rain, low light, blur, composite degradation, and multiple image domains.
- Experiment Settings: Experiments use NVIDIA A6000 GPUs, L2 loss, flipping augmentation, 256 × 256 random crops, batch size 8, learning rate 8e-5, and 300,000 steps.
- Metrics: Evaluation combines PSNR and SSIM for distortion with LPIPS and FID for perceptual similarity.
- Datasets: The benchmark suite includes unified settings such as All-in-One-3 and All-in-One-5, covering combinations of noise, haze, rain, low light, and blur.
- Datasets: The datasets include 72,135 dehazing training pairs, 200 Rain100L deraining pairs, and 5,144 clear images for denoising.
- Datasets: GoPro provides 2,111 blurry-sharp training pairs and 1,111 evaluation pairs for motion deblurring.
- Datasets: CDD-11 contains 20,790 training pairs and 2,310 testing pairs spanning 11 composite degradation conditions.
- Datasets: MNMD evaluates denoising across natural, remote-sensing, and medical image domains with Gaussian, salt-and-pepper, and Poisson noise.
C.1. Implementation Details of Methods in Experiments
The comparison experiments train DRDD and competing unified restoration methods under reported task-specific settings. Results are organized across denoising, inpainting, image-to-image translation, and implementation comparisons.
- DRDD: DRDD’s limited-data experiment initializes the denoising U-Net from pretrained parameters.
- Comparison methods: The compared methods include RDDM, DiffuIR, AdAIR, DA-CLIP, DFPIR, IR-SDE, De-IRSDE, and VLUNET.
- Results organization: Reported tables compare single-task deraining and denoising using SSIM and LPIPS.
- Results organization: Inpainting comparisons cover center and irregular 256×256 settings and center 64×64 resolution.
- Results organization: RDDM and DRDD are compared on Edges2Handbags and Edges2Shoes.
C.2. More Single I2I Tasks in Single Domain.
DRDD is evaluated across restoration, inpainting, style transfer, super-resolution, and ablation settings, with strong performance across diverse tasks and data regimes. Experiments also examine framework compatibility, isolated-dataset training, inference steps, computational resources, and unseen distributions.
- Image Inpainting: DRDD achieves the best or second-best results across inpainting configurations at 64×64 and 256×256 resolutions.The evaluation uses center and irregular masks on CelebA-HQ.
- Edges to Objects: DRDD significantly outperforms RDDM on edges2handbags and edges2shoes style transfer, while injected noise does not adversely affect performance.Results are reported in Tab. 7 and Fig. 8.
- Sampling and Efficiency: There is no significant difference between 2-step and 10-step DDIM inference on All-in-One-3.The paper also reports computational comparisons using two steps for each sub-module.
- Unified Restoration: DRDD consistently achieves the best FID and LPIPS among methods, and the best PSNR and SSIM among diffusion-based approaches on All-in-One-5.It remains highly competitive with non-diffusion methods on PSNR and SSIM.
D. More Visual Comparisons
Additional visual comparisons present DRDD outputs across restoration, inpainting, and image-to-image translation tasks. The figures cover haze restoration, noise restoration, super-resolution, and irregular-mask inpainting.
- Visual Comparisons: Figure 9 and Figure 10 provide supplementary visual examples of DRDD across restoration and inpainting tasks.These examples extend the visualizations presented in the main paper.
- Visual Comparisons: Figure 9 compares haze restoration on SOTS, noise restoration on CBSD68, and super-resolution on FFHQ.The figure provides visual results for three restoration tasks.
- Visual Comparisons: Figure 10 presents irregular-mask inpainting results for state-of-the-art methods and DRDD.The caption recommends zooming in for the best view.