PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual Denoising Diffusion Models

Jiawei Liu, Qiang Wang, Huijie Fan, Yinong Wang, Yandong Tang, Liangqiong Qu

arXiv:2308.13712v3cs.CVcs.LG

TL;DR

Image-restoration diffusion methods condition on degraded images while leaving the original denoising process uninterpretable for restoration. RDDM separates directional residual diffusion from noise diffusion, and its experiments report state-of-the-art performance on four restoration tasks in no more than five sampling steps.

  • Problem

    Denoising-based restoration conditions reverse generation on degraded images without modifying the original process, whose forward diffusion omits information about the known degraded image.

    Source: §1 · p. 1

  • Method

    RDDM decouples diffusion into residuals directed from target to conditional input and random noise, using the two components for restoration and generation.

    Source: §1 · p. 1, §4 · p. 3

  • Results

    RDDM achieves state-of-the-art performance on four image-restoration tasks in no more than five sampling steps.

    Source: §7 · p. 8

  • Takeaways & Limitations

    The authors present RDDM as a potential unified methodology for image-to-image distribution transformation.

    Source: §7 · p. 8

    Excessive disturbances can cause visual inconsistencies, so the reported partial path independence applies only to disturbances within a certain range.

    Source: §5.2 · p. 7

Abstract

from arXiv · show

We propose residual denoising diffusion models (RDDM), a novel dual diffusion process that decouples the traditional single denoising diffusion process into residual diffusion and noise diffusion. This dual diffusion framework expands the denoising-based diffusion models, initially uninterpretable for image restoration, into a unified and interpretable model for both image generation and restoration by introducing residuals. Specifically, our residual diffusion represents directional diffusion from the target image to the degraded input image and explicitly guides the reverse generation process for image restoration, while noise diffusion represents random perturbations in the diffusion process. The residual prioritizes certainty, while the noise emphasizes diversity, enabling RDDM to effectively unify tasks with varying certainty or diversity requirements, such as image generation and restoration. We demonstrate that our sampling process is consistent with that of DDPM and DDIM through coefficient transformation, and propose a partially path-independent generation process to better understand the reverse process. Notably, our RDDM enables a generic UNet, trained with only an L1 loss and a batch size of 1, to compete with state-of-the-art image restoration methods. We provide code and pre-trained models to encourage further exploration, application, and development of our innovative framework (https://github.com/nachifur/RDDM).

1. Introduction

The introduction identifies that restoration methods condition denoising on degraded images without changing a forward process that omits information about those known inputs. RDDM separates directional residual diffusion from random noise diffusion to make restoration interpretable while supporting tasks with different certainty and diversity needs.

1. Introduction

  • Unlike DDPM’s process, which omits the degraded image, RDDM represents diffusion from the target image toward the known input.The introduction frames this omission as making the forward process uninterpretable for restoration and the reverse start from noise unnecessary.Source: §1 · p. 1, Figure 1 · p. 1
  • RDDM separates directional residual diffusion from random noise diffusion, making the restoration direction explicit and supporting tasks with differing certainty or diversity needs.Residuals diffuse from the target toward the conditional input and guide reverse generation; noise represents random perturbations.Source: §1 · p. 1
  • The forward process diffuses residuals and noise simultaneously, while reverse sampling can remove residuals first and noise afterward.RDDM uses two independent coefficient schedules to control their diffusion speeds; changing schedules within a certain range or removing residuals first can preserve semantically consistent generation.Source: §1 · p. 2, Figure 2 · p. 2
  • RDDM’s sampling is consistent with DDPM and DDIM through coefficient transformation, and a generic UNet trained with only L1 loss and batch size 1 can compete with state-of-the-art restoration methods.The introduction also describes a partially path-independent generation process that decouples residuals and noise.Source: §1 · p. 2
  • Experiments demonstrate adaptation to generation, restoration, inpainting, and translation across tasks focused on certainty or diversity and using paired or unpaired data..Source: §1 · p. 2

2. Related Work

Prior restoration approaches often condition denoising on degraded images without modifying the original diffusion process. RDDM instead estimates residuals while retaining noise, extending related mixture-based approaches to independent dual diffusion.

2. Related Work

  • DDPM- and DDIM-based restoration methods [14] [39] [48] feed degraded images to a denoising network as conditional input and typically require large sampling steps and batch sizes.SDEdit [41], ColdDiffusion [2], InDI [11], and I2SB instead generate clear images directly from degraded images or noise-carrying degraded images.Source: §2 · p. 2
  • InDI [11] and I2SB use three-term mixtures but estimate the target or its linear transformation instead of noise, whereas RDDM estimates residuals and also uses noise.The passage characterizes those estimation choices as special cases of RDDM’s SM-Res.Source: §2 · p. 2
  • RDDM extends DDPM [17], DDIM [51], InDI [11], and I2SB to independent double diffusion processes, assigning residuals certainty and noise diversity.The passage also relates the work to coefficient schedules, variance strategies, image decomposition, curve integration, stochastic differential equations, and residual learning.Source: §2 · p. 2

3. Background

The background describes denoising diffusion as a forward process from target images toward Gaussian noise and a learned reverse process. It also introduces the image notation used to frame RDDM’s noisy degraded input for restoration and pure-noise input for generation.

3. Background

  • The paper uses I instead of x to describe diffusion from an image perspective.Source: §3 · p. 2
  • RDDM represents the terminal state as I_T = I_in + ϵ: setting I_in to zero gives pure noise for generation, while restoration uses a noise-carrying degraded image.Its forward process combines degradation toward I_in with injected noise.Source: Figure 3 · p. 3
  • The forward DDPM process progressively diffuses target images through a fixed Markov chain toward Gaussian noise, while the reverse process starts from noise and learns transfers between successive states.The reverse transfer probability has a fixed variance schedule, and a simplified training loss is derived from the maximum likelihood of the target-image distribution.Source: §3 · p. 3
  • A forward step mixes the preceding image with Gaussian perturbation, and reparameterization permits sampling a state directly from the target image.The passage gives the forward-step and direct-sampling forms in terms of α_t and cumulative ᾱ_t.Source: §3 · p. 3

4. Residual Denoising Diffusion Models

RDDM separates image diffusion into residual and noise processes to unify image generation and restoration. Its forward and reverse processes use these components to support task-specific sampling and prediction.

4. Residual Denoising Diffusion Models

  • RDDM sets the terminal state to I_in + noise for restoration and to 0 + noise for generation, framing the forward process as residual diffusion plus noise diffusion.Residual diffusion moves the target toward the degraded input; noise diffusion adds random perturbations.Source: §4 · p. 3

4.1. Directional Residual Diffusion Process with Perturbation

  • The forward step adds a residual shift and noise, with independent coefficient schedules controlling residual and noise diffusion.The residual is I_in − I_0; at the final restoration step, noise intensity is controlled by the cumulative noise coefficient.Source: §4.1 · pp. 3–4

4.2. Generation Process and Training Objective

  • The reverse process estimates the residual and injected noise; training can use separate losses to predict either component or both.The training input can be synthesized from the target, residual, and noise, or equivalently from the degraded input, residual, and noise.Source: §4.2 · p. 4

4.3. Sampling Method Selection Strategies

  • RDDM offers residual-prediction, noise-prediction, and joint residual-and-noise sampling, with η controlling random versus deterministic sampling.For an unknown task, the authors also describe automatic objective selection using one network and fewer than 1000 additional training iterations.Source: §4.3 · p. 4, §4.2 · p. 4
  • PSNR 11.34 for shadow and 16.30 for low-light under SM-N; SM-Res performs better on restoration, while SM-N yields better generation FID and IS.The authors recommend SM-N for tasks requiring diversity and SM-Res for tasks requiring greater certainty.Source: §4.3 · p. 4, Table 1 · p. 5
  • Residual and noise estimates have independent status; removing noise leaves residual-only interpolation, which degenerates to Rectified Flow [35].Coefficient transformation makes SM-N consistent with DDPM [17] and DDIM [51].Source: §4.3 · p. 5

5. Decoupled Dual Diffusion Framework

RDDM decouples residual and noise diffusion, giving each an independent schedule and a distinct role in generation. Its modified reverse process is partially path-independent, although excessive schedule disturbances can produce visual inconsistencies.

5.1. Decoupled Forward Diffusion Process

  • Independent schedules control residual diffusion speed and noise diffusion speed, allowing them to be adjusted separately.The reverse-process coefficients likewise correspond to the speeds of removing residuals and noise.Source: §5.1 · p. 5

5.2. Partially Path-independent Generation Process

  • The proposed schedule adjustment converts DDIM coefficients to RDDM, then changes residual diffusion speed without altering the noise schedule.The adjustment uses a normalized power function for the residual schedule.Source: §5.2 · p. 6
  • The authors use separate networks to estimate residuals and noise, with separate cumulative schedules for their time conditions.The residual and noise estimators are conditioned using ᾱt and β̄t, respectively.Source: §5.2 · p. 6
  • Modifying diffusion-speed curves can produce noise-free images with little semantic variation, but excessive disturbances cause visual inconsistencies.The authors call the property partially path-independent because it holds only for disturbances within a certain range.Source: Figure 6 · p. 7, §5.2 · p. 7
  • Removing residuals controls semantic transitions, whereas removing noise first significantly reduces diversity.The figure contrasts removing residuals before noise with the reverse order.Source: §5.2 · p. 7, Figure 6 · p. 7

6. Experiments

Across the reported restoration tasks, RDDM is competitive with state-of-the-art methods using a generic UNet and a batch size of 1. The experiments also report near-preserved generation quality after DDIM-to-RDDM conversion and visual examples of inpainting and translation.

Image Generation

  • Converted DDIM and RDDM models produce generated-image quality that is nearly the same.The comparison uses a pre-trained DDIM model converted by coefficient transformation.Source: §6 · p. 7

Image Restoration

  • RDDM is competitive with state-of-the-art restoration methods across several tasks using an identical UNet and batch size 1.The evaluation covers shadow removal, low-light enhancement, deraining, and deblurring on five datasets; the conclusion reports no more than five sampling steps.Source: §6 · pp. 7–8

Applications

  • The paper presents visual examples of image inpainting and image translation, alongside generation and restoration applications.The examples include center and irregular masks; translation is described as translating semantics before generating details.Source: §6 · p. 8, Figure 7 · p. 8

7. Conclusions

The paper presents RDDM as a unified dual-diffusion model and reports state-of-the-art results on four restoration tasks in no more than five sampling steps. Its appendices derive perturbed and deterministic sampling procedures and show how coefficient transformation aligns sampling with DDPM and DDIM.

7. Conclusions

  • RDDM reports state-of-the-art performance on four image-restoration tasks in no more than five sampling steps.The conclusion also describes the model as a three-term mixture framework beyond the prior two-term denoising-diffusion framework.Source: §7 · p. 8
  • The conclusion presents RDDM as a potential unified methodology for image-to-image distribution transformation.The stated scope includes image generation and restoration.Source: §7 · p. 8

A.1. Perturbed Generation Process

  • The perturbed-generation derivation simplifies the objective to squared-error prediction losses for residuals or noise.The residual and noise forms are shown separately, with λres and λϵ weighting their respective objectives.Source: §A.1 · p. 12

A.2. Deterministic Implicit Sampling

  • The deterministic implicit-sampling update is stated to be consistent with the paper’s sampling equation.The derivation identifies ϵt as Gaussian noise and connects Eq. 41 to Eq. 12.Source: §A.2 · p. 13

A.3. Coefficient Transformation

  • Coefficient transformation makes the RDDM sampling process consistent with DDPM and DDIM.The derivation relates the transformed update to DDIM and adjusts coefficient and variance schedules.Source: §A.3 · p. 14

A.4. Perturbed Generation Process with Sum-constrained Variance

RDDM uses a sum-constrained variance strategy for its perturbed generation process. Its η=0 sampling matches DDIM, while η=1 differs from DDPM because their variance schedules differ.

A.4. Perturbed Generation Process with Sum-constrained Variance

  • With consistent coefficient schedules, RDDM (η = 0) matches DDIM (η = 0), but RDDM (η = 1) differs from DDPM (η = 1).The difference at η = 1 is attributed to their different variance schedules; the paper says RDDM has the same total noise at different sampling steps, whereas DDPM’s total noise varies.Source: Figure 8 · p. 15, §A.4 · p. 16
  • RDDM (η = 1) has sum-constrained variance, unlike DDPM’s preserving variance.The paper states that RDDM’s variance is much smaller than DDPM’s.Source: §A.4 · p. 15
  • RDDM’s η = 1 variance is bounded by the sum of the β_t^2 terms, supporting its sum-constrained variance strategy.The stated bound is no greater than the corresponding sum of β_t^2 terms, itself bounded by 1.Source: §A.4 · pp. 14–15

A.5. Comparison With Other Methods

RDDM combines residual and noise diffusion to make restoration interpretable while retaining noise for diverse generation. Its independent schedules distinguish it from approaches that use only one target representation or a single diffusion process.

A.5. Comparison With Other Methods

  • Unlike methods that choose noise, residuals, target images, or a linear transformation term, RDDM predicts residuals while also retaining noise for generation and restoration.The paper identifies these respective choices in Shadow Diffusion, SR3, WeatherDiffusion; DvSR and Rectified Flow; ColdDiffusion and InDI; and I2SB.Source: §A.5 · p. 16
  • RDDM combines input images, target images, and noise, with residuals prioritizing certainty and noise emphasizing diversity.It preserves the DDPM generation framework through coefficient and variance transformation, allowing transfer of DDPM improvements such as IDDPM’s variance optimization.Source: §A.5 · p. 16
  • RDDM uses residuals for directional diffusion from target images to input images, avoiding task-specific degradation operators used by ColdDiffusion.The paper contrasts RDDM’s residual learning with ColdDiffusion’s task-specific transforms such as blur for deblurring and snowification for snow removal.Source: §A.5 · p. 16
  • RDDM simultaneously diffuses residuals and noise, unlike DvSR’s residual construction from a predicted clean image and noise-based diffusion.RDDM’s forward process progressively maps the target to pure noise or to a noise-carrying input image.Source: §A.5 · p. 16
  • RDDM’s independent schedules for residuals and noise extend the three-term forward-process form shared with InDI and I2SB.The paper says coefficient transformation makes the forms consistent, while RDDM’s independent schedules support a partially path-independent process with stable generation across varying diffusion rates.Source: §A.5 · p. 16

B.1. Training Details

The experiments use a consistent RDDM setup across restoration tasks and describe an automatic procedure for selecting residual or noise prediction. That procedure transitions from joint training to one prediction objective, with task-dependent outcomes and no added inference cost.

B.1. Training Details

  • For image generation, the coefficient schedules are converted for DDIM comparison, and results are evaluated with FID and IS.The paper also describes alternative schedules for RDDM training.Source: §B.1 · p. 17
  • Restoration evaluation covers shadow removal, low-light enhancement, deraining, and deblurring across five datasets, using task-specific conditions and sampling steps.The setup uses 5 steps for shadow removal and deraining, and 2 steps for low-light enhancement and deblurring; quantitative metrics include MAE, PSNR, SSIM, and LPIPS.Source: §B.1 · p. 17
  • RDDM uses the same UNet architecture and batch size 1 across restoration tasks, while many state-of-the-art methods use specialized architectures or losses.The cited examples include multi-stage, multi-branch, and GAN architectures, plus chromaticity, texture-similarity, and edge losses.Source: §B.1 · p. 17

B.2. SAMPLING Details

  • AOSA begins with simultaneous residual and noise estimation, weights both losses by a learned probability, then selects one prediction objective.The learned λ_res represents the probability that the network output is residual; the transition can occur when its tendency exceeds 51%.Source: §B.2 · pp. 18–19
  • AOSA switches to residual learning for shadow removal and low-light enhancement at about 300 iterations, and to denoising for generation at about 1000 iterations.The authors report the same inference cost as current denoising-based diffusion methods.Source: §B.2 · p. 19
  • SM-Res-N can use two separate prediction networks or one joint network, with the joint design sacrificing decoupling for dual prediction.On shadow removal, the one-network joint model reports MAE 4.57 versus 4.67 for two independent networks; a shared-encoder design reports MAE 4.72.Source: Table 5 · p. 18, §B.2 · p. 19

B.3. More Results

The reported restoration results include strong low-light and deraining comparisons, alongside competitive deblurring and additional qualitative examples. The partially path-independent process is robust to some path changes, but excessive disturbances and diffusion-speed changes can affect visual quality.

B.3. More Results

  • 4.8% PSNR and 34.2% SSIM improvement over SNR-Aware on SID-RGB are reported for RDDM.The accompanying comparison describes sharper text and more vibrant color, with less gray shading and detail blurring.Source: Table 8 · p. 21, §B.3 · pp. 22–23
  • RDDM achieves the best SSIM and LPIPS among compared methods on the LOL low-light dataset and is reported to recover better visual quality.The comparison is limited to RGB low-light datasets rather than RAW datasets such as FiveK.Source: §B.3 · p. 21
  • RDDM outperforms RainDiff128 on RainDrop while using 5 sampling steps instead of 50.RainDiff128 generates a clear image from noise; RDDM recovers from a noise-carrying input image.Source: §B.3 · p. 23
  • On GoPro deblurring, RDDM ranks second-best in LPIPS and is visually competitive with state-of-the-art methods such as Uformer-B.The paper also provides additional qualitative inpainting and image-translation examples.Source: Table 9 · p. 23, §B.3 · p. 24

B.4. Partially Path-independent Generation Process

  • The partially path-independent process remains stable across a certain range of diffusion rates and path variation, but excessive disturbances can produce visual inconsistencies.The paper also reports that generation quality changes with diffusion speed even when paths are the same.Source: §B.4 · p. 24
  • Removing residuals before noise generates faces similar to the default path, whereas removing noise first fails to reach the target in the generation setting.The reported failure is attributed to I_in = 0 for image generation.Source: §B.4 · p. 24

B.5. Ablation Studies

The ablations show that sampling choices trade restoration fidelity and certainty against generation diversity. They also examine sampling-step counts, noise perturbation, resource use, and robustness to changes in noise.

Sampling Methods

  • For shadow removal, noise-only prediction distorts colors and struggles to retain input information; more sampling steps do not improve it, while residual-only prediction avoids these problems with batch size 1.The authors suggest batch size 1 may contribute to noise-only results; for inpainting, SM-Res-N-2Net produces more realistic faces than SM-N and SM-Res.Source: §B.5 · p. 25

Sampling Steps

  • On shadow removal, RDDM performance improves with more sampling steps, with best MAE at 5 steps; this differs from InDI’s best MSE reconstruction at one step.The authors attribute the difference possibly to coefficient schedules and sampling methods.Source: §B.5 · p. 26, Table 10 · p. 26

Certainty and diversity

  • Conditional inputs increase generated-image certainty but reduce diversity; omitting them increases diversity but changes non-missing regions.The comparison is shown for image inpainting on CelebA-HQ.Source: §B.5 · p. 26, Figure 19 · p. 26

Noise Perturbation Intensity

  • Reducing the test-time noise variance from 1 to 0.01 significantly decreases generated-image diversity, while noise may aid restoration detail and perceptual quality.The variance is changed at test time by variance transformation; the generation comparison visualizes this decrease.Source: §B.5 · pp. 26–27

Sampling Methods

  • With noise variance changed during sampling, SM-Res-N-2Net can still generate meaningful faces, whereas SM-N fails; the authors find the deresidual network relatively robust to noise variations compared with the denoising network.The sampling comparison uses 10 steps.Source: §B.5 · p. 27

Resource efficiency

  • On ISTD shadow removal, RDDM’s residual-only model uses 10x fewer training iterations, parameters, and inference time than SR3, with 10% improvement in PSNR and SSIM; training requires 4.8G of GPU memory.The authors say residual prediction can increase certainty and accelerate convergence; they also report effectiveness in low-light enhancement and deraining.Source: §B.5 · pp. 27–29, Table 11 · p. 28

C. Discussions, Limitations, and Further Work

The authors frame RDDM as a unified prototype for image restoration and generation, while noting that it may fall short of task-specific state-of-the-art performance. They outline avenues for improving task performance and extending the framework.

Limitations

  • As a unified prototype for restoration and generation, RDDM may have performance limitations compared with task-specific state-of-the-art methods.The authors say further investigation is needed to attain state-of-the-art image-generation performance.Source: C. Discussions, Limitations, and… · p. 29
  • For stronger task-specific performance, the authors suggest larger UNets, larger batches, more iterations, and training strategies such as task-specific learning-rate adjustments.These are proposed avenues for exploration, not reported results.Source: C. Discussions, Limitations, and… · p. 29

Further Work

  • Proposed extensions include adaptive schedules to reduce sampling steps, one pretrained model for multiple tasks, and adaptive noise intensity for unknown tasks.Other directions include links to curve or multivariate integration, multimodal latent diffusion, and residuals in distillation.Source: C. Discussions, Limitations, and… · p. 29

Related papers

Loading 2308.13712v3…