PaperlayerView exact revision on arXiv

Source-linked AI summary

GAN-Diff : Coupling Pretrained WGAN-GP Features with Conditional Diffusion U-Nets

Saif Ahmed, Ashadulla Hil Galib, S. M. Riaz Rahman Antu, Ahmed Faizul Haque Dhrubo, Souvik Pramanik, Mohammad Abdul Qayum, Mohsin Sajjad, Mohammad Ashrafuzzaman Khan

arXiv:2608.22272v1cs.CVcs.AIcs.LG

TL;DR

The paper addresses whether a pretrained WGAN-GP can provide an effective feature-level prior for conditional diffusion image restoration. It couples frozen generator features to a diffusion U-Net through cross-attention and reports improvements on CelebA Gaussian denoising and 2× super-resolution over input baselines.

  • Problem

    The paper examines whether a pretrained WGAN-GP can provide an adequate feature-level prior for conditional diffusion-based image restoration.

    Source: §I · p. 1

  • Method

    A frozen WGAN-GP generator supplies intermediate spatial features through cross-attention to a conditional diffusion U-Net during DDIM restoration.

    Source: §I · p. 2

  • Results

    The framework consistently improves degraded and low-resolution CelebA images on Gaussian denoising and 2× super-resolution tasks.

    Source: B. Restoration Quality · p. 5, §VIII · p. 6

  • Takeaways & Limitations

    A frozen GAN feature prior can guide diffusion models toward stable and effective image restoration within the evaluated CelebA tasks.

    Source: §VI · pp. 5–6, §VIII · p. 6

    Evaluation is limited to five tracked faces, lacks final-pipeline perceptual metrics, and uses one fixed Gaussian-noise level for denoising.

    Source: §VII · p. 6

Abstract

from arXiv · show

Generative adversarial networks (GANs) can provide efficient image generation, while diffusion models offer high-quality image restoration but require iterative sampling. This paper presents a hybrid GAN-guided diffusion framework that uses a pretrained Wasserstein GAN with gradient penalty (WGAN-GP) as a feature prior for conditional diffusion-based image restoration. Intermediate features from the frozen WGAN-GP generator are incorporated into a diffusion U-Net through cross-attention and remain fixed during the DDIM sampling process. The framework is evaluated on two restoration tasks, Gaussian denoising and 2Xsuper-resolution, using CelebA face images. During development, several sources of instability were identified and addressed, including adversarial learning-rate imbalance, inappropriate diffusion initialization, excessive corruption, and insufficient parameter averaging. The resulting framework consistently improves the quality of both degraded and low-resolution images. In particular, it improves denoising performance by 4.40 dB in PSNR and super-resolution performance by 3.70 dB over their respective input baselines. These results demonstrate the potential of a frozen GAN feature prior to guide diffusion models toward stable and effective image restoration.

I. INTRODUCTION

The paper combines a frozen pretrained WGAN-GP feature prior with conditional diffusion restoration, using cross-attention to guide Gaussian denoising and 2× super-resolution. It also addresses training and inference instabilities identified during iterative development.

I. INTRODUCTION

  • Development compared DCGAN, WGAN-GP, and StyleGAN-lite, selecting WGAN-GP for stability and quality reasons.Source: §I · p. 1
  • The hybrid model uses intermediate spatial features from a frozen WGAN-GP generator as a feature-level prior for a conditional diffusion U-Net.The features are integrated through cross-attention while preserving the diffusion model’s restoration procedure.Source: §I · p. 1
  • The study diagnoses instability related to generator–critic learning rates, DDIM initialization, corruption strength, and exponential moving-average smoothing.These issues affect both training and inference in the hybrid pipeline.Source: §I · p. 2
  • The framework targets Gaussian image denoising and 2× super-resolution on CelebA face images.Source: §I · p. 2
  • The GAN feature map is calculated once per sampling chain and kept fixed throughout DDIM sampling.This design avoids changing the conditioning signal across reverse-diffusion steps.Source: §I · pp. 1–2

II. RELATED WORK

  • The approach differs from prior GAN–diffusion hybrids by conditioning the diffusion U-Net on an intermediate GAN feature map rather than replacing reverse diffusion with GAN output.Source: §II · p. 2

III. DATASET

The experiments use a 50,000-image CelebA subset prepared at 64 × 64 resolution with normalized channels. Five fixed faces support consistent checkpoint-wise comparisons.

III. DATASET

IV. METHODOLOGY

The methodology uses a sequential pipeline that extracts structural priors from a pretrained generative model and supplies them to conditional diffusion restoration paths.

IV. METHODOLOGY

  • The framework sequentially extracts robust structural priors from a pretrained generative model and uses them in conditional diffusion restoration paths.Source: A. Proposed Methodology · p. 2
  • The WGAN-GP generator maps z ∼ N(0, I128) to 64 × 64 RGB images through five transposed-convolution, BatchNorm, and ReLU blocks.Spatial resolution increases from 4 × 4 to 64 × 64, followed by a tanh output.Source: 1) Phase 1 : WGAN-GP Prior Generator · p. 3
  • The critic uses strided convolutions with spectral normalization, instance normalization, and LeakyReLU activations.Source: 1) Phase 1 : WGAN-GP Prior Generator · p. 3
  • The generator is trained with a WGAN-GP critic objective while an EMA copy is maintained for lower inference-time parameter variance.The generator updates every n_critic = 5 critic updates.Source: 1) Phase 1 : WGAN-GP Prior Generator · p. 3

2) Phase 2 : GAN-Guided Conditional U-Net and Diffusion Optimization:

The diffusion stage conditions U-Net restoration on a fixed WGAN-GP feature map through cross-attention and performs deterministic DDIM inference from task-specific initialization.

2) Phase 2 : GAN-Guided Conditional U-Net and Diffusion Optimization:

B. Experimental Setup

The restoration pipeline was evaluated using stable configurations and optimized hyperparameters, with training settings documented in Table II.

  • The implementation was evaluated across stable configurations and optimized hyperparameters to address previously identified pipeline instabilities.Source: B. Experimental Setup · p. 3
  • Table II lists the training hyperparameters used for the restoration pipeline.Source: Table II · p. 3

C. System Architecture

The system combines a pretrained generator prior with conditional diffusion restoration and evaluates outputs using PSNR and SSIM. GAN-derived features are used as a fixed conditioning signal during sampling.

System Architecture

Evaluation Metrics

  • PSNR measures reconstruction quality through the ratio between maximum image power and corrupting noise power.Source: D. Evaluation Metrics · p. 4
  • MAXI denotes the maximum possible pixel value, while MSE is the mean squared error between clean ground truth and evaluated output.Source: D. Evaluation Metrics · p. 4
  • SSIM evaluates image-quality changes by combining luminance, contrast, and structural terms.Source: D. Evaluation Metrics · p. 4
  • The metrics compare five tracked faces’ clean ground truth with degraded inputs and final diffusion-restored outputs.Source: D. Evaluation Metrics · p. 4

A. Training Dynamics

Training curves indicate stable adversarial and diffusion optimization, while restoration metrics improve across the tracked CelebA faces for both denoising and super-resolution.

Training Dynamics

  • The critic loss increased from −23.99 at epoch 1 to −4.71 at epoch 50, while generator loss decreased from 36.65 to 18.44.Source: A. Training Dynamics · p. 4
  • Denoising U-Net loss decreased from 0.079 to 0.037 in 30 epochs, while super-resolution U-Net loss decreased from 0.060 to 0.014 in 25 epochs.Source: A. Training Dynamics · p. 5
  • Generator validation showed no mode collapse across ten WGAN-GP samples, with sample standard deviation increasing from 0.430 at epoch 10 to 0.476 at epoch 50.Source: A. Training Dynamics · p. 5

Restoration Quality

C. Qualitative Observations

Visual examples show that GAN-guided diffusion removes visible denoising noise while retaining identity features and synthesizes realistic super-resolution details. The discussion attributes consistent restoration to fixed GAN conditioning during DDIM sampling and stabilized adversarial learning rates.

  • The denoising outputs eliminate visible speckle noise while retaining eyes, hair contours, and mouth-region identity features.Source: C. Qualitative Observations · p. 5
  • The super-resolution outputs synthesize realistic high-frequency details, including fine hair strands.Source: C. Qualitative Observations · p. 5
  • The GAN conditioning signal g is computed once before the reverse loop and held constant throughout all 50 DDIM steps.Resampling g at each step would make the trajectory inconsistent by providing contradictory semantic instructions.Source: §VI · pp. 5–6
  • The corrected DDIM initialization and fixed conditioning support the GAN-guided diffusion restoration shown for Gaussian denoising and 2× super-resolution.The figures compare clean references, degraded inputs, and restored outputs for five tracked faces.Source: Figure 8 · p. 6, Figure 9 · p. 6, §VI · p. 6
  • Returning to the symmetric 2 × 10−4/2 × 10−4 critic-generator learning-rate ratio resolves the instability without architectural modification.The paper attributes the instability to an overly fast critic whose gradients can become meaningless or saturated.Source: §VI · p. 6

VII. LIMITATIONS

The evaluation is restricted to five tracked CelebA faces and one Gaussian-noise level, while omitting final LPIPS/FID reporting and cross-attention ablations. The authors identify larger held-out evaluations, perceptual metrics, broader corruption testing, and conditioning ablations as needed follow-up work.

  • Evaluation is limited to five tracked faces, preventing assessment of restoration quality across a larger population.The authors suggest testing hundreds or thousands of faces in a larger test set.Source: §VII · p. 6, §VIII · p. 6
  • The final pipeline reports no LPIPS or FID scores, despite their closer alignment with human perception of face realism than PSNR/SSIM.Source: §VII · p. 6
  • Denoising robustness is tested at only one fixed Gaussian-noise level, σ = 0.15.Source: §VII · p. 6
  • No ablation study isolates the individual effect of GAN cross-attention conditioning.Source: §VII · p. 6

Related papers

Loading 2608.22272v1…