PaperlayerView exact revision on arXiv

Source-linked AI summary

PGDiff: Guiding Diffusion Models for Versatile Face Restoration via Partial Guidance

Peiqing Yang, Shangchen Zhou, Qingyi Tao, Chen Change Loy

arXiv:2309.10810v1cs.CV

TL;DR

Existing diffusion-prior restoration methods rely on degradation models, which can be hard to specify for complex real-world degradations. PGDiff instead guides pretrained diffusion using desired high-quality image properties and combines guidance for composite tasks. The paper reports improvements over diffusion-prior-based approaches and favorable performance relative to task-specific methods.

  • Problem

    Existing diffusion-prior restoration methods rely on degradation-process knowledge, while real-world mixtures of degradations can be difficult to model accurately.

    Source: §1 · pp. 1–2

  • Method

    PGDiff guides reverse diffusion with classifiers for desired high-quality image properties and combines task guidance for composite restoration.

    Source: §1 · p. 2

  • Results

    PGDiff outperforms diffusion-prior-based approaches and competes favorably with task-specific models across challenging restoration tasks.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The paper concludes that accessible high-quality image properties can guide restoration without knowing the exact degradation process, including for composite tasks.

    Source: §6 · p. 9

Abstract

from arXiv · show

Exploiting pre-trained diffusion models for restoration has recently become a favored alternative to the traditional task-specific training approach. Previous works have achieved noteworthy success by limiting the solution space using explicit degradation models. However, these methods often fall short when faced with complex degradations as they generally cannot be precisely modeled. In this paper, we propose PGDiff by introducing partial guidance, a fresh perspective that is more adaptable to real-world degradations compared to existing works. Rather than specifically defining the degradation process, our approach models the desired properties, such as image structure and color statistics of high-quality images, and applies this guidance during the reverse diffusion process. These properties are readily available and make no assumptions about the degradation process. When combined with a diffusion prior, this partial guidance can deliver appealing results across a range of restoration tasks. Additionally, PGDiff can be extended to handle composite tasks by consolidating multiple high-quality image properties, achieved by integrating the guidance from respective tasks. Experimental results demonstrate that our method not only outperforms existing diffusion-prior-based approaches but also competes favorably with task-specific models.

1 Introduction

Prior diffusion-guidance methods rely on knowing the degradation process, limiting their generalization to complex real-world degradations. PGDiff instead guides denoising with desired high-quality image properties and combines task-specific guidance for composite restoration.

1 Introduction

  • Figure 1 uses face colorization to illustrate the inference pipeline and identifies blind restoration, inpainting, and old-photo restoration as additional tasks.The figure directs readers to Table 1 for other tasks' properties, classifiers, and targets.Source: Figure 1 · p. 2
  • Real-world mixtures of degradations are difficult to model accurately, while existing methods often limit evaluation to simplified cases such as fixed-kernel downsampling.Closed-form solutions generally exist only for special cases such as linear operators, and loss-based methods require differentiable degradations.Source: §1 · pp. 1–2
  • PGDiff models desired high-quality image properties rather than the degradation process, using diffusion as a natural-image regularizer.The proposed guidance uses accessible properties such as image structure and color-channel statistics, without requiring the exact decolorization process.Source: §1 · p. 2
  • PGDiff applies classifier guidance by updating intermediate outputs using gradients from the loss between classifier outputs and target properties.Each property corresponds to a classifier; composite tasks can combine guidance from respective tasks as a weighted sum, as in old-photo restoration.Source: §1 · p. 2
  • The paper reports that PGDiff outperforms previous state-of-the-art methods across challenging tasks, including blind face and old-photo restoration.It also reports reference-based restoration as an application.Source: §1 · p. 2

2 Related Work

Related work spans GAN- and VQVAE-based generative priors and newer diffusion-prior restoration methods. The latter adapt pretrained models through guidance tied to explicit degradation models, whereas PGDiff guides desired high-quality image properties.

Generative Prior for Restoration

  • Pretrained GANs, GAN inversion, encoder-decoder architectures, and VQVAE are established generative-prior approaches to image restoration.VQVAE approaches generally train with a reconstruction objective and then fine-tune for the restoration task.Source: §2 · p. 3

Diffusion Prior

  • DDRM, DDNM, and GDP adapt pretrained diffusion models without task-specific training, but rely on fixed linear matrices or parameterized degradation models.PGDiff instead models desired high-quality image properties, avoiding the need to specify the degradation process.Source: §2 · p. 3

3 Methodology

PGDiff retains a pre-trained diffusion prior but guides reverse denoising through desired high-quality image properties rather than a degradation model. It uses property classifiers, dynamically adjustable guidance, and combined properties for composite tasks.

3.1 Preliminary

  • The forward process adds Gaussian noise, while reverse diffusion iteratively denoises from noise toward the data distribution.The reverse transition is approximated by a Gaussian whose mean is parameterized by a denoising network.Source: §3.1 · p. 3

3.2 Partial Guidance

  • PGDiff guides reverse diffusion with classifiers for desired high-quality image properties, using gradients from the property loss to update intermediate outputs.Property targets can come from the input image or a denoised intermediate output.Source: §3.2 · p. 4, Table 1 · p. 4, §3.1 · p. 4
  • A dynamic guidance scale varies with intermediate-image changes, addressing the limits of a constant scale in steering outputs toward target values.The scheme also allows multiple gradient steps per denoising step; setting the count to one recovers conventional classifier guidance.Source: §3.2 · p. 5
  • Users can increase a property's guidance strength by taking multiple gradient steps at each denoising step, improving control over each property.A single gradient step recovers conventional classifier guidance.Source: §3.2 · p. 5
  • Composite tasks combine classifiers and sum property-specific losses, extending partial guidance to complex degradations that are difficult to model accurately.Additional perceptual or adversarial losses can also be incorporated for further quality improvement.Source: §3.2 · p. 5

4 Applications

Across restoration tasks, PGDiff selects guidance for desired image properties rather than assuming a known degradation process. The applications combine these properties with a diffusion prior to address blind restoration, colorization, inpainting, old-photo restoration, reference-based restoration, and quality enhancement.

4.1 Blind Face Restoration

  • On LFW-Test, WebPhoto-Test, and WIDER-Test, PGDiff removes degradations and restores facial details, performing favorably over task-specific methods without extensive task training.GDP and DDNM produce notable artifacts on real-world degradations.Source: §4.1 · p. 6

4.2 Face Colorization

  • PGDiff guides lightness toward the input grayscale structure and color toward selected channel statistics, enabling faithful colorization and diverse styles from different reference statistics.A selected CelebA-HQ subset supplies average statistics for a universal tone.Source: §4.2 · p. 6

4.3 Face Inpainting

  • For inpainting, PGDiff preserves the input in unmasked regions and relies on diffusion to synthesize masked details, producing pleasant details coherent with visible regions.The reported CelebRef-HQ comparisons describe artifacts or unnatural outputs from GPEN, GDP, CodeFormer, and DDNM.Source: §4.3 · p. 7

4.4 Old Photo Restoration

  • For old-photo restoration, PGDiff combines restoration, colorization, and inpainting guidance; it produces sharp, colorized faces without scratches or artifacts where compared methods struggle.DDNM can complete scratches given a proper scratch map, but does not produce a high-quality face restoration result.Source: §4.4 · pp. 7–8, Figure 5 · p. 8

4.5 Reference-Based Restoration

  • Adding identity loss from a same-identity reference restores personal details such as wrinkles and eye color, and can recover fine details missed by CodeFormer alone.The reference identity features are extracted with a pretrained face-recognition network such as ArcFace.Source: §4.5 · p. 8, Figure 6 · p. 8

4.6 Quality Enhancement

  • In blind face restoration, perceptual and adversarial losses as guidance can boost performance with higher fidelity and photo-realistic details.The reported comparison uses CodeFormer with identity guidance for fine-detail reconstruction and quality-enhancement loss for fidelity.Source: §4.6 · pp. 8–9, Figure 7 · p. 9

5 Ablation Studies

The ablations test dynamic guidance weighting and multiple gradient steps. Both are reported to improve guidance toward the intended target, with distinct benefits for target similarity and detail.

5 Ablation Studies

  • Dynamic guidance weight is tested against conventional classifier guidance in face inpainting, where unmasked regions should remain similar to the input.Without the dynamic weight, plausible masked content is still generated, but unmasked-region similarity and sharpness decrease.Source: §5 · p. 9
  • The dynamic guidance weight is identified as key to maintaining high similarity to the target during guidance.Replacing constant s with snorm preserves unmasked regions in the face-inpainting comparison.Source: §5 · p. 9
  • Multiple gradient steps remove artifacts and generate finer details as the number of guidance steps increases in blind restoration.The comparison uses N = 1, 2, and 3; the authors note the benefit particularly for noisy intermediate outputs early in denoising.Source: §5 · p. 9

6 Conclusion

The conclusion presents PGDiff as a way to guide restoration through high-quality image properties rather than a modeled degradation process. The appendix describes additional implementation details, results, limitations, and societal-impact discussion.

6 Conclusion

  • PGDiff models easily accessible high-quality image properties and uses the diffusion prior to reconstruct faithful outputs without knowing the exact degradation process.The conclusion says this approach addresses the reliance of existing diffusion-prior restoration methods on degradation knowledge.Source: §6 · p. 9
  • Experiments show PGDiff outperforms diffusion-prior-based approaches on homogeneous and composite tasks and matches task-specific methods.The conclusion also states that partial guidance extends to composite tasks by aggregating multiple properties.Source: §6 · p. 9

Appendix

  • The supplementary material includes further implementation details and results, with separate sections discussing limitations and potential negative societal impacts.Its additional results cover blind-face, old-photo, reference-based, colorization, and inpainting tasks.Source: Appendix · p. 13

A Implementation Details

The appendix specifies inference resources and hyperparameter categories, then details the blind-face restorer and its quality–speed trade-off. The restorer that also receives the current intermediate image and timestep yields sharper samples than the alternative.

A.1 Inference Requirements

  • The blind-face inference model is a 512 × 512 denoising network trained on FFHQ, and inference uses an NVIDIA RTX A5000 GPU.The pretrained model is provided by [45].Source: §A.1 · p. 13

A.2 Inference Hyperparameters

  • Inference hyperparameters fall into sampling, partial-guidance, and optional quality-enhancement categories, with default choices provided for each homogeneous task.Table 2 identifies the defaults; examples include gradient scale s, property weight α, and the range for multiple gradient steps.Source: §A.2 · p. 13, Table 2 · p. 13

A.3 Restorer Design

  • The restorer is trained on FFHQ and CelebA-HQ using synthesized low-quality pairs and MSE loss to obtain smooth semantics without unnecessary high-frequency detail.Training uses four NVIDIA Tesla V100 GPUs for 200K iterations.Source: §A.3 · p. 13
  • For blind face restoration, f(y0, xt, t) produces sharper samples than f(y0), while f(y0) can provide higher inference speed.The restorer concatenates the low-quality input, intermediate image, and timestep; most computation is performed at 128 × 128 resolution after pixel unshuffling.Source: §A.3 · pp. 13–14, Figure 9 · p. 14

B.1 More Results on Blind Face Restoration

On synthetic and real-world blind-face evaluations, PGDiff is reported among the best performers across several metrics and datasets. The additional results describe old-photo restoration and identity-preserving reference-based restoration.

B.1 More Results on Blind Face Restoration

  • On synthetic CelebRef-HQ, PGDiff methods achieve best or second-best scores across FID, MUSIQ, LPIPS, and IDS.The comparison covers no-reference image-quality metrics and full-reference identity-preservation metrics.Source: §B.1 · p. 14, Table 5 · p. 15
  • Reference-based restoration achieves the highest IDS, increasing by 0.32, and a same-person high-quality reference helps recover personal characteristics similar to ground truth.The authors note that identity-preserving prediction is almost impossible from severely corrupted inputs without additional identity information.Source: §B.1 · p. 15
  • On real-world LFW-Test, WebPhoto-Test, and WIDER-Test, PGDiff achieves best or second-best scores across both FID and NIQE.GFP-GAN has the best NIQE scores across datasets, but the authors report notable artifacts; PGDiff outputs are described as robust and artifact-free.Source: §B.1 · p. 15, Figure 10 · p. 16

B.2 More Results on Old Photo Restoration

  • On challenging old-photo cases with and without scratches, PGDiff shows an advantage over GFP-GAN (v1) and DDNM in vibrant colors, fine details, and sharpness.The examples draw from CelebChild-Test and Internet images.Source: §B.2 · p. 17, Figure 11 · p. 17

B.3 More Results on Reference-Based Restoration

  • In reference-based restoration on CelebRef-HQ, adding identity loss yields personal characteristics similar to the ground truth.The dataset contains 1,005 entities, with 3 to 21 high-quality images per person.Source: §B.3 · p. 17

B.4 More Results on Face Inpainting

PGDiff is compared with diffusion-prior and task-specific restoration methods for face inpainting. It recovers challenging structures and produces diverse outputs, with the highest scores across the reported inpainting metrics.

B.4 More Results on Face Inpainting

  • PGDiff achieves the highest scores across FID, NIQE, and MUSIQ-KonIQ, compared quantitatively with DDNM and CodeFormer.The authors use these non-reference metrics because they consider diverse results important, rather than evaluating only similarity to ground truth.Source: §B.4 · p. 17
  • PGDiff recovers challenging structures such as glasses, while different random seeds produce diverse, photo-realistic outputs.Source: §B.4 · p. 17
  • On challenging cases, PGDiff produces natural outputs with details coherent to unmasked regions.Different random seeds also give varied high-quality contents.Source: Figure 13 · p. 19

B.5 More Results on Face Colorization

PGDiff is compared with diffusion-prior and task-specific methods for face colorization. It produces vibrant, detailed, diverse colorizations and achieves the best scores across almost all reported metrics, with a stated NIQE exception.

B.5 More Results on Face Colorization

  • PGDiff achieves the best scores across almost all metrics against DDNM and CodeFormer, although CodeFormer scores better on NIQE.For colorization, the authors use MUSIQ-AVA, which emphasizes aesthetic assessment.Source: §B.5 · p. 20
  • PGDiff produces more vibrant colors and finer details than DDNM and CodeFormer, with diverse outputs guided by varied color statistics.Source: §B.5 · p. 20
  • The authors report that CodeFormer alters the input identity and requires separate task-specific models, whereas PGDiff uses one pre-trained diffusion model for inpainting and colorization.Source: §B.5 · p. 20
  • The figure describes PGDiff as producing diverse colorized outputs with varied color statistics supplied as guidance.Source: Figure 14 · p. 20

C Limitations

The paper reports promising natural-image inpainting and colorization results, while warning that restoration can produce inaccurate or misused outputs.

C Limitations

  • PGDiff is extended to natural-image inpainting and colorization using an unconditional 256 × 256 diffusion model trained on ImageNet, with promising results reported.Source: Figure 15 · p. 21

D Broader Impacts

  • Heavily corrupted inputs may yield inaccurate outputs, potentially including incorrect identity recognition.Source: D Broader Impacts · p. 21
  • The authors warn that malicious users could use the restoration method for data falsification and advise care in its use.Source: D Broader Impacts · p. 21

Related papers

Loading 2309.10810v1…