PaperlayerView exact revision on arXiv

Source-linked AI summary

TSD-SR: One-Step Diffusion with Target Score Distillation for Real-World Image Super-Resolution

Linwei Dong, Qingnan Fan, Yihong Guo, Zhonghao Wang, Qi Zhang, Jinwei Chen, Yawei Luo, Changqing Zou

arXiv:2411.18263v4cs.CV

TL;DR

Real-ISR with pre-trained diffusion models is computationally expensive, while existing one-step distillation methods have limitations in restoration and detail recovery. TSD-SR combines Target Score Distillation and distribution-aware sampling in a one-step model, reporting strong restoration and the fastest inference among diffusion-based approaches. The authors identify full-reference metric assessment as a limitation for evaluating advanced Real-ISR.

  • Problem

    VSD-based one-step Real-ISR can have unreliable gradient directions for suboptimal initial outputs and insufficient fine-detail recovery under uniform timestep sampling.

    Source: §1 · p. 2

  • Method

    TSD-SR distills pre-trained text-to-image diffusion models into a one-step Real-ISR model using high-quality references for Target Score Distillation and distribution-aware sampling for detail recovery.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 4

  • Results

    TSD-SR achieves the best performance on most metrics and the fastest inference speed, running 40 times faster than SeeSR.

    Source: §1 · p. 2

  • Takeaways & Limitations

    TSD-SR outperforms existing one-step Real-ISR models in restoration quality and inference speed while using a single inference step.

    Source: §5 · p. 9

    The authors anticipate better full-reference metrics to assess advanced Real-ISR methods.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Pre-trained text-to-image diffusion models are increasingly applied to real-world image super-resolution (Real-ISR) task. Given the iterative refinement nature of diffusion models, most existing approaches are computationally expensive. While methods such as SinSR and OSEDiff have emerged to condense inference steps via distillation, their performance in image restoration or details recovery is not satisfied. To address this, we propose TSD-SR, a novel distillation framework specifically designed for real-world image super-resolution, aiming to construct an efficient and effective one-step model. We first introduce the Target Score Distillation, which leverages the priors of diffusion models and real image references to achieve more realistic image restoration. Secondly, we propose a Distribution-Aware Sampling Module to make detail-oriented gradients more readily accessible, addressing the challenge of recovering fine details. Extensive experiments demonstrate that our TSD-SR has superior restoration results (most of the metrics perform the best) and the fastest inference speed (e.g. 40 times faster than SeeSR) compared to the past Real-ISR approaches based on pre-trained diffusion priors.

1. Introduction

TSD-SR targets the unreliable guidance and detail-recovery challenges of one-step Real-ISR with Target Score Distillation and Distribution-Aware Sampling. Experiments report strong restoration and the fastest inference among diffusion-based approaches.

1. Introduction

  • VSD-based Real-ISR faces unreliable gradient directions when initial outputs are suboptimal and insufficient fine-detail recovery under uniform timestep sampling.Diffusion models' iterative denoising also makes Real-ISR computationally expensive.Source: §1 · pp. 1–2
  • TSD uses high-quality image data to provide a reliable optimization trajectory, while DASM samples distribution-based low-noise inputs to improve fine-detail recovery.The proposed framework distills multi-step text-to-image diffusion models into a one-step Real-ISR model.Source: §1 · p. 2
  • TSD-SR achieves the best performance on most metrics and runs 40 times faster than SeeSR, with one-step inference.The comparison benchmarks Real-ISR methods on DRealSR using an A100 GPU.Source: Figure 1 · p. 1, §1 · p. 2

2. Related Work

Prior Real-ISR work uses GANs or pre-trained diffusion models, with diffusion approaches pursuing both multi-step restoration and one-step distillation. The cited approaches trade off detail quality, optimization reliability, or inference speed.

2. Related Work

  • GAN-based methods can add realistic details, but training instability and mode collapse remain challenges.BSRGAN and Real-ESRGAN simulate complex degradations for restoration under unknown conditions [63].Source: §2 · p. 2
  • Multi-step diffusion Real-ISR methods use pre-trained image priors, but their 20–50 inference iterations lag far behind GAN-based inference speed.Examples include StableSR [45], DiffBiR [29], SeeSR, PASD [57], and SUPIR.Source: §2 · p. 2
  • SinSR distills ResShift to one step, AddSR uses four-step adversarial diffusion distillation, and OSEDiff distills a pre-trained text-to-image model to one step [49] [62] [55] [53].The cited approaches note constraints including limited training data, excessive unnatural details, and potentially unreliable optimization directions [49] [55] [53].Source: §2 · pp. 2–3
  • The TSD-SR pipeline trains a one-step student with pixel-space reconstruction and Target Score Distillation losses, then updates its LoRA model with diffusion loss.The teacher and LoRA model process DASM-sampled noisy latents to produce the score losses.Source: Figure 2 · p. 3

3. Methodology

TSD-SR distills a pre-trained text-to-image diffusion model into a one-step Real-ISR student, combining target-score supervision with trajectory-aware sampling. Its training also uses paired reconstruction losses and a trainable LoRA model.

3.1. Preliminaries

  • The Real-ISR objective combines reconstruction with a realism- and generalization-oriented regularizer that aligns the student output distribution with the HQ data distribution.The reconstruction loss may use L2 or LPIPS, while prior work described here uses adversarial loss or VSD for regularization.Source: §3.1 · p. 3

3.2. Overview of TSD-SR

  • The student is distilled from a pre-trained text-to-image diffusion model using a Teacher, a trainable LoRA replica, reconstruction loss, and TSD regularization.The student and HQ latents are sampled through DASM; after the student update, the LoRA model is updated with diffusion loss.Source: §3.2 · p. 4

3.3. Target Score Distillation

  • TSM addresses unreliable synthetic-latent gradients by matching Teacher predictions on synthetic and HQ latents, then blending this signal with VSD in a time-weighted TSD loss.The paper reports cosine similarity of 0.2 for synthetic latent ẑ_t versus 0.88 for HQ latent z_t; it contrasts the proposed approach with MSE, which produces over-smoothed results.Source: §3.3 · pp. 4–5

3.4. Distribution-Aware Sampling Module

  • DASM accumulates gradients from earlier-timestep samples along diffusion trajectories to focus optimization on detail recovery and reduce Stage 2 prediction divergence.The paper attributes Stage 2 divergence to missing high-frequency detail in synthetic latents and reports that trajectory-based sampling reduces this divergence.Source: §3.4 · p. 5

3.5. Training Objective

  • Student training combines pixel-space LPIPS, latent-space MSE, and TSD regularization, while LoRA training uses a diffusion loss with targets defined for DDPM or flow matching.The reconstruction-loss weight γ1 ramps from 1 to 2 during optimization while γ2 remains at its initial value of 1.Source: §3.5 · pp. 5–6

4. Experiments

Experiments compare TSD-SR with single- and multi-step Real-ISR methods, assess user preferences, and ablate its components. Results cover restoration quality, inference efficiency, and detail recovery across synthetic and real-world benchmarks.

4.1. Experimental Settings

  • Experiments train on DIV2K, Flickr2K, LSDIR, and 10K FFHQ face images, and evaluate on DIV2K-Val, RealSR, and DRealSR.The synthetic test set contains 3,000 pairs; real-world datasets comprise 128×128 LQ and 512×512 HQ images.Source: §4.1 · p. 6
  • Training first uses latent-space MSE without DASM, then removes MSE and introduces DASM; training took about 96 hours on eight V100 GPUs.The evaluation includes full-reference and no-reference image-quality metrics.Source: §4.1 · p. 6

4.2. Comparison with Existing Methods

  • TSD-SR outperforms other single-step models on the vast majority of key metrics and has advantages over multi-step models on LPIPS, DISTS, and NIQE.It also outperforms most multi-step models in FID, MUSIQ, and CLIPIQA, while other methods lead on some metrics.Source: §4.2 · p. 6, Table 1 · p. 7, Table 2 · p. 7
  • Visual comparisons describe TSD-SR as producing rich textures and realistic details with enhanced sharpness and contrast.The comparisons note over-smoothed textures for SeeSR, excessive details for SUPIR, and difficulty recovering appropriate content for PASD under more realistic degradation.Source: §4.2 · pp. 7–8, Figure 7 · p. 8
  • Over 40× faster than SeeSR, TSD-SR is the fastest among the compared one-step models.Inference comparisons use an A100 GPU and 512×512 inputs, excluding model-weight and data-loading time.Source: §4.2 · p. 8, Table 3 · p. 8

4.3. User Study

  • TSD-SR received 69.2% user approval in a 50-participant study spanning five image categories.Its category approval rates ranged from 57.6% for Animals to 84.4% for Characters.Source: §4.3 · pp. 8–9, Figure 8 · p. 9

4.4. Ablation Study

  • Removing TSM or DASM negatively affected reference-based and no-reference metrics, as well as FID.The study associates TSM removal with declines in LPIPS, DISTS, MUSIQ, and CLIPIQA, and DASM removal with declines in FID, MUSIQ, and CLIPIQA.Source: §4.4 · p. 9, Table 4 · p. 9
  • For DASM, the selected setting is N = 4 and s = 50; overall performance was better for s between 25 and 75.Performance degraded when N was larger or smaller, and larger s values degraded image quality; N = 4 was chosen partly because DASM is computationally expensive.Source: §4.4 · p. 9, Table 6 · p. 9

5. Conclusion and Limitation

TSD-SR combines Target Score Distillation and a Distribution-Aware Sampling Module in a one-step Real-ISR model. It improves restoration quality and speed over existing one-step methods, but remains substantially larger than GAN- or non-diffusion-based methods.

5. Conclusion and Limitation

  • TSD-SR uses TSD to enhance realism and DASM to sample distribution-based noisy latents and accumulate gradients for detail recovery.The method is a one-step Real-ISR model based on diffusion priors.Source: §5 · p. 9
  • Experiments show TSD-SR outperforms existing one-step Real-ISR models in restoration quality and inference speed.Source: §5 · p. 9
  • TSD-SR has significantly more parameters than previous GAN- or non-diffusion-based methods; the authors propose pruning or quantization as future compression directions.Source: §5 · p. 9

Supplementary Material

The supplementary material adds comparisons, visualizations, metric analyses, theory, and an algorithm to further examine TSD-SR.

Supplementary Material

A. Comparison with GAN-based Methods

The GAN comparison finds stronger PSNR and SSIM for GAN methods, while TSD-SR leads on no-reference metrics and produces more realistic texture details in visual comparisons.

A. Comparison with GAN-based Methods

B. More Visual Comparisons

Figures 10–12 add visual comparisons with diffusion-based methods, illustrating TSD-SR’s restoration quality.

C. Comparisons of Full-reference Metrics and Human Preference

The comparisons caution that PSNR and SSIM do not fully reflect image fidelity under complex degradation. Participants preferred TSD-SR results despite their lower PSNR and SSIM than some GAN-based results.

D. Theory of Target Score Matching

Target Score Matching compares teacher predictions for high-quality and suboptimal inputs drawn from the same distribution. Its residual-based gradient omits the computationally demanding Diffusion Jacobian term.

E. Algorithm

Algorithm 1 describes TSD-SR training, including classifier-free guidance for the Teacher and LoRA Models and diffusion-loss updates.

E. Algorithm

  • Algorithm 1 uses classifier-free guidance for both the Teacher Model and LoRA Model, with cfg weight 7.5.Source: E. Algorithm · p. 13

20 end

  • The listed training step updates θ with LG to compute the LoRA Model’s diffusion loss.Source: §20 · p. 17
  • The listed sampling step draws ϵ from N(0, I) and t from {50, · · · , 950}.Source: §20 · p. 17

Related papers

Loading 2411.18263v4…