Source-linked AI summary
SinSR: Diffusion-Based Image Super-Resolution in a Single Step
Yufei Wang, Wenhan Yang, Xinyuan Chen, Yaohui Wang, Lanqing Guo, Lap-Pui Chau, Ziwei Liu, Yu Qiao, Alex C. Kot, Bihan Wen
TL;DR
Diffusion-based SR is hindered by the many inference steps required for practical use, even though recent methods shorten the generation chain. SinSR derives a deterministic sampling process and distills its noise-to-HR mapping into a one-step student, while using ground-truth consistency during training; it achieves comparable or better performance than prior SOTA methods and the teacher, with up to ×10 faster inference.
Problem
Diffusion-based SR remains impractical when many inference steps are required; recent accelerations either depend on a precise degradation formulation or still require 15 steps.
Method
SinSR derives deterministic sampling from an accelerated diffusion-based SR teacher, distills bidirectional noise-to-HR mappings into a one-step student, and adds a ground-truth consistency-preserving loss.
Results
SinSR achieves comparable or better performance than previous SOTA methods and the teacher diffusion model while reducing inference steps from 15 to 1, with up to ×10 speedup.
Takeaways & Limitations
Diffusion-based SR can be accelerated to a single sampling step while retaining comparable or superior perceptual performance and diversity.
Takeaways & Limitations
The approach assumes that the student network can effectively capture the teacher's ODE process and that the xT-to-x0 matching is learnable in the SR setting.
Abstract
from arXiv · showhide
While super-resolution (SR) methods based on diffusion models exhibit promising results, their practical application is hindered by the substantial number of required inference steps. Recent methods utilize degraded images in the initial state, thereby shortening the Markov chain. Nevertheless, these solutions either rely on a precise formulation of the degradation process or still necessitate a relatively lengthy generation path (e.g., 15 iterations). To enhance inference speed, we propose a simple yet effective method for achieving single-step SR generation, named SinSR. Specifically, we first derive a deterministic sampling process from the most recent state-of-the-art (SOTA) method for accelerating diffusion-based SR. This allows the mapping between the input random noise and the generated high-resolution image to be obtained in a reduced and acceptable number of inference steps during training. We show that this deterministic mapping can be distilled into a student model that performs SR within only one inference step. Additionally, we propose a novel consistency-preserving loss to simultaneously leverage the ground-truth image during the distillation process, ensuring that the performance of the student model is not solely bound by the feature manifold of the teacher model, resulting in further performance improvement. Extensive experiments conducted on synthetic and real-world datasets demonstrate that the proposed method can achieve comparable or even superior performance compared to both previous SOTA methods and the teacher model, in just one sampling step, resulting in a remarkable up to x10 speedup for inference. Our code will be released at https://github.com/wyf0912/SinSR
1. Introduction
Diffusion-based SR offers strong perceptual quality but remains computationally expensive because existing approaches require many inference steps. SinSR distills a deterministic process into one-step generation while retaining comparable or superior quality and diversity.
- Motivation: Diffusion-based SR methods achieve strong perceptual quality but require substantial computation because their initial state is pure Gaussian noise.The low-resolution image is not used as prior knowledge in the initial state, so many inference steps are needed.
- Limitations of Prior Acceleration: Recent acceleration methods shorten the Markov chain, but one requires a degradation formulation while ResShift still needs 15 steps and develops artifacts with further reduction.These limitations constrain practical deployment of diffusion-based SR.
- SinSR: SinSR directly distills a deterministic mapping from input noise to HR images into a student network that performs SR in one inference step.The deterministic training pairs are derived from a recent SOTA diffusion-based SR method.
- SinSR: A consistency-preserving loss incorporates ground-truth images during distillation to improve perceptual quality beyond simply fitting the teacher mapping.The method is designed to preserve diversity and perceptual quality while reducing inference steps.
- Results: Experiments on synthetic and real-world datasets show comparable or better performance than SOTA methods and the teacher while reducing inference steps from 15 to 1.The reported speedup reaches up to ×10.
2. Related Work
SR research has progressed from regression and generative models toward diffusion models, but diffusion acceleration remains challenging because training and inference can be expensive. Distillation and alternative inference paths address this cost, motivating SinSR's direct deterministic-generation approach.
- Generative SR: Regression-based SR models can model posterior expectations but tend to produce over-smoothed images, motivating generative approaches for better perceptual quality.Autoregressive and diffusion-based models are examples of generative alternatives.
- Acceleration of Diffusion Models: Diffusion-model acceleration includes general sampling algorithms, student-model distillation, progressive distillation, and iterative exploration of improved inference paths.These approaches target reduced generation cost while retaining useful image quality.
- Acceleration of Diffusion Models: Direct diffusion distillation can incur high training overhead because solving the inference-process ODE is expensive on large datasets.Progressive distillation is used to alleviate this burden.
- Acceleration of Diffusion Models: Unlike methods that shorten the Markov chain, the proposed one-step SR directly learns the deterministic generation process.Figure 3 contrasts vanilla diffusion SR, an accelerated method, and the proposed one-step approach.
3. Motivation
Diffusion-based SR models conditional generation as a Markov chain from Gaussian noise to an HR image given an LR image. SinSR is motivated by the observation that this relationship can be converted into a deterministic mapping and distilled into one step.
- Preliminary: Existing diffusion-based SR models the conditional distribution q(x0|y) through a Markov chain starting from Gaussian noise.The diffusion model transfers standard Gaussian noise to the HR image domain conditioned on the LR image.
- Preliminary: Because the matching between xT and x0 is unknown, conventional diffusion models learn the mapping iteratively across noise levels.SinSR instead targets a deterministic mapping between the input noise and generated HR image.
- Motivation: ResShift can be reformulated as a deterministic process, producing a deterministic mapping Fθ(xT, y) from noise and the LR image to the predicted HR image.This mapping is obtained by sampling the reformulated process without retraining the pretrained model.
- Motivation: The mapping between xT and the predicted HR image is reported to be easier to learn than denoising under different noise levels.This observation supports single-step distillation into a student network.
- Motivation: The framework distills the teacher's deterministic mapping in both directions and adds a ground-truth consistency loss during training.The ground-truth image is mapped to a latent code and reconstructed to calculate an MSE loss.
4. Methodology
SinSR derives a deterministic reverse process for ResShift and distills its noise-to-HR mapping into a one-step student, while adding ground-truth consistency regularization.
- Deterministic Sampling: ResShift integrates LR-image information into the diffusion state, initializing inference from xT = y + κ√ηT ϵ rather than pure noise.The LR image y is pre-upsampled to the HR resolution, and diffusion operates in latent space.
- Deterministic Sampling: A non-Markovian reverse process preserves the marginal distributions while reformulating inference conditioned on the LR image.Its coefficients kt, mt, and jt define the reformulated reverse transitions.
- Deterministic Sampling: Sampling this reformulated process yields a deterministic mapping Fθ(xT, y) from the initial state to the teacher’s predicted HR image.The mapping can equivalently be indexed by the input noise ϵ because xT is noise-initialized through y and ϵ.
- Consistency Preserving Distillation: Vanilla distillation trains a student network to predict the teacher mapping directly from xT and y in one step.The student output f̂θ(xT, y, T) is matched to Fθ(xT, y) using the distillation objective.
- Consistency Preserving Distillation: The consistency-preserving loss uses a predicted inversion and the ground-truth HR image to regularize the student beyond teacher-only fitting.The student is reused at timestep 0 and timestep T, and the overall objective jointly minimizes Ldistill, Lreverse, and Lgt with equal weights.
5. Experiment
Experiments evaluate SinSR on real-world and synthetic datasets, efficiency, and design choices. Results indicate that one-step deterministic distillation preserves or improves perceptual performance while reducing inference steps.
- Training Details: 30K fine-tuning iterations suffice for the student model, substantially less than the 500K iterations used to train ResShift from scratch.The student is initialized from the pretrained model and converges quickly despite ODE-solving overhead during training.
- Evaluation on real-world datasets: On RealSR and RealSet65, SinSR outperforms its teacher by a large margin and achieves the best CLIPIQA performance among competitors using one inference step.Visual comparisons on real-world samples also report promising one-step results.
- Evaluation on synthetic datasets: On ImageNet-Test, reducing inference from 15 to 1 slightly decreases PSNR and SSIM, while SinSR achieves the best LPIPS and CLIPIQA performance.The one-step model is reported as on par with or slightly better than the 15-step teacher in perceptual quality.
- Evaluation of the efficiency: SinSR outperforms ResShift in efficiency and performance with one inference step, while ResShift already accelerates inference relative to LDM.Table 3 counts VQ-VAE computational cost and evaluates latent-space methods on the x4 (64→256) task.
- Ablation Studies: Deterministic teacher sampling is necessary: replacing it with ResShift’s stochastic sampling causes significant performance degradation and blurred details.Given the same random noise, stochastic sampling produces an expectation over a conditional distribution rather than a fixed paired target.
- Why does a single-step distillation work?: Directly learning the deterministic mapping between xT and x0 is easier than diffusion-loss denoising, including for a smaller model with 24.3M rather than 118.6M parameters.The denoising-trained model suffers a serious performance drop compared with direct deterministic mapping.
- Is a more sophisticated distillation strategy necessary?: Rectified Flow’s iterative reflow and distillation degrade final performance through accumulated error, so a more sophisticated strategy is not clearly beneficial here.The analysis attributes the limited benefit to the relative learnability of the deterministic SR mapping.
- Learned inversion: The proposed learned inversion provides better fidelity than DDIM inversion, indicating a more accurate estimate of xT.This inversion is used within the consistency-preserving-loss analysis.
6. Conclusion
SinSR accelerates diffusion-based super-resolution to one sampling step through bidirectional distillation of a deterministic noise-to-image mapping, while adding consistency-preserving training with ground-truth images. The resulting student model achieves performance on par with or better than its teacher in one step.
- SinSR distills a teacher diffusion model’s deterministic mapping between input noise and generated high-resolution images into a one-step student model.The method uses derived deterministic sampling to enable bidirectional distillation.
- A consistency-preserving loss lets the student learn from ground-truth images alongside information from the pretrained teacher.
- The student achieves on-par or better performance than the teacher using only one sampling step.