Source-linked AI summary
Structure-Preserving Super Resolution with Gradient Guidance
Cheng Ma, Yongming Rao, Yean Cheng, Ce Chen, Jiwen Lu, Jie Zhou
TL;DR
Perceptual-driven super-resolution methods can produce structural distortions because single-image super-resolution is ill-posed and pixel-wise optimization may blur sharp structures. SPSR uses gradient recovery and gradient-space supervision to preserve geometry while retaining perceptual quality, achieving strong perceptual metrics and reduced structural distortions.
Problem
Single-image super-resolution is ill-posed, and image-space optimization can produce averaged results that fail to recover sharp structures.
Method
SPSR combines a gradient branch that recovers high-resolution gradients with a gradient loss imposing second-order restrictions on super-resolved images.
Results
SPSR achieves the best PI and LPIPS performance, with second-best PSNR and SSIM values in most datasets, while reducing structural distortions.
Takeaways & Limitations
Gradient guidance enables SPSR to obtain excellent perceptual quality while preserving geometric structures and limiting distortions.
Abstract
from arXiv · showhide
Structures matter in single image super resolution (SISR). Recent studies benefiting from generative adversarial network (GAN) have promoted the development of SISR by recovering photo-realistic images. However, there are always undesired structural distortions in the recovered images. In this paper, we propose a structure-preserving super resolution method to alleviate the above issue while maintaining the merits of GAN-based methods to generate perceptual-pleasant details. Specifically, we exploit gradient maps of images to guide the recovery in two aspects. On the one hand, we restore high-resolution gradient maps by a gradient branch to provide additional structure priors for the SR process. On the other hand, we propose a gradient loss which imposes a second-order restriction on the super-resolved images. Along with the previous image-space loss functions, the gradient-space objectives help generative networks concentrate more on geometric structures. Moreover, our method is model-agnostic, which can be potentially used for off-the-shelf SR networks. Experimental results show that we achieve the best PI and LPIPS performance and meanwhile comparable PSNR and SSIM compared with state-of-the-art perceptual-driven SR methods. Visual results demonstrate our superiority in restoring structures while generating natural SR images.
1. Introduction
SISR recovers high-resolution images from low-resolution inputs, but its ill-posedness creates a tension between blurry PSNR-oriented outputs and structurally distorted perceptual results. SPSR uses gradient guidance to reduce these distortions while preserving perceptual quality.
- SISR recovers high-resolution images from low-resolution counterparts and supports image-analysis tasks including surveillance and satellite imagery.
- Each low-resolution input may correspond to multiple high-resolution solutions, making SISR an ill-posed problem.
- MSE optimization can produce statistical-average reconstructions that are blurry despite achieving high PSNR.
- GAN-based methods generate sharper, photo-realistic results but commonly introduce geometric distortions in edges and textures.
- SPSR restores high-resolution gradients through an auxiliary branch and applies gradient loss to preserve geometric structures during SR.
- Experimental results show that SPSR enhances SR fidelity by reducing structural distortions.
2. Related Work
Prior SISR research spans PSNR-oriented methods, perceptual-driven methods, and gradient-relevant approaches. SPSR differs by using gradient maps specifically as structural guidance for GAN-based SR, targeting geometric distortions rather than only high-frequency recovery.
- SISR methods are commonly classified as PSNR-oriented or perceptual-driven, with related work also examining gradient information.
- PSNR-oriented methods generally optimize pixel-based objectives such as MSE or L1, often producing blurry images.
- Perceptual-driven methods use perceptual, adversarial, or texture losses to improve visual quality and generate photo-realistic textures.
- Earlier gradient-relevant SR methods use edge statistics, gradient profiles, gradient fields, dictionaries, or edge detectors to recover sharpness and details.
- SPSR uses two branches, with the gradient branch super-resolving gradient maps and feeding structural guidance back into the SR process.
- Unlike prior gradient methods aimed mainly at high-frequency recovery for PSNR-oriented SR, SPSR targets geometric distortions in GAN-based SR.
3. Approach
The approach combines a structure-preserving SR branch with a gradient branch and gradient-space objectives, supplementing image-space supervision to reduce geometric distortions while retaining perceptual quality.
- 3.1. Overview: The generator contains a structure-preserving SR branch and a gradient branch that estimates HR gradient maps and guides SR reconstruction.The SR branch uses gradient guidance, while the gradient branch translates LR gradient maps to the HR modality.
- 3.2.1 Gradient Branch: Gradient maps encode local sharpness through gradient intensity, enabling image-to-image translation from LR edge sharpness to HR edge sharpness.Gradient direction is omitted because intensity is considered sufficient to reveal local sharpness.
- 3.2.1 Gradient Branch: Intermediate SR features support gradient-map estimation, whose recovered features are fused back into the SR branch to preserve structures and regulate sharpness.The gradient magnitude indicates whether a recovered region should be sharp or smooth.
- 3.2.2 Structure-Preserving SR Branch: The SR branch combines a regular generative SR network with gradient features through a fusion block, adding only one RRDB block beyond the original 23-block model.The two features are concatenated and processed by another RRDB block and convolutional layer to reconstruct final SR features.
- 3.3. Objective Functions: Image-space-only optimization can produce smooth edges or geometric distortions, whereas second-order gradient constraints encourage attention to neighboring configurations and structural sharpness.The method retains conventional pixelwise, perceptual, and adversarial losses while adding gradient-space supervision.
- 3.3. Objective Functions: Gradient loss supplements image-space losses by penalizing differences between SR and HR gradient maps through pixelwise and adversarial gradient objectives.The gradient discriminator distinguishes whether a gradient patch comes from the HR gradient map, and differentiable gradient extraction allows end-to-end training.
4. Experiments
Experiments evaluate SPSR on five benchmark datasets using perceptual and distortion metrics, component ablations, visual comparisons, and gradient analyses. Results show improved perceptual quality and structure preservation while retaining competitive distortion metrics.
- Implementation Details: Experiments use DIV2K for training, five benchmark datasets for testing, 4× bicubic degradation, and PI, LPIPS, PSNR, and SSIM evaluation.The training setup uses an ESRGAN backbone and RRDB-based gradient block.
- Quantitative Comparison: SPSR achieves the best PI and LPIPS values across all testing datasets, while obtaining second-best PSNR and SSIM values in most datasets.NatSR has higher PSNR and SSIM but substantially worse PI and LPIPS performance.
- Qualitative Comparison: Visual comparisons show that SPSR restores clearer structures and better textures than perceptual-driven methods while producing natural, realistic images.The reported examples include properly recovered brick edges and less severe structural distortions.
- Ablation Study: SPSR surpasses ESRGAN on every measurement across all testing sets, demonstrating the combined effectiveness of gradient loss and gradient-branch guidance.The ablation compares ESRGAN, SPSR without the gradient branch, SPSR without gradient loss, and complete SPSR.
- Effects of the Gradient Branch: The gradient branch recovers thin, structure-pleasing HR gradient maps from thicker LR gradients and supplies explicit structural information to the SR branch.Gradient maps identify regions requiring greater attention to sharpness and structure.
- Effects of the Gradient Branch: Removing the gradient branch produces blurrier furs and whiskers, whereas the complete model recovers sharper edges and more detailed textures.This comparison supports the branch’s contribution to perceptual fidelity.
5. Conclusion
The paper concludes that SPSR preserves geometric structures in perceptual-driven super-resolution through gradient guidance. Its experiments across five benchmark sets support the method’s effectiveness.
- Conclusion: SPSR addresses geometric distortions in perceptual-driven SR by combining a gradient branch with a gradient loss.The gradient branch recovers HR gradient maps as explicit guidance, while the gradient loss imposes second-order restrictions on recovered images.
- Conclusion: Quantitative and qualitative experiments on five popular benchmark testing sets demonstrate the effectiveness of the proposed method.
- Conclusion: Joint image-space and gradient-space supervision better captures geometric relationships in super-resolved images.
A. User Study
A user study compares SPSR with three GAN-based SR methods through randomized ranking of perceptual quality. The study collects judgments from 43 raters, totaling 1290 votes.
- User Study: The user study asks raters to rank SPSR, ESRGAN, NatSR, and SRGAN according to perceptual quality against HR references.The four SR versions are presented in randomized sequence.
- User Study: 43 human raters provide 1290 votes in the subjective evaluation of SR visual performance.
B. More Qualitative Results
Additional qualitative results compare SPSR with state-of-the-art SR methods and visualize its gradient outputs. These results report improved structural and photorealistic reconstruction, alongside successful LR-to-HR gradient conversion.
- More Qualitative Results: Across additional comparisons, SPSR performs better than competing SR methods in recovering structurally pleasing and photo-realistic images.The comparisons include EnhanceNet, SFTGAN, SRGAN, ESRGAN, and NatSR.
- More Qualitative Results: Gradient-branch visualizations show successful conversion of LR gradient maps into HR gradient maps.
- More Qualitative Results: The user-study visualization reports that SPSR outperforms state-of-the-art SR methods in generating high-quality images.