Source-linked AI summary
Toward Real-World Single Image Super-Resolution: A New Benchmark and A New Model
Jianrui Cai, Hui Zeng, Hongwei Yong, Zisheng Cao, Lei Zhang
TL;DR
Existing SISR models often rely on simulated, uniform degradations that do not represent real-world images well. This paper builds a focal-length-based RealSR dataset with aligned LR-HR pairs and proposes LP-KPN for spatially varying degradation. Models trained on RealSR produce better real-world SISR results and generalize beyond the cameras used for dataset collection.
Problem
Simulated SISR datasets use simple, uniform degradations, whereas real-world LR images have more complex authentic degradations.
Method
The paper captures paired LR-HR scenes at different DSLR focal lengths, registers them, and uses LP-KPN to predict efficient per-pixel restoration kernels.
Results
RealSR-trained models produce much better real-world SISR results than models trained on existing simulated datasets and generalize to other camera devices.
Takeaways & Limitations
RealSR provides a practical benchmark for training and evaluating SISR models under authentic degradations.
Takeaways & Limitations
The thin-lens formulation assumes static images captured beyond 3.0m, with object distance much larger than focal length.
Abstract
from arXiv · showhide
Most of the existing learning-based single image superresolution (SISR) methods are trained and evaluated on simulated datasets, where the low-resolution (LR) images are generated by applying a simple and uniform degradation (i.e., bicubic downsampling) to their high-resolution (HR) counterparts. However, the degradations in real-world LR images are far more complicated. As a consequence, the SISR models trained on simulated data become less effective when applied to practical scenarios. In this paper, we build a real-world super-resolution (RealSR) dataset where paired LR-HR images on the same scene are captured by adjusting the focal length of a digital camera. An image registration algorithm is developed to progressively align the image pairs at different resolutions. Considering that the degradation kernels are naturally non-uniform in our dataset, we present a Laplacian pyramid based kernel prediction network (LP-KPN), which efficiently learns per-pixel kernels to recover the HR image. Our extensive experiments demonstrate that SISR models trained on our RealSR dataset deliver better visual quality with sharper edges and finer textures on real-world scenes than those trained on simulated datasets. Though our RealSR dataset is built by using only two cameras (Canon 5D3 and Nikon D810), the trained model generalizes well to other camera devices such as Sony a7II and mobile phones.
1. Introduction
SISR models trained on simple simulated degradations generalize poorly to real-world images, motivating a practical paired benchmark and a model for spatially varying degradation. The paper introduces the RealSR dataset and LP-KPN, with experiments showing stronger real-world performance and cross-camera generalization.
- Simulated SISR datasets typically use simple, uniform bicubic degradation, whereas authentic real-world degradations are more complex.
- RCAN models trained with simulated degradations show no clear advantage over bicubic interpolation on the illustrated real-world image.
- RealSR captures paired LR-HR images of the same scenes at different scales by changing DSLR focal lengths.The dataset includes indoor and outdoor scenes captured with Canon 5D3 and Nikon D810 cameras.
- LP-KPN predicts restoration kernels per pixel to handle spatially variant degradation while improving kernel-prediction efficiency.The approach is motivated by depth-dependent blur and the quadratic cost of larger KPN kernels.
- The dataset addresses the lack of a general-purpose benchmark for training and evaluating real-world SISR models.
- Models trained on RealSR show good generalization to images captured by camera devices beyond the two cameras used to build the dataset.
2. Related Work
Prior SISR work largely relies on simulated degradations, while real-world paired data remain difficult to collect. Related work also includes thin-lens image formation and kernel prediction methods for restoration tasks.
- Common SISR datasets synthesize LR images using bicubic downsampling or Gaussian blurring followed by downsampling.
- Real-world SISR datasets are difficult to construct because obtaining ground-truth HR images for LR observations is challenging.
- Earlier real-world paired-data efforts used a beam splitter for face images or sensor hardware binning with postprocessing.
- Thin-lens image formation: The DSLR imaging system is modeled with a thin lens whose variables include object distance, image distance, focal length, and object and image sizes.
- Kernel prediction networks: Kernel prediction networks estimate restoration or blur kernels for individual pixels and have been applied to denoising, deblurring, and video interpolation.
3. Real-world SISR Dataset
The RealSR dataset captures same-scene images at multiple focal lengths and uses registration to produce aligned HR-LR pairs despite lens-induced differences.
- Dataset construction: Images of the same scene are captured by adjusting DSLR focal lengths, with the largest focal length providing the HR reference.The dataset uses two full-frame cameras and four focal lengths: 105mm, 50mm, 35mm, and 28mm.
- Dataset construction: The dataset records indoor and outdoor textured scenes and contains 595 HR-LR image pairs after removing captures with motion, inappropriate exposure, or blur.The image-pair counts are organized by camera and scaling factor in Table 1.
- Registration challenges: Focal-length changes introduce lens distortion, exposure differences, optical-center shifts, and slightly varying scaling factors, complicating pixel-wise alignment.These effects make direct alignment difficult even when zooming provides images at different scales.
- Dataset construction: 105mm images are cropped as ground-truth HR images, while images captured at shorter focal lengths provide LR counterparts.The LR images are registered from the 50mm, 35mm, or 28mm captures.
- Image pair registration: The registration pipeline corrects lens distortion, crops a central region, and jointly estimates luminance adjustment and an affine transformation.The objective uses a robust Lp-norm, with the affine transform initialized from the ratio of focal lengths.
- Image pair registration: The iterative optimization updates the affine transformation using weighted least squares and converges within five iterations before producing the aligned LR image.The focal-length ratio supplies a good initialization for the scaling transformation.
4. Laplacian Pyramid based Kernel Prediction Network
LP-KPN addresses spatially varying real-world degradation by predicting per-pixel restoration kernels across a Laplacian pyramid. The pyramid preserves neighborhood information while reducing the computational cost of large receptive fields.
- RealSR blur kernels vary with scene depth and focal-length-dependent depth of field, motivating per-pixel kernel prediction instead of direct pixel synthesis.
- Three k×k pyramid kernels provide an effective 4k×4k receptive field at the original resolution, reducing cost compared with directly learning one 4k×4k kernel.
- KPN predicts a k×k kernel for each input pixel and applies it to that pixel’s local neighborhood to produce the HR output.
- Large kernels capture richer neighborhood information but make the predicted kernel tensor grow quadratically with kernel size, increasing memory and computation.
- LP-KPN decomposes the LR luminance channel into a three-level Laplacian pyramid, predicts three kernel tensors, filters corresponding pyramid levels, and reconstructs the HR image.
5. Experiments
Experiments compare models trained on simulated and RealSR data, then evaluate architectures, cross-camera transfer, and performance on outside images. RealSR training and LP-KPN consistently improve real-world reconstruction quality and generalization.
- 5.1. Simulated SISR datasets vs. RealSR dataset: Models trained on simulated bicubic data perform comparably to bicubic interpolation, while multiple simulated degradations yield only marginal improvements on RealSR images.
- 5.1. Simulated SISR datasets vs. RealSR dataset: RealSR-trained models improve average PSNR by about 1.0dB at ×2 across three architectures and outperform simulated-data models at ×3 and ×4.
- 5.1. Simulated SISR datasets vs. RealSR dataset: RealSR-trained models recover clearer, more natural details, whereas simulated-data models produce blurred edges and visible artifacts.
- 5.2. SISR models trained on RealSR dataset: LP-KPN achieves the best performance among RealSR-trained models at all three scaling factors with the lowest computational cost.
- 5.3. Cross-camera evaluation: Cross-camera testing shows only 0.32dB and 0.30dB gaps for RCAN and LP-KPN, respectively, relative to in-camera settings, while both outperform bicubic interpolation.
- 5.4. Generalization to other cameras: On Sony a7II, iPhone X, and Google Pixel 2 images without HR ground truth, LP-KPN produces better visual quality and clearer, more natural details than competitors.
6. Conclusion
The paper constructs a real-world benchmark and proposes an efficient LP-KPN for SISR with authentic degradations. Experiments support improved real-world results and generalization beyond the dataset’s two cameras.
- RealSR contains 595 pixel-wise aligned HR-LR image pairs collected with Canon and Nikon cameras across indoor and outdoor scenes.
- The proposed LP-KPN uses a Laplacian pyramid based kernel prediction strategy for efficient real-world SISR.
- Models trained on RealSR produce better real-world SISR results than models trained on existing simulated datasets and generalize to other cameras.
- Future work will expand RealSR with more image pairs and camera types and investigate new training strategies.
7. Supplementary Material
The supplementary material presents the RealSR dataset, the LP-KPN architecture, and visual comparisons across models, datasets, and camera devices. Results favor RealSR-trained models and identify LP-KPN as the strongest visual performer among the compared models.
- Dataset: The RealSR dataset contains 595 HR-LR image pairs covering indoor, outdoor, and laboratory environments.Sample images illustrate the variety of image contents included in the dataset.
- Network architecture: LP-KPN uses a residual-block architecture with 3 × 3 convolutions, 64 filters in most layers, and shuffle operations for resolution changes.The final layer uses 25 filters, and the architecture is documented in Table 5.
- Visual results: Models trained on the RealSR dataset consistently obtain better visual quality than counterparts trained on simulated datasets.The comparison includes images captured by Canon 5D3 and Nikon D810 cameras.
- Visual results: Among models trained on RealSR, LP-KPN achieves the best visual quality while using fewer parameters than a 19×19 KPN.The Laplacian pyramid allows a 5×5 LP-KPN to outperform the larger-kernel KPN with lower parameter cost.
- Cross-device evaluation: Additional experiments evaluate super-resolution results on images captured by a Sony a7II, iPhone X, and Google Pixel 2.These cameras are outside the devices used to construct the RealSR dataset.