Source-linked AI summary
Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution
Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, Ming-Hsuan Yang
TL;DR
Single-image super-resolution must reconstruct an HR image from one LR input without the computational cost and artifacts associated with pre-upscaling. LapSRN progressively predicts residuals with learned transposed-convolution upsampling, deep supervision, and robust loss optimization. It achieves favorable speed and accuracy while producing multi-scale outputs for resource-aware use.
Problem
CNN-based super-resolution methods rely on pre-defined upsampling and ℓ2 optimization, which increase cost, introduce artifacts, or produce blurry predictions.
Method
LapSRN uses a Laplacian pyramid of CNNs to extract LR-space features, progressively predict sub-band residuals, and learn transposed-convolution upsampling with deep supervision and Charbonnier loss.
Results
LapSRN performs favorably against state-of-the-art SR algorithms in visual quality and run time, with real-time speed on most evaluated datasets and significantly better reconstruction accuracy than several CNN models.
Takeaways & Limitations
Progressive reconstruction provides multiple scale predictions in one feed-forward pass, enabling the same network to adapt its output scale to available computational resources.
Takeaways & Limitations
At 8× scaling, LapSRN does not hallucinate fine details lost in the LR input, and its relatively large model size remains a limitation.
Abstract
from arXiv · showhide
Convolutional neural networks have recently demonstrated high-quality reconstruction for single-image super-resolution. In this paper, we propose the Laplacian Pyramid Super-Resolution Network (LapSRN) to progressively reconstruct the sub-band residuals of high-resolution images. At each pyramid level, our model takes coarse-resolution feature maps as input, predicts the high-frequency residuals, and uses transposed convolutions for upsampling to the finer level. Our method does not require the bicubic interpolation as the pre-processing step and thus dramatically reduces the computational complexity. We train the proposed LapSRN with deep supervision using a robust Charbonnier loss function and achieve high-quality reconstruction. Furthermore, our network generates multi-scale predictions in one feed-forward pass through the progressive reconstruction, thereby facilitates resource-aware applications. Extensive quantitative and qualitative evaluations on benchmark datasets show that the proposed algorithm performs favorably against the state-of-the-art methods in terms of speed and accuracy.
1. Introduction
LapSRN addresses the cost, artifact, and capacity limitations of CNN-based super-resolution by progressively reconstructing residuals from LR inputs. It combines learned upsampling, deep supervision, and robust loss optimization, while providing speed and multi-scale flexibility.
- Existing CNN-based SR methods often upscale inputs with bicubic interpolation before prediction, increasing computational cost and causing reconstruction artifacts.
- LapSRN extracts features directly from LR images and progressively predicts sub-band residuals using convolutional and transposed convolutional layers.The architecture applies deep supervision at each pyramid level.
- The model jointly learns upsampling filters and uses deep supervision with the Charbonnier loss to handle outliers and reduce visual artifacts.
- LapSRN is faster than several CNN-based SR models, achieves real-time speed on most evaluated datasets, and provides significantly better reconstruction accuracy.
- Progressive reconstruction produces multiple SR predictions in one feed-forward pass, allowing an 8× model to perform 2× or 4× SR by bypassing finer residual levels.
2. Related Work and Problem Context
CNN-based super-resolution methods differ in how they upscale images, reconstruct residuals, and optimize their networks. LapSRN extends these approaches with learned progressive reconstruction, robust loss optimization, and shared representations across pyramid levels.
- Example-based SR methods learn LR-to-HR mappings from image databases using approaches including dictionary learning, local linear regression, and random forests.
- CNN-based SR methods include direct and progressive reconstruction strategies, with layer counts covering both convolutional and transposed convolutional layers.
- CNN-based methods commonly optimize an ℓ2 loss, which can produce overly smooth results that correlate poorly with human perception.
- Earlier CNN-based methods use bicubic or learned upsampling, while LapSRN jointly learns residuals and upsampling filters with convolutional and transposed convolutional layers.
- Unlike LAPGAN, LapSRN predicts a particular HR image from an LR input rather than synthesizing diverse images from random noise and sample inputs.
- LapSRN connects pyramid levels through transposed convolutions and shared lower-level feature representations, while LAPGAN uses independent sub-networks without weight sharing.
- LapSRN extracts features in LR space, increasing the receptive field and accelerating processing compared with designs that convolve on upsampled images.
3. Deep Laplacian Pyramid Network for SR
LapSRN uses a Laplacian-pyramid cascade to reconstruct super-resolved images progressively, with deep supervision and robust loss optimization at multiple scales.
- Network architecture: LapSRN takes an LR image and progressively predicts residual images across log2 S pyramid levels, where S is the scale factor.For 8× super-resolution, the network uses three sub-networks.
- Network architecture: At each level, convolutional layers extract coarse-resolution features, transposed convolution upsamples them, and reconstruction layers predict residual images or finer-level features.The architecture separates feature extraction from image reconstruction while generating finer-resolution feature maps with one transposed convolutional layer.
- Loss function: The target image at level s is modeled as ys = xs + rs, combining the upscaled LR image with the residual image.Here xs is the upscaled LR image and rs is the residual at level s.
- Loss function: LapSRN uses a robust Charbonnier loss instead of mean squared error to handle outliers, with ε empirically set to 1e−3.The Charbonnier penalty is described as a differentiable variant of the ℓ1 norm.
- Deep supervision: Deep supervision assigns each pyramid level its own loss and corresponding ground-truth image, enabling multi-scale output predictions.The supervision uses different image scales at corresponding pyramid levels.
- Implementation and training details: Training uses 91 Yang et al. images and 200 BSD training images, with randomly sampled 128 × 128 patches and data augmentation.Augmentation includes random scaling, rotations by 90°, 180°, or 270°, and flipping.
- Ablation analysis: Figure 2 analyzes convergence for the pyramid structure, loss functions, and residual learning, while Table 2 reports PSNR drops after replacing these components.The reported Table 2 comparisons use SET5 and SET14.
4. Experiment Results
Experiments analyze LapSRN's components and compare its accuracy, speed, and visual quality across benchmark datasets and real-world images and videos. The results show favorable performance, efficient processing, artifact suppression, and clear limits when the input lacks recoverable structure.
- Model analysis: Removing the pyramid structure reduces performance by 0.7 dB on SET5 and 0.4 dB on SET14.The comparison uses a residual-learning network with matched convolutional depth.
- Model analysis: Deeper networks generally improve performance at the expense of increased computational cost, motivating depth d = 10 for 2× and 4× models.The selected depth balances performance and speed relative to d = 3, 5, 10, and 15.
- Benchmark comparisons: LapSRN performs favorably against existing methods on most datasets using PSNR, SSIM, and IFC across 2×, 4×, and 8× SR.The reported comparisons cover SET5, SET14, BSDS100, URBAN100, and MANGA109.
- Visual comparisons: At 4× SR, progressive reconstruction and robust loss suppress artifacts while accurately reconstructing parallel lines and grid patterns.Visual comparisons include URBAN100, BSDS100, and MANGA109.
- Efficiency: LapSRN is faster than the evaluated methods except FSRCNN on SET14 at 4× SR, while achieving a speed-and-accuracy trade-off.Runtime measurements use the same machine, with results reported in terms of PSNR.
- Real-world applications: Real-world photo experiments show sharper and more accurate reconstructions without ground-truth images or known downsampling kernels.The application targets historical photographs with JPEG compression artifacts.
- Video applications: Both FSRCNN and LapSRN achieve real-time performance on 1200 × 800 video frames, while LapSRN extracts features from the LR space.Frames are evaluated for 2×, 4×, and 8× SR after 8× downsampling.
- Limitations: LapSRN cannot recover fine details when the LR input lacks sufficient structure, a limitation shared by parametric SR methods.The paper also notes the proposed network has a relatively large model size.
5. Conclusions
The paper presents LapSRN, a deep convolutional network within a Laplacian pyramid framework for fast and accurate single-image super-resolution. It progressively predicts high-frequency residuals and favorably compares with state-of-the-art algorithms in visual quality and run time.
- LapSRN uses a Laplacian pyramid framework for fast and accurate single-image super-resolution.
- The model progressively predicts high-frequency residuals in a coarse-to-fine manner.
- Learned transposed convolutional layers replace bicubic interpolation, alleviating undesired artifacts and reducing computational complexity.
- Benchmark evaluations show favorable performance against state-of-the-art super-resolution algorithms in visual quality and run time.