Source-linked AI summary
Deep Back-Projection Networks For Super-Resolution
Muhammad Haris, Greg Shakhnarovich, Norimichi Ukita
TL;DR
Single-image super-resolution is an ill-posed LR-to-HR reconstruction problem that feed-forward architectures struggle to represent, especially at large scaling factors. DBPN uses mutually connected iterative up- and down-sampling with error feedback and feature concatenation, achieving superior results, including state-of-the-art performance at 8× enlargement.
Problem
Feed-forward super-resolution networks do not fully address the mutual dependencies between low- and high-resolution images, limiting LR-to-HR representation especially for large scaling factors.
Method
DBPN is an end-to-end trainable architecture that alternates mutually connected up- and down-projection units and uses feedback from projection errors to guide reconstruction.
Results
DBPN outperforms other state-of-the-art methods on large scaling factors such as 8× enlargement, while its SS network also outperforms several baselines with fewer parameters.
Takeaways & Limitations
Iterative error feedback and accumulated HR features provide an effective basis for super-resolution across large scaling factors and model sizes.
Takeaways & Limitations
The reported settings were selected using general intuition and preliminary experiments, and DBPN was optimized with MSE rather than adversarial loss.
Abstract
from arXiv · showhide
The feed-forward architectures of recently proposed deep super-resolution networks learn representations of low-resolution inputs, and the non-linear mapping from those to high-resolution output. However, this approach does not fully address the mutual dependencies of low- and high-resolution images. We propose Deep Back-Projection Networks (DBPN), that exploit iterative up- and down-sampling layers, providing an error feedback mechanism for projection errors at each stage. We construct mutually-connected up- and down-sampling stages each of which represents different types of image degradation and high-resolution components. We show that extending this idea to allow concatenation of features across up- and down-sampling stages (Dense DBPN) allows us to reconstruct further improve super-resolution, yielding superior results and in particular establishing new state of the art results for large scaling factors such as 8x across multiple data sets.
1. Introduction
Single-image super-resolution is an ill-posed LR-to-HR reconstruction problem, and feed-forward networks struggle to represent this relationship at large scaling factors. DBPN addresses this with iterative, mutually connected up- and down-sampling and error feedback.
- Single-image super-resolution recovers a high-resolution image from a low-resolution image, an ill-posed inverse problem.
- Feed-forward networks learn nonlinear LR-to-HR mappings but have difficulty representing the relationship, especially for large scaling factors.
- DBPN uses iterative up- and down-sampling stages with error feedback to guide reconstruction and characterize projection errors.
- Mutually connected up- and down-sampling stages represent the relationship between LR and HR images while generating HR features at different depths.
- Deep concatenation combines HR feature maps from all up-sampling steps, while dense connections encourage feature reuse.
2. Related Work
Related work includes predefined, single, and progressive upsampling, while feedback and back-projection motivate DBPN’s iterative design. DBPN extends these ideas with learned, mutually connected up- and down-sampling stages.
- Image super-resolution using deep networks: Deep SR networks are categorized into predefined, single, progressive, and iterative up- and downsampling approaches.
- Image super-resolution using deep networks: Predefined upsampling interpolates LR inputs before learning an MR-to-HR nonlinear mapping, but may produce new noise.
- Image super-resolution using deep networks: Single upsampling replaces predefined operators but can lack capacity for complicated mappings; EDSR requires many filters and lengthy training.
- Image super-resolution using deep networks: Progressive upsampling reconstructs multiple scales in one feed-forward network but relies on limited LR features.
- Image super-resolution using deep networks: DBPN distributes reconstruction-error calculation across iterative stages and preserves HR components through learned up- and down-sampling operators.
- Feedback networks: Earlier feedback and back-projection methods iteratively correct predictions or minimize reconstruction error, but can depend on fixed parameters such as blur operators and iteration counts.
3. Deep Back-Projection Networks
DBPN uses mutually connected up- and down-projection units to iteratively map between LR and HR feature spaces and feed projection errors back into reconstruction. Dense DBPN extends this design with concatenated features from preceding units, while reconstruction concatenates outputs from all up-projection units.
- Projection units: Projection units are trained end-to-end to map LR feature maps to HR maps or HR feature maps to LR maps.
- Projection units: Each up-projection unit maps a previous LR feature map to HR, back-projects it to LR, and remaps the residual to HR before summing outputs.
- Projection units: Up- and down-projection units alternate across stages, forming a self-correcting procedure that feeds projection errors back to the sampling layers.
- Dense projection units: D-DBPN concatenates outputs from all preceding projection units as each unit’s input, using 1 × 1 convolutions to merge them before projection.
- Network architecture: The D-DBPN architecture comprises initial feature extraction, alternating back-projection stages, and reconstruction from concatenated HR feature maps.
- Network architecture: For T stages, the network uses one initial extraction stage, T up-projection units, T −1 down-projection units, and a reconstruction layer.
4. Experimental Results
Experiments show that DBPN variants achieve strong super-resolution performance across scaling factors, while dense connections improve reconstruction and lightweight models offer favorable parameter trade-offs.
- Depth analysis: 31.86 dB: the L network outperforms VDSR, DRCN, LapSRN, and DRRN on Set5 at 4× enlargement.Its gains are 0.51, 0.33, 0.32, and 0.18 dB, respectively.
- Depth analysis: The S, M, and L networks outperform current state-of-the-art methods on Set5 at 8× enlargement.The performance difference between the L and M networks is only 0.04 dB.
- Parameter efficiency: 72% and 37% fewer parameters: SS outperforms VDSR at 4× and 8× enlargement, respectively.SS is a lighter version of the S network designed for low computation and real-time processing.
- Parameter efficiency: 76% fewer parameters with approximately the same PSNR: D-DBPN compares with EDSR at 4× enlargement; at 8×, it uses 47% fewer parameters with better PSNR.The authors report that these results provide the best trade-off between performance and parameter count.
- Dense connection: 0.13 dB and 0.05 dB: D-DBPN-L exceeds DBPN-L on Set5 and Set14 at 4× enlargement.At 8×, the corresponding gains increase to 0.23 dB and 0.19 dB.
- Comparison with state of the art: At 8× enlargement, D-DBPN reaches 25.50 dB on Manga109, 0.61 dB above EDSR, and 23.25 dB on Urban100, 0.13 dB above EDSR.The authors report stronger performance on fine-structure images such as manga characters.
- Qualitative comparison: D-DBPN preserves high-resolution components better than other networks in qualitative 8× comparisons.The authors attribute this to extracting features and creating contextual information from the low-resolution input.
5. Conclusion
Deep Back-Projection Networks revise super-resolution features through iterative up- and down-sampling with error feedback, outperforming other state-of-the-art methods, especially at 8× enlargement.
- DBPN uses multiple up- and down-sampling stages with error feedback to revise sampling results and accumulate self-correcting features.The network creates the SR image from features accumulated across upsampling stages.
- DBPN outperforms other state-of-the-art methods at large scaling factors such as 8× enlargement.The paper reports this conclusion alongside quantitative and qualitative evaluations.
- At 8× super-resolution, Ours reaches 23.1 dB versus EDSR at 19.1 dB in the second qualitative comparison.