Source-linked AI summary
Feedback Network for Image Super-Resolution
Zhen Li, Jinglei Yang, Zheng Liu, Xiaomin Yang, Gwanggil Jeon, Wei Wu
TL;DR
Image super-resolution must reconstruct HR images from LR inputs, while deep and feedforward recurrent architectures leave limitations in parameter efficiency and backward information access. SRFBN uses recurrent hidden-state feedback through a specialized feedback block, with curriculum training for complex degradations, and reports competitive or better performance with fewer parameters.
Problem
Image-SR networks face parameter, storage, and overfitting costs as depth grows, while feedforward recurrent structures cannot provide later-layer information to earlier layers.
Method
SRFBN uses an RNN with a feedback block that refines low-level representations using high-level hidden-state information, with losses tied across iterations and curriculum targets for complex degradations.
Results
SRFBN and SRFBN+ achieve the best results on almost all quantitative comparisons for BD and DN degradation models, while delivering competitive or better performance with fewer parameters overall.
Takeaways & Limitations
Feedback enables step-by-step SR reconstruction, and curriculum learning makes SRFBN suitable for LR images corrupted by complex degradation models.
Takeaways & Limitations
The experiments use specified degradation settings, including bicubic downsampling, Gaussian-blurred downsampling, and bicubic downsampling with Gaussian noise level 30.
Abstract
from arXiv · showhide
Recent advances in image super-resolution (SR) explored the power of deep learning to achieve a better reconstruction performance. However, the feedback mechanism, which commonly exists in human visual system, has not been fully exploited in existing deep learning based image SR methods. In this paper, we propose an image super-resolution feedback network (SRFBN) to refine low-level representations with high-level information. Specifically, we use hidden states in an RNN with constraints to achieve such feedback manner. A feedback block is designed to handle the feedback connections and to generate powerful high-level representations. The proposed SRFBN comes with a strong early reconstruction ability and can create the final high-resolution image step by step. In addition, we introduce a curriculum learning strategy to make the network well suitable for more complicated tasks, where the low-resolution images are corrupted by multiple types of degradation. Extensive experimental results demonstrate the superiority of the proposed SRFBN in comparison with the state-of-the-art methods. Code is avaliable at https://github.com/Paper99/SRFBN_CVPR19.
1. Introduction
Image super-resolution reconstructs HR images from LR inputs, but deeper networks increase parameters and feedforward recurrent structures cannot send later-layer information back to earlier layers. SRFBN addresses this with recurrent feedback, high-level representations, and curriculum-based training.
- Image super-resolution reconstructs a high-resolution image from a low-resolution counterpart, an ill-posed problem because multiple HR images can produce the same LR image.
- Deeper networks improve representation capacity but increase parameters, storage demands, and overfitting risk.
- Feedforward recurrent networks share information forward but prevent previous layers from accessing useful information from following layers.
- SRFBN uses an RNN with a feedback block to refine low-level information using high-level information carried by hidden states across iterations.Losses are connected to each iteration so hidden states contain information about the HR image.
- The feedback structure provides strong early reconstruction ability while requiring few parameters.
- SRFBN combines up- and downsampling with dense skip connections in its feedback block and uses curriculum targets of increasing reconstruction difficulty for complex degradation.The curriculum strategy is described as learning complex degradation models step by step.
2. Related Work
Prior image-SR work explored deep, recurrent, and feedback architectures, while curriculum learning gradually increases target difficulty. SRFBN extends these ideas with an SR-specific feedback block and an easy-to-hard curriculum for multiply degraded inputs.
- Deep image-SR networks use depth and skip connections, but their increasing parameter counts create storage and overfitting concerns.
- Recurrent structures reduce parameters and support feedback in SRFBN, but earlier recurrent SR methods retained feedforward LR-to-SR information flow.
- Feedback mechanisms transmit high-level information to earlier processing stages, and SRFBN adapts this principle to image SR with a dedicated feedback block instead of ConvLSTM.
- Curriculum learning gradually increases the difficulty of learned targets to improve the training procedure.
- SRFBN applies an easy-to-hard curriculum within one query to progressively restore LR images corrupted by multiple degradation types.
3. Feedback Network for Image SR
SRFBN uses an iterative feedback system in which hidden states carry high-level information back to refine low-level representations. Its shared LR feature extraction, feedback, and reconstruction blocks produce progressively refined SR outputs, with curriculum targets for complex degradations.
- Network structure: SRFBN unfolds an RNN for T iterations, sharing LR feature extraction, feedback, and reconstruction-block weights across time.Each iteration is trained with a tied loss so its hidden state carries output information.
- Network structure: Each iteration recovers a residual image from LR input while a residual skip connection supplies an upsampled image around the sub-network.The SR image combines the recovered residual with the upsampled LR input.
- Reconstruction block: The reconstruction block upsamples LR features with Deconv(k, m) and uses Conv(3, cout) to generate a residual image.A bilinear upsample kernel is used for the skip path, and the final result is taken from the last iteration.
- Feedback block: The feedback block receives the previous hidden state, applies sequential up- and down-sampling projection groups with dense skip connections, and fuses their LR features.Its output provides refined high-level representations to the next iteration and reconstruction block.
- Curriculum learning strategy: The network uses L1 loss across T outputs, with identical HR targets for single degradation and difficulty-ordered targets to enforce a curriculum for complex degradations.Each iteration receives equal loss weight through W_t=1.
4. Experimental Results
Experiments evaluate SRFBN on standard and multiple-degradation settings, examining feedback iterations, feature representations, curriculum learning, parameter efficiency, and reconstruction quality.
- Experimental settings: Experiments use PSNR and SSIM on Set5, Set14, B100, Urban100, and Manga109, evaluated on the luminance channel.
- Experimental settings: The study evaluates bicubic degradation alongside blur-downsampling and noisy-downsampling models, with Gaussian blur kernel size 7x7 and noise level 30.
- Study of T and G: Feedback connections improve reconstruction over T=1, and increasing iterations T continues to raise reconstruction quality.
- Feedback vs. feedforward: SRFBN outperforms its feedforward counterpart at every iteration, demonstrating stronger early predictions and a benefit from feedback beyond recurrence alone.
- Feedback vs. feedforward: Feedback feature maps suppress smooth regions more strongly at early iterations, supporting more accurate residual-image recovery than feedforward feature maps.
- Curriculum learning: Curriculum learning assists SRFBN on BD and DN degradations during both scratch training and BI-pretrained fine-tuning, with fine-tuning producing higher PSNR.
- Network parameters: SRFBN-S achieves the best results among networks below 1000K parameters, while SRFBN and SRFBN+ remain competitive with larger models using 35% and 7% of D-DBPN and EDSR parameters.
- Reconstruction results: Under BI degradation, SRFBN and SRFBN+ outperform almost all compared methods, while SRFBN produces clearer and more faithful ×4 reconstructions in qualitative examples.
5. Conclusion
The paper proposes SRFBN, a feedback-based image super-resolution network that enhances low-level representations with high-level information. Its feedback block manages feedback information flow and feature reuse, while curriculum learning targets more complicated degradation settings.
- SRFBN reconstructs super-resolution images by enhancing low-level representations with high-level ones.
- The feedback block handles feedback information flow and supports feature reuse within the network.
- A curriculum learning strategy is introduced to make the network suitable for more complicated tasks involving corrupted low-resolution images.
Supplementary Material
The supplementary material provides discussions, additional quantitative and qualitative results, and runtime comparisons for SRFBN.
- The supplement discusses the feedback block and provides additional insights into the feedback mechanism.
- It includes quantitative results using DIV2K training images, running-time comparisons, and additional qualitative results.
A. Study of Feedback Block
The feedback-block study examines up- and down-sampling layers, dense skip connections, and alternative basic blocks. Results indicate that these design choices affect reconstruction performance and representation ability.
- Feedback-block components: Replacing up- and down-sampling layers with 3 × 3 convolutions dramatically decreases PSNR on Set5 at scale factor ×4.The study attributes the difference to contextual information captured by large-kernel up- and down-sampling operations.
- Feedback-block components: Adding dense skip connections further improves reconstruction performance by enabling information flow across hierarchy layers and time.
- Alternative block designs: The study compares SRFBN’s feedback block with projection units, RDB, and ConvLSTM under matched block settings.
- Alternative block designs: All compared SR custom blocks outperform ConvLSTM by a large margin, while SRFBN’s feedback block obtains the best quantitative results.The passage links ConvLSTM’s gate mechanisms to loss of contextual information for the next iteration.
B. Additional Insights on Feedback Mechanism
Additional analyses show how feedback refines feature representations across iterations. Compared with feedforward processing, feedback introduces richer frequency information early and makes the second iteration more similar to the final representation.
- Feature-map visualization: After the first iteration, refined low-level feature maps show bright activations along image contours and outline edges.The analysis interprets this as high-level representations being added to initial feature maps.
- Spectral analysis: Feedback networks estimate more mid-frequency and high-frequency information than feedforward networks at early iterations.
- Spectral analysis: At t=2, the feedback network’s mid- and high-frequency information is more similar to the final t=4 representation than at t=1.
- Spectral analysis: Feedforward networks gradually recover mid- and high-frequency components across iterations, whereas feedback networks refine already well-developed information.
C. Sanity Check
The sanity check isolates architecture effects using the same DIV2K training data and compares SRFBN with recurrent and state-of-the-art networks. SRFBN-S improves over MemNet with substantially fewer parameters, while the final SRFBN remains competitive with D-DBPN.
- The comparison uses DIV2K training images for all networks to isolate the effect of architecture design.
- SRFBN-S achieves better quantitative results than MemNet with 71% fewer parameters.
- The final SRFBN obtains competitive results against D-DBPN, especially on Urban100 and Manga109.
- The reported comparison is presented in Table 8, while Table 9 separately captions average running time on Urban100 at scale factor 4.
D. Running Time Comparison
The running-time comparison evaluates SRFBN-S and SRFBN against five state-of-the-art networks on Urban100 at scale factor ×4. Both proposed models are reported as fastest, with SRFBN using fewer parameters and fewer convolutional layers than RCAN.
- SRFBN-S and SRFBN have the fastest evaluation time among six compared networks on Urban100 at scale factor ×4.The comparison includes MemNet, EDSR, D-DBPN, RDN, and RCAN.
- SRFBN uses about 100 convolutional layers and 77% fewer parameters than RCAN.The reported parameter counts are 3,631K for SRFBN versus 15,592K for RCAN.
- The timing evaluation uses official codes on the same 4.2GHz Intel i7 CPU, 16G RAM, and NVIDIA 1080Ti GPU.
E. More Qualitative Results
Figures 11–22 provide additional visual results for SRFBN across BI, BD, and DN degradation models at scale factor ×4. The paper presents these examples as evidence of the proposed network’s superiority.
- Figures 11–14 show visual results for the BI degradation model at scale factor ×4.
- Figures 15–18 show visual results for the BD degradation model at scale factor ×4.
- Figures 19–22 show visual results for the DN degradation model at scale factor ×4.
- The visual comparisons include IRCNN, SRMD or SRMDNF, RDN, and SRFBN.