Source-linked AI summary
Residual Local Feature Network for Efficient Super-Resolution
Fangyuan Kong, Mingxi Li, Songwei Liu, Ding Liu, Jingwen He, Yang Bai, Fangmin Chen, Lean Fu
TL;DR
Efficient SISR must balance reconstruction quality with physical-device inference speed, since parameter and FLOP reductions do not directly guarantee faster deployment. The paper proposes RLFN with simplified local feature learning, an improved contrastive-loss feature extractor, and multi-stage warm-start training; its overall scheme achieves a balance of quality and inference speed and RLFN outperforms state-of-the-art efficient models in runtime while maintaining PSNR and SSIM.
Problem
Efficient SR models often emphasize reducing parameters or FLOPs and use complex feature connections, while practical deployment requires higher physical-device inference speed.
Method
RLFN simplifies feature aggregation with residual local feature learning, selects shallow detail-preserving intermediate features for contrastive loss, and reuses previous-stage weights through multi-stage warm-start training.
Results
RLFN outperforms other state-of-the-art efficient SR models in runtime while maintaining PSNR and SSIM, with comparable model-size performance advantages reported across benchmark datasets.
Takeaways & Limitations
The combined network structure and training method achieves a balance between SR quality and inference speed for lightweight models.
Takeaways & Limitations
The feature-distillation connections and concatenation in RFDB reduce parameters and improve restoration performance but severely deteriorate inference speed.
Abstract
from arXiv · showhide
Deep learning based approaches has achieved great performance in single image super-resolution (SISR). However, recent advances in efficient super-resolution focus on reducing the number of parameters and FLOPs, and they aggregate more powerful features by improving feature utilization through complex layer connection strategies. These structures may not be necessary to achieve higher running speed, which makes them difficult to be deployed to resource-constrained devices. In this work, we propose a novel Residual Local Feature Network (RLFN). The main idea is using three convolutional layers for residual local feature learning to simplify feature aggregation, which achieves a good trade-off between model performance and inference time. Moreover, we revisit the popular contrastive loss and observe that the selection of intermediate features of its feature extractor has great influence on the performance. Besides, we propose a novel multi-stage warm-start training strategy. In each stage, the pre-trained weights from previous stages are utilized to improve the model performance. Combined with the improved contrastive loss and training strategy, the proposed RLFN outperforms all the state-of-the-art efficient image SR models in terms of runtime while maintaining both PSNR and SSIM for SR. In addition, we won the first place in the runtime track of the NTIRE 2022 efficient super-resolution challenge. Code will be available at https://github.com/fyan111/RLFN.
1. Introduction
SISR quality has improved, but computation-heavy models and proxy efficiency measures remain poorly suited to resource-constrained deployment. RLFN addresses this by simplifying feature aggregation, refining contrastive-loss features, and using multi-stage warm-start training.
- SISR reconstructs high-resolution images from low-resolution inputs, but advanced models often require computation costs that hinder deployment on resource-constrained devices.
- Parameters and FLOPs are only proxies for physical-device performance, motivating efficient SR models that prioritize inference speed.
- RLFN simplifies feature aggregation to reduce network fragments, improve compactness, and accelerate inference without sacrificing SR restoration accuracy.
- Shallow intermediate features preserve details and textures important for PSNR-oriented models, motivating an improved contrastive-loss feature extractor.
- Multi-stage warm-start training reuses trained weights from previous stages to boost SR performance.
2. Related Work
Lightweight SISR research targets real-time reconstruction by reducing computational demands through architectural and training choices. Prior work includes interpolation and deconvolution upsampling, recursive parameter sharing, and training refinements.
- Lightweight SISR seeks real-time reconstruction on resource-constrained mobile devices, where efficiency has practical and business relevance.
- SCRNN upscales with bicubic interpolation before its three-layer network, while FSRCNN moves upsampling to a terminal deconvolution layer.
- DRCN uses deep recursive convolution to reduce parameter counts through recursive computation.
- Prior SISR work mainly optimized network architecture, while training strategies received comparatively little exploration.
- Fine-tuning with l2 loss, initializing 4x models from pretrained 2x models, and increasing training iterations were reported to improve PSNR.
3. Method
The method simplifies efficient super-resolution through residual local feature learning, while revisiting feature selection in contrastive loss and introducing improved feature extraction and training strategies. RLFN removes hardware-unfriendly distillation connections, analyzes feature depth, and uses a multi-stage training design to balance restoration quality and inference speed.
- RLFN: RLFN combines initial feature extraction, cascaded residual local feature blocks, feature smoothing, and reconstruction to generate super-resolved images.The reconstruction module uses a 3 × 3 convolution and a nonparametric sub-pixel operation.
- Rethinking RFDB: Removing feature distillation connections reduces inference time while retaining comparable restoration performance in the RFDB analysis.RFDB R 48 reduces inference time by 25% versus the original RFDB, while RFDB R 52 achieves comparable results with greater speed.
- Residual Local Feature Block: RLFB replaces multiple distillation connections with a few stacked convolution-ReLU layers for local feature extraction and residual refinement.The block adds the final refined features to the skipped input features before applying a 1 × 1 convolution and ESA block.
- ESA Redundancy: Pruning analysis identifies the three ESA ConvGroup convolution layers as highly redundant, so each ESA block keeps one without performance degradation.The layers rank top-1, top-3, and top-4 in redundancy, and the modification slightly improves inference time and parameter count.
- Revisiting Contrastive Loss: The contrastive-loss feature extractor is revised because shallow features preserve edges and textures needed by PSNR-oriented models, whereas deeper features provide more semantic guidance.Features are extracted from selected intermediate VGG-19 layers, and the authors use difference-map analysis to motivate greater reliance on shallow information.
- Revisiting Contrastive Loss: A randomly initialized feature extractor produces stronger difference-map responses and captures more details and textures than the pre-trained VGG-19 extractor.The observation supports the paper’s proposed feature extractor and indicates that pre-training is not necessary for capturing some structural information.
4. Experiments
Experiments show that RLFN's simplified architecture preserves restoration quality while improving inference speed, and that contrastive loss and warm-start training improve performance. The proposed methods also generalize to existing SISR models and support strong challenge results.
- Quantitative Results: RLFN-S and RLFN achieve superior PSNR and SSIM compared with other efficient super-resolution models on benchmark datasets.RLFN-S matches or exceeds RFDN with 80K fewer parameters, while RLFN outperforms similarly sized methods.
- Architecture Optimization: RLFB maintains RFDB's restoration performance while providing obvious speed advantages after removing feature-distillation connections and intensive additions.The ablation constructs RFDB R 48, RFDB R 52, and RLFB to evaluate progressively simplified blocks.
- Architecture Optimization: Reducing ESA ConvGroups from three convolution layers to one preserves similar restoration performance while accelerating inference.The comparison evaluates RLFB esa g3 against the proposed RLFB under the same setting.
- Contrastive Loss: Contrastive loss consistently improves PSNR and SSIM on four benchmark datasets.The ablation compares warm-start training with and without contrastive loss in the second warm-start stage.
- Warm-Start Strategy: The proposed warm-start strategy outperforms alternative learning-rate strategies in PSNR and SSIM.The authors report that warm-start training helps optimization jump out of local minima and improve overall performance.
- Generalization: The contrastive loss and warm-start strategy are generic and can be applied to existing SISR models such as EDSR.Their generalization is evaluated by applying each method individually to EDSR.
5. Conclusion
The paper proposes RLFN for efficient SISR by simplifying layer connections and combining the architecture with revised contrastive loss and warm-start training. Experiments report a balance between image quality and inference speed.
- Conclusion: RLFN reduces layers and simplifies connections to produce a lighter and faster efficient SISR network.The conclusion presents this architectural simplification as part of the overall scheme.
- Conclusion: Revised contrastive loss and warm-start training are included to improve training of lightweight super-resolution models.The paper changes the feature extractor and intermediate features used by contrastive loss.
- Conclusion: The overall architecture and training method achieve a balance between reconstruction quality and inference speed.The conclusion characterizes this balance as the outcome of extensive experiments.