PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual Feature Distillation Network for Lightweight Image Super-Resolution

Jie Liu, Jie Tang, Gangshan Wu

arXiv:2009.11551v1eess.IVcs.CV

TL;DR

CNN-based super-resolution achieves strong performance but often requires computation unsuitable for edge devices, leaving efficient model design as an important challenge. The paper proposes FDCs and SRBs, builds RFDN around them, and reports state-of-the-art results with a favorable performance–complexity trade-off; E-RFDN won the AIM 2020 efficient super-resolution challenge.

  • Problem

    CNN-based super-resolution methods can require heavy computation, while information distillation’s channel splitting is not sufficiently understood for efficient design and can limit lightweight flexibility.

  • Method

    The paper replaces channel-splitting-based distillation with more lightweight, flexible FDCs and combines them with SRBs in the RFDN architecture.

  • Results

    RFDN achieves state-of-the-art quantitative and qualitative results with a better performance–model-complexity trade-off, while E-RFDN won first place in the AIM 2020 efficient super-resolution challenge.

  • Takeaways & Limitations

    RFDN provides a fast and lightweight image super-resolution model that maintains strong reconstruction performance while using fewer parameters than competing lightweight methods.

  • Takeaways & Limitations

    The paper notes that the channel-splitting-based progressive refinement module is inefficient and inflexible because distilled features use parameter-heavy 3 × 3 convolutions and constrain identity connections.

Abstract

from arXiv · show

Recent advances in single image super-resolution (SISR) explored the power of convolutional neural network (CNN) to achieve a better performance. Despite the great success of CNN-based methods, it is not easy to apply these methods to edge devices due to the requirement of heavy computation. To solve this problem, various fast and lightweight CNN models have been proposed. The information distillation network is one of the state-of-the-art methods, which adopts the channel splitting operation to extract distilled features. However, it is not clear enough how this operation helps in the design of efficient SISR models. In this paper, we propose the feature distillation connection (FDC) that is functionally equivalent to the channel splitting operation while being more lightweight and flexible. Thanks to FDC, we can rethink the information multi-distillation network (IMDN) and propose a lightweight and accurate SISR model called residual feature distillation network (RFDN). RFDN uses multiple feature distillation connections to learn more discriminative feature representations. We also propose a shallow residual block (SRB) as the main building block of RFDN so that the network can benefit most from residual learning while still being lightweight enough. Extensive experimental results show that the proposed RFDN achieve a better trade-off against the state-of-the-art methods in terms of performance and model complexity. Moreover, we propose an enhanced RFDN (E-RFDN) and won the first place in the AIM 2020 efficient super-resolution challenge. Code will be available at https://github.com/njulj/RFDN.

1 Introduction

Image super-resolution reconstructs high-resolution images from low-resolution inputs, but lightweight models are needed for time-sensitive and real-world applications. The paper analyzes information distillation and proposes RFDN, using FDCs and SRBs to improve the performance–complexity trade-off.

  • Image super-resolution reconstructs a high-resolution image from its low-resolution counterpart, an ill-posed task because multiple high-resolution images can share one low-resolution image.
  • Deeper CNNs improve super-resolution quality but are unsuitable for real-world scenarios, motivating fast and lightweight models with better performance–complexity trade-offs.The cited discussion contrasts 20-layer VDSR and over-160-layer EDSR with deployment needs.
  • Recursive models reduce parameters through sharing but require greater depth or width, increasing operations and inference time.
  • Information distillation retains part of the extracted features while refining the remainder, aggregating current information with partially retained local short-path information.
  • The paper identifies information distillation as insufficiently efficient and inflexible, then proposes lighter, more flexible FDCs and the RFDN architecture.
  • RFDN combines FDCs with SRBs, while the stated contributions target state-of-the-art super-resolution performance with fewer parameters and identity connections without extra parameters.

2 RELATED WORK

Related work develops CNN-based super-resolution models through increasing depth, parameter sharing, progressive reconstruction, persistent memory, and efficient upsampling. Lightweight variants target mobile deployment, including CARN-M, IDN, and IMDN.

  • SRCNN introduced a three-layer CNN for end-to-end mapping from low-resolution to high-resolution images, while VDSR improved performance with 20 convolutional layers.
  • DRCN reduced model complexity by recursively applying a feature-extraction layer 16 times, extending parameter sharing in super-resolution networks.
  • LapSRN progressively reconstructs high-resolution sub-band residuals from original low-resolution inputs to address speed and accuracy.
  • MemNet addresses long-term dependencies in image restoration, while efficient sub-pixel convolution reduces computational cost and testing time for upscaling.
  • CARN-M targets mobile devices, and IDN and IMDN use information distillation to develop fast and lightweight image super-resolution models.

3 METHOD

RFDN rethinks information distillation by replacing channel-splitting operations with lightweight, flexible feature distillation connections and combines them with shallow residual blocks. Multiple RFDBs progressively refine features before feature assembly and reconstruction.

  • 3.1 Information multi-distillation block: The original IMDB progressively refines features by repeatedly splitting preceding features into retained distilled features and coarse features for further processing.Distilled features are concatenated as the progressive refinement module's output.
  • 3.2 Rethinking the IMDB: The channel-splitting operation is inefficient and inflexible because distilled features use redundant 3×3 filters and couple refinement with splitting.This coupling makes it difficult to use identity connections in the refinement pipeline.
  • 3.2 Rethinking the IMDB: The rethought IMDB decouples channel splitting into concurrent 3×3 distillation and refinement convolutions, providing a clearer and more flexible architecture.The distillation layer produces distilled features, while the refinement layer processes coarse features.
  • 3.3 Residual feature distillation block: RFDB replaces most 3×3 channel-reduction convolutions with 1×1 convolutions while retaining a 3×3 convolution for spatial feature refinement.These outer connections are called feature distillation connections (FDC).
  • 3.3 Residual feature distillation block: SRB combines a 3×3 convolution, identity connection, and activation unit to add deeper residual connections without extra parameters.RFDN uses RFDBs as its main stacked blocks.

4 EXPERIMENTS

Experiments train on DIV2K and evaluate ×2, ×3, and ×4 super-resolution using PSNR and SSIM on five benchmark datasets. The study also investigates FDC and SRB and distillation rate at ×4.

  • DIV2K provides 800 high-quality RGB training images, while Set5, Set14, BSD100, Urban100, and Manga109 serve as test datasets.
  • PSNR and SSIM measure super-resolved image quality on the Y channel converted from RGB to YCbCr.
  • Training inputs are formed by bicubic down-sampling at ×2, ×3, and ×4, with 64 randomly cropped 64 × 64 LR patches per minibatch.
  • Table 1 investigates FDC and SRB, while Table 2 investigates distillation rates on benchmark datasets with scale factor ×4.

4.3 Model analysis

The ablation study evaluates FDC and SRB separately and jointly, showing that both components improve the network’s super-resolution performance without adding parameters for SRB.

  • FDC improves the base method’s Manga109 PSNR from 30.28 to 30.47, a gain of +0.19dB.
  • SRB improves Manga109 performance by +0.12dB PSNR and +0.0024 SSIM without introducing extra parameters.
  • Adding FDC and SRB enables the RFDB to significantly outperform the base block.

Ablation study

The distillation-rate study finds that rate 0.5 offers a favorable balance between super-resolution performance and parameter count, and uses it in RFDN and RFDN-L.

  • A distillation rate of 0.5 provides a good trade-off between SR performance and the number of parameters.
  • The selected 0.5 distillation rate is adopted as the final setting in both RFDN and RFDN-L.
  • Figure 4 compares PSNR and parameter count on the Set5 ×4 dataset.

Model complexity analysis

RFDN and RFDN-L achieve stronger PSNR–complexity trade-offs than lightweight baselines, using fewer parameters or calculations than IMDN while retaining comparable inference speed.

  • RFDN achieves better performance than VDSR, MemNet, IDN, and IMDN with fewer parameters on Set5 ×4.
  • RFDN and RFDN-L achieve higher PSNR than IMDN while using fewer calculations in the PSNR-versus-Mult-Adds comparison.
  • RFDN uses 44 FPS versus IMDN’s 49 FPS, indicating comparable inference speed.

4.4 Comparison with state-of-the-arts

RFDN delivers a favorable performance–complexity trade-off across ×2, ×3, and ×4 super-resolution scales, using substantially fewer parameters than competing lightweight models.

  • 534/541/550K parameters for ×2/×3/×4 SR support RFDN’s comparable or better results against state-of-the-art lightweight methods.
  • RFDN achieves a better trade-off than IMDN across the evaluated lightweight super-resolution comparisons.
  • RFDN-L achieves the best results in most quantitative comparisons, particularly at larger scaling factors, with slightly more parameters.

4.5 About the experimental settings

Under matched training settings, RFDN outperforms IMDN across five benchmark datasets while using fewer parameters, isolating the effect of its network design.

  • Table 3 reports average PSNR/SSIM across Set5, Set14, BSD100, Urban100, and Manga109 for scale factors ×2, ×3, and ×4.
  • RFDN outperforms IMDN on all five benchmark datasets in both PSNR and SSIM with much fewer parameters.
  • Both RFDN and IMDN are trained from scratch under the same experimental settings for ×4 super-resolution.

4.6 Enhanced RFDN for AIM20 challenge

E-RFDN extends RFDN for the AIM 2020 efficient super-resolution challenge and won first place, with the evaluation emphasizing efficiency as well as reconstruction quality.

  • E-RFDN won first place in the AIM 2020 efficient super-resolution challenge.
  • E-RFDN replaces the RFDB CCA layer with the ESA block and uses four enhanced RFDBs.
  • Compared with IMDN’s AIM 2019 constrained-SR challenge entry, E-RFDN provides gains in runtime, parameters, FLOPs, and activations.
  • The challenge table includes the top five methods and reports efficient super-resolution challenge results.

5 CONCLUSION

The paper analyzes information distillation for lightweight image super-resolution and builds RFDN from feature distillation connections and shallow residual blocks. Experiments report strong quantitative and qualitative results with modest computational complexity for mobile deployment.

  • FDCs are presented as more lightweight and flexible than the information distillation mechanism used in IMDN.
  • RFDN combines shallow residual blocks with feature distillation connections for fast and lightweight image super-resolution.
  • The proposed method achieves state-of-the-art quantitative and qualitative results with a modest number of parameters and mult-adds.
  • The model can be easily ported to mobile devices within the paper’s stated lightweight-complexity scope.
Loading 2009.11551v1…