PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual Dense Network for Image Restoration

Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, Yun Fu

arXiv:1812.10477v2cs.CV

TL;DR

Deep image-restoration models often underuse hierarchical features from the original low-quality image. The paper proposes residual dense networks that densely connect and adaptively fuse local and global features, achieving favorable performance across several restoration tasks while sometimes failing to recover proper details in challenging cases.

  • Problem

    Most deep CNN-based image-restoration models do not fully use hierarchical features from the original low-quality image.

  • Method

    RDN uses residual dense blocks with contiguous memory, local feature fusion, and global feature fusion to extract and adaptively combine hierarchical features.

  • Results

    Extensive benchmark and real-world evaluations show favorable or superior performance against existing methods across several image-restoration tasks.

  • Takeaways & Limitations

    RDN provides a unified framework applicable to super-resolution, denoising, compression artifact reduction, and deblurring.

  • Takeaways & Limitations

    In challenging cases such as large scaling factors, RDN may fail to recover proper details or structures from limited input information.

Abstract

from arXiv · show

Convolutional neural network has recently achieved great success for image restoration (IR) and also offered hierarchical features. However, most deep CNN based IR models do not make full use of the hierarchical features from the original low-quality images, thereby achieving relatively-low performance. In this paper, we propose a novel residual dense network (RDN) to address this problem in IR. We fully exploit the hierarchical features from all the convolutional layers. Specifically, we propose residual dense block (RDB) to extract abundant local features via densely connected convolutional layers. RDB further allows direct connections from the state of preceding RDB to all the layers of current RDB, leading to a contiguous memory mechanism. To adaptively learn more effective features from preceding and current local features and stabilize the training of wider network, we proposed local feature fusion in RDB. After fully obtaining dense local features, we use global feature fusion to jointly and adaptively learn global hierarchical features in a holistic way. We demonstrate the effectiveness of RDN with several representative IR applications, single image super-resolution, Gaussian image denoising, image compression artifact reduction, and image deblurring. Experiments on benchmark and real-world datasets show that our RDN achieves favorable performance against state-of-the-art methods for each IR task quantitatively and visually.

1 INTRODUCTION

The paper identifies underuse of hierarchical features and training difficulties in deep, wide restoration networks, then proposes RDN with dense local and global feature fusion. It extends the framework across multiple restoration tasks and reports favorable performance against existing approaches.

  • Image restoration is an ill-posed inverse problem because degradation from high-quality to low-quality images is irreversible.
  • Many restoration methods neglect hierarchical features from the original low-quality image, while interpolation-based processing can add blur and quadratically increase computation.
  • Higher growth rates can improve dense-network performance, but wider networks become harder to train and numerically unstable.
  • RDN extracts hierarchical features from the original low-quality image through residual dense blocks with contiguous memory, local feature fusion, and global feature fusion.
  • The unified RDN framework uses local dense connections and adaptively preserves accumulated features through local feature fusion.
  • The authors extend RDN to denoising, compression artifact reduction, and deblurring, with experiments reporting outperformance of existing approaches on these tasks.

2 RELATED WORK

Related work spans model-based and learning-based restoration methods, including CNN architectures for super-resolution, denoising, compression artifact reduction, and other tasks. The paper positions RDN as a response to the limited use of hierarchical features in prior learning-based approaches.

  • Image restoration methods are broadly categorized as model-based and learning-based approaches.
  • Early CNN restoration work introduced SRCNN for super-resolution and ARCNN for compression artifact reduction, followed by deeper residual, recursive, and encoder-decoder architectures.
  • Several super-resolution methods extract features from the original low-resolution input and upscale final features using transposed or sub-pixel convolution.
  • Residual learning, gradient clipping, recursive supervision, and memory mechanisms were developed to ease training of deeper restoration networks.
  • Most prior deep learning restoration methods lose useful hierarchical features from the original low-quality image, motivating RDN's adaptive fusion of features from all layers in low-quality space.

3 RESIDUAL DENSE NETWORK FOR IR

RDN extracts and fuses hierarchical features from the low-quality image through residual dense blocks and dense feature fusion. Its task-specific structure uses upscaling for super-resolution and residual learning for denoising, artifact reduction, and deblurring.

  • Network Structure: RDN for image restoration combines shallow feature extraction, residual dense blocks, dense feature fusion, and an up-sampling net.For super-resolution, two convolutional layers extract shallow features before the RDB stack.
  • Task-specific Structures: For denoising, compression artifact reduction, and deblurring, RDN keeps input and output resolutions equal and removes the upscaling module.Residual learning connects the input and output for faster training; the super-resolution structure is chosen to reduce GPU memory and runtime.
  • Residual Dense Block: Residual dense blocks produce local features by connecting each convolutional layer to preceding RDB states and earlier layers within the current block.This contiguous-memory design preserves feed-forward connections while extracting local dense features.
  • Residual Dense Block: Local feature fusion uses a 1×1 convolution to adaptively combine preceding-RDB and current-block features while reducing their feature number.The paper notes that larger growth rates improve performance but make very deep dense networks harder to train without LFF.
  • Dense Feature Fusion: Dense feature fusion globally combines features from all RDBs through global feature fusion and global residual learning.A 1×1 convolution adaptively fuses levels, followed by a 3×3 convolution for further feature extraction.

4 DIFFERENCES WITH PRIOR WORKS

RDN differs from prior restoration networks by directly exploiting hierarchical features from the original low-resolution input through residual dense blocks and feature-fusion mechanisms.

  • RDN’s design differences from prior methods are evaluated through analyses of block connections, convergence, and parameter-performance trade-offs.
  • RDN targets image restoration without DenseNet operations such as batch normalization and max pooling, using local feature fusion and local residual learning between convolutional layers.
  • Compared with SRDenseNet, RDN adds contiguous memory connections from each preceding RDB to every layer of the current RDB.The RDB also uses local feature fusion to support wider networks and local residual learning to improve information flow.
  • Unlike MemNet, RDN extracts features directly from the original low-resolution input rather than interpolating it to the desired size.RDN then fuses dense local features from its RDBs to exploit hierarchical information more fully.

5 NETWORK INVESTIGATIONS

Network investigations show that deeper and wider RDNs improve performance, while CM, LRL, and GFF improve training and jointly produce the strongest ablation result and a favorable accuracy–runtime trade-off.

  • 5.1 Study of D, C, and G.: Larger D, C, and G improve performance, while smaller values can reduce training performance; RDN still outperforms SRCNN.D denotes the number of RDBs, C the convolutional layers per RDB, and G the growth rate.
  • 5.2 Ablation Investigation: Without CM, LRL, or GFF, the baseline reaches PSNR = 34.87 dB and performs very poorly because the deep network is difficult to train.All eight ablation networks use D = 20, C = 6, and G = 32, with LFF retained by default.
  • 5.2 Ablation Investigation: Adding CM, LRL, or GFF individually improves the baseline by contributing to information and gradient flow.
  • 5.2 Ablation Investigation: Combining two components performs better than using one, and the configuration with CM, LRL, and GFF performs best.
  • 5.2 Ablation Investigation: Convergence curves show that CM, LRL, and GFF stabilize training without obvious performance drop.The curves are based on PSNR on Set5 (×2) over 200 epochs.
  • 5.3 Model Size, Performance, and Running Time: RDN uses half as many parameters as EDSR while achieving better results, and offers comparable or shorter test time than other methods.On Set14 (×2), the reported comparison indicates a favorable performance–running-time trade-off.

6 EXPERIMENTAL RESULTS

The experiments evaluate RDN across multiple restoration tasks using benchmark and real-world data under specified degradation models, training settings, and metrics.

  • 6.1 Settings: DIV2K provides 800 training, 100 validation, and 100 test images for most evaluated restoration applications.For bicubic super-resolution, the study also investigates larger training data using Flickr2K together with DIV2K.
  • 6.1 Settings: Testing uses standard datasets spanning super-resolution, denoising, compression artifact reduction, and deblurring.The listed datasets include Set5, Set14, B100, Urban100, Manga109, Kodak24, BSD68, LIVE1, Classic5, and McMaster18.
  • 6.1 Settings: Quantitative evaluation uses PSNR and SSIM on the luminance channel.
  • 6.1 Settings: The study simulates super-resolution with bicubic, blurred-downsampled, and blurred-downsampled-plus-noise degradation models.Other tasks use additive Gaussian noise, JPEG compression, or Gaussian blur with additive noise as specified.
  • 6.1 Settings: Training uses random 48 × 48 LQ RGB patches, augmentation by flips and 90° rotations, and Adam optimization.The initial learning rate is 10^-4 for all layers.

6.2 Image Super-Resolution

RDN is evaluated for super-resolution under BI, BD, and DN degradation models, where it consistently achieves strong quantitative and visual results. Its hierarchical-feature design helps recover sharper structures, details, and edges across degradation settings.

  • BI degradation model: RDN uses D = 16, C = 8, and G = 64 for BI-model comparison, with Flickr2K added to the training data.The setting is chosen for fair comparison because most competing methods use about 64 filters per convolutional layer.
  • BI degradation model: RDN performs best on all datasets and scaling factors against persistent CNN models under the BI degradation model.The comparison covers ×2, ×3, and ×4 super-resolution and indicates stronger effectiveness for RDB than for the compared dense and memory blocks.
  • BI degradation model: RDN recovers sharper, clearer, and more faithful edges than most compared methods at ×4 and ×8 scaling under the BI model.It also alleviates degradation artifacts and recovers more correct structures in the reported example.
  • BD and DN degradation models: RDN and RDN+ achieve the best results on all listed datasets under both BD and DN degradation models at scaling factor ×3.The reported gains are consistent with the visual comparisons in Figures 8 and 9.
  • BD and DN degradation models: Under BD degradation, RDN suppresses blurring artifacts and recovers sharper edges, while under DN degradation it removes noise and recovers more details.These comparisons support RDN's applicability to jointly denoising and super-resolving images.
  • Real-world images: On historical images with JPEG artifacts, RDN reconstructs sharper and more accurate results than SRMDNF and D-DBPN at ×4 scaling.The comparison indicates robustness to different or unknown degradation models.

6.3 Image Denoising

RDN is evaluated for grayscale, color, and real-world image denoising across multiple noise levels and datasets. It reports higher quantitative performance and preserves more details and sharper edges than the compared methods.

  • Grayscale denoising: At σ = 50, RDN gains 0.28 dB, 0.22 dB, 0.11 dB, and 0.88 dB over FFDNet on four test sets, respectively.For σ = 30, 50, and 70, RDN's gains over BM3D are larger than 0.7 dB.
  • Grayscale denoising: RDN removes high-level grayscale noise while recovering details that competing methods either fail to remove or oversmooth.The visual comparison focuses on noise level σ = 50 and details in smooth regions and tiny lines.
  • Color denoising: RDN produces fewer edge and smooth-region artifacts and preserves clearer edges than the compared color denoising methods.The visual comparison is reported at noise level σ = 50.

6.4 Image Compression Artifact Reduction

RDN is evaluated for JPEG compression artifact reduction on LIVE1 and Classic5 across four quality settings. It achieves higher PSNR and SSIM than the compared methods and better preserves content structure at very low quality.

  • Quantitative results: RDN and RDN+ achieve higher PSNR and SSIM values than all compared methods on LIVE1 and Classic5 across JPEG qualities q = 10, 20, 30, and 40.The comparison focuses on the Y channel in YCbCr space.
  • Quantitative results: At q = 10, RDN improves PSNR over DnCNN by 0.48 dB and 0.60 dB on the two reported datasets.The paper describes q = 10 as a very low compression-quality setting.
  • Visual results: At q = 10, RDN removes compression artifacts while preserving details and content structures that competing methods oversmooth or leave corrupted.The visual comparison covers the first through fourth rows of Figure 13.

6.5 Image Deblurring

RDN is tested for image deblurring with a 25×25 Gaussian blur kernel and additive Gaussian noise. It achieves large improvements over IRCNN and reconstructs sharper edges and finer details.

  • Experimental setting: The deblurring setting applies a 25×25 Gaussian blur kernel with standard deviation 1.6 and additive Gaussian noise σ = 2.The paper evaluates this commonly used synthetic blur-and-noise setting.
  • Quantitative results: RDN achieves large improvements over IRCNN on McMaster18, Kodak24, and Urban100.The quantitative comparison is reported using PSNR and SSIM.
  • Visual results: RDN reconstructs sharper edges and tiny details than IRCNN in the visual deblurring results.IRCNN is reported to retain some blurry structures.

7 DISCUSSIONS

RDN combines local and global feature fusion to achieve strong representational ability, favorable model-size/performance tradeoffs, and promising performance across image restoration tasks. Its limitations become evident when inputs are highly challenging or insufficient to support correct structure recovery.

  • Benefits of RDN: RDB combines local and global feature fusion, giving RDN strong representational ability and a favorable model-size/performance tradeoff.
  • Benefits of RDN: RDN can be directly applied or generalized to several image restoration tasks with promising performance.
  • Limitations of RDN: At large scaling factors, RDN may fail to recover proper details.
  • Limitations of RDN: With limited input information, RDN may generate likely texture patterns from training data instead of recovering correct structures.
  • Challenges in Image Restoration: Extreme degradation and complex real-world processes make restoration, data preparation, and network training more difficult.
  • Future Works: Future work includes stronger local and global feature representations, broader restoration applications, and possible benefits from adversarial training.

8 CONCLUSIONS

The paper proposes RDN for image restoration based on residual dense blocks that connect and fuse features across blocks and layers. Evaluations show superiority over state-of-the-art methods across several restoration tasks.

  • RDN uses residual dense blocks with contiguous memory, local feature fusion, and local residual learning to improve feature and gradient flow.
  • Extensive benchmark and real-world evaluations demonstrate RDN’s superiority over state-of-the-art methods for several image restoration tasks.
Loading 1812.10477v2…