Source-linked AI summary
Dual Residual Networks Leveraging the Potential of Paired Operations for Image Restoration
Xing Liu, Masanori Suganuma, Zhun Sun, Takayuki Okatani
TL;DR
Image restoration needs architectures that exploit effective paired operations across tasks. The paper introduces dual residual connections and modular DuRB blocks with task-specific paired operations, then evaluates the resulting networks across five tasks and nine datasets, reporting consistent improvement over previous methods.
Problem
The paper studies how to design residual-network architectures that better exploit paired operations for image restoration across different tasks.
Method
The authors introduce dual residual connections and DuRB modules with two containers for arbitrary paired operations, then build task-specific networks by stacking these blocks.
Results
The proposed approach consistently works better than previous methods across five image-restoration tasks evaluated using nine datasets.
Takeaways & Limitations
Properly chosen paired operations within the dual-residual design provide a common modular basis for networks addressing multiple image-restoration tasks.
Abstract
from arXiv · showhide
In this paper, we study design of deep neural networks for tasks of image restoration. We propose a novel style of residual connections dubbed "dual residual connection", which exploits the potential of paired operations, e.g., up- and down-sampling or convolution with large- and small-size kernels. We design a modular block implementing this connection style; it is equipped with two containers to which arbitrary paired operations are inserted. Adopting the "unraveled" view of the residual networks proposed by Veit et al., we point out that a stack of the proposed modular blocks allows the first operation in a block interact with the second operation in any subsequent blocks. Specifying the two operations in each of the stacked blocks, we build a complete network for each individual task of image restoration. We experimentally evaluate the proposed approach on five image restoration tasks using nine datasets. The results show that the proposed networks with properly chosen paired operations outperform previous methods on almost all of the tasks and datasets.
1. Introduction
The paper seeks a residual-network design that exploits paired operations across image-restoration tasks. Its dual residual connection increases interactions between paired operations while retaining their pairing in possible paths.
- Motivation: Prior work found repetitive up-/down-sampling and large-/small-kernel convolution pairs effective for image restoration.The paper uses these findings as motivation for integrating paired operations into residual networks.
- Dual residual connection: The proposed dual residual connection allows f_i to pair with g_j for any i ≤ j, rather than only matching operations at the same depth.For three blocks, the possible paths include six combinations: (f1,g1), (f2,g2), (f3,g3), (f1,g2), (f1,g3), and (f2,g3).
- Dual residual connection: The connection guarantees that the two operation families remain paired along possible paths, unlike the alternative connection style shown in Fig. 1(c).This preserves paired-operation structure while expanding potential cross-block interactions.
- Proposed module: The Dual Residual Block (DuRB) implements the proposed connection using two containers for paired operations.The block is intended as a modular component for constructing task-specific restoration networks.
- Evaluation: The authors evaluate DuRB-based networks on five image-restoration tasks using nine datasets.The paper frames the architecture as a design intended to be shared across different restoration tasks.
2. Related Work
The related work covers established neural and classical approaches for noise, blur, haze, raindrop, and rain-streak restoration. It also motivates comparing residual-connection designs and paired-operation implementations.
- Gaussian noise removal: Noise-removal research includes REDNet, MemNet, and searched convolutional autoencoders using repetitive large- and small-kernel pairs.The cited architectural-search results report that these repetitive paired convolutions outperform REDNet and MemNet by a good margin.
- Motion blur removal: Motion-blur methods range from latent-kernel estimation to CNN, modified-residual-block, and GAN-based approaches.The literature also introduced new datasets for motion-blur removal.
- Haze removal: Haze-removal methods estimate atmospheric light and transmission or directly estimate the clear image from the hazy input.One cited CNN method jointly estimates transmission and atmospheric light, while others directly estimate the clear image.
- Raindrop detection and removal: Raindrop-removal work includes PCA templates, K-Means with median filtering, and a hybrid convolutional-LSTM/CNN trained in a GAN framework.These approaches address detection, localization, and clear-image generation in different ways.
- Rain-streak removal: Rain-streak removal methods use guided filtering, rain-density estimation, recurrent networks, and non-locally enhanced dense blocks.The cited methods model or separate rain components before restoring clear images.
- Connection comparison: Table 1 compares three residual-connection types, with dashes marking applications considered infeasible.The comparison fixes the number of blocks, operations, and other configurations across the evaluated structures.
3. Dual Residual Blocks
DuRB is a modular residual block whose two containers hold paired operations selected for each restoration task. Different DuRB variants and network comparisons test whether this connection style is effective across tasks.
- Block structure: DuRB uses containers T_l^1 and T_l^2 for the paired first and second operations in block l.Normalization and ReLU layers can be incorporated when necessary.
- DuRB variants: The authors define four DuRB variants: DuRB-P, DuRB-U, DuRB-S, and DuRB-US.The variants combine convolution, sampling, and channel-wise attention in different paired-operation configurations.
- DuRB variants: DuRB-P pairs convolutions, DuRB-U pairs up-sampling with down-sampling, DuRB-S pairs convolution with channel-wise attention plus convolution, and DuRB-US combines sampling with attention.The paper assigns these variants to different restoration tasks.
- Experimental comparison: The connection-style experiment evaluates three base structures with fixed block counts, operations, and other configurations across five restoration tasks.The compared task-specific designs include DuRB-P, DuRB-U, DuRB-US, DuRB-S with DuRB-P, and DuRB-S.
4. Five Image Restoration Tasks
The paper applies dual-residual designs with task-specific paired operations to five image-restoration problems, reporting strong results across noise removal, deblurring, dehazing, raindrop removal, and rain-streak removal.
- The experiments cover Gaussian and real-world noise removal, motion-blur removal, haze removal, raindrop removal, and rain-streak removal.Different DuRB variants are used to construct networks for these restoration tasks.
- Noise Removal: DuRN-P uses paired convolutions with large- and small-size receptive fields for Gaussian noise removal.The network stacks six DuRBs and uses an outermost residual connection.
- Noise Removal: The proposed network outperforms previous methods for all three additive Gaussian noise levels: 30, 50, and 70.Table 2 reports PSNR/SSIM results on BSD200-grayscale.
- Noise Removal: On real-world noise, DuRN-P achieves the best result despite using fewer parameters than the compared methods.The dataset contains 40 instance-image and mean-image pairs, and batch normalization is removed for this experiment.
- Motion Blur Removal: For motion-blur removal, DuRN-U pairs up- and down-sampling within a symmetric encoder-decoder network.The model restores recognizable details and outperforms DeBlurGAN in object-detection accuracy on the Car Dataset.
- The proposed networks report favorable outcomes on haze, raindrop, and rain-streak removal, including realistic dehazing, three of four raindrop comparisons, and best rain-streak performance.DuRN-US handles strong ambient light in the reported dehazing examples, while DuRN-S achieves the best performance on two rain-streak benchmarks.
5. Summary and Discussions
The paper proposes dual residual connections and the DuRB modular block to exploit paired operations for image restoration. Networks built from these blocks were evaluated across five tasks and nine datasets, consistently outperforming previous methods.
- Dual residual connection is a residual-connection style designed to exploit paired operations for image restoration.
- The DuRB modular block provides two containers for inserting arbitrary paired operations.
- DuRN networks stack DuRBs with task-specific operation choices for five image restoration tasks.
- Nine datasets were used to evaluate the proposed approach, which consistently worked better than previous methods.
Supplementary material for “Dual Residual Networks Leveraging
The supplementary material explains the experimental settings used for the paper’s five image restoration tasks. Its stated purpose is to provide additional task-specific experimental details.
- The supplementary material accompanies the paper titled “The Potential of Paired Operations for Image Restoration.”
- It provides additional explanations about the experimental setting for each of the five image restoration tasks.
- The material is organized around task-specific experimental settings rather than a new restoration task.
A.1. Details of Training Method
The proposed DuRNs use a common Adam optimizer and weighted SSIM-plus-l1 loss, with task-specific exceptions for grayscale Gaussian noise removal and raindrop removal.
- All proposed DuRNs are trained with Adam using (β1, β2) = (0.9, 0.999) and ϵ = 1.0 × 10−8.
- The standard loss is 1.1 × SSIM + 0.75 × l1 across tasks.
- Grayscale Gaussian noise removal on BSD500 uses l2 loss instead of the standard weighted loss.
- Raindrop removal switches from the weighted loss to single l1 loss after 4,000 epochs for 100 additional epochs.
A.2. Additional Results
Additional visual-quality comparison results for the five restoration tasks are intended to be provided with the authors’ GitHub code repository because of file-size limitations.
- Additional visual-quality comparisons cover all five image restoration tasks.
- The additional results are provided with the code in the authors’ GitHub repository because of file-size limitations.
- The supplementary visual results are qualitative comparisons rather than newly specified quantitative metrics.
A.3. Noise Removal
The noise-removal configuration uses paired convolutions with differing receptive fields, while training relies on cropped noisy-image patches and specified dataset construction procedures.
- Architecture: The receptive-field sizes increase with block depth, except at l = 5 to avoid an excessively large receptive field.The design aims to make successive blocks examine the input at increasing scales.
- Architecture: DuRB-P uses paired convolutions with large and small receptive fields.The receptive field is defined as dilation rate × (kernel size - 1) + 1.
- Training: Gaussian-noise training uses batch size 100 and randomly cropped 64 × 64 regions from original noisy training images.Noisy images were generated following the procedure in reference [39].
- Training: Real-world noise training selects 30 of 40 noisy-image and mean-image pairs, using randomly cropped 128×128 patches and batches of 30 patches.The remaining 10 image pairs are reserved for testing.
A.4. Motion Blur Removal
The experiments specify task-specific DuRB configurations and compare alternative block designs across restoration tasks, while activation visualization examines intermediate network representations.
- Motion Blur Removal: DuRB-U uses identical 3 × 3 convolutions with dilation rate 1 and stride 2 for its second operation to perform down-sampling.The cited configuration is used across all DuRB-U blocks.
- Haze Removal: DuRB-US uses the same 3 × 3, dilation-rate-1, stride-2 second operation across all blocks for haze removal.The first operation is specified separately in Table 3.
- Activation Visualization: Activation maps are summed across channels at selected DuRB-US inputs and outputs; around l = 3, the DuRN-US map resembles a transmission map.The visualization compares blocks at l = 0, 3, 6, and 12 for different input images.
- Raindrop Removal: The DuRN-S-P for raindrop removal combines DuRB-S and DuRB-P, using 3 × 3 operations for DuRB-S and 5 × 5 operations for DuRB-P.Both configurations use dilation rate 1.
- Cross-task Evaluation: The study evaluates every combination of four DuRB versions and five restoration tasks to test whether task-specific designs transfer to non-target tasks.The compared versions are DuRB-P, DuRB-U, DuRB-S, and DuRB-US.