PaperlayerView exact revision on arXiv

Source-linked AI summary

Path-Restore: Learning Network Path Selection for Image Restoration

Ke Yu, Xintao Wang, Chao Dong, Xiaoou Tang, Chen Change Loy

arXiv:1904.10343v2cs.CV

TL;DR

Very deep CNNs improve image restoration but raise computational costs, and regions within an image can differ in content and distortion. Path-Restore combines a multi-path CNN with a reinforcement-learning pathfinder and difficulty-regulated reward to choose region-specific routes. It reports comparable or superior performance with faster speed across restoration tasks, including state-of-the-art performance on the Darmstadt Noise Dataset.

  • Problem

    Very deep CNNs have improved restoration performance, but their increasing computational cost limits practical use, while regions in one image vary in content and distortion and can differ in restoration difficulty.

    Source: §1 · p. 1

  • Method

    Path-Restore uses a multi-path CNN and a reinforcement-learning pathfinder with a difficulty-regulated reward to select routes for image regions.

    Source: §1 · pp. 1–2

  • Results

    Path-Restore reports comparable or superior performance to existing approaches with faster speed across restoration tasks and state-of-the-art performance on the Darmstadt Noise Dataset.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Path-Restore is particularly effective for real-world denoising when noise distribution varies across regions of an image.

    Source: §1 · p. 2

Abstract

from arXiv · show

Very deep Convolutional Neural Networks (CNNs) have greatly improved the performance on various image restoration tasks. However, this comes at a price of increasing computational burden, hence limiting their practical usages. We observe that some corrupted image regions are inherently easier to restore than others since the distortion and content vary within an image. To leverage this, we propose Path-Restore, a multi-path CNN with a pathfinder that can dynamically select an appropriate route for each image region. We train the pathfinder using reinforcement learning with a difficulty-regulated reward. This reward is related to the performance, complexity and "the difficulty of restoring a region". A policy mask is further investigated to jointly process all the image regions. We conduct experiments on denoising and mixed restoration tasks. The results show that our method achieves comparable or superior performance to existing approaches with less computational cost. In particular, Path-Restore is effective for real-world denoising, where the noise distribution varies across different regions on a single image. Compared to the state-of-the-art RIDNet, our method achieves comparable performance and runs 2.7x faster on the realistic Darmstadt Noise Dataset.

1 INTRODUCTION

Very deep CNNs can improve restoration but impose computational costs, while regions within one image differ in content and distortion and therefore in restoration difficulty. Path-Restore selects region-specific paths with a multi-path CNN and a pathfinder, using a difficulty-regulated reward to balance restoration performance and computation.

1 INTRODUCTION

  • Path-Restore pairs a multi-path CNN with a pathfinder trained through reinforcement learning to select paths for regions based on their content and distortion.The pathfinder uses a reward that encourages high performance with fewer computations.Source: §1 · pp. 1–2
  • The difficulty-regulated reward scales regional performance gains by restoration difficulty, encouraging more computation for hard regions while favoring short paths for easy ones.The paper relates difficulty to loss: the blue region receives a higher final reward than the green region after difficulty regulation, despite a smaller raw loss decrease.Source: §1 · p. 2
  • A dynamic block combines a shared path with several dynamic paths of different complexity, with path number and complexity configurable for the task.The number of paths is proportional to the number of distortion types the system aims to address.Source: §1 · p. 2
  • PathRestore-Mask predicts a low-resolution policy mask and routes regional features through selected paths jointly, achieving a better performance-complexity trade-off than independent processing.The basic version instead restores overlapping patches separately and merges them.Source: §1 · p. 2
  • Path-Restore reports comparable or superior performance to existing approaches with faster speed across restoration tasks, and state-of-the-art performance on the Darmstadt Noise Dataset.It highlights real-world denoising where noise distribution varies spatially within an image.Source: §1 · p. 2
  • The paper reports that adjusting a reward-function hyperparameter during training can balance performance and complexity.Source: §1 · p. 2

2 RELATED WORK

Prior restoration work includes CNNs for individual degradations and methods addressing varied distortions, while dynamic-network approaches route computation by image or task. Path-Restore targets region-level routing that accounts for both image content and restoration task, using dynamic paths and reinforcement learning.

Advances in Image Restoration

  • Many restoration methods address a specific degradation, while DnCNN, CBDNet, and other approaches handle varied distortions; applying one path to all regions can be less efficient.The passage describes DnCNN for Gaussian-noise levels, CBDNet for diverse real noise, and RIDNet and PRIDNet as capacity-expanding denoisers.Source: §2 · p. 2
  • Anwar et al. combine internal denoising for smooth regions with category-specific external denoising for textured regions, which the paper contrasts with its region-wise routing approach.Source: §2 · p. 2

Dynamic Networks

  • RL-Restore selects sequences of CNNs with fewer resources than a single deep network, while Path-Restore uses one network for dynamic processing and can adjust its trade-off through the reward.The paper also states Path-Restore runs faster and produces more spatially consistent results than RL-Restore.Source: §2 · p. 2
  • Earlier dynamic networks route computation based on whole-image content or task, whereas Path-Restore selects paths for different distorted-image regions using both content and restoration task.The paper identifies region-level path selection as distinct from task-level routing, where path choice is unrelated to image content within the task.Source: §2 · p. 3

Handling Non-Differentiable Operations

  • Path-Restore uses reinforcement learning because its reward need not be differentiable, unlike the differentiable target required by Gumbel Softmax estimators.The paper notes differentiable estimators may be worth exploring for their training efficiency.Source: §2 · p. 3

3 METHODOLOGY

Path-Restore routes image regions through a shared path and a selected path of varying complexity. Its pathfinder learns route selection with a difficulty-regulated reward, while Path-Restore-Mask jointly processes regions and training proceeds in two stages.

3.1 Architecture of Path-Restore

  • Each dynamic block sends features through a shared path and one selected path among alternatives of differing complexity.The pathfinder selects the dynamic path; the first is a bypass connection and the others contain residual blocks.Source: §3.1 · p. 3

3.2 Pathfinder

  • The pathfinder uses reinforcement learning, with a reward penalizing complex paths and scaling final-block L2 improvement by restoration difficulty.Difficulty decreases as output loss approaches zero, reducing the reward for spending computation on easy regions.Source: §3.2 · p. 4

3.3 Path-Restore-Mask

  • Path-Restore-Mask predicts low-resolution per-path masks, upsamples them, and combines path features so each region uses one path without overlapping-patch processing.This image-level approach addresses the selection accuracy and computation costs associated with independently processing overlapping regions.Source: §3.3 · pp. 4–5

3.4 Training Algorithm

  • Training has two stages: random-route CNN initialization with final and intermediate losses, then joint CNN/pathfinder optimization using REINFORCE and an individual bypass baseline.The intermediate loss encourages consistency among states observed by the pathfinder across dynamic blocks.Source: §3.4 · p. 5

4 EXPERIMENTS

The experiments evaluate Path-Restore on real-world and blind Gaussian denoising, mixed distortions, and category-specific denoising, considering restoration performance, computational cost, and learned routing across image regions. They also examine Path-Restore-Mask and the effects of reward regulation, training data, and the pathfinder.

4.1 Evaluation on Real-World Denoising

  • Against CBDNet, Path-Restore gains nearly 1 dB with 19% fewer FLOPs and 27% faster GPU speed, while matching RIDNet at about 40% of computation and runtime.Path-Restore-Ext further improves by 0.7 dB and achieves state-of-the-art DND performance. On SIDD, Path-Restore is reported as the most efficient existing method; its real-world policy uses longer paths for dark, severely noisy regions and shorter paths for brighter regions with slight noise.Source: Table 2 · p. 6, §4.1 · pp. 6–7, Table 3 · p. 6

4.2 Evaluation on Blind Gaussian Denoising

  • Path-Restore uses about 25% fewer FLOPs than DnCNN with comparable performance and is about 20% faster on average in practice.For spatially variant Gaussian noise, it improves some fine-textured regions by selecting complex paths; its policies respond to both texture and noise level, favoring shorter paths for smooth regions and longer paths for textured or noisier regions.Source: Table 4 · p. 8, §4.2 · pp. 8–9, Figure 7 · p. 8

4.3 Evaluation on Mixed Distortions

  • On 2K images, Path-Restore is 0.2 dB higher than RL-Restore-re and RL-Restore-im with a tiny FLOPs increase, and runs in 0.954s versus 1.34s and 1.09s, respectively.On 63×63 sub-images, it performs slightly better than RL-Restore. Its unified network avoids model-switching overhead during inference.Source: Table 5 · p. 9, §4.3 · p. 9
  • Path-Restore’s dynamic blocks produce spatially consistent mixed-distortion results, routing according to content and distortion; severe noise favors longer paths, while JPEG paths are often used at the first block.The comparison reports inconsistent boundaries for RL-Restore-re and missed severe distortions for RL-Restore-im, whereas Path-Restore handles differing regional distortions in one network.Source: Figure 11 · p. 10, §4.3 · p. 10

4.4 Evaluation of Path-Restore-Mask

  • On DND and SIDD, Path-Restore-Mask improves PSNR by more than 0.2 dB over Path-Restore while using 10% fewer FLOPs and shorter GPU runtime.Joint image-level processing exploits adjacent-region features and avoids computation on overlapping patches; its path-selection policy is more diverse and spatially smoother, with fewer artifacts in severely noisy regions.Source: Table 6 · p. 10, §4.4 · p. 11, Figure 13 · p. 11

4.5 Discussions

  • Using two paths for mixed distortions increases FLOPs by nearly 10% at comparable performance; lowering reward penalty increases both PSNR and FLOPs, while difficulty regulation improves the PSNR-FLOPs trade-off.A smaller penalty encourages longer paths, and the difficulty-regulated reward saves computation on easy regions while allowing longer paths for hard regions.Source: Table 7 · p. 11, §4.5 · pp. 11–12, Figure 14 · p. 11
  • Using RIDNet training data improves Path-Restore-Mask PSNR by 0.19 dB on DND and 0.08 dB on SIDD with inference time almost unchanged; removing the pathfinder raises runtime by 39% on DND and 13% on SIDD.Without the pathfinder, DND PSNR is unchanged, while SIDD PSNR decreases by 0.19 dB.Source: Table 8 · p. 12, §4.5 · p. 12
  • On mixed distortions, learned routing outperforms shortest, longest, and random path policies; Path-Restore has fewer parameters than RL-Restore, while PRIDNet has 140 times its denoising parameters.The reported denoising parameter count is slightly higher than FFDNet’s but lower than those of other compared methods.Source: §4.5 · p. 12, Table 9 · p. 13
  • Category-specific denoising uses costly PatchMatch external denoising and is less effective on natural images than on regular-pattern images.The passage attributes this limitation to similar patches not always being available in natural images, and notes particularly high cost on high-resolution images.Source: §4.5 · p. 13
  • Path-Restore’s difficulty-regulated pathfinder assigns textured regions to long paths and smooth regions to short paths, matching the category-specific map and human prior.Qualitative results show Path-Restore producing clear, sharp output while category-specific denoising produces blurry output; that method manually defines its textured-region criterion.Source: §4.5 · p. 13
  • Path-Restore is more than 5 dB higher in PSNR and 1,000 times faster than category-specific denoising.The comparison uses grayscale images on CBSD68 with Gaussian noise σ = 30.Source: Table 10 · p. 13, §4.5 · p. 13

5 CONCLUSION

Path-Restore uses a multi-path CNN and pathfinder to select restoration routes for image regions, aiming to reduce computation while maintaining performance. It achieves comparable or superior performance on restoration tasks, particularly real-world denoising.

5 CONCLUSION

  • Path-Restore achieves comparable or superior performance to existing methods on varied restoration tasks with much fewer computations.The method is especially effective for real-world denoising, where noise varies across image regions.Source: §5 · p. 13
  • A multi-path CNN provides routes for different distortions, while a pathfinder selects a route for each image region.The pathfinder uses a difficulty-regulated reward that encourages short paths for regions that are easy to restore.Source: §5 · p. 13
  • The framework supports independent or joint processing of image regions, and region-based path selection may apply to other tasks where regions require different treatment.Source: §5 · p. 13

Related papers

Loading 1904.10343v2…