PaperlayerView exact revision on arXiv

Source-linked AI summary

Connecting Image Denoising and High-Level Vision Tasks via Deep Learning

Ding Liu, Bihan Wen, Jianbo Jiao, Xianming Liu, Zhangyang Wang, Thomas S. Huang

arXiv:1809.01826v1cs.CV

TL;DR

The paper addresses the fragile separation between image denoising and high-level vision by connecting them in a unified deep-learning framework. It combines a multiscale denoiser with a cascaded high-level task network and joint-loss training, reporting improved denoising quality, high-level-task performance, and cross-task robustness.

  • Problem

    Image denoising and high-level vision are conventionally handled separately, while their mutual influence and the use of semantics to guide denoising remain insufficiently studied.

  • Method

    The paper proposes a multiscale denoising CNN and cascades it with a high-level vision network, using joint-loss back-propagation to update only the denoiser.

  • Results

    The cascaded joint-loss network improves denoised-image perceptual quality and high-level vision accuracy, while the trained denoiser generalizes across tasks without fine-tuning.

  • Takeaways & Limitations

    Image semantics can be exploited simultaneously for image denoising and high-level vision within a feasible and robust unified framework.

Abstract

from arXiv · show

Image denoising and high-level vision tasks are usually handled independently in the conventional practice of computer vision, and their connection is fragile. In this paper, we cope with the two jointly and explore the mutual influence between them with the focus on two questions, namely (1) how image denoising can help improving high-level vision tasks, and (2) how the semantic information from high-level vision tasks can be used to guide image denoising. First for image denoising we propose a convolutional neural network in which convolutions are conducted in various spatial resolutions via downsampling and upsampling operations in order to fuse and exploit contextual information on different scales. Second we propose a deep neural network solution that cascades two modules for image denoising and various high-level tasks, respectively, and use the joint loss for updating only the denoising network via back-propagation. We experimentally show that on one hand, the proposed denoiser has the generality to overcome the performance degradation of different high-level vision tasks. On the other hand, with the guidance of high-level vision information, the denoising network produces more visually appealing results. Extensive experiments demonstrate the benefit of exploiting image semantics simultaneously for image denoising and high-level vision tasks via deep learning. The code is available online: https://github.com/Ding-Liu/DeepDenoising

I. INTRODUCTION

The paper unifies image denoising and high-level vision by studying how denoising affects semantic tasks and how semantic information can guide denoising. It proposes multiscale denoising and cascaded joint training, reporting improved denoising quality, high-level-task accuracy, and cross-task robustness.

  • Conventional computer vision separates low-level image processing from high-level vision, leaving their mutual influence insufficiently studied.
  • The proposed denoising CNN processes features at multiple spatial scales through downsampling and upsampling, then fuses them to exploit broader context.Downsampling also reduces feature-map sizes and computation cost.
  • The cascaded framework connects an image-denoising module with a high-level vision module and updates only the denoiser using gradients from a joint loss.Keeping the high-level network untouched is intended to support denoiser generalization across tasks while preserving semantic details.
  • Joint training improves denoised-image perceptual quality and high-level vision accuracy while making a denoiser trained for one task usable with other tasks without fine-tuning.The paper reports consistent performance for noisy and noiseless images and reduced training effort across tasks.
  • The paper presents a unified deep-learning framework for simultaneously exploiting image semantics in denoising and high-level vision tasks.The authors identify this as a first attempt and support the framework with extensive experiments, including real-noise removal and subjective visual-quality evaluation.

II. RELATED WORK

Related work progresses from classical local and non-local denoising priors to deep neural networks, while the paper positions its contribution as a unified denoising and high-level-vision framework.

  • Classical denoising methods recover clean signals by exploiting local structures, non-local patch similarities, or both.Examples include coefficient shrinkage methods and globally grouped similar patches with structural priors.
  • Deep-learning denoising methods include multilayer perceptron and convolutional neural network approaches.The literature includes stacked denoising autoencoders, plain MLPs, CNNs, unfolded iterative algorithms, skip-connected networks, and dilated convolutions.
  • The proposed framework first introduces its denoising network and then cascades it with a high-level vision module using a joint-loss training strategy.

A. Denoising Network

The proposed denoising network uses nested multiscale feature contraction and expansion to combine spatial context, recover details, and produce reconstructed images of arbitrary size.

  • The network takes a noisy image as input and outputs a reconstructed image through feature contraction and expansion using downsampling and upsampling.
  • Each downsampling–upsampling pair creates a new spatial scale, enabling the network to process information across multiple resolutions.Nested pairs build deeper networks with more spatial scales and generally better restoration performance.
  • A long-distance skip connection sums the learned residual with the input image to generate the reconstructed image and recover details.
  • Feature encoding operates separately at each scale, while feature decoding fuses representations from adjacent scales.The paper tests concatenation and element-wise summation, finding similar denoising performance before choosing concatenation to accommodate differing channel numbers.
  • Downsampling progressively enlarges convolutional receptive fields and reduces computation by decreasing feature-map sizes.The larger receptive field incorporates broader spatial context for denoising.
  • Spatially invariant operations allow the denoising network to handle input images of arbitrary size.

B. When Image Denoising Meets High-Level Vision Tasks

The paper cascades denoising and high-level vision networks so semantic-task information guides denoising while the denoised output supports downstream vision tasks.

  • A noisy image passes first through the denoising network, whose output is then processed by a high-level vision network.
  • The cascade aims to reconstruct visually pleasing images guided by high-level vision information while maintaining accuracy across various tasks after training on one task.
  • Training Strategy: Training fixes the high-level vision network and updates only the denoising network using error back-propagated through the cascade.
  • Loss: The reconstruction loss is mean squared error between the denoising output and the noiseless reference image.
  • Loss: Perceptual loss compares feature representations of the reconstructed and reference images, using early layers of the high-level vision network as the extractor.
  • Loss: The joint loss combines reconstruction, perceptual, and high-level-task losses as a weighted sum.

IV. EXPERIMENTS

The experiments first evaluate the proposed denoising network, then examine its relationship with high-level vision networks and extend it to real noise removal.

  • The experimental section evaluates the proposed denoising network, studies its relation to high-level vision networks, and considers real noise removal.

A. Image Denoising

The denoising network reconstructs RGB images from Gaussian-noisy inputs, while controlled architecture experiments select a multiscale design with a global skip connection.

  • The denoiser takes RGB images and directly outputs reconstructed images after Gaussian noise with zero mean is added during training.
  • Training uses 432 Berkeley segmentation dataset color images, 48×48 patches, SGD with batch size 32, and 1,500,000 iterations.
  • Architecture experiments vary spatial scales, encoding and decoding modules, and convolutional kernel configurations to analyze their contributions to denoising performance.
  • Higher computation cost from more spatial scales and parameters improves performance, while a global skip connection further improves results.
  • The selected architecture uses three spatial scales, a global skip connection, and encoding kernels (128, 32, 32, 128) with decoding kernels (256, 64, 64, 256).
  • Table I reports average PSNR in dB and model-parameter counts for denoising models evaluated on Kodak images with σ = 35.

2) Comparisons with State-of-the-Art Methods:

The proposed denoiser is evaluated against established color image denoising methods across noise levels and datasets, with quantitative results favoring the proposed method. Comparisons also include visual examples and high-level-task settings.

  • Denoising comparisons: The evaluation compares CBM3D, MCWNNM, DnCNN, FFDNet, and the proposed method at noise levels σ = 25, 35, and 50 on Kodak and CBSD68.PSNR is measured on Kodak whole images and CBSD68 images.
  • Quantitative results: The proposed method outperforms all competing approaches quantitatively across different noise levels.It achieves the highest PSNR in almost every Kodak image and the highest average PSNR on CBSD68.
  • High-level vision evaluation: The study uses VGG-16 for classification and DeepLab-LargeFOV for semantic segmentation in the high-level vision experiments.The experiments follow the preprocessing protocols of the corresponding networks.
  • Visual comparisons: Visual denoising examples compare the ground truth, noisy image, CBM3D, DnCNN, and proposed-model outputs with and without high-level guidance.The examples use Kodak images with noise level 50.

1) High-Level Vision Information Guided Image Denoising:

High-level vision guidance changes denoising toward preserving semantically important details rather than optimizing reconstruction error alone. The paper evaluates this effect through gradient visualizations, visual comparisons, and subjective quality ratings.

  • Evaluation rationale: PSNR may correlate poorly with human visual quality, so the experiments emphasize qualitative differences between joint-loss and MSE-trained denoisers.The authors state that MSE training should outperform the joint loss on PSNR, making PSNR an unsuitable sole objective here.
  • Joint-loss guidance: The cascade combines image reconstruction, perceptual, and classification losses to guide the denoising network.Gradient visualizations analyze each loss component separately on ILSVRC2012 validation images.
  • Gradient analysis: Classification-loss gradients concentrate on class-specific, texture-rich regions that conventional denoising can oversmooth, while perceptual-loss gradients focus on local features.Reconstruction-loss gradients are spatially distributed because the reconstruction loss is not spatially biased.
  • Gradient analysis: Joint-loss training denoises class-specific regions differently from the rest of the image, reflecting complementary contributions from the three losses.The classification signal provides auxiliary supervision for preserving semantic details.
  • Visual quality: On Kodak examples, jointly trained outputs preserve more details and textures with better visual quality than CBM3D, DnCNN, and the separately trained denoiser.The comparison uses zoom-in regions from images with noise level 50.
  • Subjective evaluation: A 15-rater MOS test compares four denoising methods, and the high-level-guidance method achieves the highest score.Scores range from 1 for low image quality to 4 for high image quality.

2) Generality of the Denoiser for High-Level Vision Tasks:

The denoiser improves downstream classification and segmentation on noisy images when trained jointly with high-level vision networks. Cross-task tests indicate that the learned denoising generalizes across different high-level tasks.

  • Evaluation settings: The evaluation compares direct VGG processing, CBM3D+VGG, Separate+VGG, Joint Training, and Joint Training (Cross-Task) on noisy images.Classification uses ILSVRC2012, while segmentation uses Pascal VOC 2012.
  • Training assumption: The high-level vision network weights are initialized from noiseless-setting models and remain fixed during denoiser training.This setting isolates updates to the denoising module.
  • Downstream performance: The baseline VGG approach performs much worse than the other cases, while denoising without high-level semantics can still introduce artifacts that reduce accuracy.The proposed Joint Training approach maintains sufficiently high accuracy across noise levels.
  • Cross-task generality: A denoiser jointly trained for segmentation improves classification when connected cross-task, and a classification-trained denoiser improves segmentation mean IoU over Separate+VGG.Cross-task performance is comparable to task-dedicated Joint Training in classification and better than Separate+VGG in segmentation.
  • Semantic segmentation: The separately trained denoiser produces lower segmentation accuracy than joint-loss approaches and shows oversmoothing artifacts in inaccurate regions.Joint Training and Joint Training (Cross-Task) achieve finer segmentation results in the visual examples.

C. Extension to Real Noise Removal

The method is additionally tested on realistic noise from the Darmstadt Noise Dataset, whose corruption is more complicated than i.i.d. Gaussian noise. It preserves local structure better than the compared methods in the reported examples.

  • Real-noise evaluation: On real noisy DND images, the proposed method preserves more local edges and high-frequency components than CBM3D and DnCNN.The compared methods are prone to generate oversmoothing regions.
  • Real-noise evaluation: The authors report that the real-noise results demonstrate robustness to the noise statistical distribution and good generalization to realistic corruption.DND images are captured at higher ISO and contain more complicated corruption than i.i.d. Gaussian noise.

V. CONCLUSION

The paper presents a unified, practical framework connecting low-level image processing with high-level semantic tasks through information flow between them. It reports improved denoising and high-level task performance, while identifying broader semantic guidance and additional low-level applications as future directions.

  • V. CONCLUSION: High-level semantic information flows back to low-level image processing, enabling joint handling of denoising and multiple high-level vision tasks.The authors describe this as a simple and efficient deep learning solution.
  • V. CONCLUSION: The joint approach achieves superior performance in both image denoising and various high-level vision tasks.
  • V. CONCLUSION: A denoiser trained for one high-level vision task also shows robustness to other high-level vision tasks.
  • V. CONCLUSION: Future work will extend high-level semantic guidance to tasks such as super-resolution and incorporate more types of semantical information.
Loading 1809.01826v1…