PaperlayerView exact revision on arXiv

Source-linked AI summary

FFDNet: Toward a Fast and Flexible Solution for CNN based Image Denoising

Kai Zhang, Wangmeng Zuo, Lei Zhang

arXiv:1710.04026v2cs.CV

TL;DR

Existing discriminative denoisers often require separate models for different noise levels and lack spatially variant-noise flexibility. FFDNet addresses this with a tunable noise level map and downsampled sub-image processing, and experiments report effective, efficient denoising across synthetic and real noisy images.

  • Problem

    Existing discriminative denoisers commonly require multiple noise-specific models and lack flexibility for spatially variant noise, limiting practical denoising applications.

  • Method

    FFDNet uses a tunable noise level map as input and processes downsampled sub-images to provide flexible denoising with efficient inference.

  • Results

    FFDNet achieves state-of-the-art results on synthetic AWGN, robustly controls noise reduction versus detail preservation, handles spatially variant AWGN, and produces perceptually appealing real-noise results.

  • Takeaways & Limitations

    A single FFDNet model supports varied noise levels and spatially variant noise, making it attractive for practical denoising applications.

Abstract

from arXiv · show

Due to the fast inference and good performance, discriminative learning methods have been widely studied in image denoising. However, these methods mostly learn a specific model for each noise level, and require multiple models for denoising images with different noise levels. They also lack flexibility to deal with spatially variant noise, limiting their applications in practical denoising. To address these issues, we present a fast and flexible denoising convolutional neural network, namely FFDNet, with a tunable noise level map as the input. The proposed FFDNet works on downsampled sub-images, achieving a good trade-off between inference speed and denoising performance. In contrast to the existing discriminative denoisers, FFDNet enjoys several desirable properties, including (i) the ability to handle a wide range of noise levels (i.e., [0, 75]) effectively with a single network, (ii) the ability to remove spatially variant noise by specifying a non-uniform noise level map, and (iii) faster speed than benchmark BM3D even on CPU without sacrificing denoising performance. Extensive experiments on synthetic and real noisy images are conducted to evaluate FFDNet in comparison with state-of-the-art denoisers. The results show that FFDNet is effective and efficient, making it highly attractive for practical denoising applications.

I. INTRODUCTION

Existing denoisers trade flexibility, efficiency, and robustness differently, motivating FFDNet as a single CNN controlled by a tunable noise level map. The paper evaluates this design on synthetic and real noisy images, including spatially variant noise.

  • Motivation: Image denoising is important for visual quality, Bayesian prior evaluation, and sequential image-restoration methods.Noise arises during sensing and can heavily degrade acquired images.
  • Motivation: Practical denoisers should use one model, remain efficient and effective, handle spatially variant noise, and control noise reduction versus detail preservation.These requirements apply when noise is known, estimated, or difficult to estimate.
  • Limitations of Existing Methods: Model-based methods such as BM3D and WNNM handle varied noise levels but are time-consuming and cannot directly remove spatially variant noise.The passage also notes their reliance on hand-crafted image priors.
  • Limitations of Existing Methods: Existing discriminative methods use fast CNN inference but generally require preset noise-specific models and lack flexibility for spatially variant noise.The cited preset range example is [0, 55].
  • Proposed Solution: FFDNet models the noise level map as an input while keeping network parameters invariant, enabling one network to handle different noise levels.This contrasts with DnCNN, whose parameters vary with noise level.
  • Proposed Solution: FFDNet uses a tunable noise level map to control the trade-off between noise reduction and detail preservation, while downsampled sub-images accelerate training and testing.Orthogonal initialization is adopted to alleviate visual degradation from overly large noise levels.
  • Evaluation: Experiments report superior denoising performance and computational efficiency against state-of-the-art methods, favorable spatially variant AWGN results, and perceptually convincing real-noise outputs.Compared methods include BM3D, WNNM, TNRD, and DnCNN.
  • Contributions: FFDNet is presented as a fast, flexible denoiser with perceptually appealing results on both synthetic AWGN and real-world noisy images.The authors identify practical image denoising as its potential application.

II. RELATED WORK

Prior work spans optimization-guided and plain discriminative denoisers, with CNNs achieving strong performance but existing discriminative models remaining limited in noise-level flexibility and spatially variant denoising.

  • Method Categories: Related methods divide into MAP inference-guided discriminative learning and plain discriminative learning.The distinction concerns whether optimization-inspired inference is incorporated into learning.
  • MAP-Guided Learning: MAP-guided methods offer efficiency and interpretability but are constrained by MAP model forms and generally underperform state-of-the-art CNN denoisers.Their architectures derive from optimization algorithms such as HQS and gradient descent.
  • Plain Discriminative Learning: Plain discriminative methods learn a direct mapping that implicitly models image priors rather than explicitly modeling them.MLPs and CNNs are examples of this category.
  • Plain Discriminative Learning: Early MLP and CNN denoisers could not compete with BM3D, while later plain MLP and DnCNN methods achieved comparable or state-of-the-art performance.DnCNN benefited from residual learning and batch normalization.
  • Open Problem: Existing discriminative methods still require multiple models for different noise levels and cannot handle spatially variant noise.The related-work discussion identifies a single flexible model as an unaddressed issue.

III. PROPOSED FAST AND FLEXIBLE DISCRIMINATIVE CNN DENOISER

FFDNet targets fast, flexible, and robust denoising with a tunable noise level map and reversible downsampling, using a CNN that processes four sub-images plus the map.

  • Design Objectives: FFDNet is designed around three objectives: fast speed, flexibility across noise conditions, and robustness in controlling denoising without visual artifacts.The flexibility objective includes spatially variant noise.
  • Noise-Level Conditioning: A tunable noise level map M is supplied as input so the model can handle different noise levels and spatially variant noise.The map also controls the balance between noise reduction and detail preservation.
  • Downsampling: Reversible downsampling reshapes a W × H × C image into four sub-images of size W/2 × H/2 × 4C before CNN processing.The operator improves efficiency while preserving the representation needed for reconstruction.
  • Network Architecture: The downsampled sub-images are concatenated with M to form the CNN input tensor of size W/2 × H/2 × (4C + 1).For spatially invariant AWGN, M is a uniform map whose elements equal σ.
  • Network Architecture: The CNN uses convolutional layers with ReLU and batch normalization, followed by upscaling as the reverse of downsampling.The first, middle, and final layers use Conv+ReLU, Conv+BN+ReLU, and Conv configurations, respectively.
  • Network Configuration: The architecture uses 15 convolution layers and 64 feature channels for grayscale images, versus 12 layers and 96 channels for color images.The settings are chosen empirically to balance complexity and performance.
  • Network Configuration: Different color-image settings exploit inter-channel dependencies and provide more feature channels, yielding an average gain of 0.15dB.The reported gain is attributed to the color-specific configuration.

B. Noise Level Map

FFDNet converts noise-level control into an input map, allowing a CNN to represent denoising across noise levels and spatially varying degradation. The map also preserves the model-based role of controlling noise reduction versus detail preservation.

  • Noise-level control: Model-based denoising uses σ in the data-fidelity term, while λ balances noise reduction against detail preservation.Too-small λ leaves noise, whereas too-large λ smooths details along with noise.
  • Noise-level map: CNN flexibility is obtained by stretching scalar σ into a noise level map M with the same spatial dimensions as the noisy image.This resolves the dimensional mismatch between image input y and scalar noise level σ.
  • Noise-level map: FFDNet models denoising as x = F(y, M; Θ), keeping Θ invariant while M specifies the noise level.This allows one network to handle different noise levels instead of changing network parameters with σ.
  • Generalized degradation maps: M can be extended to multichannel degradation maps, including multivariate Gaussian noise with covariance Σ in RGB space.A non-uniform M supports spatially variant noise within a single CNN model.

C. Denoising on Sub-images

FFDNet improves efficiency by denoising reversible downsampled sub-images, expanding the effective receptive field without requiring a deeper network. Experiments show a substantial speed gain while maintaining comparable denoising accuracy, though mismatched noise maps can produce artifacts.

  • Efficiency design: Dilated convolution was rejected because FFDNet empirically produced artifacts around sharp edges.The issue arises while seeking efficiency without sacrificing modeling capacity and receptive field.
  • Efficiency design: FFDNet applies a reversible downsampling layer and performs CNN denoising on small sub-images with downsampling factor 2.The operation reshapes the input into sub-images to improve computational efficiency.
  • Receptive field: A 15-layer FFDNet with 3 × 3 convolutions reaches a 62 × 62 receptive field, versus 31×31 for a plain 15-layer CNN.Downsampled processing expands the effective receptive field while permitting moderate network depth.
  • Experimental trade-off: Nearly 3 times faster, FFDNet is also more memory-friendly than the baseline CNN while maintaining denoising performance on BSD68.At σ = 15, the baseline leads by 0.02dB; at σ = 50, FFDNet leads by 0.09dB.
  • Noise-level-map robustness: A noise level map matched to the input avoids artifacts, whereas a much higher mismatched map may create visual artifacts.The map is intended to control the trade-off between smoothing details and reducing noise.

E. FFDNet vs. a Single Blind Model

FFDNet’s non-blind formulation differs from a single blind model in generalization, AWGN performance, and applicability to broader restoration algorithms. The supplied discussion reports advantages for the non-blind model while noting its design and training choices.

  • Generalization: A blind model can perform favorably on synthetic AWGN without the noise level but generalizes poorly to more complex real noisy images.The discussion attributes this boundary to the importance of accurately modeling the degradation process.
  • Generalization: The non-blind FFDNet model acts like multiple denoisers anchored to noise levels, enabling control of noise removal and detail preservation.This control facilitates removal of real noise to some extent in the reported comparison.
  • AWGN performance: About 0.1dB average gain on BSD68 is reported for non-blind AWGN removal over the blind model.The passage suggests the noise level map provides additional information to the input.
  • Application range: The non-blind model can be plugged into variable splitting algorithms for image deblurring, SISR, and image inpainting, unlike the blind model.These algorithms use a denoising subproblem with a current noise level.
  • Network design: FFDNet uses batch normalization for training acceleration and omits residual learning in its network design.The authors report that batch normalization accelerates training regardless of residual-learning strategy.

G. Un-clipping vs. Clipping of Noisy Images for Training

The paper distinguishes un-clipped AWGN training from clipping-based training, while evaluating both FFDNet variants across synthetic and real-image settings. Un-clipped training is the default, whereas FFDNet-Clip separately models quantized noisy images.

  • Training settings: Un-clipped noisy images preserve the ideal AWGN assumption used by most denoisers and are therefore the paper’s default training setting.The noisy input is not clipped or quantized into 8-bit values.
  • Training settings: Clipping makes synthetic noisy images more realistic because real noisy images are integer-valued and range-limited.At high noise levels, clipping can also make the noise non-zero-mean and reduce reliability in variable-splitting restoration algorithms.
  • Training settings: FFDNet-Clip is trained and tested with noisy images quantized into 8-bit format under the clipping setting.The quantized noisy image is generated from a clean image with MATLAB’s imnoise function.
  • Dataset generation: FFDNet training uses input-output pairs {(y_i, M_i; x_i)}, where y_i is formed by adding AWGN to latent image x_i and M_i is the noise-level map.The paper uses AWGN because it is appropriate without specific noise-source information and can approximate real-world noise locally.
  • Dataset generation: The training corpus combines 400 BSD images, 400 ImageNet validation images, and 4,744 Waterloo Exploration Database images.Each epoch randomly crops 128×8,000 patches, using 70×70 grayscale patches and 50×50 color patches with σ ∈ [0, 75].
  • Evaluation datasets: The experiments evaluate grayscale denoising on BSD68, Set12, and RNI6, and color denoising on CBSD68, Kodak24, McMaster, and RNI15.RNI6 contains six real noisy grayscale images without ground truth; the color experiments use four named datasets.

B. Experiments on AWGN Removal

On spatially invariant AWGN, FFDNet is compared with model-based and discriminative denoisers across grayscale and color benchmarks. It surpasses several grayscale baselines, performs best perceptually in the reported visual comparison, and consistently beats CBM3D for color images while competing with CDnCNN.

  • Grayscale AWGN: FFDNet is slightly inferior to DnCNN for σ ≤ 25 but gradually outperforms DnCNN when σ > 25.The reported explanation links this pattern to FFDNet’s larger receptive field and DnCNN’s greater modeling capacity.
  • Grayscale AWGN: FFDNet outperforms WNNM on “House” but is inferior on “Barbara,” whose repetitive structures favor WNNM’s nonlocal self-similarity prior.The visual comparison in Fig. 3 is reported to show the best perceptual quality for FFDNet overall.
  • Clipping setting: Under clipping, FFDNet-Clip performs better than DCGRF and RBDN on Clip300 when the noise level is matched.Its BSD68 PSNR is reported as similar to FFDNet trained with the un-clipping setting.
  • Color AWGN: For color denoising, FFDNet consistently outperforms CBM3D across CBSD68, Kodak24, and McMaster at different noise levels and has competing performance with CDnCNN.The comparison uses both quantitative results and visual comparisons.

C. Experiments on Spatially Variant AWGN Removal

FFDNet uses a downsampled noise-level map to denoise spatially variant AWGN, preserving performance when the map remains spatially smooth. A non-uniform map better balances strong-noise removal and detail preservation than a uniform map.

  • FFDNet receives a bilinearly downsampled noise-level map for spatially variant AWGN denoising.Because the map is spatially smooth, downsampling generally has little effect on final denoising performance.
  • The experiments compare FFDNet with methods on BSD68, CLIP300, and color-image datasets using average PSNR across multiple noise levels.The supplied table captions identify the datasets, PSNR metric, and tested noise levels but do not provide the table values.
  • A non-uniform noise-level map removes spatially variant AWGN more flexibly than a uniform map.Using the mean noise level leaves stronger noise insufficiently removed and can reduce detail preservation in lower-noise regions.
  • Figure 5 reports 30.08dB for FFDNet with the ground-truth noise map versus 27.45dB with a uniform mean-valued map.The noisy input in the example is reported at 20.55dB.

D. Experiments on Noise Level Sensitivity

Noise-level mismatch affects denoising quality and the trade-off between noise removal and detail preservation. FFDNet shows sensitivity behavior comparable to BM3D and DnCNN while allowing users to tune the input noise level.

  • Users may prefer a higher input noise level when the real noise level is uncertain, although this can remove image details with the noise.
  • FFDNet achieves denoising results similar to BM3D and DnCNN when their input noise levels match.
  • With a fixed input noise level, PSNR stays similar below the ground-truth noise level and decreases when the ground-truth level becomes higher.This behavior is reported for BM3D, DnCNN, and FFDNet.
  • The best visual quality occurs when the input noise level matches the ground-truth level.Higher input levels progressively smooth noise and image structures, with FFDNet potentially removing low-contrast line structures.
  • A slightly higher input noise level generally produces better visual results than a lower one, with little visual difference when the excess is small.The paper concludes that FFDNet has noise-level sensitivity similar to BM3D and DnCNN for balancing noise reduction and detail preservation.

E. Experiments on Real Noisy Images

FFDNet is evaluated on real noisy images whose noise is unknown and often non-Gaussian, signal-dependent, and spatially variant. Across grayscale, color, and challenging spatially variant examples, it produces visually appealing results and handles several non-AWGN noise types.

  • Real-image evaluation lacks known clean ground truth and noise level, while real noise often violates the AWGN assumption.The paper describes real noise as potentially non-Gaussian, spatially variant, and signal-dependent.
  • FFDNet and DnCNN remove structured noise in the grayscale “Building” example without losing underlying image textures.Noise Clinic and BM3D leave that structured noise unremoved, while blind DnCNN-B performs poorly on non-AWGN real noise.
  • FFDNet removes structured color noise in “Boy,” whereas CBM3D leaves it unremoved.The passage contrasts this with stronger performance of CDnCNN-B on AWGN-like noise such as “Frog.”
  • FFDNet handles JPEG lossy-compression noise and video noise in additional real-image examples.
  • A proper non-uniform map preserves low-noise-region details while removing strong noise in spatially variant real images.Uniform low noise leaves strong noise, whereas uniform high noise smooths details in low-noise regions.
  • FFDNet trained on unquantized image data performs well on 8-bit quantized real noisy images.

F. Running Time

FFDNet is designed for practical denoising with competitive performance, flexibility, and lower computational cost. Its multi-threaded CPU implementation is about three times faster than DnCNN and BM3D, while GPU speed also exceeds DnCNN.

  • The running-time evaluation covers BM3D, DnCNN, and FFDNet on grayscale and color images of three sizes.The experiments use 256×256, 512×512, and 1,024×1,024 images.
  • FFDNet takes nearly the same time on grayscale and color images, unlike BM3D, whose color processing requires additional chrominance computation.
  • FFDNet with multi-threaded CPU computation is about three times faster than DnCNN and BM3D.Even single-threaded FFDNet is faster than BM3D, and FFDNet is much faster than DnCNN on GPU.
  • The paper concludes that FFDNet provides a practical CNN denoising solution by combining flexibility, efficiency, and effectiveness.
Loading 1710.04026v2…