Source-linked AI summary
Deep Image Demosaicking using a Cascade of Convolutional Residual Denoising Networks
Filippos Kokkinos, Stamatios Lefkimmiatis
TL;DR
Joint demosaicking and denoising must recover missing image information from noisy sensor data, an ill-posed task in camera pipelines. This paper builds a network around majorization-minimization and denoising, and reports better reconstruction than the current CNN-based state of the art with fewer parameters and less training data.
Problem
Joint demosaicking and denoising is ill-posed because two-thirds of sensor information are missing and the rest are corrupted by noise.
Source: §1 · p. 1Method
Results
The network outperforms the current CNN-based state of the art in reconstruction quality while using fewer trainable parameters and a small fraction of the training data.
Source: §1 · p. 3Takeaways & Limitations
The approach generalizes well on small datasets and also surpasses state-of-the-art performance for Fuji X-Trans CFA data in linRGB and sRGB.
Source: §8 · p. 13, §7 · p. 13The initial observation model assumes i.i.d. Gaussian noise, a simplification; the network is also evaluated with noise statistics that better approximate real noisy conditions.
Source: §2 · p. 4
Abstract
from arXiv · showhide
Demosaicking and denoising are among the most crucial steps of modern digital camera pipelines and their joint treatment is a highly ill-posed inverse problem where at-least two-thirds of the information are missing and the rest are corrupted by noise. This poses a great challenge in obtaining meaningful reconstructions and a special care for the efficient treatment of the problem is required. While there are several machine learning approaches that have been recently introduced to deal with joint image demosaicking-denoising, in this work we propose a novel deep learning architecture which is inspired by powerful classical image regularization methods and large-scale convex optimization techniques. Consequently, our derived network is more transparent and has a clear interpretation compared to alternative competitive deep learning approaches. Our extensive experiments demonstrate that our network outperforms any previous approaches on both noisy and noise-free data. This improvement in reconstruction quality is attributed to the principled way we design our network architecture, which also requires fewer trainable parameters than the current state-of-the-art deep network solution. Finally, we show that our network has the ability to generalize well even when it is trained on small datasets, while keeping the overall number of trainable parameters low.
1 Introduction
Joint demosaicking and denoising is difficult because much sensor information is missing and the remaining data are noisy, while many demosaicking methods assume a specific CFA pattern. The paper introduces a learned joint approach and reports improved reconstruction with fewer parameters and less training data than the current CNN-based state of the art.
1 Introduction
- Joint recovery is ill-posed: two-thirds of the underlying data are missing and the remainder are perturbed by noise.Errors at this early stage can lead to unsatisfying final camera images.Source: §1 · p. 1
- Many existing demosaicking methods assume a specific Bayer pattern, limiting applicability to cameras with other CFA patterns.Methods that generalize across CFA patterns are preferred.Source: §1 · p. 2
- The MSR dataset provides realistic reference images and includes Bayer and Fuji X-Trans CFA patterns for training and evaluation.Images are demosaicked in linRGB before conversion to sRGB.Source: §1 · p. 2
- Prior work found that joint denoising and demosaicking yields better results, motivating the paper’s joint treatment of noisy raw sensor data.Sensor data are affected by shot noise and read noise, which can under certain circumstances be approximated by heteroscedastic Gaussian noise.Source: §1 · pp. 2–3
- The proposed network outperforms the current CNN-based state of the art while using fewer trainable parameters and only a small fraction of the training data.The authors report higher-quality reconstruction in both linRGB and sRGB.Source: §1 · p. 3
2 Problem Formulation
The paper models raw sensor measurements as a degraded version of an unknown image plus noise, with the degradation operator encoding the CFA pattern. Because the operator is singular and the data are noisy, recovery is ill-posed; the formulation separates efficient optimization from learning an appropriate regularizer.
2 Problem Formulation
- The observation model relates raw measurements y to the unknown image x through degradation matrix M, which represents the device’s spatial response and CFA pattern.Zero diagonal entries in M mark missing spatial and channel color information.Source: §2 · p. 3
- The model assumes i.i.d. Gaussian measurement noise as a starting point, while later training and evaluation use noise statistics that better approximate real noisy conditions.The text explicitly calls the Gaussian assumption a simplification.Source: §2 · p. 4
- Because M is singular and measurements are noisy, recovering x is ill-posed and does not have a unique solution.The paper frames restoration as a MAP estimation problem and an equivalent objective minimization.Source: §2 · p. 4
- The formulation requires both an efficient minimization algorithm and a suitable regularizer, whose form the paper proposes to infer indirectly from training data.A non-quadratic regularizer prevents solving the problem simply as a system of linear equations.Source: §2 · pp. 4–5
3 Majorization-Minimization Framework
The paper uses majorization-minimization to replace a difficult objective with simpler surrogate minimizations that upper-bound it. Its key architectural insight is that the resulting subproblem is a denoising problem, allowing a denoising network to perform each iteration without explicitly choosing the regularizer.
3 Majorization-Minimization Framework
- Majorization-minimization avoids extra tuning parameters required by some splitting methods by iteratively minimizing simpler surrogate functions that upper-bound the original objective.The paper notes that these surrogates are simpler to handle than the original objective.Source: §3 · p. 5
- A majorizer touches the objective at the current iterate and lies above it elsewhere, so minimizing it produces an iterate that decreases the original objective.The surrogate need not be minimized exactly for the objective to decrease.Source: §3 · p. 5
- Choosing α greater than 1 makes the constructed distance term positive definite and yields the paper’s valid majorizer.This follows from M being a binary diagonal, idempotent matrix with maximum eigenvalue 1.Source: §3 · p. 6
- The surrogate decouples M from x and becomes a denoising problem, so a denoising network can compute each iteration and implicitly learn the regularizer from data.Its noisy input z has noise variance σ²/α.Source: §3 · p. 6
4 Residual Denoising Network (ResDNet)
ResDNet serves as the denoising solver for the paper’s surrogate optimization problem. It estimates noise from a distorted image while conditioning on the noise variance, then subtracts the estimate from the input.
4 Residual Denoising Network (ResDNet)
- ResDNet predicts the noise realization in its input and subtracts that estimate to produce a denoised image.The network is designed to solve the denoising surrogate in the overall method.Source: §4 · p. 6
- Providing noise variance as a second input lets one parameter set handle a wide range of noise levels.The paper contrasts this with training a separate non-blind network for each noise level.Source: §4 · p. 7
- ResDNet uses five fundamental blocks, including convolutional layers, PReLU nonlinear blocks with shortcuts, and a transposed convolution reducing channels to three.The nonlinear stage contains D × 2 blocks, with a shortcut connection every second block.Source: §4 · p. 7
- The implementation uses reflexive padding and omits batch normalization; its convolutional filters are constrained to have zero mean and norm s.The filters are parameterized so their ℓ2-norm equals the trainable scalar s.Source: §4 · p. 8
- A projection layer uses the noise variance to normalize the estimated noise before subtraction, and the output is clipped to [0, 255].Clipping enforces the prior that valid pixel intensities lie in that range.Source: §4 · p. 8
5 Demosaicking Network Architecture
The demosaicking network unrolls an MM procedure, using a denoiser to solve each iteration’s surrogate problem. It shares denoiser parameters across iterations and learns acceleration and noise-level settings.
5 Demosaicking Network Architecture
- Each unrolled step combines an extrapolated estimate with the measured CFA data before applying ResDNet.The update is x^(i+1) = ResDNet((I − M)u + y, σ_i), with ResDNet parameters shared across iterations.Source: §5 · pp. 8–9
- The same ResDNet parameters are used at all K iterations, keeping the overall parameter count low.The method uses a denoiser that accepts a wide range of noise levels.Source: §5 · p. 9
- Trainable extrapolation weights address the MM framework’s potentially slow convergence.The weights are learned from data rather than fixed to the values originally suggested by Beck and Teboulle.Source: §5 · p. 9
- A continuation strategy starts with larger noise levels and gradually decreases them toward the target value.The trainable noise vector is initialized evenly on a log scale from σ_max to σ_min, then fine-tuned by back-propagation.Source: §5 · p. 9
- Unlike P3 and RED, the method requires no user-tuned λ and uses a trainable universal denoiser within its iterative network.The cited approaches use static denoisers, whereas this method can train ResDNet on available training data.Source: §5 · p. 10
6 Network Training
The denoiser is first trained for image denoising, then the full network is trained end-to-end for joint denoising and demosaicking. Joint training uses shared iteration weights and backpropagation through time.
6.1 Image Denoising
- The ResDNet pretraining uses 400 BSDS images for training and 100 for validation, with patches corrupted by noise σ ∈ [0, 15].The denoiser has depth D = 5 and is optimized for mean squared error.Source: §6.1 · p. 10
6.2 Joint Denoising and Demosaicking
- Joint training fine-tunes the pretrained denoiser end-to-end with averaged L1 loss and backpropagation through time because iteration weights are shared.The loss compares rasterized inputs and ground-truth images to the network output.Source: §6.2 · p. 10
- Joint training uses batch size 4, K = 10 steps, and noise levels initialized from σ_max = 15 to σ_min = 1.Training takes 3 hours on the MSR Demosaicking Dataset and 5 days on a small MIT Demosaicking Dataset subset.Source: §6.2 · pp. 10–11
- Table 1 compares methods by PSNR in the demosaick-only scenario; the MIT result uses a 40,000-image subset of 2.6 million images.The Kodak dataset is resized to 512×768 for evaluation.Source: Table 1 · p. 11
7 Experiments
Experiments report strong reconstruction performance on noise-free and noisy data, including with limited training data and the non-Bayer Fuji X-Trans pattern. The network also uses fewer parameters than the compared state-of-the-art system and has a reported runtime of 0.05 seconds for a 220 × 132 color image.
7 Experiments
- 40,000 MIT images yielded equal and often better performance than trained on 2.6 million images in the noise-free comparison.The MIT dataset contains only noise-free samples; the system trained on 40,000 images because of the small-batch-size constraint.Source: §7 · pp. 11–12, Table 2 · p. 12
- On noisy data, training on 200 MSR images generalized better than the state-of-the-art system.The model was optimized in linRGB and evaluated in both linRGB and sRGB.Source: Table 2 · p. 12, §7 · p. 12
- 380,356 trainable parameters compare with 559,776 for the current state-of-the-art system.Source: §7 · p. 13
- 0.05 seconds are required for a 220 × 132 color image on an Nvidia GTX 1080Ti, with runtime scaling linearly by image size.Source: §7 · p. 13
- For Fuji X-Trans, the network surpassed state-of-the-art performance in both linRGB and sRGB.The evaluation used the non-Bayer X-Trans pattern.Source: Table 3 · p. 13, §7 · p. 13
8 Conclusion
The conclusion presents a high-quality joint denoising and demosaicking system that compares favorably with the current state-of-the-art network while generalizing from small datasets with relatively few parameters. The accompanying figures show refinement across network steps and compare noisy-image reconstructions.
8 Conclusion
- The system produces high-quality joint denoising and demosaicking results, outperforming the current state-of-the-art network quantitatively and qualitatively while generalizing from small datasets with fewer parameters.Source: §8 · p. 13
- Step 1 gives a rough approximation, while Steps 3 and 10 show progressively more refined reconstructions.Source: Figure 2 · p. 14
- On noisy MSR images, the network removes noise while retaining fine detail, whereas competing methods either fail to denoise or oversmooth.Source: Figure 3 · p. 14