Source-linked AI summary
Burst Denoising with Kernel Prediction Networks
Ben Mildenhall, Jonathan T. Barron, Jiawen Chen, Dillon Sharlet, Ren Ng, Robert Carroll
TL;DR
Handheld burst denoising must combine multiple noisy frames despite motion and difficult ground-truth acquisition. The paper uses a kernel-prediction CNN trained on physically modeled synthetic data and optimized with an annealed loss. It matches or outperforms state-of-the-art methods on synthetic and real data across noise levels, with evaluation presented for grayscale raw-derived images.
Problem
Handheld burst denoising must handle scene and camera motion, while real noisy and noise-free burst ground-truth pairs are impractical to collect at neural-network training scale.
Method
A convolutional network predicts spatially varying kernels that align and denoise burst frames, using physically modeled synthetic data and an annealed loss during optimization.
Results
The model matches or outperforms the state-of-the-art on both synthetic and real datasets across noise levels.
Takeaways & Limitations
Kernel prediction provides a burst-denoising approach that uses multi-image information while addressing small unknown misalignments.
Takeaways & Limitations
The noise estimate supplied to the network is necessarily approximate because it substitutes observed intensity for true intensity, and experiments use grayscale images derived from Bayer data.
Abstract
from arXiv · showhide
We present a technique for jointly denoising bursts of images taken from a handheld camera. In particular, we propose a convolutional neural network architecture for predicting spatially varying kernels that can both align and denoise frames, a synthetic data generation approach based on a realistic noise formation model, and an optimization guided by an annealed loss function to avoid undesirable local minima. Our model matches or outperforms the state-of-the-art across a wide range of noise levels on both real and synthetic data.
1. Introduction
Burst denoising combines multiple noisy frames, but handheld motion makes simple temporal averaging unreliable. The paper proposes a CNN-based approach with synthetic training data and kernel prediction to retain multi-image denoising benefits while handling misalignment.
- Handheld-camera bursts are difficult to denoise because scene and camera motion makes simple temporal averaging unreliable.
- The method combines multi-image denoising benefits with the capacity and generality of convolutional neural networks.
- Synthetic training data converts post-processed internet images into data resembling raw linear camera captures, avoiding the need for camera-captured ground-truth pairs.
- The network predicts a unique 3D denoising kernel for each output pixel, improving performance over direct pixel synthesis and exposing how burst frames are used.
- Providing the noise level during training and testing lets the network generalize across a wider noise range than a blind denoising network.
2. Related work
Prior work spans classical single- and multi-image denoising, alignment and robust averaging for mobile bursts, and neural operators for related restoration tasks. Kernel-predicting networks provide a related strategy by constraining outputs through per-pixel filters.
- Classical methods recover clean signals using analytical priors, while multi-image methods average similar patches across time.
- The paper’s qualitative comparison contrasts noisy reference frames, burst averaging, and the proposed method on handheld low-light bursts.
- Mobile burst methods typically align image patches within a few pixels before jointly denoising them through robust averaging.
- Neural networks have been applied to multi-image denoising, demosaicking, deblurring, and superresolution.
- Kernel-Predicting Networks generate a filter for every desired output pixel, constraining the output space and thereby preventing artifacts.
3. Problem specification
The task is to produce one clean reference image from a noisy handheld burst while accounting for raw-sensor noise, motion, and imperfect correspondence. Synthetic bursts model small and severe misalignments alongside shot and read noise derived from camera behavior.
- The goal is to denoise a selected reference image X1 using alternate burst frames X2, . . . , XN in the raw linear domain.
- Raw sensor noise combines signal-dependent shot noise with approximately Gaussian read noise, whose parameters can vary across images as sensor gain changes.
- Handheld bursts contain global hand motion, local scene motion, and possible disocclusions that can make accurate correspondence impossible.
- Synthetic training data is used because real noisy and noise-free burst pairs are impractical to collect at the scale required by deep networks.
- Synthetic bursts include alternate-frame translations of up to ±2 pixels, while separate random shifts of up to ±16 pixels simulate complete alignment failure.
- Training samples shot and read factors from ranges observed in real data after intensity scaling and approximate linearization of source images.
4. Model
The KPN predicts spatially varying per-pixel kernels that jointly align, average, and denoise burst frames. An annealed intermediate loss helps train the network to use alternate frames before learning their final relative weighting.
- Kernel prediction architecture: The model generates per-pixel filter kernels that jointly align, average, and denoise the burst into a clean reference frame.It combines kernel-prediction ideas from video interpolation and Monte Carlo denoising.
- Kernel prediction architecture: An encoder-decoder produces K^2N channels, reshaped into N spatially varying K×K filters at each pixel.The filters are applied to corresponding image neighborhoods, then averaged across time; experiments use K = 5 and N = 8.
- Noise-aware input: The network receives the raw burst plus a per-pixel noise estimate based on the observed intensity and known read and shot noise parameters.The estimate is approximate because observed intensity substitutes for the unknown true intensity.
- Kernel prediction architecture: Predicted kernels may contain negative values because the model omits softmax normalization, which otherwise produced unstable training gradients.This differs from the normalization used by Bako et al.
- Basic loss function: The basic loss combines L2 intensity error with L1 gradient error after white-level restoration and sRGB gamma correction.The transfer function makes the loss more perceptually relevant and avoids optimization problems associated with straightforward gamma correction.
- Annealed loss term: Annealing adds image-space losses for each individually filtered frame, initially encouraging alignment and denoising before optimizing the full burst output.With β = 100 and α = .9998, the additional term is phased out around t = 40,000.
- Annealed loss term: After annealing, the network retains alternate-frame alignment and reweights filters so well-aligned frames contribute strongly while poorly aligned frames are ignored.This behavior remains after the individual-frame constraint becomes negligible.
- Implementation: At test time, the implementation processes about 0.7 megapixels/sec on an NVIDIA GTX 1080 Ti.Training uses Adam for one million iterations on an NVIDIA K40 GPU.
5. Experiments
Experiments evaluate the model on synthetic and real bursts, testing its noise generalization, kernel behavior, annealed loss, and robustness to motion. Across these settings, the method preserves detail, matches strong baselines, and adapts its frame weighting to noise and scene content.
- The evaluation combines synthetic test sets, ablations, and qualitative comparisons on real mobile-phone bursts.The experiments assess the annealed loss, noise model, kernel-prediction architecture, and generalization to real data.
- Synthetic test set: The direct synthesis ablation tends to produce oversmoothed results and is favorable only under the highest-noise conditions.Figure 4 illustrates oversmoothing from both VBM4D and the direct synthesis network on the synthetic test set.
- Synthetic test set: The model matches VBM4D across all tested noise levels on additive white Gaussian noise using both PSNR and SSIM.This test uses gamma-corrected synthetic data, a setting for which VBM4D is specifically designed.
- Generalization: Predicted spatially varying kernels reject large scene motions while combining frames for static backgrounds and relying more heavily on the reference frame for moving subjects.On real bursts, the method recovers detail under significant noise without producing artifacts from large scene motion.
- Ablations: Annealing helps the network escape a local minimum that ignores alternate frames and learn shifted kernels for ±2 pixel training misalignments.With the reported settings, the annealing schedule effectively reaches zero after 3–5% of training iterations.
- Generalization: Noise-aware KPN generalizes beyond its training noise range better than the blind variant and can tune denoising strength through its noise-level input.As σ′ increases, the network automatically reweights filters to use more information from alternate frames.
6. Conclusion
The paper presents a learning-based method for jointly denoising handheld-camera image bursts, using physical-model-based synthetic training data and an annealed loss. The method outperforms the state of the art on synthetic and real datasets.
- The method jointly denoises bursts captured by handheld cameras using a learning-based approach.
- The training data are synthesized from a physical image formation model to support training on real images.
- An annealed loss function, guided by how kernels handle motion, is a key component of successful kernel prediction network training.
- The model outperforms the state of the art on both synthetic and real datasets.
A. Derivation of noise model parameters
The noise model expresses measured output and variance in terms of analog gain, digital gain, photoelectron count, and read noise. Read noise is fixed, while the camera controls the analog and digital gains.
- Analog gain ga scales the initial photoelectron count q before digital gain gd is applied.
- Measured output is z = gdgaq, with variance (gdga)z + (gdr)2.
- Read noise r is fixed, whereas analog and digital gains are controlled by the camera.
B.1. VBM4D
The VBM4D comparisons estimate one noise level for each burst and evaluate the method under different synthetic data and preprocessing settings. Parameter sweeps select k = 1 for linear data and gamma-corrected PSNR evaluation.
- VBM4D comparisons estimate a single noise level for each burst, with p varying over all pixels in the reference frame.
- For the linear synthetic test set, all methods receive data with added shot and read noise.
- Gamma-corrected PSNR is better when VBM4D runs on original linear data rather than gamma-corrected data first.
- k = 1 produces the best VBM4D results in a sweep over kσrms for k ∈ [.5, 1, 2, 3].
- Table 2 uses gamma-corrected color data with added constant-variance Gaussian noise, a setting intended to suit VBM4D.
B.2. Nonlocal means
The NLM comparisons tune its single noise parameter by sweeping multipliers of σrms and select k = 2 as the best setting.
- k = 2 produces the best NLM results among multipliers kσrms with k ∈ [.5, 1, 2, 3, 4].
B.3. HDR+
HDR+ merging uses a single free parameter, c, with c ≈102.5 performing best on average for linear data in [0, 1].
- B.3. HDR+: HDR+ merging uses one free parameter, c.The parameter is referenced in equation 7 in.
- B.3. HDR+: c ≈102.5 works best on average for the linear data tested.
- B.3. HDR+: The reported data lie in the interval [0, 1].