Deep Burst Denoising

1. Introduction

• Smartphone apertures, photosites, and A/D conversion are compromised relative to larger cameras, making noise a greater problem in smartphone images.Source: §1 · p. 1
• Burst capture shifts light integration from hardware to computation and is less likely than long exposures to blur moving subjects or blow out highlights.The authors describe burst photography as available on inexpensive hardware and useful for capturing moving subjects.Source: §1 · p. 2
• The computational pipeline stabilizes burst frames, denoises each frame with a fully convolutional CNN, then recurrently integrates information across frames.Source: §1 · p. 2
• The authors report state-of-the-art burst-dataset denoising, performance on par with or better than recent multi-frame methods, and generalization to super-resolution.Source: §1 · p. 2

2. Related work

• Image restoration maps corrupted observations to a restored target; the paper treats denoising and super-resolution as examples of these inverse problems.Source: §2 · p. 2

Denoising

• Single-image denoising ranges from block-statistics methods such as Non-Local Means and BM3D to feed-forward, recurrent, and hybrid learning methods.Source: Denoising · p. 2
• Multi-image denoisers combine aligned burst observations through block-based methods, VBM3D/VBM4D, homography-flow compositing, or end-to-end optimization systems.FlexISP and ProxImaL are examples of end-to-end pipeline formulations.Source: Denoising · p. 2

Super-Resolution

• Super-resolution produces a higher-resolution image from one or more fixed-resolution inputs, and prior work includes both single-image and multi-image methods.The cited literature includes convolutional single-image models and methods for multi-image super-resolution.Source: Super-Resolution · p. 3

2.1. Neural Architectures

• Recurrent networks can expand effective depth in single-image models or integrate observations across frames; this work uses the latter principle.Source: §2.1 · p. 3
• Rather than use skip connections directly, the method feeds activations from each network level into corresponding RNNs to integrate burst information.Source: §2.1 · p. 3

3.1. Goals

• The design goals include arbitrary frame counts, single-frame competitiveness, motion robustness, denoising every frame, temporal coherence, and transfer to other restoration tasks.Source: §3.1 · p. 3

3.2. Single frame denoising

• The SFD predicts a pixel-aligned denoised image from a noisy input and is trained by minimizing L1 distance to the clean target.The base architecture uses 8 layers, 64 channels, 3×3 convolutions, and ReLU activations except at the last layer.Source: §3.2 · pp. 3–4

3.3. Multi-frame denoising

• The multi-frame model uses recurrent connections to aggregate each frame’s SFD activations, supporting arbitrary sequence lengths in a two-track design.The upper track performs single-frame denoising; the lower track fuses its results for multi-frame prediction.Source: §3.3 · p. 4, Figure 2 · p. 4
• Decoupling per-frame feature extraction from recurrent aggregation enables SFD pretraining, which accelerates learning and yields better PSNR than training the full MFD from scratch.The SFD weights remain trainable during MFD training rather than being frozen.Source: §3.3 · p. 4
• The recurrent design aggregates information both across frames and laterally across physical scales and abstraction levels.Source: §3.3 · p. 4

4. Implementation and Results

• Evaluation covers single-image denoising, multi-frame denoising, and single-image super-resolution.Source: §4 · p. 4

4.1. Data

• The dataset contains approximately 54.5k successfully stabilized sequences from roughly 73k public Apple Live Photos, with 50k for training, 3.5k for validation, and 1k for testing.The sequences are from iPhone 6S and later and have 360×480 resolution.Source: §4.1 · p. 4

4.2. Stabilization

• Stabilization uses tracked correspondences and a rotation-only motion model to estimate homographies, but residual scene and camera motion remains for the network to handle.Frames are warped to a reference pose and cropped to valid regions.Source: §4.2 · pp. 4–5

4.3. Training details

• The single-frame model is trained first, then initializes an 8-frame multi-frame model trained with back-propagation through time.The first stage takes 500 epochs; multi-frame training continues for 125 epochs.Source: §4.3 · p. 5

4.4. Noise modelling

• For benchmark comparisons, the authors evaluate additive white Gaussian noise at σ = 15, 25, 50, and 75.To model real sensor noise, they train separate models with Poisson noise, simulate Bayer filtering, and reconstruct RGB by bilinear interpolation.Source: §4.4 · p. 5

4.5. Single frame denoising

• The comparison trains single-frame denoisers on additive white Gaussian noise, using PASCAL VOC2010 images and evaluating on BSD68.The methods include SFD, DenoiseNet, DnCNN, BM3D, and TNRD.Source: §4.5 · p. 5
• On BSD68, single-frame denoising shows diminishing PSNR returns over time, consistent with Levin et al..The evaluation tests on 68 natural images from the Berkeley Segmentation Dataset.Source: §4.5 · p. 5
• Pre-training gives the multi-frame model a constant 0.5dB lead over training from scratch and reaches a stable state sooner.Figure 3 tracks average PSNR by frame position in sequences with Gaussian noise σ = 50.Source: §4.5 · p. 5, Figure 3 · p. 5
• At 20 layers, SFD underperforms DenoiseNet and DnCNN by ∼0.2dB; Table 1 says SFD matches BM3D at 8 layers and gets close to both at 20.The comparison uses single-frame additive white Gaussian noise denoising on BSD68, measured by PSNR.Source: §4.5 · p. 5, Table 1 · p. 5

4.6. Burst denoising

• The evaluation compares the architecture with single-frame models and VBM4D on 1,000 test sequences, each 16 frames long, using average PSNR.Table 3 reports multi-frame denoising on Live Photo sequences with additive white Gaussian noise.Source: §4.6 · p. 5, Table 3 · p. 6
• The concatenation model performs significantly worse than the recurrent model; 16- and 20-layer variants do not surpass 8- and 12-layer models after 125 epochs.The 12-layer network adds 0.18dB over the 8-layer model, but the authors choose 8 layers to avoid a 50% increase in memory and computation time.Source: Ablation study · p. 6, Table 2 · p. 6
• Models trained on 2- or 4-frame sequences fail to generalize beyond their training length, while the 8-frame model denoises longer test sequences.Figure 4 tests models trained with F = 2, 4, and 8 on 16-frame sequences.Source: Ablation study · p. 6, Figure 4 · p. 6
• Feeding sequences forward then backward raises average PSNR and avoids burn-in and flickering; this ordering is used in all experiments.Figure 5 compares first-pass and repeat-pass behavior.Source: Ablation study · p. 6, Figure 5 · p. 6

4.7. FlexISP

• Each sequence is denoised using only the first 8 frames, with models trained to replicate the synthetic sequences’ noise conditions.FLICKR DOLL and KODAK FENCE use different noise settings, and KODAK FENCE also simulates a Bayer filter.Source: §4.7 · pp. 6, 8
• On stabilized Live Photo data, multi-frame denoising produces a sharper image than single-frame denoising and VBM4D, while multi-frame 4× super-resolution improves sharpness over the single-frame model.The HDR+ examples also show denoising that preserves sharp details and highlight information.Source: Figure 6 · p. 7, Figure 7 · p. 7, Figure 9 · p. 8
• On FLICKR DOLL, the method reaches 29.39dB versus FlexISP’s 29.41dB; on KODAK FENCE, it reaches 34.976dB versus FlexISP’s 34.44dB.The KODAK FENCE result is a 0.5dB advantage over FlexISP, but demosaicing artifacts remain, likely because training data lacked high-frequency demosaicing artifacts.Source: Figure 8 · p. 7, §4.7 · p. 8

• The model was trained for 4× super-resolution with the denoising pipeline’s other training procedures unchanged.Source: §4.8 · p. 8
• Input patches were downsampled 4× using pixel area resampling, then resized to their original size with bilinear sampling.Source: §4.8 · p. 8
• Figure 7 shows two super-resolution results, with additional results in the supplemental material.Source: §4.8 · p. 8

• The single-frame architecture uses stride-1 convolutions for full-resolution processing, but is memory- and computationally expensive and has a small receptive field for its depth.Source: §5 · p. 8
• Multiscale architectures such as U-Nets could reduce computation and memory while increasing the receptive field.Source: §5 · p. 8
• Although trained on pre-stabilized sequences, the network’s accuracy drops significantly on unstabilized sequences and it is unstable on longer sequences.The authors suggest training it to stabilize sequences by warping inside the network.Source: §5 · p. 8

• The architecture achieves state-of-the-art performance on the Live Photos dataset and matches existing multi-frame denoisers on challenging datasets with real camera noise.Source: §6 · p. 8
• The recurrent architecture vastly outperforms single-frame models on multi-frame data, where single-frame models are reaching performance limits for denoising.Source: §6 · p. 8
• The authors improve a simple single-frame architecture by adding recurrent connections to process bursts of images.Source: §6 · p. 8