Source-linked AI summary
Unprocessing Images for Learned Raw Denoising
Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, Jonathan T. Barron
TL;DR
Learned denoising often trains on synthetic data even though evaluation uses real raw sensor readings, and image-processing effects are frequently overlooked. The paper unprocesses Internet images by inverting camera-pipeline steps and includes raw processing in the training loss. On the Darmstadt Noise Dataset, the resulting network achieves 14%-38% lower error rates and is 9×-18× faster than the previous state of the art, while generalizing beyond modeled sensors.
Problem
Learned denoising commonly uses synthetic data for real raw-image evaluation, while camera image-processing effects are less explored despite transforming raw measurements and noise.
Method
The method inverts camera-pipeline transformations to synthesize realistic raw data from Internet images and applies raw processing to model outputs and targets before loss computation.
Results
14%-38% lower error rates and 9×-18× faster runtime than the previous state of the art are reported on the Darmstadt Noise Dataset.
Takeaways & Limitations
The approach generalizes to sensors outside those explicitly modeled during synthetic training-data generation.
Takeaways & Limitations
Tone mapping is approximated with a simple smoothstep curve because more complex edge-aware local tone mapping is difficult to reverse-engineer.
Abstract
from arXiv · showhide
Machine learning techniques work best when the data used for training resembles the data used for evaluation. This holds true for learned single-image denoising algorithms, which are applied to real raw camera sensor readings but, due to practical constraints, are often trained on synthetic image data. Though it is understood that generalizing from synthetic to real data requires careful consideration of the noise properties of image sensors, the other aspects of a camera's image processing pipeline (gain, color correction, tone mapping, etc) are often overlooked, despite their significant effect on how raw measurements are transformed into finished images. To address this, we present a technique to "unprocess" images by inverting each step of an image processing pipeline, thereby allowing us to synthesize realistic raw sensor measurements from commonly available internet photos. We additionally model the relevant components of an image processing pipeline when evaluating our loss function, which allows training to be aware of all relevant photometric processing that will occur after denoising. By processing and unprocessing model outputs and training data in this way, we are able to train a simple convolutional neural network that has 14%-38% lower error rates and is 9x-18x faster than the previous state of the art on the Darmstadt Noise Dataset, and generalizes to sensors outside of that dataset as well.
1. Introduction
Learned denoising methods often rely on synthetic paired data yet underperform on real noisy raw images. This paper attributes part of that gap to unrealistic data and overlooked image-processing steps, proposing modeled unprocessing and processing-aware training.
- Motivation: Synthetic paired datasets are commonly used because neural denoising models require large training sets.Capturing real noisy and clean pairs is difficult, expensive, and time consuming, and new sensors may require new datasets.
- Motivation: Recent benchmarks show learned models are often outperformed by traditional hand-engineered algorithms on real noisy raw images.
- Research gap: More realistic noise models address sensor-physics mismatch but omit many image-processing steps that transform raw measurements into finished images.
- Approach: The paper inverts key pipeline operations to synthesize realistic raw data from generic Internet images and incorporates conventional processing into neural-network training.
- Reported outcome: 14%-38% lower error rates and 9×-18× faster runtime are reported than the previous state of the art on the Darmstadt Noise Dataset.The approach also generalizes to devices not explicitly modeled during synthetic-data generation.
2. Related Work
Prior denoising research modeled noise analytically or collected real noisy/clean pairs, while learned methods trained on synthetic data struggled on real raw images. This work focuses on the less-explored role of the image-processing pipeline in that mismatch.
- Denoising methods: Classic denoising methods use hand-engineered statistical assumptions, whereas later approaches use nonparametric self-similarity to recover clean signals.
- Noise modeling: Real sensor noise is better represented by shot and read noise or signal-dependent Gaussian and Poissonian components than by additive white Gaussian noise.
- Real paired data: Real noisy/clean image-pair datasets exposed that learned methods trained on synthetic data could be outperformed by older models such as BM3D.
- Real paired data: Collecting real paired photographs for training is promising but carries substantial dataset and labor requirements.
- Pipeline modeling: Image-processing pipelines alter intensity scaling, input noise, and rendered appearance, making their modeling and inversion a less-explored issue in learned raw denoising.
3. Raw Image Pipeline
The raw-image pipeline transforms noisy sensor measurements through noise modeling, gain, white balance, demosaicing, color correction, gamma, and tone mapping. The paper defines inverses for these stages to generate synthetic raw data whose intensity distributions resemble real raw images.
- 3.1. Shot and Read Noise: Raw sensor noise combines shot noise from photon arrival statistics with approximately Gaussian read noise, modeled together as signal-dependent heteroscedastic Gaussian noise.The variance depends on true signal intensity, with parameters determined by sensor analog and digital gains.
- 3.1. Shot and Read Noise: Synthetic noise levels are sampled from the joint distribution of shot/read noise parameter pairs observed in the Darmstadt dataset.The distribution is visualized in Figure 3, where circle size indicates pair frequency.
- 3.2. Demosaicing: Demosaicing reconstructs three color measurements from Bayer-filtered sensor pixels; its inverse retains only the filter-appropriate color value at each pixel.The procedure adopts the Darmstadt dataset’s bilinear interpolation convention for demosaicing.
- 3.4. White Balance: White balance applies per-channel red and blue gains, sampled from Darmstadt metadata distributions, to compensate for illumination and camera color responses.The paper samples red gains in [1.9, 2.4] and blue gains in [1.5, 1.9], uniformly and independently.
- 3.3. Digital Gain: Digital and white-balance gains are inverted using dataset-informed scaling and sampled channel gains, while a highlight-preserving transformation avoids systematically removing highlights and clipped pixels.The transformation is linear below threshold t = 0.9 and uses a cubic form for high-gain, high-intensity values.
- 3.5. Color Correction and 3.6. Gamma Compression: Color correction uses a camera-specific 3 × 3 matrix, gamma compression uses a clamped standard curve, and tone mapping is approximated by an invertible smoothstep curve.The unprocessed intensity histograms are similar to real raw intensity histograms across color channels.
4. Model
The model is trained on synthetically unprocessed noisy raw data, while loss computation processes both predictions and targets through a simplified camera pipeline. Its U-Net uses Bayer-domain inputs plus per-pixel noise estimates to produce denoised mosaics.
- Training data: Synthetic training pairs are created by inverting image-processing transformations, then adding shot and read noise to the raw ground truth.Unprocessing reverses tone mapping, gamma decompression, color correction, white balance, and digital gain.
- Processing-aware loss: The loss processes both the network output and noise-free ground truth through white balance, demosaicing, color correction, and gamma compression.This exposes training to downstream photometric effects after denoising.
- Network: The network maps a noisy Bayer-domain raw image to a reduced-noise image in the same domain.A U-Net with skip connections is used for the mapping.
- Network: A per-pixel noise standard-deviation map is concatenated as four additional channels, one for each R-G-G-B Bayer plane.The estimate is based on the input image’s shot and read noise parameters.
- Training data: Training uses one-million-image MIR Flickr data, downsampled and randomly cropped into 128 × 128 patches with flips for augmentation.Five percent is reserved for validation and five percent for testing.
- Training objectives: The sRGB model computes L1 loss after processing outputs and targets into sRGB, whereas the Raw model computes L1 loss directly in raw space.The two variants target different evaluation domains.
5. Results
On the Darmstadt Noise Dataset, both model variants outperform published baselines across raw and sRGB outputs, with substantial error reductions and a reported 22ms runtime on 512 × 512 images. Ablations show that realistic noise modeling and unprocessing both matter, while evaluation on HDR+ tests generalization beyond Darmstadt.
- Evaluation: The Darmstadt benchmark contains 50 real high-resolution images pairing noisy high-ISO captures with nearly noise-free low-ISO ground truth.Images come from four consumer cameras and are presented as raw unprocessed linear intensities.
- Benchmark performance: Both model variants produce higher PSNR and SSIM than all baselines across outputs, with each variant best in its targeted domain.The benchmark reports PSNR and SSIM for raw and sRGB outputs.
- Benchmark performance: 14% and 25% lower error are achieved on the two raw metrics than N3Net, while 21% and 38% lower error are achieved on the two sRGB metrics than N3Net and CBDNet.The reductions are reported relative to the next-best techniques for the corresponding metrics.
- Generalization: Evaluation on raw HDR+ images is used to test whether the method generalizes to other datasets and devices.Additional HDR+ results are provided in Figure 7 and the supplement.
- Ablations: Removing any proposed component reduces quality, with Gaussian noise causing the largest sensitivity in the ablation study.The study reports that realistic noise modeling matters more than the other tested components.
- Ablations: Unprocessing contributes substantially, especially for sRGB metrics, and a four-times-larger network does not compensate for omitting it.Removing the residual output block causes the smallest performance decrease among architectural ablations.
- Runtime: 22ms is the reported runtime for 512 × 512 Darmstadt images on a single NVIDIA GeForce GTX 1080Ti GPU, excluding GPU transfer time.The value is the mean over 100 runs.
6. Conclusion
The paper presents an unprocessing technique that creates realistic raw training data from generic images and incorporates image processing into denoising training. On the Darmstadt Noise Dataset, the resulting model achieves lower error rates and faster runtimes than the previous state of the art.
- 14%-38% lower error rates than the previous state of the art on the Darmstadt Noise Dataset.
- 9×-18× faster runtimes than the previous state of the art on the Darmstadt Noise Dataset.
- Unprocessing models and inverts each camera image-processing step to synthesize realistic noisy/clean raw training pairs from Internet images.