Source-linked AI summary
A Physics-based Noise Formation Model for Extreme Low-light Raw Denoising
Kaixuan Wei, Ying Fu, Jiaolong Yang, Hua Huang
TL;DR
Realistic training data remains difficult for extreme low-light raw denoising because learned models generalize poorly when synthetic noise misses camera-electronics effects. This paper proposes a CMOS-based noise model and camera-parameter calibration method, then evaluates generality across multiple devices. Training only on the resulting synthetic data enables a network to compete with or sometimes outperform one trained with paired real data.
Problem
Learned denoisers generalize poorly when training data do not resemble real raw images, while electronic-camera noise remains under-modeled in extreme low light.
Method
The paper models CMOS electronic imaging and calibrates its noise parameters for new cameras, incorporating photon shot, read, row, and other sensor noise sources.
Results
Training only with the proposed synthetic data lets a convolutional network compete with or sometimes outperform one trained with paired real data.
Takeaways & Limitations
The model provides a physics-based route to realistic noisy raw-data synthesis for evaluating and training extreme low-light denoisers across camera devices.
Takeaways & Limitations
The quantitative evaluation is limited to indoor scenes of the SID Sony set, although the authors state that the method can be applied more broadly.
Abstract
from arXiv · showhide
Lacking rich and realistic data, learned single image denoising algorithms generalize poorly to real raw images that do not resemble the data used for training. Although the problem can be alleviated by the heteroscedastic Gaussian model for noise synthesis, the noise sources caused by digital camera electronics are still largely overlooked, despite their significant effect on raw measurement, especially under extremely low-light condition. To address this issue, we present a highly accurate noise formation model based on the characteristics of CMOS photosensors, thereby enabling us to synthesize realistic samples that better match the physics of image formation process. Given the proposed noise model, we additionally propose a method to calibrate the noise parameters for available modern digital cameras, which is simple and reproducible for any new device. We systematically study the generalizability of a neural network trained with existing schemes, by introducing a new low-light denoising dataset that covers many modern digital cameras from diverse brands. Extensive empirical results collectively show that by utilizing our proposed noise formation model, a network can reach the capability as if it had been trained with rich real data, which demonstrates the effectiveness of our noise formation model.
1. Introduction
Extreme low-light denoising is constrained by limited photons, realistic sensor noise, and costly real-data collection. The paper addresses these constraints with a physics-based raw-noise model, camera-specific calibration, and evaluation across diverse devices.
- Motivation: Limited photon counts make low-light photography noisy, while larger apertures, longer exposures, and flash introduce depth-of-field, blur, or color tradeoffs.These acquisition strategies are additionally constrained on smartphones, dynamic scenes, and distant subjects.
- Motivation: Burst capture can increase signal-to-noise ratio but is vulnerable to ghosting in dynamic scenes, motivating single-image neural denoising.Neural approaches require abundant labelled data resembling real low-light photographs, which is labor-intensive to collect across cameras.
- Problem: Heteroscedastic Gaussian noise approximates moderate low-light noise but misses sensor noise under severe darkness, including visually noticeable banding artifacts.The paper identifies banding as an unmodeled component exacerbated in dim environments.
- Approach: The proposed raw-image model follows CMOS electronic imaging and models photon shot, pixel circuit, and quantization noise to synthesize realistic noisy data.The model avoids effects introduced by the ISP’s raw-to-sRGB conversion and targets the physical noise-formation process.
- Contributions: The model is designed to synthesize realistic noisy images whose quality matches real data under extreme low-light conditions.This contribution complements the camera-adaptation procedure and multi-device evaluation dataset.
- Contributions: The paper introduces a reproducible method to calibrate noise parameters for a given camera.It also collects data from varied camera devices to assess the model’s effectiveness and generality.
2. Related Work
Single-image denoising methods face a gap between synthetic training noise and real camera noise. Prior work uses paired real data or increasingly realistic synthetic models, but data collection remains costly and some sensor effects remain incompletely modeled.
- Single-image denoising: Modern single-image denoising methods are largely data-driven, but synthetic training noise can differ substantially from realistic evaluation noise.Traditional methods instead rely on analytical image priors such as smoothness, sparsity, self-similarity, or low rank.
- Real-data training: Paired real-data training reduces the synthetic-to-real gap but requires sufficient labelled samples, making data collection expensive and time-consuming.Noise2Noise and Noise2Void reduce the need for clean targets but do not substantially remove the labor burden for capturing real data.
- Synthetic noise modeling: Heteroscedastic Gaussian models represent photon shot noise and sensor readout effects in raw data, while later work adds dynamic streak noise, channel heterogeneity, and clipping.These models improve synthetic realism by incorporating signal dependence and additional high-sensitivity effects.
- Synthetic noise modeling: Noiseflow still requires paired real data because it obtains noise data by subtracting ground-truth images from noisy images.This requirement limits its independence from labelled real acquisitions.
3. Physics-based Noise Formation Model
The model follows CMOS image formation from photons to digital numbers and represents extreme-low-light noise through multiple physical and statistical components. It also calibrates camera-specific parameters using flat-field and bias frames, then evaluates whether the resulting model reproduces real sensor noise.
- 3. Physics-based Noise Formation Model: The raw-image model represents digital sensor output using photoelectrons, system gain, and aggregate noise from light and camera processes.The aggregate noise includes physically distinct sources modeled throughout the electronic imaging pipeline.
- 3.1. Sensor Raw Image Formation: Photon shot noise models uncertainty in photon-generated electron counts with a signal-dependent Poisson distribution.The paper treats shot noise as unavoidable because it arises from the quantum nature of light.
- 3.1. Sensor Raw Image Formation: Read noise combines multiple circuit-related sources and is modeled with a Tukey lambda distribution to capture its observed long-tailed shape.The model sets the location parameter to zero under the zero-mean noise assumption; λ and σ_TL represent shape and scale.
- 3.1. Sensor Raw Image Formation: Row noise is simulated by sampling a zero-mean Gaussian offset and adding that offset to every pixel in one row.The model considers horizontal stripes while treating column noise as generally negligible in the measured noise data.
- 3.1. Sensor Raw Image Formation: Quantization noise is introduced when the analog voltage is converted into discrete digital codes by an ADC.The quantization component uses a uniform distribution whose range depends on the quantization step q.
- 3.1. Sensor Raw Image Formation: The complete formation model combines photon shot, read, row, and quantization noise with the overall system gain.These four components are identified explicitly in the model summary.
- 3.2. Sensor Noise Evaluation: Calibration estimates camera-specific parameters from flat-field and bias frames collected at different ISO settings.Flat-field frames support Photon Transfer estimation of K, while bias frames characterize light-independent read and pattern noise; parameters are generally stationary at fixed ISO.
- 3.2. Sensor Noise Evaluation: Residual bias-frame read noise exhibits long tails, and Tukey lambda fitting is selected through PPCC and probability plots; across cameras, the model produces more realistic noise than Gaussian fitting.The evaluation reports camera-dependent noise parameters and improved goodness-of-fit and visual similarity for simulated versus real noise.
4. Extreme Low-light Denoising (ELD) Dataset
The ELD dataset is designed to test generalization across extreme low-light scenes and diverse camera devices. It includes calibrated noise data and controlled exposure reductions to simulate severe low-light conditions.
- The dataset covers 10 indoor scenes and four camera devices from multiple brands: SonyA7S2, NikonD850, CanonEOS70D, and CanonEOS700D.
- Bias and flat-field frames were recorded for each camera to calibrate the proposed noise model.
- Each scene-camera pair uses a base-ISO reference image followed by images with deliberately reduced exposure times.
- A second reference image was captured afterward to check for accidental illumination or camera-scene changes.
5. Experiments
Experiments evaluate training schemes, noise models, and generalization on Sony SID data and the four-camera ELD dataset. The proposed model produces visually strong results, adapts across cameras, and benefits from additional clean raw images, though quantitative evaluation is limited to indoor SID scenes.
- Implementation details: The learning pipeline uses a U-Net and clean raw images, while paired noisy inputs are generated with the proposed noise model.
- Competing methods: The comparison includes paired real data, Noise2Noise, homoscedastic and heteroscedastic Gaussian models, BM3D, and Anscombe-BM3D.
- Ablation study on noise models: The proposed model accounts for extreme-low-light effects omitted by Gaussian models, including non-Poisson behavior, horizontal banding, and long-tailed read noise.
- Results on SID Sony dataset: Quantitative evaluation is restricted to indoor scenes from the SID Sony set, although visual comparisons also include outdoor scenes.
- Results on SID Sony dataset: The final model produces visually appealing SID results comparable to those obtained with paired real data, whereas G+P distorts colors and leaves conspicuous banding.
- Results on our ELD dataset: On the ELD dataset, synthetic-data training often outperforms paired-real-data training because the latter can overfit Sony-specific noise patterns.
- Results on our ELD dataset: Additional clean raw samples from MIT5K improve color and brightness restoration, while calibration can be simplified using parameter samples from various cameras.
6. Conclusion
The paper presents a physics-based CMOS noise model and a calibration method for extreme low-light raw denoising. Across a four-camera dataset, synthetic-data training can compete with or sometimes outperform training with paired real data.
- The proposed model revisits the electronic imaging pipeline and includes influential noise sources overlooked by existing models.
- The calibration method adapts the noise model to available camera devices.
- The model synthesizes noisy raw data that better matches the physical process of noise formation.
- The evaluation dataset covers four representative camera devices.
- Training only with synthetic data enables a convolutional neural network to compete with or sometimes outperform training with paired real data.