Source-linked AI summary
Noise Flow: Noise Modeling with Conditional Normalizing Flows
Abdelrahman Abdelhamed, Marcus A. Brubaker, Michael S. Brown
TL;DR
Real sensor noise is more complex than widely used Gaussian and signal-dependent models, motivating a more expressive noise model. Noise Flow combines parametric noise structure with conditional normalizing flows, achieving a 0.42 nats/pixel improvement over camera-calibrated NLFs while remaining compact.
Problem
Widely used Gaussian and signal-dependent noise models provide only coarse approximations because real sensor noise includes additional spatial and nonlinear sources.
Method
Noise Flow combines established parametric noise models with conditional normalizing flows conditioned on intensity, camera type, and gain.
Results
0.42 nats/pixel improvement over camera-calibrated noise level functions corresponds to 52% higher likelihood, using a model with fewer than 2500 parameters.
Takeaways & Limitations
Noise Flow provides a compact model that accurately models and generates realistic noise distributions across cameras and gain factors.
Takeaways & Limitations
The SIDD provides gain-amplified clean images rather than the true latent clean images, so the learned gain parameter is used to estimate the latent signal when needed.
Abstract
from arXiv · showhide
Modeling and synthesizing image noise is an important aspect in many computer vision applications. The long-standing additive white Gaussian and heteroscedastic (signal-dependent) noise models widely used in the literature provide only a coarse approximation of real sensor noise. This paper introduces Noise Flow, a powerful and accurate noise model based on recent normalizing flow architectures. Noise Flow combines well-established basic parametric noise models (e.g., signal-dependent noise) with the flexibility and expressiveness of normalizing flow networks. The result is a single, comprehensive, compact noise model containing fewer than 2500 parameters yet able to represent multiple cameras and gain factors. Noise Flow dramatically outperforms existing noise models, with 0.42 nats/pixel improvement over the camera-calibrated noise level functions, which translates to 52% improvement in the likelihood of sampled noise. Noise Flow represents the first serious attempt to go beyond simple parametric models to one that leverages the power of deep learning and data-driven noise distributions.
1. Introduction
Real sensor noise is more complex than commonly used Gaussian and signal-dependent models can represent. Noise Flow combines parametric noise insights with normalizing flows to model and synthesize realistic noise compactly.
- Motivation: Existing Gaussian and signal-dependent models omit important real-noise properties, including spatial non-uniformity, amplification noise, and quantization.Gaussian models miss signal-dependent photon noise, while NLF models also miss several spatial and nonlinear noise sources.
- Evidence: Noise Flow produces synthetic noise most similar to real noise among the Gaussian, NLF, and Noise Flow models.Figure 1 compares similarity qualitatively and using KL divergence relative to real noise.
- Contribution: Noise Flow combines established parametric noise models with the expressiveness of generative normalizing-flow models.The model is designed to capture both known noise structure and more complex data-driven behavior.
- Contribution: Noise Flow conditions its normalizing-flow model on intensity, camera type, and gain settings to represent observed noise distributions.The resulting model is a strict generalization of the camera NLF.
- Contribution: Fewer than 2500 parameters produce a single compact model that represents multiple cameras and gain factors.The model is reported as considerably more accurate than existing models.
2. Background and Related Work
Image noise arises from deviations between measurements and the underlying signal, with physical and sensor-related causes. Prior models capture some components, while normalizing flows provide expressive invertible transformations for density estimation and synthesis.
- Image noise: Image noise is modeled as measurement deviation from the actual signal caused by phenomena such as photon noise and fixed-pattern sensor noise.These causes include both physical imaging processes and electronic sensor characteristics.
- Classical models: The common homoscedastic Gaussian model assumes independent, identically distributed pixel noise with zero mean and fixed variance.This is also called additive white Gaussian noise.
- Classical models: Poisson and heteroscedastic Gaussian models incorporate signal-dependent noise variance, while Poisson-Gaussian models combine signal-dependent and signal-independent components.The heteroscedastic Gaussian model is commonly called the noise level function.
- Limitations: Prior models still miss sources including fixed-pattern, defective-pixel, clipped-intensity, spatially correlated, amplification, and quantization noise.A GAN-based synthesis attempt also lacked clear quantitative evaluation according to the related-work discussion.
- Normalizing flows: A normalizing flow transforms a tractable base random variable through differentiable, invertible mappings for density estimation and generation.Likelihood evaluation uses the change-of-variables formula and log-determinant volume corrections.
- Normalizing flows: Linear and affine coupling transformations provide efficiently invertible building blocks whose Jacobian determinants can be computed tractably.Affine coupling increases expressiveness by conditioning one partition of dimensions on another.
- Application: Normalizing flows can map real noise distributions to simple tractable distributions and directly synthesize realistic noise through inverse transformations.The paper adapts them to noise modeling using conditional bijections.
3. Noise Flow
Noise Flow combines parametric noise transformations with expressive normalizing-flow layers to model camera noise conditioned on signal intensity, camera type, and gain. Its bijective architecture supports both noise simulation and likelihood-based density estimation while remaining compact.
- 3. Noise Flow: Noise Flow mixes signal-dependent and gain transformations with affine coupling flows to capture both known noise structure and more complex behavior.The architecture uses novel signal-dependent and gain layers alongside expressive affine coupling transformations.
- 3.1. Noise Modeling using Normalizing Flows: The model estimates noise distributions from observed camera-noise layers using a Normal base distribution and negative-log-likelihood training.Noise layers are obtained by subtracting clean images from corresponding noisy images.
- 3.1. Noise Modeling using Normalizing Flows: Noise Flow uses Glow-derived affine coupling and 1 × 1 convolutional layers to model pixel and cross-channel correlations.The affine coupling layers capture correlations between image dimensions, while 1 × 1 convolutions capture cross-channel correlations.
- 3.1. Noise Modeling using Normalizing Flows: The architecture sequences a signal-dependent layer, K unconditional flow steps, a gain layer, and another K unconditional flow steps; experiments use K = 4.Each unconditional step contains an affine coupling layer followed by a 1 × 1 convolutional layer, and the model is fully bijective.
- 3.2.1 Signal-Dependent Layer: The signal-dependent layer is initialized near identity while enforcing positive, intensity-increasing noise scales through exponential parameterization.The parameters β1 and β2 are represented as exponentials, with b1 = −5.0 and b2 = 0 producing β1 ≈0, β2 = 1.0, and s ≈1.0 initially.
- 3.2.2 Gain Layer: The gain layer scales noise according to ISO, permits deviations from strict ISO scaling, and adds camera-specific factors to capture inter-camera gain differences.The camera-specific scaling factor ψm ∈R+ was found effective for capturing differences in gain factors between cameras.
4. Experiments
Experiments on SIDD evaluate Noise Flow for density estimation and noise synthesis against Gaussian and camera NLF baselines. The model improves likelihood and marginal-distribution fidelity while learning signal-, gain-, and camera-specific behavior.
- Experimental setup: Noise Flow is trained and evaluated on SIDD raw-RGB noise, using negative log likelihood for density estimation and marginal KL divergence for generated-sample fidelity.SIDD contains noisy and ground-truth images from five cameras across lighting conditions and ISO levels; evaluation uses held-out test data.
- Noise density estimation: 0.69 and 0.42 nats/pixel NLL improvements over Gaussian and camera NLF models, respectively, translate to 99.4% and 51.6% likelihood improvements.Testing NLL compares Noise Flow with both baselines on the SIDD dataset.
- Noise synthesis: 97.9% and 84.1% marginal KL-divergence improvements over Gaussian and camera NLF models, respectively, make Noise Flow the strongest sampling model.The metric compares pixel-wise marginal distributions of generated samples with corresponding test-set noise samples.
- Noise synthesis: Across ISO levels and lighting conditions, Noise Flow samples are closest to real noise and show greater visual similarity than Gaussian and camera NLF samples.The comparison covers ISO levels 100 through 1600 under normal and low-light conditions.
- Learned parameters: Learned signal-dependent parameters preserve a dominant Poisson component and smaller additive Gaussian component, while gain factors track their corresponding ISO levels.The model uses positive parameterizations for the signal-dependent terms and learns gain factors for the evaluated ISO levels; ISO 200 was omitted for insufficient SIDD images.
- Camera-specific modeling: Camera-specific gain scales differ across devices and correlate with camera-specific β1 values; adding these parameters and affine-coupling blocks improves modeling performance.For a new camera, the camera-specific gain parameters can be fine-tuned while the signal-dependent and affine-coupling layers remain non-camera-specific.
5. Application to Real Image Denoising
The denoising experiment evaluates DnCNN models trained with synthetic Gaussian, camera NLF, and Noise Flow noise, alongside real-noise training. Noise Flow produces the strongest reported denoising metrics and avoids some artifacts seen with the other synthetic models.
- Quantitative results: Noise Flow training yields the highest PSNR and SSIM, slightly exceeding DnCNN-Real despite the limited real-noise training set.
- Visual results: DnCNN-Gauss tends to over-smooth noise, whereas DnCNN-CamNLF frequently produces artifacts and pixel saturation.These behaviors are illustrated in rows 3 and 5 for Gaussian training and rows 1 and 5 for camera NLF training.
- Quantitative results: DnCNN-NF achieves higher PSNR and SSIM than models trained on Gaussian noise, camera NLFs, and real noise.Table 3 reports the best testing PSNR and SSIM values for each training strategy.
- Visual results: Figure 8 compares denoised outputs from DnCNN models trained on Gaussian, camera NLF, Noise Flow, and real noise with noisy and ground-truth images.
6. Conclusion
The paper presents a compact conditional normalizing flow for modeling and synthesizing realistic image noise. It reports improved likelihood over camera-calibrated noise level functions and positions the model as useful for computer vision and image processing tasks.
- Method: Noise Flow combines established noise models with the expressiveness of normalizing flows in a conditional model for image noise modeling and synthesis.
- Model properties: Fewer than 2500 parameters yield a compact model that accurately models and generates realistic noise distributions.
- Results: 0.42 nats/pixel improvement, equivalent to 52% higher likelihood, is reported over camera-calibrated noise level functions.