PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Denoising Algorithms with Real Photographs

Tobias Plötz, Stefan Roth

arXiv:1707.01313v1cs.CV

TL;DR

The paper addresses the lack of realistic ground truth for denoising by developing a benchmark methodology based on real photographs. It captures low- and high-ISO image pairs, corrects their residual differences with post-processing, and constructs the DND dataset. On real noise, BM3D outperforms several methods favored by synthetic-noise evaluations, while the benchmark exposes important differences from those settings.

  • Problem

    Denoising is usually evaluated with synthetic i.i.d. Gaussian noise because realistic noise-free ground truth for real photographs is difficult to obtain.

  • Method

    The paper captures same-scene image pairs at different ISO values with adjusted exposure times and post-processes them using alignment, intensity correction, and low-frequency bias removal.

  • Results

    On photographs with real noise, BM3D outperforms several recent techniques previously presumed superior on synthetic i.i.d. Gaussian noise.

  • Takeaways & Limitations

    The benchmark shows that realistic evaluations can yield conclusions that differ strongly from those based on synthetic noise.

  • Takeaways & Limitations

    The post-processing reference image retains some smoothing from interpolation, although the authors report this effect is not critical.

Abstract

from arXiv · show

Lacking realistic ground truth data, image denoising techniques are traditionally evaluated on images corrupted by synthesized i.i.d. Gaussian noise. We aim to obviate this unrealistic setting by developing a methodology for benchmarking denoising techniques on real photographs. We capture pairs of images with different ISO values and appropriately adjusted exposure times, where the nearly noise-free low-ISO image serves as reference. To derive the ground truth, careful post-processing is needed. We correct spatial misalignment, cope with inaccuracies in the exposure parameters through a linear intensity transform based on a novel heteroscedastic Tobit regression model, and remove residual low-frequency bias that stems, e.g., from minor illumination changes. We then capture a novel benchmark dataset, the Darmstadt Noise Dataset (DND), with consumer cameras of differing sensor sizes. One interesting finding is that various recent techniques that perform well on synthetic noise are clearly outperformed by BM3D on photographs with real noise. Our benchmark delineates realistic evaluation scenarios that deviate strongly from those commonly used in the scientific literature.

1. Introduction

The paper addresses the lack of realistic ground truth for denoising by benchmarking algorithms on real photographs. It develops a capture and post-processing methodology, then shows that real-noise evaluations can reverse conclusions drawn from synthetic Gaussian noise.

  • Motivation: Real photographs are rarely used quantitatively because their true noise-free images are difficult to determine.Most evaluations instead add synthetic i.i.d. Gaussian noise, although real photographic noise is not i.i.d. Gaussian.
  • Method: The proposed methodology captures same-scene image pairs at different ISO values with inversely adjusted exposure times.A novel heteroscedastic Tobit regression handles inaccurate gain and exposure settings while accounting for clipping and signal-dependent noise.
  • Dataset: The Darmstadt Noise Dataset contains 50 scenes captured with four consumer cameras spanning smartphone to full-frame sensors.The publicly available dataset covers varied ISO values and realistic photographic situations.
  • Findings: On real photographs, BM3D outperforms several methods previously presumed superior under synthetic i.i.d. Gaussian noise.The paper also reports lower consumer-camera noise strengths than commonly assumed and emphasizes denoising before nonlinear camera processing.

2. Related Work

Prior work models sensor noise or validates denoising on real data only partially. The paper distinguishes its benchmark through refined error correction, linear raw intensities, and estimation from just two images.

  • Noise modeling: Sensor noise includes shot noise with intensity-dependent variance, alongside electronic and discretization sources.Shot noise follows a Poisson distribution, so its variance is not stationary across an image.
  • Noise modeling: Most denoising evaluations assume stationary i.i.d. Gaussian noise despite intensity-dependent noise variance.Related approaches use heteroscedastic Gaussian models or variance-stabilizing transforms.
  • Existing real-data validation: Small-scale real-data studies use temporal averaging but overlook clipping-induced bias and later nonlinear processing of raw intensities.The paper identifies both omissions as relevant to constructing denoising ground truth.
  • Benchmark distinction: Compared with RENOIR, the proposed benchmark models heteroscedastic noise, corrects spatial misalignment, removes low-frequency bias, and uses linear raw intensities.RENOIR is based on 8-bit demosaiced images, whereas this work uses untainted linear raw intensities.
  • Capture efficiency: The proposed Tobit regression estimates noise-process parameters from just two images instead of aggregating multiple exposures temporally.This provides a less tedious capture protocol than repeated measurements at each pixel site.

3. Image Model and Data Acquisition

The acquisition model represents noisy and reference observations as clipped, intensity-dependent noisy images of related latent scenes. The protocol uses low-ISO references and compensating exposure times, while practical capture controls reduce scene variation.

  • Image formation: A noisy observation is formed by adding Poisson-Gaussian noise to a latent image and clipping intensities to sensor saturation limits.The noise strength depends on the noise-free intensity and is approximated with a heteroscedastic Gaussian model.
  • Reference acquisition: The reference image uses low ISO to reduce noise, while the unavailable noisy-image latent scene is approximated through a second capture of the same scene.The reference is intended to provide a nearly noise-free proxy for denoising ground truth.
  • Residual errors: The image-formation process relates latent reference and noisy images through linear intensity scaling, camera translation, and residual low-frequency variation.Post-processing reverses these transformations to produce the denoising ground truth.
  • Exposure compensation: The protocol keeps noise-free intensity theoretically constant by pairing a base-ISO reference with n-times longer exposure when the noisy image uses n-times larger ISO.Aperture, white balance, and focus remain constant.
  • Capture controls: Rapid WiFi-controlled capture, tripod mounting, and mirrorless cameras are used to reduce camera shake and scene variation.The protocol requires a static camera and stable illumination for noise-free intensities to remain constant.
  • Raw data processing: The dataset is extracted as normalized linear raw intensities, using black and white levels to scale values into [0, 1].The camera collection spans sensor sizes from 1/2.3 inch to full frame.

4. Post-Processing

The post-processing pipeline corrects object motion, spatial misalignment, linear intensity changes, and residual low-frequency bias to produce a realistic denoising reference. Its heteroscedastic Tobit regression models clipped observations and intensity-dependent noise when estimating the intensity transform.

  • Error correction: Four error sources require correction: object motion, subpixel misalignment, lighting changes, and exposure-related intensity changes.Moving objects are masked; translation and intensity scaling are inverted, while remaining low-frequency bias is filtered.
  • Linear intensity changes: The low-ISO image is linearly transformed using estimated slope α1 and offset α2 to account for imperfect gain, exposure, and black-level parameters.The offset α2 captures inaccuracies in the recorded black level.
  • Heteroscedastic Tobit regression: Heteroscedastic Tobit regression jointly accommodates clipped observations and intensity-dependent noise, extending basic Tobit regression with homoscedastic noise.The model estimates α1, α2, and noise-variance parameters by maximizing the log-likelihood.
  • Final residual: After linear correction, intensity-dependent bias is removed; guided filtering yields a mostly zero-mean residual whose variance increases with image intensity.The guided filter uses a large 40-pixel support and can leave small edge-localized residuals.
  • Spatial misalignment: Lucas-Kanade alignment estimates global 2D translation, after which bilinear interpolation shifts the low-frequency reference image.The noisy image is not interpolated because its high-frequency noise would be smoothed.
  • Low-frequency residual correction: Low-frequency bias is estimated by low-pass filtering the debiased residual and subtracted before inverting the debiasing step.The bias can arise from illumination changes or rolling-shutter flicker under artificial lighting.

5. Experimental Validation

Experiments validate the post-processing pipeline on simulated and real image pairs, showing accurate transformation recovery and realistic residual-noise statistics. The full pipeline closely matches ground-truth behavior and achieves a 53.7 dB PSNR reference quality.

  • Experimental setup: The evaluation simulates translations, linear intensity changes, low-frequency patterns, noise, and clipping using captured low-ISO images as latent references.Eleven noise settings cover consumer-camera parameters, with 100 trials per setting.
  • Intensity estimation: At medium to high noise levels, heteroscedastic Tobit regression significantly outperforms homoscedastic Tobit, least-squares, and the clipped-observation baseline.At low noise levels, all methods perform similarly because translation and low-frequency bias dominate estimation difficulty.
  • Alignment: Translation estimation remains roughly constant across most noise levels and increases only under severe noise.Figure 5c reports RMSE in pixels for recovering the simulated translation.
  • Bias removal: The full low-frequency correction produces higher PSNR than omitting debiasing, although PSNR decreases as noise increases because filtering leaks some noise into the reference.The remaining gap between noisy and latent-image PSNR remains large enough to measure denoising performance.
  • Quality of ground truth: Full post-processing nearly removes residual correlation and autocorrelation, while all three correction steps improve upon combinations of only two steps.The post-processed residual variance is also significantly closer to the ground-truth variance when all steps are applied.
  • Real-data validation: Synthetic and real-image test statistics differ only marginally, indicating that translations, intensity scaling, and additive low-frequency patterns model the real transformation accurately.The full pipeline reaches a highest PSNR of 53.7 dB against the latent reference.
  • Noise-parameter calibration: Tobit-regression noise estimates differ from spatial-averaging estimates by a mean error of 0.003, or 0.3% on average.The estimates remain stable across exposure times, with an average error of 0.5% for fixed camera and ISO combinations.

6. Benchmark

The benchmark evaluates denoising methods across raw, variance-stabilized, and sRGB spaces using 50 realistic scenes and 1000 cropped test images. On real noise, BM3D performs best overall, while raw-space denoising outperforms sRGB processing.

  • Dataset: The DND benchmark contains 50 scenes selected as typical yet challenging photographs.The dataset includes images from four consumer cameras and supports realistic evaluation across different camera conditions.
  • Evaluation setup: The evaluation compares seven conventional methods and two discriminative deep methods across multiple image representations.Algorithms are tested on linear raw intensities, after VST, and after conversion to sRGB.
  • Evaluation setup: 1000 test crops of 512 × 512 pixels are extracted from the benchmark images for evaluation.Each scene contributes 20 crops, with limited overlap and exclusion of pixels that changed between exposures.
  • Results: BM3D is overall the best-performing method, followed by WNNM, across the benchmark’s noise levels.This reverses earlier conclusions from synthetic Gaussian-noise evaluations, where other methods were considered superior to BM3D.
  • Results: Denoising in sRGB produces significantly worse results than denoising in raw space because sRGB noise is spatio-chromatically correlated.The noisy images have mean PSNR values of 39.39 dB in linear raw space and 29.98 dB in sRGB space.
  • Results: The benchmark’s realistic noise is lower than noise levels commonly assumed in synthetic evaluations.The mean PSNR of noisy raw images is 39.38 dB, compared with 52.76 dB for reference images.

7. Conclusion

The paper introduces a real-photograph acquisition and post-processing pipeline for denoising benchmarks. Its DND dataset shows that BM3D outperforms recent methods on real photographs, challenging conclusions from synthetic-noise settings.

  • Conclusion: The acquisition procedure pairs same-scene images captured with different analog gains and exposure times.The paired captures are used to construct realistic noisy and nearly noise-free references.
  • Conclusion: Heteroscedastic Tobit regression corrects intensity errors caused by inaccurate gain and exposure changes while modeling clipping and intensity-dependent noise.The pipeline also corrects lighting changes in transformed space and spatial misalignments.
  • Conclusion: The post-processing is necessary on real photographs and effective on simulated data.The authors report that accurate ground truth requires correcting residual errors from the acquisition process.
  • Conclusion: BM3D continues to outperform recent denoising methods on real photographs, unlike results from previously used synthetic settings.The benchmark therefore exposes a mismatch between conventional synthetic evaluation and realistic image noise.
  • Conclusion: The analysis finds that common scientific evaluation practice has rather limited relevance for real-photograph denoising.This conclusion follows from the benchmark’s real-noise comparisons and ground-truth construction procedure.

– Supplemental Material –

The supplemental material proves a linear-uncorrelation property and details the heteroscedastic Tobit regression used for realistic ground-truth construction. It derives likelihood terms for unclipped and clipped observations and describes parameter estimation.

  • A. Uncorrelated debiased images: The debiased image A(y_n) and debiased residual image R(y_n) are linearly uncorrelated.The proof proceeds by showing that the conditional expectation of the residual given the debiased image is zero, which implies zero covariance.
  • B. Heteroscedastic Tobit Regression: The supplemental material derives the log-likelihood and gradient for the proposed heteroscedastic Tobit regression model.It also explains the approximation of the noise term and parameter notation used in the derivation.
  • B.1. Log-likelihood: Model parameters are estimated by minimizing the negative log-likelihood over the high-ISO observations.The derivation first considers unclipped intensities and then incorporates clipping through the conditional distribution.
  • B.1. Log-likelihood: The clipped conditional distribution distinguishes unclipped observations from values clipped below or above the valid intensity range.Clipped observations are represented using probability mass from latent noisy values that would map to the observed boundary.
  • B.1. Log-likelihood: For stationary noise, the model reduces to Tobit regression with clipping at 0 and 1.The paper’s model extends this setting by allowing the standard deviation to vary with the transformed intensity.

B.2. Log-likelihood Gradient

The gradient derivation differentiates the likelihood across all clipping cases and specifies practical optimization safeguards. These include positive parameterization, edge-pixel exclusion, outlier truncation, and calibrated noise parameters.

  • B.2. Log-likelihood Gradient: Partial derivatives of terms involving (c−x̃)/σ_r,n(x̃) provide the building blocks for the likelihood gradients.The resulting derivatives are used for each of the three observation cases.
  • B.2. Log-likelihood Gradient: The gradient is derived separately for unclipped observations and both clipping cases.The second case uses the same derivative identities, while the third case is analogous.
  • Implementation: The optimization uses β′ = log β_r,n so that β_r,n remains positive.This reparameterization enforces the positivity required for the noise parameter.
  • Implementation: Pixels near image edges are excluded and gradients are set to zero when log T(x_i|x_r) < −10.These choices make the regression more robust to boundary effects and outliers.
  • Implementation: Noise parameters β are obtained from full Tobit regression on controlled color-checker images before estimating α for dataset image pairs.The controlled calibration provides previously recorded noise parameters for the pairwise intensity transformation.

B.3. Approximation of Noise Term

The paper evaluates approximating the noise term under imperfect ISO and exposure settings, finding that the approximation introduces negligible error on the dataset. Clipping-related bias also prevents simple averaging from recovering the noise-free signal.

  • Approximation accuracy: The approximation converges to the exact relation only when ISO and exposure time are set with perfect accuracy.In practice, those parameters cannot be controlled perfectly.
  • Approximation accuracy: 1.4 · 10^-4 average normalized RMSE means approximating the noise standard deviation introduces only 0.014% relative error.The paper describes this error as insignificant compared with overall estimation accuracy.
  • Clipping bias: Clipped noisy observations strongly deviate from noise-free intensities near clipping boundaries.This clipping bias means spatial or temporal averaging cannot directly recover the noise-free signal.
  • Clipping bias: The post-processing pipeline therefore models intensity changes and removes residual low-frequency bias rather than relying on direct averaging.The supplied passages support the need to address intensity changes and clipping-related bias, while the full smoothing step is truncated.

D. Simulation of Poisson-Gaussian Noise

The paper validates a Poisson-Gaussian simulation by comparing its first two moments with those of the heteroscedastic Gaussian noise model. The simulated noisy signal matches the target moments and therefore provides a suitable noise simulation under the stated model.

  • Moment matching: The simulation models noise on unclipped signals before comparing mean and variance with the target noise model.The passages define the unclipped noisy signals and derive their mean and variance from the Poisson distribution.
  • Validation: The noise-model validation uses a normalized RMSE over pixel intensities to quantify approximation error.The error metric is defined using a uniform distribution over pixel intensities.
  • Moment matching: The Poisson-Gaussian simulation matches the first two moments of the heteroscedastic Gaussian noisy signal.The agreement applies to both the simulated noisy signal and the reference-image simulation.

E. Additional Results

Additional DND results show broad coverage of real-image noise levels and strong benchmark headroom, while performance varies across methods, noise strengths, and display spaces. BM3D and WNNM lead the SSIM results, whereas several other methods exhibit oversmoothing, undersmoothing, or weak noise-level generalization.

  • Dataset coverage and benchmark headroom: 39.38 dB mean PSNR for noisy images versus 52.76 dB for reference images leaves substantial headroom for denoising improvements.The dataset spans a wide range of noise levels, enabling evaluation across many situations.
  • Benchmark results: NCSR has severe problems with weak intensity-dependent noise but becomes more competitive after variance stabilization.The result is reported for denoising performance aggregated by noise level.
  • Benchmark results: BM3D and WNNM show the best SSIM performance, with scores differing only marginally.SSIM scores are high across all tested methods.
  • Visual comparisons: Visual examples show MLP and FoE oversmoothing fine structures, while TNRD undersmooths and leaves substantial noise.The comparisons are shown in linear raw and sRGB spaces after the camera processing pipeline.
Loading 1707.01313v1…