Source-linked AI summary
Practical Deep Raw Image Denoising on Mobile Devices
Yuzhi Wang, Haibin Huang, Qin Xu, Jiaming Liu, Yiqun Liu, Jue Wang
TL;DR
Deep denoisers have succeeded on benchmarks, but their computational cost makes direct mobile deployment impractical, especially for noisy low-light mobile images. The paper estimates sensor noise, trains a lightweight model on sensor-specific synthetic data, and uses a k-Sigma Transform across ISO settings; it reports compatible performance with state-of-the-art methods that use much larger networks.
Problem
Deep neural denoisers have outperformed many traditional methods, but their computational demands make direct deployment on resource-limited mobile devices impractical.
Source: §1 · p. 1Method
The method estimates sensor noise to train a lightweight network on sensor-specific synthetic pairs and uses a k-Sigma Transform to normalize different ISO settings.
Source: §1 · p. 2Results
The proposed method achieves compatible performance with state-of-the-art methods, which typically employ much larger networks that cannot be directly applied for mobile applications.
Source: §6 · p. 14Takeaways & Limitations
The ISO-independent processing allows a network to handle a wide range of noise levels without increasing network complexity.
Source: §6 · p. 14Further reducing computational complexity remains future work for applying the method to video streams in real time.
Source: §6 · p. 14
Abstract
from arXiv · showhide
Deep learning-based image denoising approaches have been extensively studied in recent years, prevailing in many public benchmark datasets. However, the stat-of-the-art networks are computationally too expensive to be directly applied on mobile devices. In this work, we propose a light-weight, efficient neural network-based raw image denoiser that runs smoothly on mainstream mobile devices, and produces high quality denoising results. Our key insights are twofold: (1) by measuring and estimating sensor noise level, a smaller network trained on synthetic sensor-specific data can out-perform larger ones trained on general data; (2) the large noise level variation under different ISO settings can be removed by a novel k-Sigma Transform, allowing a small network to efficiently handle a wide range of noise levels. We conduct extensive experiments to demonstrate the efficiency and accuracy of our approach. Our proposed mobile-friendly denoising model runs at ~70 milliseconds per megapixel on Qualcomm Snapdragon 855 chipset, and it is the basis of the night shot feature of several flagship smartphones released in 2019.
1 Introduction
Mobile images are especially noisy in low light, while heavyweight denoisers are impractical on device. The paper combines sensor-specific training with an ISO-normalizing transform to make raw denoising efficient across noise levels.
1 Introduction
- Mobile photos are more vulnerable to noise than DSLR images, especially in low light, while heavyweight DNN denoisers are impractical on mobile devices.The paper attributes the higher noise to relatively low-cost mobile sensors and lenses, and the deployment constraint to limited computational resources.Source: §1 · p. 1
- Measuring sensor-specific noise supports synthetic clean/noisy training pairs, and the resulting lightweight network remains effective on real images from the same smartphone model.The paper characterizes sensor noise as consistent and measurable with sufficient accuracy.Source: §1 · p. 2
- The k-Sigma Transform maps images from different ISO settings into an ISO-invariant signal-noise space, letting one small network handle varying noise levels.This avoids training separate models for each ISO setting or one larger model for varied noise levels.Source: §1 · p. 2
- The method achieves compatible performance with state-of-the-art approaches using significantly fewer computational resources, while retaining more fine detail than the default ISP in low-light examples.The paper presents it as a practical deep-learning denoiser for mobile devices.Source: §1 · pp. 2–3, Figure 1 · p. 2
2 Related work
Prior denoising work has advanced image quality, but mobile deployment remains constrained by complexity and limited attention to device adaptability. Raw-image research also faces costly real-pair data construction and limitations of multi-frame approaches.
2 Related work
- Classical denoisers such as BM3D often have high algorithmic complexity and generally do not exploit known sensor-noise characteristics, complicating smartphone implementation.BM3D is described as a leading method in accuracy and robustness.Source: §2 · p. 3
- CNN denoisers have improved accuracy through network architectures, but their work has paid limited attention to adaptability for mobile devices.Earlier MLP work achieved results comparable with BM3D.Source: §2 · p. 3
- Raw-image CNN methods lead on public benchmarks, but constructing high-quality real clean/noisy pairs is tedious, motivating study of synthetic noise for training.Prior work reports that properly trained synthetic-data networks can generalize to real data.Source: §2 · p. 3
- Multi-frame merging is the existing mobile best practice, but alignment is difficult with moving objects and averaging may leave residual noise at high noise levels.The paper describes its approach as the first single-frame deep-learning raw denoiser designed specifically for mobile devices.Source: §2 · p. 3
3 Method
Section 3 presents a sensor noise model and parameter-estimation procedure, then uses the k-Sigma Transform to synthesize ISO-independent training data for a small network handling varied noise levels.
3.1 The Noise Model
- The sensor model combines Poisson photon variation with Gaussian noise before and after analog gain, yielding a Poisson-Gaussian model whose k and σ² vary with ISO.Source: §3.1 · pp. 4–5
3.2 Parameter Estimation
- Repeated RAW bursts of a static grayscale chart provide luminance means and equal-luminance variances for linear regression estimating k and σ².The estimates are for a specified ISO setting of a specific sensor.Source: §3.2 · p. 5
3.3 The k-Sigma Transform
- The k-Sigma Transform addresses ISO-driven noise variation by letting one network process transformed inputs and invert its output to estimate the true image.In the transformed space, the noisy-image distribution depends only on the transformed true image value.Source: §3.3 · p. 6
4 Learning to Denoise
The paper pairs a mobile-oriented U-Net-like denoiser with sensor-specific synthetic training. Its training pipeline transforms noisy and clean images into an ISO-independent space before optimization.
4.1 Mobile-friendly Network Architecture
- Four encoder and four decoder stages use separable convolutions to reduce computation, with a residual output added to the input image.The U-Net-like design uses skip connections, 5×5 encoder kernels, and 3×3 decoder convolutions.Source: §4.1 · pp. 6–7, Figure 5 · p. 7
4.2 Training Dataset
- The training set uses 214 high-quality Sony α7s II RAW images as clean targets, with noisy inputs synthesized from target-sensor noise parameters.The clean-image subset comes from 10 s and 30 s long exposures; sampled Bayer patches are augmented and transformed with k-Sigma.Source: §4.2 · p. 8, §4.3 · p. 8
4.3 Settings and Details
- Training uses ℓ1 loss and Adam, with batch size 1 and convergence at 8000×214 iterations.The triangular cyclical learning-rate schedule has a maximum of 1e-3 and decays its base rate to 1e-5 after 4000×214 iterations.Source: §4.3 · p. 8
5 Experiments
Experiments estimate the Reno-10x sensor’s noise parameters and evaluate denoising on a controlled real-world dataset. The proposed models combine competitive denoising with low mobile computation, and ablations test noise synthesis and ISO handling.
5.1 Noise Parameters Estimation
- The fitted value–variance line at ISO 4800 estimates k from its slope and σ2 from its intercept; across ISO settings, k and σ2 follow linear and quadratic fits.The fitted curves allow noise parameters at any ISO setting to be calculated for data synthesis and the k-Sigma Transform.Source: §5.1 · p. 9, Figure 6 · p. 9
5.2 Test Dataset and Metrics
- The test set uses four static scenes at two luminance levels and five ISO–exposure combinations, with 64-frame means as ground truth and PSNR/SSIM measured after sRGB processing.The ISO–exposure pairs keep brightness similar while varying noise; Fig. 7 marks the metric regions of interest.Source: §5.2 · pp. 9–10, Figure 7 · p. 10
5.3 Results
- 70.70 ms per 1 MP on Snapdragon 855; synthetic-data models beat SIDD-trained models, while our 3.6G model matches UNet-36G at 10% complexity (3.6G vs 36.3G).The reported mobile-model time is for a 1024 × 1024 Bayer input; the comparable 21G and 36G models take 292 ms and 383 ms, respectively.Source: §5.3 · pp. 10–11, Figure 8 · p. 11, Table 1 · p. 11
5.4 Ablation Studies
- 40.06 dB PSNR and 0.9335 SSIM are upper bounds from testset overfitting; testset images with synthetic noise achieve 39.97 dB PSNR, while inaccurate noise parameters degrade results.Using the SID dataset achieves 39.76 dB PSNR.Source: Data synthesis method · p. 12, Table 2 · p. 13
- 39.65 dB PSNR for concat-variance is about 0.09 dB above iso-augmentation; k-Sigma achieves the highest PSNR, and performs consistently across ISO settings.Single-ISO methods perform much worse overall, though they can be competitive when test ISO matches their training ISO.Source: Table 3 · p. 13, Robustness to ISO · p. 13, Table 4 · p. 13
6 Conclusion
The paper presents a mobile raw-image denoiser using sensor-specific synthetic training and k-Sigma processing to handle varied noise levels without increasing network complexity. It reports integration in several flagship phones’ night-shot modes and identifies further complexity reduction for real-time video as future work.
6 Conclusion
- Sensor-specific noise estimation and synthetic training support a lightweight model that generalizes to real noise, while k-Sigma processing handles a wide range of noise levels without added network complexity.The method achieves performance comparable to state-of-the-art methods that use much larger networks.Source: §6 · p. 14
- The denoiser can replace an existing camera-pipeline denoising component, and its output provides a base for more aggressive ISP post-processing.The method was featured in the night-shot modes of several flagship phones released in 2019.Source: §6 · p. 14
- Further complexity reduction is identified as future work for applying the method to real-time video streams.Source: §6 · p. 14