Source-linked AI summary
Learning to See in the Dark
Chen Chen, Qifeng Chen, Jia Xu, Vladlen Koltun
TL;DR
Extreme low-light imaging is hindered by low photon counts, noise, and low SNR, while existing processing methods perform poorly in demanding conditions. The paper introduces the SID dataset and an end-to-end raw-data pipeline, reporting successful noise suppression and color transformation, with amplification up to 300 times.
Problem
Extreme low-light imaging has low photon counts and SNR, while existing processing methods have limited effectiveness and lacked a public real-world dataset with ground truth for fast low-light imaging.
Method
The paper collects paired raw short-exposure and long-exposure reference images in the SID dataset and trains a fully-convolutional network end-to-end to process low-light raw data.
Results
The pipeline achieves successful noise suppression and correct color transformation on SID data, amplifying low-light images by up to 300 times.
Takeaways & Limitations
SID supports data-driven development for extreme low-light imaging, including video-rate imaging in sub-lux conditions.
Takeaways & Limitations
The SID dataset does not contain humans or dynamic objects, limiting the represented scene content.
Abstract
from arXiv · showhide
Imaging in low light is challenging due to low photon count and low SNR. Short-exposure images suffer from noise, while long exposure can induce blur and is often impractical. A variety of denoising, deblurring, and enhancement techniques have been proposed, but their effectiveness is limited in extreme conditions, such as video-rate imaging at night. To support the development of learning-based pipelines for low-light image processing, we introduce a dataset of raw short-exposure low-light images, with corresponding long-exposure reference images. Using the presented dataset, we develop a pipeline for processing low-light images, based on end-to-end training of a fully-convolutional network. The network operates directly on raw sensor data and replaces much of the traditional image processing pipeline, which tends to perform poorly on such data. We report promising results on the new dataset, analyze factors that affect performance, and highlight opportunities for future work. The results are shown in the supplementary video at https://youtu.be/qWKUFK7MWvg
1. Introduction
Extreme low-light imaging remains difficult because short exposures have severely limited photons and SNR, while longer exposures introduce blur and are impractical for fast capture. The paper addresses this setting with a learned raw-image pipeline and a dataset pairing short-exposure inputs with long-exposure references.
- Low photon counts make low-light images noisy, while high ISO and postprocessing cannot resolve the underlying low signal-to-noise ratio.High ISO increases brightness but also amplifies noise; scaling and histogram stretching do not recover missing signal.
- Longer exposure, wider aperture, and flash can increase signal, but each introduces practical drawbacks such as blur from camera or object motion.
- Existing denoising, deblurring, and enhancement methods generally target dim environments with moderate noise rather than extreme low light at video rates.
- At less than 0.1 lux and 1/30-second exposure, ISO 8,000 produced an essentially black image, while ISO 409,600 revealed content but remained dim, noisy, and color-distorted.
- Burst methods can also fail because alignment is unreliable in extreme low light and burst pipelines are not designed for video capture.
- The proposed pipeline learns color transformation, demosaicing, noise reduction, and enhancement directly from raw data through end-to-end training.
- The SID dataset provides real fast-exposure low-light images paired with long-exposure references, enabling training and evaluation with ground truth.The paper reports amplification of up to 300 times with successful noise reduction and correct color transformation.
2. Related Work
Prior low-light processing spans classical and learned denoising, enhancement, and burst methods, but evaluations commonly rely on synthetic noise, missing ground truth, or conditions less extreme than the paper’s target.
- Denoising methods use priors such as smoothness, sparsity, low rank, and self-similarity, while learned models can compete with classical methods at selected noise levels.
- Most denoising evaluations use synthetic Gaussian or salt&pepper noise, and real-image evaluation found BM3D stronger than newer techniques.
- Joint denoising and demosaicing methods have been evaluated on synthetic Bayer patterns and synthetic noise rather than real extreme-low-light images.
- Burst denoising can improve results by collecting more scene information but often requires reference selection and dense correspondence estimation.
- Low-light enhancement includes histogram equalization, gamma correction, inverse dark channel priors, wavelets, Retinex models, and illumination-map estimation.
- Existing noisy-image datasets include real-noise benchmarks, but reported spatial misalignment or lack of reliable ground truth limits their suitability for this setting.
3. See-in-the-Dark Dataset
SID is a real low-light dataset for raw short-exposure processing, pairing thousands of inputs with long-exposure references across indoor and outdoor scenes and multiple sensors. Its reference images require static scenes, and its target is perceptual image quality rather than exhaustive noise removal.
- SID contains 5094 raw short-exposure images and 424 distinct long-exposure reference images.Multiple short-exposure images may correspond to one reference, with each counted separately for training and testing.
- The dataset includes outdoor scenes at 0.2–5 lux and darker indoor scenes at 0.03–0.3 lux.
- Input exposures range from 1/30 to 1/10 seconds, while reference exposures are 100–300 times longer, spanning 10–30 seconds.
- Because reference exposures are necessarily long, all dataset scenes are static; approximately 20% form the test set and 10% the validation set.
- Images were captured with Sony α7S II and Fujifilm X-T2 cameras using different sensor filter arrays.The Sony uses a full-frame Bayer sensor, while the Fuji uses an APS-C X-Trans sensor.
- The capture procedure kept the camera fixed between long- and short-exposure images and collected short-exposure sequences for burst-denoising comparisons.
- Long-exposure references may retain some noise, and the dataset targets perceptually good images rather than exhaustive noise removal or maximum contrast.
4. Method
The method replaces much of the conventional low-light processing chain with an end-to-end fully convolutional network operating on packed, amplified raw sensor data. External amplification controls output brightness, while camera-specific training maps short-exposure inputs to sRGB references.
- Pipeline: Traditional processing applies sequential modules including white balance, demosaicing, denoising, sharpening, color conversion, and gamma correction, but struggles in fast low-light imaging.
- Pipeline: The proposed approach trains a fully-convolutional network to perform the entire image-processing pipeline directly on raw data rather than processed sRGB images.
- Pipeline: For Bayer inputs, raw data is packed into four channels; for X-Trans inputs, it is packed into nine channels after arranging 6×6 blocks.
- Pipeline: The pipeline subtracts black level, scales data by the desired amplification ratio, and reconstructs full resolution using a sub-pixel layer.
- Network architecture: The default network architecture is a U-net, selected after comparing it with a multi-scale context aggregation network and finding residual connections unhelpful in this setting.
- Amplification: The amplification ratio is supplied externally like a camera ISO setting, controlling output brightness while the network performs blind noise suppression and color transformation.
- Training: Training uses camera-specific networks, L1 loss, Adam optimization, random 512×512 crops, augmentation, and exposure-difference amplification ratios such as x100, x250, or x300.
5. Experiments
Experiments compare the learned pipeline with traditional processing, BM3D, and burst denoising, then evaluate cross-sensor generalization and controlled design choices. The learned approach performs strongly in challenging low-light settings, while raw-domain processing and architectural choices affect quality.
- Qualitative results: The traditional pipeline produces severe noise and color distortion in extreme low-light conditions.
- Perceptual experiments: The learned single-image pipeline significantly outperforms BM3D and burst denoising on Sony x300 and is on par on Sony x100.The perceptual experiment used 1,180 comparisons from 10 workers, and the baselines received oracle preprocessing and, for burst denoising, eight perfectly aligned images.
- Cross-sensor generalization: A Sony-trained network applied to iPhone 6s raw images produces good contrast, low noise, and well-adjusted color compared with traditional processing.The smartphone example uses an amplification ratio of x100.
- Network structure: The U-net has higher PSNR on both evaluated sets than CAN, although CAN sometimes produces higher SSIM and incorrect colors.Figure 8 shows color recovery failure for CAN on a Fuji x300 patch.
- Input color space: Operating directly on raw sensor data is more effective in extreme low light than processing sRGB images produced by the traditional pipeline.
- Data arrangement: Packing raw colors performs better than masking Bayer data or using 6×6 packing for X-Trans data, which can lose hues, color, and detail.
- Postprocessing: Excluding histogram stretching from training avoids artifacts and yields darker but cleaner outputs, with stretching optionally applied during postprocessing.Training on histogram-stretched references significantly reduces accuracy and is prone to overfitting global histogram statistics.
6. Discussion
The See-in-the-Dark dataset and end-to-end fully-convolutional pipeline produce promising low-light results, including noise suppression and correct color transformation. The discussion identifies remaining quality, generalization, and runtime limitations, alongside opportunities for improvement.
- Contributions: The SID dataset supports data-driven processing of extreme low-light images with low photon counts and low SNR.It was created to support approaches to imaging in the dark at video rates and sub-lux conditions.
- Results: An end-to-end fully-convolutional network improves traditional low-light processing, with successful noise suppression and correct color transformation on SID data.The pipeline is based on end-to-end training of a fully-convolutional network.
- Limitations: The pipeline remains imperfect: the x300 subset is particularly challenging, and artifacts and loss of detail appear in extreme low-light outputs.Figure 10(d) demonstrates artifacts in the network output, while Figure 10 notes apparent detail loss upon close examination.
- Limitations: The amplification ratio must be chosen externally, and the current approach assumes a dedicated network for each camera sensor.Cross-sensor generalization experiments are described as preliminary and encouraging.
- Future work: Future work includes optimizing network architecture, training, runtime, amplification-ratio inference, and cross-sensor generalization.The authors expect these directions to yield further image-quality and deployment improvements.
- Limitations: Full-resolution processing takes 0.38 seconds for Sony images and 0.66 seconds for Fuji images, preventing real-time full-resolution processing.A low-resolution preview can nevertheless be produced in real time.