PaperlayerView exact revision on arXiv

Source-linked AI summary

Handheld Mobile Photography in Very Low Light

Orly Liba, Kiran Murthy, Yun-Ta Tsai, Tim Brooks, Tianfan Xue, Nikhil Karnad, Qiurui He, Jonathan T. Barron, Dillon Sharlet, Ryan Geiss, Samuel W. Hasinoff, Yael Pritch, Marc Levoy

arXiv:1910.11336v1cs.CVcs.GR

TL;DR

Low-light mobile photography must contend with sensor noise and motion in handheld scenes. The paper extends burst imaging with motion-aware capture and merging, low-light white balancing, and specialized tone mapping. Its pipeline produces photographs with more detail, low noise, pleasing colors, and good global contrast, while the authors note that autofocus remains a practical limitation.

  • Problem

    Low-light images have low signal relative to sensor noise, while handheld capture and moving subjects make longer exposures prone to blur.

    Source: §1 · pp. 1–2

  • Method

    The system extends burst imaging with motion metering and motion-robust merging, learning-based low-light white balancing, and specialized tone mapping.

    Source: §7 · p. 15

  • Results

    The final pipeline produces more detail with low noise, pleasing colors, and good global contrast than the baseline, and higher-quality output than the compared CNN.

    Source: §6 · p. 13

  • Takeaways & Limitations

    The system overcomes many constraints of small sensors and runs efficiently on a modern mobile device.

    Source: §7 · p. 15

    At about 0.4 lux, the camera’s autofocus fails almost 50% of the time; the system falls back to hyperfocal focus when autofocus reports failure.

    Source: §6.1 · p. 15

Abstract

from arXiv · show

Taking photographs in low light using a mobile phone is challenging and rarely produces pleasing results. Aside from the physical limits imposed by read noise and photon shot noise, these cameras are typically handheld, have small apertures and sensors, use mass-produced analog electronics that cannot easily be cooled, and are commonly used to photograph subjects that move, like children and pets. In this paper we describe a system for capturing clean, sharp, colorful photographs in light as low as 0.3~lux, where human vision becomes monochromatic and indistinct. To permit handheld photography without flash illumination, we capture, align, and combine multiple frames. Our system employs "motion metering", which uses an estimate of motion magnitudes (whether due to handshake or moving objects) to identify the number of frames and the per-frame exposure times that together minimize both noise and motion blur in a captured burst. We combine these frames using robust alignment and merging techniques that are specialized for high-noise imagery. To ensure accurate colors in such low light, we employ a learning-based auto white balancing algorithm. To prevent the photographs from looking like they were shot in daylight, we use tone mapping techniques inspired by illusionistic painting: increasing contrast, crushing shadows to black, and surrounding the scene with darkness. All of these processes are performed using the limited computational resources of a mobile device. Our system can be used by novice photographers to produce shareable pictures in a few seconds based on a single shutter press, even in environments so dim that humans cannot see clearly.

1 INTRODUCTION

Low-light photography is difficult because sensor noise, handheld motion, and the limitations of small mobile cameras constrain image quality. The paper extends a mobile burst-imaging system with motion-aware capture and merging, low-light white balance, and tone mapping for scenes as dim as about 0.3 lux.

1 INTRODUCTION

  • Single-frame denoising can lose detail at very high noise, while longer exposures and stronger burst merging risk motion blur, leaving SNR improvement alone short of an overall solution.The system targets handheld photography without flash, manual controls, or more than about six seconds of scene acquisition.Source: §1 · p. 2
  • The design addresses mobile-device constraints, including limited compute resources and processing time of about two seconds.The pipeline’s capture time is between one and six seconds after the shutter press.Source: §1 · p. 2, Figure 2 · p. 3
  • The system extends Hasinoff et al. (2016), whose mobile burst pipeline could produce high-quality photographs down to about 3 lux, to target handheld scenes as dim as about 0.3 lux.The paper identifies noise-versus-motion-blur tradeoffs as the additional challenge at still lower light levels.Source: §1 · p. 2
  • Motion metering selects burst exposure settings to balance noise and motion blur, while motion-robust merging increases contributions in static regions and reduces them where blur may occur.Exposure limits are also set using on-device gyroscope measurements of camera stability.Source: §1 · p. 3
  • A learning-based white-balancing algorithm targets accurate color under strongly tinted lighting, and specialized tone mapping reveals detail while retaining the appearance of a dark scene.The paper describes the color-constancy algorithm as trained and evaluated with a novel metric.Source: §1 · p. 3
  • The authors released several hundred raw bursts spanning varied scenes and conditions, captured and processed with their pipeline.Source: §1 · p. 3

2 MOTION METERING

Motion metering predicts scene and camera motion to set burst exposure time and gain, balancing noise against blur. It combines motion-magnitude estimation, future-motion prediction, gyro-based stability detection, and a dynamic exposure schedule.

2.1 Efficient motion magnitude estimation

  • Bounded Flow estimates motion magnitude from intensity changes and spatial gradients, masks low-gradient regions relative to noise, and refines estimates using 90th-percentile motion per bin.On the MPI-Sintel evaluation, it was as accurate as non-neural methods and 2× faster than the next-fastest method.Source: §2.1 · p. 4, Table 1 · p. 5

2.2 Future motion prediction

  • The GMM-based predictor estimates an upper bound on the minimum motion across the next K frames, assuming future motion follows the pre-shutter distribution; it had the lowest error among three filters.The evaluation used 777 sequences, with N = 10 past estimates and K = 4 future frames.Source: §2.2 · pp. 5–6, Table 2 · p. 6

2.3 Camera stability detection

  • Gyroscope measurements identify camera stability, enabling exposures up to 1 s when stabilized versus 333 ms handheld; readings nearest the shutter press are masked to improve discrimination.The system averages angular speeds over 1.466 s before capture and masks the nearest 0.400 s because shutter-press vibration can resemble handheld motion.Source: §2.3 · p. 6

2.4 Exposure time selection

  • The dynamic exposure schedule factorizes auto-exposure target sensitivity into exposure time and gain, using predicted motion to limit blur and camera stability to cap exposure time.As scenes darken, gain may become excessive, so the schedule increases exposure time and gain together; the dynamic schedule helps most at intermediate light levels.Source: §2.4 · p. 7
  • 0.5 f-stop longer exposures reduced noise on a still scene, while 1 f-stop shorter exposures reduced blur on a moving scene, without noticeable added noise.These paired captures show the dynamic schedule choosing different exposure times than the static schedule according to scene motion.Source: Figure 8 · p. 7, §2.4 · p. 7

3 MOTION-ADAPTIVE BURST MERGING

The section extends mobile burst denoising with motion-aware, spatially varying merging to reduce noise while limiting motion artifacts. It also adapts alignment and spatial denoising to noisy bursts.

3.2 Spatially varying temporal merging

  • Mismatch maps let Fourier-domain merging increase in static regions and decrease where motion or alignment errors may cause artifacts.The maps use aligned tile differences; a mismatch value of zero means no mismatch and one means high mismatch, with s = 0.5 found to suppress noise while detecting motion.Source: §3 · p. 8, §3.2 · p. 9

3.1 Alignment

  • Alignment uses a coarse-to-fine, tile-based method on raw-image pyramids, with larger tiles as noise rises.Tiles range from 16 to 64 pixels; the largest are used in dark scenes.Source: §3.1 · p. 8

3.2 Spatially varying temporal merging

  • Temporal strength controls the denoising–motion-robustness tradeoff: stronger merging reduces noise but can increase motion blur.The method varies this strength by tile according to mismatch, using a sharp reference frame and reducing contributions from mismatched tiles.Source: §3.2 · pp. 8–9, Figure 9 · p. 9

3.3 Spatially varying denoising

  • Spatial denoising increases in tiles where limited frame merging raises the estimated noise variance, compensating for reduced temporal denoising.Static regions are not affected by the spatially varying adjustment.Source: §3.3 · p. 9, Figure 10 · p. 10
  • The modified merge reduces noise further and preserves more detail than the prior merge on dSLR bursts, while reducing motion blur from short exposures.The method also merges a 13-frame mobile burst in a dark indoor scene with substantial motion.Source: §3.3 · pp. 9–10, Figure 11 · p. 10

4 LOW-LIGHT AUTO WHITE BALANCE

Low-light white balancing is difficult because scenes are noisier and illuminants more diverse, often leaving one or more color channels with little intensity. The section adapts learning-based white balance with a photographer-labeled dataset and an error metric designed for these conditions.

4 LOW-LIGHT AUTO WHITE BALANCE

  • The low-light adaptation modifies FFCC using a new dataset and error metric because low-light imagery is noisy and its illuminants are diverse.Existing color-constancy datasets target neutralization and often lack extreme low-light images and spectral calibration.Source: §4 · p. 10, §4.1 · p. 10

4.1 Dataset

  • The authors collected 5,000 calibrated mobile images across varied light levels and had professional photographers choose aesthetically preferred white balances.Annotators viewed the final camera-pipeline rendering while selecting white points.Source: §4.1 · p. 10

4.2 Error metrics

  • ARE evaluates the appearance of the average image patch under the recovered illumination, reducing sensitivity to errors in low-intensity channels.Its sensitivity to a channel decreases with that channel’s mean true-image intensity.Source: §4.2 · p. 11
  • Minimizing ARE during training improves validation performance across all reported error metrics.The comparison uses three-fold cross-validation on the authors’ dataset.Source: §4.2 · p. 12, Table 3 · p. 12

5 TONE MAPPING

The tone-mapping strategy aims for colorful low-light images that retain a nighttime appearance rather than simulating daylight. It combines shadow and highlight adjustments with contrast, saturation, and vignette heuristics.

5 TONE MAPPING

  • The tone-mapping intent is to render vibrant, colorful dark scenes despite the difficulty of simulating rod-mediated human vision.The paper frames this as bringing a capability developed for digital astronomy to handheld mobile cameras.Source: §5 · p. 12, Figure 13 · p. 12
  • The final-look heuristics limit shadow boosting, compress highlights, increase saturation as scene brightness falls, and add brightness-dependent vignetting.Saturation increases by up to 20%; vignetting is applied below 5 lux and increases in darker scenes.Source: §5 · pp. 12–13
  • Shadow gain increases as illuminance falls, reaching over one additional stop (2.2×) at the 0.1-lux threshold.The method also applies up to 20% of the additional shadow gain to highlights, in inverse proportion to normalized scene dynamic range.Source: §5 · pp. 12–13
  • The proposed tone mapping reveals shadow detail while retaining global contrast and a natural nighttime appearance, unlike the darker prior rendering or flattened CLAHE result.Figure 14 compares the three renderings in a scene at about 0.4 lux.Source: Figure 14 · p. 13, §5 · p. 13

6 RESULTS

The system’s burst pipeline improves detail, noise, color, and contrast over a baseline burst pipeline, and compares favorably with a CNN and specialized low-light modes. Its practical limitations include autofocus failures, preview and shutter delays, black-level handling, and sky rendition.

6 RESULTS

  • The final pipeline improves detail, noise, color rendition, and global contrast over the baseline burst pipeline.Figure 15 shows tone mapping alone makes noise more apparent; motion metering and motion-robust merging improve noise and detail, and auto white balance improves color rendition.Source: §6 · p. 13, Figure 15 · p. 14
  • Against Chen et al. (2018), the pipeline produces more detail and fewer artifacts, processes bursts on-phone in under 2 seconds, and uses under 400 MB versus an estimated 3 GB.The comparison uses Sony α7S II images; Figure 16 also shows more detail and less noise for the 13-frame burst than for single-frame processing.Source: §6 · pp. 13, 15, Figure 16 · p. 14
  • Compared with specialized low-light device modes, the system produces a balanced rendition with good brightness, pleasing color, and few artifacts.The comparison scenes are illuminated by dim tungsten light at 0.4 lux.Source: §6 · p. 15, Figure 17 · p. 15

6.1 Limitations

  • At about 0.4 lux, autofocus fails almost 50% of the time; the system falls back to hyperfocal focus and also permits manual focus adjustment.The device’s out-of-box autofocus is not optimized for very low light.Source: §6.1 · p. 15
  • Low-light preview is constrained by the exposure and processing latency needed to meet the viewfinder refresh rate of at least 15 fps.The system uses the camera’s hardware-accelerated preview pipeline and adaptively gains its output with a GPU shader.Source: §6.1 · p. 15
  • Raw-frame capture begins only after shutter press, causing noticeable shutter lag; viewfinder-rate frames are not used because their short exposures leave read noise dominant.Source: §6.1 · p. 15
  • The system uses black levels supplied by the camera driver, although a good sensor can approach within 0.25 DN of a 10-bit signal.The paper notes that a gain-dependent lookup table can further improve black-level accuracy.Source: §6.1 · p. 15
  • Global mid-tone adjustment can brighten the sky along with foreground details, so some scenes do not convey darkness adequately.The paper says a different tonal adjustment specifically for the sky would be needed to account for the perception of darkness.Source: §6.1 · p. 15

7 CONCLUSIONS

The paper presents a computational photography system for high-quality pictures in very low light. It combines motion-aware burst capture and merging with learned white balance and specialized tone mapping, while running efficiently on modern mobile devices.

7 CONCLUSIONS

  • The system is an implementation designed to meet stated requirements for high-quality pictures in very low-light conditions.Source: §7 · p. 15
  • Motion metering selects burst parameters to trade off signal-to-noise ratio and motion blur, while motion-robust merging limits artifacts in extremely noisy raw images.Source: §7 · p. 15
  • The system incorporates learning-based auto white balancing and specialized tone mapping, overcomes many small-sensor constraints, and runs efficiently on a modern mobile device.Source: §7 · p. 15

A VISUALIZATION OF MOTION-ADAPTIVE BURST MERGING

The appendix visualizes burst merging in examples with extensive motion and in scenes with fast motion or changing illumination.

A VISUALIZATION OF MOTION-ADAPTIVE BURST MERGING

B MERGING USING MOTION-ROBUST SUPER-RESOLUTION

The appendix discusses spatial-domain motion-robust super-resolution as an alternative to Fourier-domain merging, with different strengths and implementation requirements.

B MERGING USING MOTION-ROBUST SUPER-RESOLUTION

  • Spatial merging uses locally adaptive kernels and per-pixel analysis to handle motion, misalignment, and occlusions, and can preserve more detail in faces and textures.It requires a specialized implementation suited to target GPU hardware and operating systems, which may not apply to all mobile devices.Source: B MERGING USING MOTION-ROBUST… · p. 17
  • Spatial-domain merging reproduces sharp bright light sources more accurately, whereas Fourier-domain denoising can cause ringing artifacts.Source: B MERGING USING MOTION-ROBUST… · p. 17
  • Fourier-domain denoising is more effective in very low-light scenes and can produce more perceptually pleasing noise texture in uniform regions such as skies.Source: B MERGING USING MOTION-ROBUST… · p. 17

C.1 Dataset

The authors visualize some of the training data and the tool used to annotate it.

C.1 Dataset

  • Figure 21 shows examples of the training data and the tool used to annotate it.Source: §C.1 · p. 17

C.2 Additional improvements to FFCC

The white-balance system adds face-specific image information and adapts its model to scene brightness. Its frequency-domain denoising has different detail, artifact, and noise trade-offs from spatial-domain merging.

C.2 Additional improvements to FFCC

  • Adding face-only pixel and edge histograms lets FFCC reason separately about face and non-face content when estimating illumination neutralization.Source: §C.2 · p. 17
  • The spatial-domain method preserves detail better amid motion but costs more computation; the frequency-domain method reduces low-frequency noise better but can artifact around bright lights.Source: Figure 20 · p. 17
  • The model interpolates four brightness-based histograms to approximate separate daylight, indoor, night, and extreme-low-light models.Scene brightness uses image content and metadata; the four bin locations are training-set percentiles, and each receives one fourth of model capacity.Source: §C.2 · p. 18
  • The training set is large and diverse, and a separate tool annotates the desired white balance for each image.Source: Figure 21 · p. 18
  • Figure 22 visualizes a smooth RBF-regression mapping between calibrated reference white points from the IMX520 and IMX378 sensors.Source: Figure 22 · p. 18

C.3 Sensor Calibration

Sensor calibration maps camera-specific imagery into a canonical color space so one cross-camera white-balance model can be learned and applied.

C.3 Sensor Calibration

  • Because sensor spectral sensitivities vary across generations and within a camera model, the system maps each camera's imagery into a canonical color space.Source: §C.3 · p. 18
  • The calibration mapping uses Gaussian-kernel RBF interpolation on observed and canonical log-UV coordinates, with regularization toward a near-identity RGB-gain warp.The kernel scale is σ = 0.3 and the regularization weight is λ = 0.1.Source: §C.3 · p. 18
  • The system trains one cross-camera FFCC model in canonical space, then maps new-image colors into that space and transforms estimated gains back to the sensor space.The inverse transformation uses Gauss-Newton and usually converges in three or four iterations.Source: §C.3 · p. 19

D COMPARISONS TO OTHER SYSTEMS

The comparisons show differences from prior burst processing, commercial low-light modes, and an end-to-end CNN. Outcomes vary with scene lighting and camera setup, and the CNN comparison uses color-matched pipeline results.

D COMPARISONS TO OTHER SYSTEMS

Related papers

Loading 1910.11336v1…