PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Optics for Single-shot High-dynamic-range Imaging

Christopher A. Metzler, Hayato Ikoma, Yifan Peng, Gordon Wetzstein

arXiv:1908.00620v1eess.IVcs.CV

TL;DR

Single-shot HDR imaging is difficult because sensor saturation destroys bright-scene details that CNNs operating on one LDR image may only hallucinate. The paper jointly trains an optical PSF encoder and CNN decoder, fabricates the optimized lens as a camera add-on, and reports better results than purely CNN-based and other PSF-engineering approaches. The approach preserves bright-scene information optically but makes post-processing necessary and remains vulnerable to large saturated regions and optical imperfections.

  • Problem

    Sensor saturation irreversibly loses bright-scene intensity information, while conventional single-image CNN approaches may hallucinate saturated details rather than faithfully recover them.

  • Method

    The paper jointly optimizes a diffractive optical element, parameterized by its PSF, and a CNN decoder so the optics encode HDR information into limited-range sensor measurements.

  • Results

    The proposed deep-optics system generally outperforms alternative single-shot HDR approaches in extensive simulations and is validated with a fabricated optical element and physical prototype.

  • Takeaways & Limitations

    Optically encoding bright-scene information before CNN reconstruction provides a practical single-shot HDR imaging route using a fabricated add-on lens for a conventional camera.

  • Takeaways & Limitations

    Post-processing is necessary, and reconstruction can be limited by extremely large saturated regions, PSF-calibration artifacts, blur, glare, and reduced effective field of view.

Abstract

from arXiv · show

High-dynamic-range (HDR) imaging is crucial for many computer graphics and vision applications. Yet, acquiring HDR images with a single shot remains a challenging problem. Whereas modern deep learning approaches are successful at hallucinating plausible HDR content from a single low-dynamic-range (LDR) image, saturated scene details often cannot be faithfully recovered. Inspired by recent deep optical imaging approaches, we interpret this problem as jointly training an optical encoder and electronic decoder where the encoder is parameterized by the point spread function (PSF) of the lens, the bottleneck is the sensor with a limited dynamic range, and the decoder is a convolutional neural network (CNN). The lens surface is then jointly optimized with the CNN in a training phase; we fabricate this optimized optical element and attach it as a hardware add-on to a conventional camera during inference. In extensive simulations and with a physical prototype, we demonstrate that this end-to-end deep optical imaging approach to single-shot HDR imaging outperforms both purely CNN-based approaches and other PSF engineering approaches.

1. Introduction

Single-shot HDR imaging must recover scene details lost when sensor pixels saturate. The paper addresses this by jointly learning an optical encoder and CNN decoder, then fabricating the optimized optical element for a conventional camera.

  • Sensor saturation irreversibly loses bright-scene intensity information, a problem worsened by shrinking pixel sizes and their lower full well capacity.
  • The system jointly optimizes an optical PSF encoder and CNN decoder so the optics preserve saturated-pixel information in nearby low-dynamic-range measurements.
  • The optimization addresses phase retrieval, saturated-region inpainting, and deconvolution as coupled inverse problems.
  • Extensive simulations show better results than alternative single-shot HDR approaches, while a fabricated diffractive element validates the system experimentally as a camera add-on.
  • The proposed multiplexing approach uses a learned, grating-like diffractive optical element to create shifted and scaled image copies for reconstructing bright scene regions.
  • The approach makes computational processing integral to image formation and may fail when measurements contain extremely large saturated regions.

2. Related Work

Prior HDR systems use multiple exposures, multiple sensors, specialized sensor patterns, optical coding, or CNN hallucination to address limited sensor dynamic range. This work instead jointly optimizes an optical filter and CNN, combining encoded scene information with learned reconstruction.

  • Multi-shot HDR methods can fail under motion, while multi-sensor systems introduce calibration, cost, and form-factor challenges.
  • Single-shot alternatives include reverse tone mapping, spatially varying exposures or ISO, coded PSFs, modulo cameras, and CNN-based hallucination.
  • The proposed method encodes HDR content into the sensor image with an optimized PSF rather than hallucinating missing values from a conventional LDR image.
  • Unlike prior optical filtering, the optical element and reconstruction CNN are trained end-to-end, and the fabricated lens demonstrates single-shot HDR capture with a prototype camera.
  • The paper extends deep optics beyond earlier applications by evaluating end-to-end optics and image processing for single-shot HDR imaging with a custom prototype.

3. End-to-end HDR Imaging

The method jointly optimizes a lens PSF and CNN decoder to recover HDR scenes from clipped, noisy single-shot sensor measurements. A differentiable optical model simulates image formation, while fabrication constraints and batchwise loss design shape the learned optical element.

  • Image formation: The camera model maps scene irradiance x through PSF convolution and read noise, then applies a linear response with saturation clipping to produce sensor image y.The model assumes a shift-invariant PSF and ignores quantization; nonlinear response functions can be calibrated to approximate linear response.
  • Optimization objective: The goal is to jointly optimize the optical PSF h and reconstruction algorithm G so that HDR scenes with ∥x∥∞≫1 can be recovered from LDR measurements.The optical system includes a conventional lens with a custom DOE add-on whose surface profile determines the PSF.
  • Optical model: The DOE surface profile φ induces wavelength-dependent phase delays, and Fourier-optics propagation produces the sensor PSF h from the resulting wave field.The DOE uses a binary circular aperture mask, while the compound lens is approximated by a thin-lens model.
  • CNN reconstruction: The CNN decoder uses a five-scale U-Net with skip connections, four downsampling operations, four upsampling operations, and 64 feature maps per layer.ReLU activations, BatchNorm, and bilinear-initialized transposed convolutions are included in the architecture.
  • Loss and fabrication constraints: The training objective uses batchwise γ-corrected ℓ2 loss with γ = 1/2, equivalent to minimizing RMSE over batches rather than typical MSE.This formulation is intended to improve robustness to outlier reconstructions, while clipping and a Laplacian smoothness term constrain the lens profile for fabrication.

4. Analysis and Evaluation

Evaluation compares conventional LDR reconstruction, CNN baselines, star-PSF coding, and end-to-end deep optics. The learned PSF multiplexes shifted and scaled image copies, and the end-to-end approaches achieve the strongest reported image quality.

  • Optical coding: The optimized deep-optics PSF contains several peaks that multiplex shifted and scaled copies of the image, with chromatic channel offsets caused by the single DOE design.The unconstrained variant mainly blurs green while focusing red and blue, whereas the physically realizable design produces distinct chromatic peaks.
  • Qualitative evaluation: In Figure 5, the physically realizable end-to-end deep-optics approach achieves the best results among the compared reconstruction methods.The comparison includes conventional LDR, HDR-CNN, a smaller U-Net, a star-PSF U-Net, and the proposed constrained deep-optics system.
  • Quantitative evaluation: The end-to-end approaches achieve the best image quality across HDR-VDP-2 perceptual difference and PSNR measurements reported for the entire test set.The unconstrained approach is usually better than the physically realizable version, while its PSNR-γ is slightly lower.

5. Fabrication and Implementation

The optimized phase profile is fabricated as a diffractive optical element using PDMS replica molding. The fabricated DOE has a diameter of 5 mm and is evaluated against its optimized height profile.

  • Lens fabrication: The optimized diffractive optical element is fabricated from polydimethyl-siloxane using replica molding and has a diameter of 5 mm.Figure 6 compares the optimized height profile with profilometer measurements and a 3D rendering of the fabricated DOE.

LDR

The system combines a fabricated diffractive optical element with CNN reconstruction, and comparisons show strong HDR reconstruction quality. The measured optical response broadly matches simulation despite blur and glare from fabrication and modeling limitations.

  • LDR: Figure 5 compares conventional LDR, CNN, U-Net, star-shaped-PSF, and end-to-end deep optics reconstructions.The comparisons use full images at -1 stop and regions of interest at -3 stops.
  • LDR: The end-to-end approach achieves the best PSNR, observed image quality, and HDR-VDP-2 Q value across the examples.Color-coded insets indicate the predicted probability of visible differences from the ground-truth HDR images.
  • LDR: The fabricated DOE produces a PSF with a central peak and lower-amplitude satellite peaks at different locations across color channels.The grating-like lens surface multiplexes shifted and scaled image copies, with visible chromatic aberrations.
  • LDR: The measured PSF is slightly blurrier than the simulated PSF, with additional glare attributed to fabrication errors and interreflections.These differences affect the recovered optical response but do not eliminate the general correspondence between measured and simulated shapes.
  • LDR: The DOE is mounted as a hardware add-on to a conventional SLR camera, with its exact distance from the compound lens not modeled precisely.The setup uses a Canon Rebel T5, a Nikon Nikkor 35 mm lens, and a Thorlabs-mounted DOE.

6. Experimental Results

Prototype experiments cover three indoor scenes and one outdoor night scene containing saturated bright details. The deep-optics measurements preserve information about those details for CNN-based reconstruction, unlike direct LDR reconstruction that must inpaint them.

  • 6. Experimental Results: The prototype was evaluated on three laboratory scenes and one outdoor scene captured at night.The experiments compare captured measurements, CNN reconstructions, reference LDR and HDR images, and HDR-CNN results.
  • 6. Experimental Results: Captured LDR images contain saturated regions whose scene details, including a light-bulb filament and wall-light structure, are lost without optical encoding.These examples include details that remain present in the reference HDR images.
  • 6. Experimental Results: Direct HDR-CNN reconstruction hallucinates smooth bright regions because the saturated LDR measurements provide no information about the missing detail.The paper identifies these scenes as cases where inpainting cannot reproduce the actual saturated content.
  • 6. Experimental Results: The prototype optically encodes bright detail by superimposing shifted and scaled image copies, which the CNN uses to recover saturated scene content.The reconstruction digitally removes the image copies after using their encoded information.

7. Discussion

The paper frames deep optics as joint optical and electronic design for preserving saturated HDR information, with simulations and a prototype supporting its effectiveness. It also identifies computational, calibration, optical, and training-distribution limits.

  • 7. Discussion: The method jointly trains a PSF-based optical encoder and CNN decoder so bright scene details are encoded into LDR measurements.Its multiplexing PSF superimposes multiple shifted exposures while the CNN reconstructs the HDR image.
  • 7. Discussion: The framework is presented as the first application of end-to-end optics-and-image-processing optimization to single-shot HDR imaging.This extends the emerging end-to-end deep-optics methodology to HDR capture.
  • 7. Discussion: Unlike direct LDR CNN reconstruction, deep optics preserves bright-region information optically rather than relying solely on inpainting.Prototype results show recovery of details that direct HDR-CNN reconstruction replaces with smooth regions.
  • 7. Discussion: The method makes computational processing integral to image formation and may fail when extremely large saturated regions cause even lower-intensity copies to saturate.Training data must represent the HDR values expected during inference because untrained conditions can produce low-quality results.
  • 7. Discussion: Fabrication-related blur and glare complicate deconvolution and can make the PSF shift variant, limiting the effective field of view.The paper suggests aperture-plane integration, more general optical modeling, or anti-reflection coatings as possible improvements.

8. Conclusion

The paper presents end-to-end optics and reconstruction design as a paradigm for domain-specific computational cameras, demonstrating benefits for single-shot HDR imaging and potential for richer visual information.

  • End-to-end optics and reconstruction design benefits single-shot HDR imaging and may enable domain-specific cameras with richer visual information and new imaging modalities.
Loading 1908.00620v1…