PaperlayerView exact revision on arXiv

Source-linked AI summary

Single-shot Hyperspectral-Depth Imaging with Learned Diffractive Optics

Seung-Hwan Baek, Hayato Ikoma, Daniel S. Jeon, Yuqi Li, Wolfgang Heidrich, Gordon Wetzstein, Min H. Kim

arXiv:2009.00463v3eess.IVcs.CV

TL;DR

Existing HS-D approaches combine separate spectral and depth imaging systems, with added size, capture time, cost, and alignment burdens. This paper uses a learned DOE and jointly optimized reconstruction network to recover depth and spectrum from one image, reporting better form factor and accuracy than state-of-the-art single-shot methods and alternative optical designs.

  • Problem

    Existing HS-D systems combine separate spectral and depth imagers, increasing form factor, cost, capture time, and alignment or registration problems.

    Source: §1 · p. 1

  • Method

    A differentiable imaging pipeline jointly optimizes a DOE and CNN to reconstruct depth and spectrum from a single captured image.

    Source: §1 · p. 1

  • Results

    The HS-D method outperforms state-of-the-art single-shot methods and alternative optical designs in form factor and accuracy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments with simulation and a real system validate the method’s quality and accuracy, supporting compact and portable HS-D imaging without compromising them.

    Source: §7 · p. 8

    The method struggles with low-light or intensity-saturated scenes, and its DOE is customized to a target camera configuration.

    Source: §6 · p. 8

Abstract

from arXiv · show

Imaging depth and spectrum have been extensively studied in isolation from each other for decades. Recently, hyperspectral-depth (HS-D) imaging emerges to capture both information simultaneously by combining two different imaging systems; one for depth, the other for spectrum. While being accurate, this combinational approach induces increased form factor, cost, capture time, and alignment/registration problems. In this work, departing from the combinational principle, we propose a compact single-shot monocular HS-D imaging method. Our method uses a diffractive optical element (DOE), the point spread function of which changes with respect to both depth and spectrum. This enables us to reconstruct spectrum and depth from a single captured image. To this end, we develop a differentiable simulator and a neural-network-based reconstruction that are jointly optimized via automatic differentiation. To facilitate learning the DOE, we present a first HS-D dataset by building a benchtop HS-D imager that acquires high-quality ground truth. We evaluate our method with synthetic and real experiments by building an experimental prototype and achieve state-of-the-art HS-D imaging results.

1. Introduction

Existing HS-D systems combine separate spectral and depth imagers, while this work proposes a compact single-shot alternative using a learned DOE and jointly optimized reconstruction. The authors also introduce a benchtop HS-D dataset and report improved form factor and accuracy over prior single-shot methods and alternative optical designs.

1. Introduction

  • Separate spectral and depth imagers can acquire both accurately, but increase form factor, cost, and capture time and create alignment and registration problems.Source: §1 · p. 1
  • A single DOE and conventional camera encode depth and spectrum in one image, with a CNN reconstruction jointly optimized with the DOE by backpropagation.The DOE’s shape varies with both depth and spectrum.Source: §1 · pp. 1–2, Figure 2 · p. 2
  • A benchtop system combining structured-light 3D scanning and bandpass-filter hyperspectral imaging supplies ground-truth HS-D data for supervised joint optimization.The authors describe it as the first HS-D dataset.Source: §1 · p. 2
  • The method outperforms state-of-the-art single-shot HS-D methods and alternative optical designs in both form factor and accuracy.The reported comparison is for the method trained on the HS-D dataset.Source: §1 · p. 2
  • The proposed compact monocular method reconstructs a depth map and hyperspectral image from a single shot.Source: Figure 1 · p. 1, §1 · p. 2

2. Related Work

Prior work has studied hyperspectral and depth imaging separately, and HS-D systems have generally combined separate modalities or restricted snapshot systems. This work extends spectrum-from-PSF imaging by jointly learning a DOE and reconstruction network for compact monocular HS-D capture.

Hyperspectral Imaging

  • Spectrum-from-PSF methods estimate hyperspectral images using spectrally varying point spread functions; this work extends that approach to capture depth as well as spectrum.Source: §2 · p. 2

Depth Imaging

  • Depth estimation has used depth-dependent PSFs, coded apertures, and phase masks, while traditional depth-from-defocus analyzes two or more images.Diffractive depth-dependent spectral PSFs have also been used for all-in-focus particle imaging velocimetry.Source: §2 · p. 2

Hyperspectral-depth Imaging

  • Hyperspectral-depth systems combine separate spectral and depth modalities, while some snapshot monocular systems pair CASSI with light-field or time-of-flight imaging but are restricted to indoor scenes and large optical-table setups.The cited approaches can also face cross-modality parallax and alignment problems.Source: §2 · p. 2

Differentiable Optical Simulation

  • The authors jointly optimize a DOE and CNN for snapshot hyperspectral-depth imaging, extending differentiable optical-design work to this task.Source: §2 · p. 3

3. Diffraction-based HS-D Encoding

The method models how a DOE’s point spread function varies with wavelength and scene depth, encoding both in a single image. A differentiable Fourier-optics simulator models this image formation for joint hyperspectral and depth reconstruction.

3. Diffraction-based HS-D Encoding

  • The differentiable PSF simulator computes a PSF from the DOE height map, wavelength, and scene depth.The wave field is modeled at the aperture plane, passes through the aperture and DOE, and propagates to the sensor, where the PSF is the squared magnitude of the complex wave field.Source: §3 · p. 3
  • The PSF varies with wavelength and depth, so its optical cues can in principle support recovering both from a single image.The model sums scene, DOE, and focal phase delays and transforms them into the frequency domain.Source: §3 · p. 3
  • The image simulator combines dataset hyperspectral images and depth maps with the PSF to synthesize RGB sensor images.It uses a convolutional PSF model and layered scene representation; the model adds signal-independent Gaussian noise and excludes Poisson noise.Source: §3 · pp. 3–4
  • Figure 4 shows PSF variation across 420–660 nm and 0.4–2.0 m, with the reconstruction network trained end-to-end.Figure 5 depicts an encoder taking the RGB sensor image and two decoders producing a hyperspectral image and depth map, with skip connections and residual learning via spectral upsampling.Source: Figure 4 · p. 4, Figure 5 · p. 4
  • Sampling uses 25 spectral channels from 420 to 660 nm and seven depth levels from 0.4 to 2.0 m, linearly spaced in disparity.The paper states that dense sampling supports estimating HS-D images from PSF spectral cues and accurately simulating image formation.Source: §3 · p. 4

4. End-to-End HS-D Reconstruction

A depth- and spectrum-dependent point spread function lets the system reconstruct a spectral image and depth map from one captured image. A differentiable simulator and reconstruction network are optimized together.

4. End-to-End HS-D Reconstruction

  • A U-Net-based dual-decoder network reconstructs depth and the hyperspectral image, with camera-response-weighted RGB upsampling added to the spectral decoder output.The spectral output has 25 channels versus three input RGB channels.Source: §4 · pp. 4–5
  • The method outperforms the state-of-the-art HS-D imaging method [12], which combines light-field imaging and compressive spectral sensing.The comparison is reported in Table 1.Source: Table 1 · p. 5
  • The optimized DOE outperforms Fresnel, spiral [19], and Fisher [37] DOE designs for HS-D imaging.Table 2 reports the design comparison.Source: Table 2 · p. 5
  • Training minimizes mean absolute error on inverse depth and the hyperspectral image, plus a depth total-variation regularizer.The balancing weights are α = 1, β = 10^-2, and γ = 10^-2.Source: §4 · p. 5
  • The differentiable simulator and reconstruction network are jointly optimized by backpropagation, using 256×256 patches and a DOE height initialized by Fisher information.The DOE height is h and the network parameters are θ.Source: §4 · p. 5
  • Learned PSFs vary with spectrum and depth while remaining compact, balancing discriminative cues against spatial occupancy and metameric ambiguity.The reconstruction combines low-level optical PSF cues with higher-level image context.Source: §4 · p. 5

5. HS-D Dataset

The authors build a benchtop HS-D imager to create aligned hyperspectral reflectance and depth ground truth for training and evaluation.

5. HS-D Dataset

  • The dataset contains 18 aligned reflectance-image and depth-map pairs, with objects arranged to avoid occluding structured illumination.Background masks support patch sampling from valid object regions.Source: §5 · p. 5
  • Reflectance is augmented using 29 CIE standard illuminants, producing 522 hyperspectral images across 420–680 nm at 10 nm intervals.Source: §5 · p. 5
  • Depth spans 0.4–2.5 m, and patch-wise training uses 20,000 distinctive patches.The ground truth is acquired by combining structured-light scanning with LCTF-based hyperspectral imaging.Source: §5 · p. 5

6. Results

Across synthetic comparisons and real prototype captures, the method reconstructs hyperspectral and depth information with a compact DOE-based camera. Comparisons report gains over prior HS-D and alternative DOE designs, alongside spectral accuracy and limitations in challenging scenes.

HS-D Imaging

  • The compact method outperforms the single-shot HS-D comparator in Table 1 and Figure 7 while producing higher-accuracy spectral and depth data with a smaller form factor.The comparator combines compressive spectral imaging and light-field imaging.Source: §6 · p. 5, Figure 7 · p. 6

DOE Design

  • The learned DOE outperforms Fresnel, spiral, and Fisher designs, preserving high-frequency structure where alternatives show washed colors or degraded spatial resolution.All DOE designs use the same reconstruction network for a fair comparison.Source: §6 · p. 6, Figure 8 · p. 6

Hyperspectral Imaging

  • The method outperforms two hyperspectral-only systems in spectral accuracy while achieving second-best spatial-structure performance, and also reconstructs depth.The compared systems show smooth edges or overly smoothed spectral variation.Source: §6 · pp. 6–7, Figure 10 · p. 7

Depth Imaging

  • The learned DOE produces clearer depth maps than two DOE-based depth-only methods while also reconstructing spectral information.The alternatives are not designed for hyperspectral imaging.Source: §6 · p. 7, Figure 12 · p. 8

PSF Dependency and Discriminating Power

  • The learned PSF varies insignificantly between zero and eight degrees of incident angle, while Fisher-DOE initialization was chosen for its higher CRLB discriminating power.The reported CRLBs for Fresnel, Spiral, and Fisher DOEs are 2.73, 1.89, and 1.36; the authors note CRLB was designed for a single 3D point and impulse spectral peak.Source: §6 · p. 8

Spectral Evaluation

  • 0.0478 mean reflectance RMSE was measured across 24 ColorChecker patches under CIE D65 in simulation, despite excluding those images from training.The reconstructed spectral power distributions closely match ground truth.Source: Figure 11 · p. 7, §6 · p. 8

HS-D Imaging Prototype

  • The prototype captured five real-world scenes, with ground-truth spectrum and depth measured using a spectroradiometer and laser distance meter.Scenes were captured under sunlight and office-LED lighting; Figure 9 compares normalized radiance at indicated points with spectroradiometer measurements.Source: Figure 9 · p. 7, §6 · p. 8

Limitations

  • The method struggles with low-light and intensity-saturated scenes; memory demands constrain larger reconstruction networks, and DOE generalization beyond a target camera configuration remains future work.Fabrication-related long-tail PSFs also reduce contrast in real captures.Source: §6 · p. 8

7. Conclusion

The paper presents a single-shot HS-D system that jointly exploits depth and spectral dependence in a learned DOE and creates an HS-D dataset for joint learning. Simulation and real-system experiments validate the method, which the authors present as enabling compact, portable imaging without compromising quality and accuracy.

Conclusion

  • The system combines a learned DOE with a conventional DSLR camera to capture hyperspectral and depth information in a single shot.It exploits the depth and spectral dependence of the point spread function through end-to-end learning.Source: §7 · p. 8
  • The authors created the first HS-D dataset to enable the joint learning procedure and report validation through simulation and real-system experiments.Source: §7 · p. 8
  • The method is presented as reducing HS-D imaging form factor to support compact, portable imaging without compromising quality and accuracy.Source: §7 · p. 8

Related papers

Loading 2009.00463v3…