PaperlayerView exact revision on arXiv

Source-linked AI summary

Spectral DiffuserCam: lensless snapshot hyperspectral imaging with a spectral filter array

Kristina Monakhova, Kyrollos Yanny, Neerja Aggarwal, Laura Waller

arXiv:2006.08565v2eess.IVcs.CVphysics.optics

TL;DR

Hyperspectral imaging is valuable but existing systems can be slow, expensive, bulky, or resolution-limited. The paper introduces a compact lensless camera that combines a sensor-mounted spectral filter array with a diffuser and sparse inverse reconstruction. The prototype demonstrates sub-super-pixel spatial resolution across 64 spectral channels, while the design remains spectrally configurable.

  • Problem

    Existing hyperspectral imagers are costly or slow, while snapshot systems can be bulky or trade spatial and spectral resolution.

  • Method

    A tiled spectral filter array and nearby diffuser multiplex spatio-spectral information for sparsity-based reconstruction from a single measurement.

  • Results

    64 spectral channels spanning 386-898nm achieve approximately 0.19 super-pixels for two-point resolution and 0.3 super-pixels for multi-point resolution.

  • Takeaways & Limitations

    Decoupled spectral and spatial responses allow contiguous or non-contiguous filters with user-selected bandwidths in a compact, inexpensive architecture.

  • Takeaways & Limitations

    The system has low light throughput from narrow-band filters and is scene-dependent because the light is spread by the diffuser.

Abstract

from arXiv · show

Hyperspectral imaging is useful for applications ranging from medical diagnostics to agricultural crop monitoring; however, traditional scanning hyperspectral imagers are prohibitively slow and expensive for widespread adoption. Snapshot techniques exist but are often confined to bulky benchtop setups or have low spatio-spectral resolution. In this paper, we propose a novel, compact, and inexpensive computational camera for snapshot hyperspectral imaging. Our system consists of a tiled spectral filter array placed directly on the image sensor and a diffuser placed close to the sensor. Each point in the world maps to a unique pseudorandom pattern on the spectral filter array, which encodes multiplexed spatio-spectral information. By solving a sparsity-constrained inverse problem, we recover the hyperspectral volume with sub-super-pixel resolution. Our hyperspectral imaging framework is flexible and can be designed with contiguous or non-contiguous spectral filters that can be chosen for a given application. We provide theory for system design, demonstrate a prototype device, and present experimental results with high spatio-spectral resolution.

1. INTRODUCTION

The paper targets hyperspectral imaging systems that are costly, slow, bulky, or resolution-limited, and introduces a compact snapshot architecture combining a diffuser with a tiled spectral filter array. The framework supports flexible spectral sampling and is evaluated through theory, simulations, and a prototype.

  • Motivation: Hyperspectral imaging captures spectral information at each spatial location, supporting material detection and classification beyond RGB imaging.Applications include crop monitoring, medical diagnostics, microscopy, and food quality analysis.
  • Motivation: Commercial hyperspectral cameras cost $25,000 - $100,000, while their price and size limit widespread use.
  • Motivation: Traditional scanners are slow and require precise moving parts, while snapshot methods can sacrifice spatial resolution or remain bulky.Computational approaches recover spectral cubes from encoded measurements but commonly use table-top optical systems.
  • Proposed approach: The proposed system places a tiled spectral filter array on the sensor and a diffuser nearby to multiplex each world point across many camera pixels.This multiplexing supports recovery of the full spatio-spectral cube without the resolution loss of a non-multiplexing optic.
  • Proposed approach: The encoding decouples spectral and spatial responses, allowing contiguous or non-contiguous user-selected spectral filters and bandwidths.Under scene-sparsity and diffuser-randomness conditions, filters determine spectral sampling while diffuser autocorrelation determines spatial resolution.
  • Contributions: The paper contributes a compressive snapshot framework, resolution theory and simulations, and a prototype demonstrating recovery on natural-scene data.

2. RELATED WORK

Prior snapshot hyperspectral systems use filter arrays, coded apertures, speckle, or dispersion, but often trade spatial resolution, spectral flexibility, or compactness. Spectral DiffuserCam combines a filter array with a lensless diffuser and sparse reconstruction to recover high-resolution hyperspectral information from a single measurement.

  • Existing approaches: Snapshot hyperspectral methods include spectral filter arrays, coded apertures, speckle-based systems, and dispersive approaches.
  • Existing approaches: Tiled spectral filter arrays increase spectral resolution as filters multiply but reduce spatial resolution, while photonic-crystal demonstrations remain limited to 10×10 spatial pixels.
  • Existing approaches: Coded-aperture systems capture hyperspectral images and videos but typically require large table-top assemblies with multiple lenses and optical components.The proposed system instead uses a sensor-attached spectral filter array and a nearby thin diffuser.
  • Existing approaches: Speckle systems can be compact, but wavelength-correlated speckle limits spectral resolution and complicates application-specific design.
  • Existing approaches: Dispersive methods can preserve spatial resolution but may encode spectra only at object edges, producing an ill-conditioned problem and lower spectral accuracy.
  • Positioning of this work: Spectral DiffuserCam combines a lensless diffuser, spectral filter array, and sparsity assumptions to reconstruct 64-wavelength hyperspectral data with decoupled spatial and spectral design.The design supports custom, non-contiguous spectral bands and achieves close to full sensor spatial resolution through diffuser multiplexing.

3. SYSTEM DESIGN OVERVIEW

The system uses a diffuser to spread each scene point across many spectral-filter pixels, avoiding the measurement gaps of a high-NA lens while improving on the spatial resolution of a low-NA lens. A forward model describes this multiplexed measurement for inverse reconstruction.

  • Architecture: The system deposits a spectral filter array on the sensor and uses a diffuser as the multiplexing optic for compact, inexpensive lensless imaging.
  • Architecture: The design compares high-NA and low-NA lenses with a diffuser-based architecture using a repeated 3 × 3 spectral filter array.The high-NA case matches the diffraction-limited spot to one filter pixel, whereas the low-NA case matches it to a super-pixel.
  • Forward model: The image-formation model sums contributions from spectral filter bands after scene information is spatially multiplexed across the array.The forward model supports iterative inverse reconstruction and analysis of spatial and spectral resolution.
  • Architecture: A high-NA lens can miss a point source outside a filter’s passband, while a low-NA lens records the spectrum at the cost of spatial resolution.
  • Architecture: A multiplexing optic avoids high-NA measurement gaps and provides better resolution than the low-NA case.
  • Multiplexing and recovery: The diffuser spreads each point source across many filter pixels and spectral bands, enabling compressed-sensing recovery whose spatial resolution depends on PSF autocorrelation and scene complexity.

4. IMAGING FORWARD MODEL

The forward model combines spectral filtering with diffuser-based spatial multiplexing to map a hyperspectral scene into a 2D sensor measurement. It assumes wavelength-invariant blur and depth-independent imaging beyond the hyperfocal distance.

  • Spectral filter model: The spectral filter array measures each sensor pixel as wavelength-wise point-wise products summed across K bands.The filter function Fλ[x,y] describes spectral transmittance and incorporates the sensor response.
  • Diffuser model: A smooth pseudorandom diffuser spatially multiplexes each wavelength by convolving the scene with an on-axis point-spread function.An aperture limits higher-angle rays, and compressed sensing can recover detail finer than the super-pixel size.
  • Diffuser model: The model uses a wavelength-invariant PSF, while allowing extension to a spectrally varying PSF when dispersion is stronger.The wavelength-invariant assumption is experimentally validated in the paper.
  • Diffuser model: For objects beyond the hyperfocal distance, negligible depth variance makes a 2D convolutional model valid.Objects within the hyperfocal distance require a 3D model to account for depth-dependent PSF variation.

C. Combined model

The combined forward model applies diffuser convolution and spectral filtering to produce the measured image through a linear operator A. Its intermediate variable is the blurred scene before wavelength-dependent filtering.

  • Combined model: The combined model first convolves each spectral scene slice with the diffuser PSF, then applies the filter function and sums across wavelengths.The intermediate w[x,y,λ] represents the PSF-convolved scene before point-wise filtering.
  • Combined model: The complete linear measurement process is represented by the matrix operator A.The final measurement is written as b = Av.

5. HYPERSPECTRAL RECONSTRUCTION

Hyperspectral reconstruction treats the 2D measurement as an underdetermined compressive-sensing inverse problem. The method combines 3D total variation, non-negativity, and spectral low-rank regularization, using a calibrated prototype and iterative optimization.

  • Reconstruction method: The reconstruction solves an underdetermined inverse problem using l1 minimization for the incoherent, multiplexed measurement.The optimization includes a non-negativity constraint, weighted 3D total variation, and a low-rank spectral prior.
  • Reconstruction method: FISTA with weighted anisotropic 3DTV solves the regularized reconstruction problem.The tuning parameters control the 3DTV and low-rank priors.
  • Prototype: The prototype uses a 28×20 array of 8×8 filter super-pixels spanning 386-898nm on a cropped 448×320 CMOS sensor.The diffuser is positioned 1cm from the sensor, and the array provides 64 filter channels per super-pixel.
  • Calibration: The diffuser PSF is experimentally calibrated across wavelengths and found to remain similar, supporting single-wavelength PSF calibration.Figure 4 separately characterizes the diffuser PSF and the filter-array spectral responses.
  • System performance: 1ms-13ms exposure times support video-rate acquisition, while computational reconstruction takes 12-24 minutes for 500-1000 iterations on an RTX 2080-Ti GPU.The reported reconstruction timing is for MATLAB implementation.

A. Filter Function Calibration

The system calibrates both the spectral filter response and diffuser point-spread function before reconstruction. Its measured non-idealities constrain spectral calibration and effective spectral-band performance.

  • Filter response calibration: A motorized monochromator produces a 5nm-FWHM narrow-band source for calibrating the filter response.The source is swept from 386nm to 898nm in 8nm increments.
  • Diffuser calibration: The diffuser PSF is calibrated separately because its smooth, large-scale features make the response nearly wavelength invariant.This allows calibration at a single wavelength using one point-source image.
  • System non-idealities: Undefined automatic contrast stretching makes measurements nonlinear and impedes imaging of dim objects.It also prevents reliable normalization of calibration images, potentially making some reconstructed wavelength bands appear too bright or dim.
  • System non-idealities: 49 effective spectral bands are supported instead of 64 because uniformly sampled calibration wavelengths do not match the filters’ 5-12nm center spacing and 6-23nm bandwidths.The paper displays all 64 bands, but some have overlapping spectral responses.
  • Resolution analysis: The system’s expected spatial resolution is approximately 0.3 super-pixels for complex scenes, based on condition-number analysis.The analysis compares diffuser performance with high-NA and low-NA lenses.

a. Theoretical Spatial Resolution

The system’s spatial resolution is characterized by two-point reconstructions and local condition-number analysis. It achieves 0.19 super-pixel two-point resolution, while approximately 0.3 super-pixels describes performance for more complex scenes.

  • Theoretical spatial resolution: 0.19 super-pixels is the theoretical two-point resolution, defined by the half-width of the autocorrelation peak at 70% of its maximum.This corresponds to approximately 3 sensor pixels.
  • Experimental spatial resolution: 0.19 super-pixel resolution is experimentally demonstrated across wavelengths using two-point reconstructions.The measured result matches the theoretical resolution.
  • Spectral resolution: Recovered spectral peaks match their true wavelengths within 5nm for narrow-band point sources.The spectral-resolution analysis overlays reconstructed spectra with ground-truth spectra.
  • Scene-dependent resolution: Because nonlinear regularization makes resolution object dependent, two-point measurements represent a best-case characterization.Local condition numbers estimate invertibility as scene complexity changes.
  • Scene-dependent resolution: The diffuser has a condition number below 40 for separations greater than approximately 0.3 super-pixels, outperforming the compared lenses.The low-NA lens requires about 1 super-pixel, while the high-NA lens has an erratic condition number due to missing information.
  • Scene-dependent resolution: Beyond 0.3 super-pixels, the diffuser condition number does not worsen arbitrarily as scene complexity increases.This establishes approximately 0.3 super-pixels as the expected spatial resolution for complex scenes.

D. Simulated Resolution Target Reconstruction

Simulated resolution-target reconstructions evaluate diffuser performance against low-NA and high-NA lenses. The diffuser resolves features at 0.3 super-pixels, whereas the low-NA lens needs roughly 1 super-pixel and the high-NA lens produces gaps.

  • Condition-number analysis: The analysis evaluates 2D spatial and 3D spatio-spectral condition numbers as separation distance varies.The 3D case varies both spatial and spectral positions, and a condition number below 40 is considered good.
  • Condition-number analysis: The diffuser consistently performs better at small separations than low-NA and high-NA lenses.The diffuser resolves objects as close as 0.3 super-pixels apart in more complex scenes.
  • Resolution-target reconstruction: 0.3-super-pixel features are resolved in simulated reconstructions, while the low-NA lens resolves roughly 1-super-pixel features and the high-NA lens produces gaps.The simulations add Gaussian noise with variance 1 × 10^-5 and use 2,000 FISTA iterations with 3DTV.

8. EXPERIMENTAL RESULTS

Experiments demonstrate that Spectral DiffuserCam reconstructs spatially and spectrally resolved scenes, while its filter design supports application-specific spectral sampling. The prototype currently remains constrained by light throughput, low-light performance, and scene sparsity.

  • Experimental reconstructions: The RGB LED reconstruction produced spectral profiles that closely matched spectrometer measurements.The scene contained four red, four green, and two blue LEDs.
  • Design flexibility: The spectral filters can be chosen for a specific application, including nonlinear sampling across a wide wavelength range.The authors identify this flexibility as a key advantage over previous compact snapshot hyperspectral imagers.
  • Simulated comparisons: The diffuser achieved higher spatial resolution and better accuracy than low-NA and high-NA lens designs in simulation.The comparison used reconstructions of a target illuminated at 634 nm, 570 nm, 474 nm, and broadband wavelengths.
  • Limitations: The current system has low light throughput and reduced SNR, making it unsuitable for low-light conditions.Narrow-band filters reject much of the light, while the diffuser spreads light over many pixels.
  • Limitations: Reconstruction performance depends on scene sparsity, and insufficiently sparse scenes may produce artifacts.The limitation arises from the reconstruction algorithm’s nonlinear regularization term and reliance on sparse gradients.

10. CONCLUSION

The paper presents an ultra-compact, inexpensive hyperspectral modality that combines a spectral filter array with lensless diffuser imaging. A prototype reconstructs complex spatio-spectral scenes with high resolution across 64 spectral bands.

  • Conclusion: The proposed modality combines a color filter array and lensless imaging for an ultra-compact and inexpensive hyperspectral camera.The filter array encodes spectral information, while the diffuser multiplexes incoming light across many spectral filters.
  • Conclusion: Compressive sensing reconstructs high spatio-spectral resolution from a single 2D measurement.The multiplexed measurement maps each world point to many spectral filters on the sensor.
  • Conclusion: The prototype achieved up to 0.19 super-pixel spatial resolution across 64 spectral bands.The paper also reports characterization of the system’s two-point and multi-point resolution.

Reconstructions

The reconstruction figures document simulated and experimental hyperspectral outputs, alongside cited background and related-work passages. The supplied reconstruction evidence includes spectral-profile comparisons and lens-design comparisons.

  • Experimental reconstructions: Experimental reconstructions display raw measurements, false-color images, xλ sum projections, and spectral line profiles for four spatial points.Ground-truth spectral line profiles measured with a spectrometer are plotted for reference.
Loading 2006.08565v2…