Source-linked AI summary
Virtual Wave Optics for Non-Line-of-Sight Imaging
Xiaochun Liu, Ibón Guillén, Marco La Manna, Ji Hyun Nam, Syed Azer Reza, Toan Huu Le, Diego Gutierrez, Adrian Jarabo, Andreas Velten
TL;DR
The paper formulates NLOS imaging as virtual wave imaging by representing intensity variations with a phasor field and modeling propagation with Rayleigh-Sommerfeld operators. The resulting framework supports configurable virtual cameras and transient capture, while practical boundaries include difficulty implementing full H without an array sensor.
Problem
NLOS systems need methods that can handle complex light transport, including multiply scattered and ambient light, without relying on restrictive scene assumptions.
Method
The method models NLOS transport as virtual wave propagation using phasor fields, configurable projector-camera functions, and Rayleigh-Sommerfeld operators.
Results
The method produces cleaner contours and improved contrast on confocal data and performs automatically on noisy data without explicit noise-level estimation.
Takeaways & Limitations
The framework enables virtual camera systems for refocusing and transient visualization of multibounce NLOS light transport.
Takeaways & Limitations
Capturing a full H remains difficult without an array sensor, which the authors leave for future work.
Abstract
from arXiv · showhide
Non-Line-of-Sight (NLOS) imaging allows to observe objects partially or fully occluded from direct view, by analyzing indirect diffuse reflections off a secondary, relay surface. Despite its many potential applications, existing methods lack practical usability due to several shared limitations, including the assumption of single scattering only, lack of occlusions, and Lambertian reflectance. We lift these limitations by transforming the NLOS problem into a virtual Line-Of-Sight (LOS) one. Since imaging information cannot be recovered from the irradiance arriving at the relay surface, we introduce the concept of the phasor field, a mathematical construct representing a fast variation in irradiance. We show that NLOS light transport can be modeled as the propagation of a phasor field wave, which can be solved accurately by the Rayleigh-Sommerfeld diffraction integral. We demonstrate for the first time NLOS reconstruction of complex scenes with strong multiply scattered and ambient light, arbitrary materials, large depth range, and occlusions. Our method handles these challenging cases without explicitly developing a light transport model. By leveraging existing fast algorithms, we outperform existing methods in terms of execution speed, computational complexity, and memory use. We believe that our approach will help unlock the potential of NLOS imaging, and the development of novel applications not restricted to lab conditions. For example, we demonstrate both refocusing and transient NLOS videos of real-world, complex scenes with large depth.
A.1 The phasor field Rayleigh-Sommerfeld integral
The phasor-field model derives NLOS light propagation from time-varying intensity and expresses it using a Rayleigh-Sommerfeld diffraction integral. An approximation converts the source integral into the RSD form while introducing only a slow-varying amplitude error.
- Point-source phasor field: The phasor field is derived from sinusoidally modulated light intensity emitted by a point source.The source has amplitude L0 and modulation frequency ω.
- Point-source phasor field: Propagation to a destination accounts for travel time and radial intensity falloff before forming the point-source phasor field.Travel time is tp = |xd − xs|/c, with c denoting propagation speed.
- Surface-source propagation: For incoherent sources distributed over a surface S, the destination phasor field is obtained by integrating contributions from the entire surface.The resulting expression contains the source-to-destination distance in both phase and amplitude terms.
- RSD approximation: The source-distance approximation factors out the average source-to-destination distance, making the expression equivalent to the Rayleigh-Sommerfeld propagator.The approximation replaces |xs − xd|^2 with |xs − xd||<S̄> − xd|.
- RSD approximation: The approximation preserves the phase term but causes a slow-varying amplitude error that can be precomputed for a known source plane.The method generally does not invert the 1/r falloff, and further formulations may address this error.
A.2 Linearity of the RSD integral
The Rayleigh-Sommerfeld operator is linear and commutes with temporal convolution, enabling multibounce propagation and reconstruction from impulse-response measurements.
- Operator properties: The RSD propagation operator is linear, so propagating a weighted sum of waves equals summing their individually propagated waves.For waves P1 and P2, R(aP1 + bP2) = aR(P1) + bR(P2).
- Operator properties: Temporal convolution can be applied either before RSD propagation or after propagation through the operator.This property follows directly from the integral representation of R.
- Multibounce transport: Multibounce NLOS transport is modeled by applying the RSD operator once for each light-transport bounce.The scene response therefore results from repeated wave propagation across successive surfaces.
- Impulse-response reconstruction: The scene response H is measured using spatial and temporal impulse functions, after which arbitrary input fields are represented as superpositions of these impulses.The continuous formulation corresponds to integrating over source points.
A.3.1 Phase operator of an ideal lens
An ideal lens focuses a planar wavefront to a point by canceling the spherical-wave phase, and this phase operation can equivalently be represented as a time shift.
- Ideal-lens behavior: An ideal lens converts a planar wavefront into a point at focal distance f, or reverses this process for light from that point.A point source produces spherical wavefronts whose phase varies across the lens plane.
- Phase operation: The lens applies a phase shift that cancels the spherical-wave phase term across the lens plane.This produces the phase profile required for focusing or collimation.
- Phase operation: For a wave expressed as a superposition of monochromatic components, the lens phase shift acts on each component of the wavefront.The resulting operation can be formulated for broadband waves through their monochromatic decomposition.
- Time-shift interpretation: The ideal-lens phase shift is equivalent to shifting the wave in time.This equivalence provides the basis for implementing the lens operation in the virtual wave framework.
- Modeling choice: The phasor-field lens cancels propagation phase but leaves the 1/r falloff uncorrected to avoid added complexity unless quantitative albedo reconstruction is required.The authors state that correcting this falloff is possible but may only be warranted for quantitative albedo information.
A.3.2 Example Projector and Camera Functions
The framework defines virtual projectors and camera operators computationally, enabling several LOS-style imaging systems for NLOS reconstruction. The examples include photography, transient femtophotography, and confocal imaging, each with distinct capabilities and trade-offs.
- Virtual camera framework: The model implements arbitrary virtual cameras by defining a projector function and camera operator computationally.These functions are based on physical LOS systems but are not constrained by actual hardware.
- Photography camera: The conventional photography system reconstructs 2D NLOS images without requiring the light source position or timing.Its resolution depends on the temporal bandwidth of the projector function.
- Transient camera: The femtophotography system captures videos of light transport through the NLOS scene using a short illumination pulse and a camera focused at the illuminated depth.It can visualize multibounce transport and separate direct and global components, but video generation is computationally expensive.
- Confocal camera: The confocal system images selected voxels in a volume using a focused ultrashort pulse.Increasing the virtual pulse width worsens depth resolution but improves signal-to-noise ratio; σ = 6 ω was found effective in practice.
- Capture arrangement: The current capture arrangement uses laser scanning and one stationary SPAD by exploiting reciprocity to interchange source and camera positions.Multiple SPADs could remove scanning or provide more data points in future systems.
A.4 Resolution Limits
The paper derives NLOS resolution limits from wave-based imaging criteria and shows that the limit depends on the virtual projector-camera configuration. Focusing on both illumination and detection improves resolution relative to one-sided focusing.
- Resolution criteria: NLOS resolution can be estimated using the Rayleigh criterion for wave-based systems, with limits depending on aperture, distance, pulse duration, and light speed.A proposed limit is Δx = 1.22cτd/L.
- Virtual focusing: Focusing on both the light-source and detector sides doubles resolution compared with focusing on only one side.The paper relates this behavior to confocal and structured-illumination microscopy.
- Method comparison: Table 2 compares the reconstruction complexity of FBP, CNLOS, Fast FBP, and the proposed method.
A.5 Complexity analysis
The complexity analysis decomposes the point-scanner reconstruction into temporal convolutions and back-projection. With fast back-projection, both operations share the same asymptotic complexity, yielding O(NpNaNt).
- A.5 Complexity analysis: The point-scanner reconstruction consists of a set of 1D convolutions followed by back-projection.
- A.5 Complexity analysis: The 1D convolutions cost O(NpNaNt) when the kernel size Nk is treated as constant.Replacing the discrete Fourier transform with an FFT does not reduce complexity because the kernel is small and fixed.
- A.5 Complexity analysis: Naive back-projection has complexity O(NpNcNv) and dominates the convolution cost under the stated sampling assumptions.
- A.5 Complexity analysis: Fast back-projection reduces the cost to O(NpNaNv^1/3), which is at most O(NpNaNt) when Nt ≥ Np, Na.
- A.5 Complexity analysis: Under these assumptions, convolution and back-projection have matching complexity, so the proposed method costs O(NpNaNt).
A.6 Fresnel Approximation
The method uses the exact Rayleigh-Sommerfeld diffraction propagator and then adopts the Fresnel approximation for efficient virtual-camera computation. The approximation assumes propagation between parallel planes with sufficiently large separation.
- A.6 Fresnel Approximation: The Rayleigh-Sommerfeld propagator provides an exact wave-equation solution but is computationally expensive for optical imaging.The framework therefore uses more efficient Fourier-optics methods, including the Fresnel approximation.
- A.6 Fresnel Approximation: For parallel planes separated by z, the plane-to-plane propagator is defined between source and destination coordinates.
- A.6 Fresnel Approximation: The approximation assumes z is much larger than the transverse coordinate differences, allowing the slowly varying distance factor to be replaced by 1/z.
- A.6 Fresnel Approximation: The rapidly oscillating phase term is approximated using a binomial expansion of the propagation distance.
- A.6 Fresnel Approximation: With the Fresnel approximation, propagation between planes can be interpreted as a 2D convolution and used to reconstruct a focal plane.Validity depends on aperture radius d, propagation distance L, and wavelength λ.
A.7 Comparison with other methods using the confocal dataset
The confocal-dataset comparison evaluates CNLOS deconvolution, LOG filtered backprojection, and virtual wave imaging on shared public datasets. The virtual wave method improves contrast and contour cleanliness on simple scenes and performs automatically on noisy datasets without explicit noise estimation.
- Comparison setup: The comparison uses CNLOS deconvolution, LOG filtered backprojection, and virtual wave imaging on identical public datasets without preprocessing.Each dataset occupies a row, with methods arranged from left to right in the comparison figures.
- Configuration scope: The virtual wave and filtered-backprojection methods are not limited to confocal data and can support simpler capture configurations and more complex scenes.The comparison section distinguishes these methods from the confocal-specific dataset configuration.
- Simple scenes: For simple isolated geometries, all methods approach their resolution limits, while virtual wave imaging produces higher contrast and cleaner contours.The comparison uses the published confocal dataset and the LOG-filtered backprojection configuration.
- Simple scenes: Virtual wave imaging improves contrast by handling multiply scattered light that contaminates the other reconstructions.Thresholding can reduce scattering in simple datasets, but that strategy fails in more complex scenes.
- Noisy scenes: On noisy datasets, filtered backprojection fails, whereas the phasor-field virtual wave method performs well without explicitly estimating the noise level.CNLOS deconvolution removes uniform background noise with a Wiener filter but requires explicit noise-level estimation.