PaperlayerView exact revision on arXiv

Source-linked AI summary

Cascaded Non-Line-of-Sight Imaging

Diego Royo, María Peña, Forrest B. Peterson, Andreas Velten, Julio Marco, Diego Gutierrez

arXiv:2609.16017v1cs.GRcs.CV

TL;DR

Most time-of-flight NLOS methods assume third-bounce paths, leaving higher-order illumination underused and limiting supported scene configurations. This paper computes virtual impulse responses at hidden walls from a visible-wall capture, then demonstrates fourth- and fifth-bounce imaging in simulation and with a prototype. The approach also explores rough walls and multiple hidden walls to address visibility limits and reveal features from different viewpoints.

  • Problem

    Most existing NLOS methods assume third-bounce-only illumination, while higher-order paths remain underused and prior approaches cover restricted locations, orientations, or specular paths.

    Source: §1 · p. 1

  • Method

    The method computes an analogous virtual impulse response at a hidden wall from the captured relay-wall response, concatenating a second virtual NLOS system from one capture.

    Source: §1 · p. 1

  • Results

    Simulation and prototype results demonstrate fourth- and fifth-bounce imaging, including objects around two corners and in challenging orientations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Exploring rough-wall interactions and combining secondary walls provides different viewpoints for imaging hidden objects and observing previously unseen features.

    Source: §1 · p. 2

    Each concatenation imposes resolution loss from reduced signal, limited aperture sizes, longer focus distances, and increased multipath interference.

    Source: §9 · p. 11

Abstract

from arXiv · show

Time-of-flight non-line-of-sight (NLOS) imaging recovers information from hidden objects by analyzing the time of flight of indirect photons scattered on a visible (relay) wall. Most methods make the simplifying assumption that photons travel exclusively three-bounce paths, thus ignoring other useful information encoded in higher-order photons (with, e.g., four- or five-bounce paths). We present a novel cascaded NLOS imaging approach that leverages higher-order information and allows imaging a broader range of single- and multi-corner scenarios. We combine ultra-fast laser scanning with recent time-gated 2D sensor arrays to capture the scene's impulse response on a visible relay wall. From the captured impulse response, our method computes an analogous virtual impulse response at any other hidden wall. This effectively allows us to concatenate a second, virtual NLOS imaging system that leverages higher-order illumination. We validate our cascaded imaging method both in simulation and with a real prototype, demonstrating NLOS imaging with fourth- and fifth-bounce illumination of objects in challenging orientations and hidden around two corners. We also analyze how wave-based NLOS imaging interacts with rough hidden walls, which explains and helps overcome existing visibility limitations. We further illustrate how to image hidden objects from different perspectives, thus observing previously unseen features, by relying on multiple hidden walls.

1 Introduction

Most NLOS methods use only third-bounce illumination, limiting imaging to single-corner setups with favorable surface orientations. Cascaded NLOS imaging computes a virtual impulse response at a hidden wall from one relay-wall capture, enabling higher-order imaging in more varied scenes.

1 Introduction

  • Third-bounce-only methods miss higher-order photons and limit imaging to single-corner setups with favorable surface orientations.Earlier higher-order approaches also restrict objects to particular locations and orientations or precise specular paths.Source: §1 · p. 1
  • Cascaded NLOS imaging treats the captured relay-wall response as a virtual aperture, extending wave-based NLOS imaging to hidden walls.The approach builds on wave-based systems that interpret relay-wall measurements as virtual waves arriving at an aperture.Source: §1 · p. 1
  • A virtual projector and camera compute an analogous impulse response at a hidden wall, concatenating a second NLOS system from one capture.The second system can image objects hidden around two corners without capturing additional data.Source: §1 · pp. 1–2
  • Simulation and prototype results use fourth- and fifth-bounce illumination to image objects around two corners and in challenging orientations.The study also explores rough-wall interactions and combines multiple secondary walls to image hidden objects from different perspectives.Source: §1 · p. 2

2 Related work

Time-of-flight NLOS research spans physical and learned reconstruction methods, while phasor-field methods cast relay-wall measurements as virtual waves. Prior higher-order approaches address narrower tasks or path configurations than the cascaded imaging approach described here.

2 Related work

  • Time-of-flight NLOS methods reconstruct hidden scenes from indirect-photon travel times using physical or deep-learning models.Related work also reduces computational cost through confocal acquisition, non-uniform sampling, or efficient implementations.Source: §2 · p. 2
  • Phasor-field methods interpret relay-wall measurements as virtual waves at an aperture, enabling optics-based imaging operators for NLOS scenes.This work extends that formulation to reconstruct time-resolved illumination at hidden surfaces and exploit higher-order illumination.Source: §2 · p. 2
  • Earlier higher-order methods optimize object position, separate bounce illumination, or visualize interreflections, while cascaded imaging targets scene reconstruction.Other approaches use polarization or require planar hidden walls and objects on subsets of specular paths; cascaded imaging can use rough walls and combine third-, fourth-, and fifth-bounce information.Source: §2 · p. 2

3 Background on NLOS imaging

Conventional time-of-flight NLOS imaging records a time-resolved impulse response on a visible relay wall and applies imaging operators to estimate hidden geometry. Backprojection shifts and sums the response for third-bounce paths, while wave-based operators provide a frequency-domain formulation.

3 Background on NLOS imaging

  • A pulsed laser illuminates relay-wall points, and a camera records the returned time-resolved impulse response between illumination and sensing points.The response H(l1, s1, t) records total time of flight from l1 to s1.Source: §3 · p. 2
  • Imaging operators transform the impulse response into an image of hidden geometry at points in a bounding volume.The backprojection operator temporally shifts and adds the response; its travel-time shift corresponds to the third-bounce path and estimates hidden geometry.Source: §3 · p. 3
  • Higher-order illumination in the response causes artifacts for methods assuming third-bounce-only paths, so common strategies filter the image or impulse response.The phasor-field framework applies a temporal band-pass filter to impulse responses; equations use H to denote already-filtered responses.Source: §3 · p. 3
  • Wave-based operators interpret the relay wall as a computational lens that focuses light from illumination and camera apertures onto scene points.The confocal camera operator focuses both apertures simultaneously for third-bounce illumination, and imaging can convert back from the frequency domain by integrating over frequencies.Source: §3 · p. 3

4 Cascaded NLOS imaging

Cascaded NLOS imaging creates secondary virtual imaging systems at hidden walls from one relay-wall capture, allowing higher-order illumination to be used. Transient camera and projector operators compute virtual impulse responses that conventional imaging operators can then image.

4 Cascaded NLOS imaging

  • Secondary virtual imaging systems at hidden walls let the method use illumination beyond the third bounce.Third-, fourth-, and fifth-bounce images can be combined; secondary walls can be found with third-bounce methods without prior room-layout knowledge.Source: §4 · p. 3, §4.3 · p. 5

4.1 Transient camera and transient projector operators

  • Transient camera Φtc focuses light from camera-aperture points S1 to hidden-scene points s_b, while transient projector Φtp focuses illumination-aperture points L1 to points l_a.Together, the operators model light paths to and from the hidden scene and reconstruct time-resolved light transport.Source: §4.1 · p. 4

4.2 Virtual impulse response function

  • The virtual impulse response H′_a,b represents the reconstructed temporal response at s_b on wall ℜ_b for light focused at l_a on wall ℜ_a.It is computed from the relay-wall response H by applying the transient camera and projector operations; ℜ_a and ℜ_b may be different walls.Source: §4.2 · pp. 4–5
  • For paths through hidden walls, the virtual response re-expresses higher-order illumination as third-bounce illumination for conventional imaging operators.In the two-corner T example, fifth-bounce illumination in H becomes third-bounce illumination in H′_2,2, which is then used to image T.Source: §4.2 · p. 5, §4.3 · p. 5

4.3 Cascading virtual imaging systems

  • A conventional confocal camera operator can image H′_a,b, forming a secondary virtual imaging system from the computed response.The method can further apply transient camera and projector operators to H′ to compute another cascaded response.Source: §4.3 · p. 5
  • Effective resolution increases with the area and resolution of all apertures and decreases with average distance along the paths from apertures through the imaged volume.The method uses third-bounce imaging to locate secondary relay walls, with reported 1–3 cm error in simulated and real captures that does not affect cascaded results.Source: §4.3 · p. 5

5 Imaging methods beyond the third bounce

The methods use secondary walls to make fourth- and fifth-bounce information available to NLOS imaging, where conventional third-bounce imaging may miss favorably oriented targets. The resulting third-, fourth-, and fifth-bounce images can be combined.

5 Imaging methods beyond the third bounce

  • In the ABC example, conventional third-bounce imaging can image B, while A and C have limited visibility because of their orientations and the finite relay-wall size.The captured impulse response also contains fourth- and fifth-bounce illumination involving secondary walls ℜ2 and ℜ3.Source: §5 · p. 5, Figure 5 · p. 6

5.1 Fourth-bounce imaging method

  • Fourth-bounce imaging computes a virtual impulse response by focusing either the illumination or camera side at a secondary wall, then applies the confocal camera operator.The two path formulations give the same results in the ABC scene because their time of flight is the same.Source: §5.1 · pp. 5–6

5.2 Fifth-bounce imaging method

  • Fifth-bounce imaging focuses both illumination and camera sides at secondary walls to compute a virtual response, then uses the confocal camera operator to form an image.The illustrated example focuses at points on the right wall ℜ3.Source: §5.2 · p. 6, Figure 5 · p. 6

5.3 Combining multiple images

  • The combined image adds the third-, fourth-, and fifth-bounce images by summing their amplitudes.The method is phasor-based.Source: §5.3 · p. 6, Figure 5 · p. 6

6 Virtual surface reflectance and surface visibility

Virtual reflectance determines which hidden-wall areas return phasor waves to the sensor aperture, and therefore which areas are visible. Roughness can widen coverage, while filter parameters and higher-order paths shape visibility and imaging quality.

6.1 Virtual specular and diffuse behavior

  • Rough walls return light to the aperture from a larger area than planar walls, but their virtual diffuse reflection introduces speckle-like artifacts.Planar walls reflect specularly, so only the area that redirects light toward the aperture is recovered.Source: §6.1 · p. 6, Figure 6 · p. 7
  • When the filter wavelength is near the rough wall’s facet size, the wall appears virtually diffuse; larger wavelengths make both walls virtually specular.Increasing σ narrows the filter bandwidth and further enhances diffuseness near the facet size.Source: §6.1 · p. 7, Figure 7 · p. 7

6.2 Surface visibility under higher-order illumination

  • Under higher-order illumination, every surface along the path must reflect light toward the next surface, with the last reflecting it back to the aperture.Visibility therefore depends on the virtual reflectance of all surfaces along the path, not only the imaged surface.Source: §6.2 · p. 7
  • Rough walls can widen angular visibility enough to serve as secondary virtual imaging systems, but speckle-like artifacts limit their resolution.Rough walls are not required: transient projector and camera operators can also form apertures on virtually specular secondary walls.Source: §6.2 · p. 7

7 Results in simulation

Simulations show cascaded imaging extending visibility beyond third-bounce methods: additional virtual imaging systems recover otherwise missing object features and support imaging and tracking around two corners. The results also show a coverage–resolution trade-off for rough secondary walls.

7 Results in simulation

  • The simulations use realistic noise and matched SNR, calibration error, and temporal uncertainty; the impulse response needs to be captured only once.The reported scenarios cover visibility limits, multi-view geometry, and two-corner tracking.Source: §7 · p. 7

7.1 Overcoming third-bounce visibility limits

  • Combining third-, fourth-, and fifth-bounce images recovers all three letters, whereas the third-bounce image shows only B.Third-bounce photons account for 90% of the simulated signal, fourth-bounce for 7.5%, and fifth-bounce for 2.5%.Source: §7.1 · pp. 7–8

7.2 Enhanced multi-view imaging of complex geometry

  • Depth RMSE for the bunny falls from 3.5 cm to 2.9 cm when images from two virtual viewpoints are combined.The combined view recovers features such as the ears and tail that were missing from the third-bounce image.Source: §7.2 · p. 8, Figure 8 · p. 8

7.3 Imaging and tracking around two corners

  • With a rough secondary wall, the method accurately follows a hidden object’s trajectory, while virtual mirrors fail to track it accurately.Roughness decreases spatial resolution but can reveal scene regions that other NLOS methods leave invisible.Source: §7.3 · pp. 8–9, Figure 9 · p. 8

8 Results with our real prototype

Real-prototype results demonstrate fourth- and fifth-bounce imaging, including targets that conventional methods cannot see and scenes with rough secondary walls. The prototype’s limited sensor resolution and aperture reduce image fidelity relative to simulation.

8.1 Fourth-bounce imaging results

  • Fourth-bounce illumination images the hidden A-shaped target that the third-bounce result misses, while third-bounce imaging shows the 2-shaped target.The prototype-matched simulation reproduces the degradation and resolution of the real capture.Source: §8.1 · pp. 9–10, Figure 11 · p. 9, Figure 12 · p. 9
  • The 16 × 16 SPAD array and its 50 cm × 32.5 cm relay-wall coverage constrain fidelity, contributing to higher noise, reduced resolution, and the simulation-to-real gap.The prototype uses a lower-resolution array and smaller coverage than the 32 × 32 SPAD, 2 m × 2 m simulated setup.Source: §8.1 · p. 9

8.2 Fifth-bounce imaging results

  • The prototype also images around two corners through a rough secondary wall without prior knowledge of that wall.The cascaded method identifies the target where conventional methods and virtual mirrors fail.Source: §8.2 · pp. 10–11, Figure 15 · p. 11
  • The fifth-bounce method images a target around two corners at its correct location, although third-bounce imaging and virtual mirrors fail.The real and simulated impulse responses and resulting image closely match, with small differences attributed to geometric discrepancies.Source: §8.2 · pp. 10–11, Figure 13 · p. 10, Figure 14 · p. 10

9 Conclusions and discussion

Cascaded virtual imaging systems extend NLOS imaging beyond three-bounce paths, while repeated concatenation and current prototype resolution constrain the detail obtained. The section also describes faster convolution-based propagation and resolution limits.

9 Conclusions and discussion

  • Cascaded systems use fourth- and fifth-bounce illumination beyond the conventional three-bounce assumption, and further cascades are theoretically possible.A tertiary relay wall could use fifth- to seventh-bounce illumination.Source: §9 · p. 11
  • Each added cascade reduces resolution through weaker signal, limited apertures, longer focus distances, and increased multipath interference.Source: §9 · p. 11
  • The current 16 × 16 prototype yields virtual-impulse-response spatial resolution of about 20–40 cm, limiting resolvable features.The estimated resolution depends on the setup.Source: §9 · p. 11
  • The real prototype images the L-shaped target at the correct location and depth but does not resolve its asymmetric features.The conclusion contrasts this result with existing NLOS methods, which fail to image the target.Source: §9 · p. 11, Figure 16 · p. 11

A Fast computation of our virtual 𝐻′ using convolutions

B.1 Resolution of our virtual impulse response

The virtual impulse response is evaluated against an equivalent direct capture, and its spatial and temporal resolution are modeled from aperture, wavelength, and depth. Those bounds also constrain the resolution of the cascaded system.

B.1 Resolution of our virtual impulse response

  • The Rayleigh criterion models virtual-response lateral resolution from aperture size, wavelength, and distance from the relay wall.The temporal resolution is empirically modeled as Δt = Δx/c.Source: §B.1 · pp. 13–14
  • With the 16 × 16 prototype, λ ≥ 10 cm and a 32.5 cm aperture side yield Δx ≈ 22 cm at z = 1 m; the 32 × 32 simulation yields Δx ≈ 3.5 cm.Even a larger covered area does not remove the resolution cost of longer wavelengths.Source: §B.1 · p. 14
  • The virtual impulse-response peak aligns in time with the equivalent direct capture and corresponds to fifth-bounce illumination.The comparison uses a T-shaped object and secondary wall ℜ2.Source: §B.1 · p. 14, Figure S.1 · p. 14
  • 40 cm FWHM for the virtual impulse-response peak compares with 11 cm FWHM for the equivalent direct-capture peak.The virtual response uses λc = 15 cm; its predicted spatial resolution of about 39.1 cm closely matches the measured 40 cm.Source: §B.1 · p. 14

B.2 Resolution of our cascaded imaging systems

  • Cascaded-system resolution also depends on virtual-response resolution and the effective secondary aperture, which can be roughly half the nominal aperture in tested scenes.The estimated lower bound is Δx′ ≥ cΔt/2 ≈ Δx/2; exact resolution depends on scene layout.Source: §B.2 · pp. 14–15

C Impact of estimation error of secondary relay walls

The section tests whether errors in locating a secondary relay wall affect final cascaded images and introduces virtual surface reflectance as a frequency-dependent property of phasor-field imaging.

C Impact of estimation error of secondary relay walls

  • Third-bounce estimates of the secondary wall fall within ±3 cm, and the resulting cascaded image is similar to one using positions from a 3D scan.Both reconstructions show a similarly shaped object at the expected location.Source: C Impact of estimation error of… · p. 15

D Virtual surface reflectance and the missing cone

  • Virtual surface reflectance depends on the imaging frequencies and can differ from real reflectance because phasor-field processing selects centimeter-range wavelengths.The impulse response is convolved with a Morlet-wavelet virtual illumination function whose spectrum is centered at Ωc = c/λc.Source: D Virtual surface reflectance and the… · p. 15

D.1 Missing cone under higher-order illumination

The cascaded system changes its apertures across hidden walls to sample different Fourier regions of the NLOS measurement function. In the tested orientations, this samples components relevant to the target that the compared fixed-aperture approach misses.

D.1 Missing cone under higher-order illumination

  • Only the cascaded system samples the Fourier components relevant to the hidden target across the tested cases, while Royo et al. [2023] remains susceptible to the missing-cone problem.Its transient projector and camera form apertures on secondary and tertiary walls, adapting to target orientation.Source: §D.1 · pp. 15–16, Figure S.3 · p. 16

E Realistic capture noise models for simulation

The simulations model realistic capture noise and show that both third-bounce and fifth-bounce imaging remain robust to noise, with resolution loss driven more by the difficulty of fifth-bounce imaging. The section also documents the real-prototype setup.

E Realistic capture noise models for simulation

F Setup photographs

  • Setup scans and photographs show the prototype aimed at the relay wall, with a 16×16 SPAD array and laser covering different areas of that wall.The sensor is focused on a 50 cm × 32.5 cm area, while the laser covers 1.9 m × 1.9 m.Source: F Setup photographs · p. 17, Figure S.5 · p. 18
Loading 2609.16017v1…