PaperlayerView exact revision on arXiv

Source-linked AI summary

Memory-efficient GPU pipelines for real-time non-line-of-sight reconstruction

Alfonso López-Ruiz, Diego Royo

arXiv:2608.28183v1cs.DCcs.CV

TL;DR

NLOS reconstruction is a large, memory-intensive inverse problem whose demands increase as SPAD arrays raise acquisition throughput. The paper rebuilds f-k migration and phasor-fields as GPU pipelines for streaming and offline processing, using compact phasor-fields kernels and dataflow optimizations. The implementations achieve up to 42× streaming speedup and up to 14× volumetric speedup while using as little as 2.5% of baseline memory.

  • Problem

    NLOS reconstruction must process billions of photon timestamps, and higher-throughput SPAD arrays are making reconstruction a limiting stage.

  • Method

    The paper rebuilds f-k migration and phasor-fields GPU execution for streaming and offline processing, constructing phasor-fields ring-and-radius kernels offline and reorganizing GPU dataflow.

  • Results

    The implementations are up to 42× faster than the reference streaming pipeline and up to 14.0× faster than PttR for volumetric reconstruction, using 2.5% of PttR’s peak memory on average.

  • Takeaways & Limitations

    The memory savings permit substantially larger reconstructions on the same hardware or comparable reconstructions on lower-memory GPUs.

  • Takeaways & Limitations

    Streaming experiments read recorded photons from disk rather than directly from live acquisition hardware.

Abstract

from arXiv · show

Non-line-of-sight (NLOS) imaging reconstructs scenes hidden around a corner from indirect light recorded by a single-photon avalanche diode (SPAD). A single reconstruction is a large inverse problem: billions of photon timestamps must be binned, moved through memory, transformed and inverted. As SPAD arrays raise acquisition throughput, reconstruction becomes the limiting stage. We rebuild the GPU execution of two established wave-based algorithms, f-k migration and phasor-fields, for both streaming and offline processing. On the phasor-fields side we assemble the ring-and-radius kernels of previous work once and offline, using the analytic Fourier transform of a ring, so the propagation kernel never exists in dense form at runtime, reducing the memory and bandwidth. We reorganize the pipeline of both algorithms with fused kernels, warp-level photon binning, batched transforms, CUDA graph replay, and FP16 storage applied only where it reduces the actual bottleneck. Our implementations are up to 42x faster than the reference streaming pipeline and up to 14x faster than the fastest published GPU baseline, all while using a fraction of the memory (down to 2.5%), enabling vastly larger and finer reconstructions on the same hardware, or comparable ones within a much lower memory budget. We report an ablation of each implementation choice and propose three denoising strategies enabled by the resulting frame budget for next-generation NLOS video processing.

1. Introduction

NLOS reconstruction is becoming a computational bottleneck as higher-throughput SPAD arrays produce more data, motivating memory-efficient GPU pipelines for wave-based methods. The paper rebuilds f-k migration and phasor-fields for streaming and offline processing, achieving substantial speedups and memory reductions while adding denoising strategies.

  • Motivation: A single NLOS frame can contain billions of photon timestamps that must be binned, transferred, transformed, and inverted.The computational cost grows with relay-wall sampling, sensor temporal resolution, and reconstruction-volume size.
  • Motivation: SPAD arrays alleviate acquisition bottlenecks by recording photon events in parallel, but future arrays will increase reconstruction demands.Higher pixel-level capture parallelism makes reconstruction throughput increasingly important.
  • Approach: The paper rebuilds f-k migration and phasor-fields GPU execution for both streaming and offline reconstruction.The evaluation covers performance and memory across reconstruction sizes, precision modes, padding settings, and prior accelerated implementations.
  • Results: 42× faster streaming reconstruction uses 8.2% of the reference pipeline’s memory, while volumetric reconstruction is 7.7×–14.0× faster than PttR using 2.5% of its peak memory on average.The streaming comparison is 0.27 versus 3.34 GB; the volumetric comparison is 0.22–0.29 versus 8.9–12.7 GB.
  • Contributions: The phasor-fields pipeline constructs ring-and-radius kernels offline from an analytic Fourier transform of a ring, removing runtime dense kernel reconstruction.The contributions also include CUDA implementations, benchmarking, and three strategies for removing noise in streamed reconstructions.

2. Related Work

Prior NLOS systems use GPU, FPGA, CUDA, and high-level GPU acceleration, but published methods generally target smaller volumes or limited reconstruction settings. This work compares against established streaming and offline baselines while addressing GPU dataflow and memory bottlenecks.

  • Algorithmic optimizations: Wave-based methods exploit convolution, efficiency approximations, memory reductions, phase information, and radial symmetry to reduce computational or memory costs.Jiang et al. use ring- or radius-wise propagation kernels based on radial symmetry.
  • Scaling limitation: Prior accelerated approaches target relatively small reconstruction volumes and do not scale well.This limitation motivates the paper’s focus on throughput and memory capacity.
  • Baseline comparison: PttR reports high throughput for single-plane phasor-fields reconstruction, whereas full-volume reconstruction requires multiple planes at different depths.The paper reports 7.7× and 14.0× speedups over PttR on volumetric Z and office scenes, respectively, using 2.5% of its peak memory on average.
  • Evaluation scope: The evaluation separates experiments rerun on the authors’ hardware from literature-reported values because algorithms, hardware, and depth settings differ across studies.Each evaluated method uses its own published reconstruction operator.
  • GPU pipelines: GPU pipeline bottlenecks include short kernel launches, device feeding, and volumes exceeding available memory.The implementation records frame processing in CUDA graphs, overlaps capture and computation, and restructures memory-bound execution.

3. Background: non-line-of-sight imaging

Active NLOS imaging estimates hidden scenes from time-resolved indirect light measured at a visible relay wall. The paper reviews f-k migration and phasor-fields as wave-based reconstruction methods, including their capture assumptions and radial-kernel optimization.

  • Imaging setup: In active NLOS imaging, a pulsed laser illuminates a visible relay wall and a sensor records time-resolved indirect light from the hidden scene.Repeated measurements form an impulse response indexed by illumination position, sensing position, and time of flight.
  • Capture modalities: Confocal capture uses the same relay-wall point for illumination and sensing, while non-confocal capture allows those points to differ.Confocal capture is slower but produces a more structured reconstruction problem.
  • f-k migration: f-k migration assumes confocal data and uses Stolt interpolation in the frequency domain to migrate the measured field back to the hidden-scene time slice.The hidden scene is modeled as a wave field sampled at relay-wall positions and later times.
  • Phasor-fields: Phasor-fields modulates an illumination signal on the impulse response, transforms it in time, and propagates the resulting field from the relay wall to the hidden scene.The relay wall is treated as a virtual camera aperture, and the reconstruction sums contributions over temporal frequencies.
  • Radial optimization: When relay-wall and reconstruction planes are parallel and aligned, radial symmetry permits ring- or radius-wise kernels that reduce computation and memory while preserving the propagation model.A radial Fourier profile can carry the same information as the corresponding two-dimensional radial function.

4. Proposed GPU implementation

The implementation reorganizes NLOS reconstruction into a GPU-oriented pipeline for streaming and offline processing, combining producer-consumer data flow with memory-conscious optimizations for f–k migration and phasor-fields. It also introduces temporal denoising strategies for consecutive reconstructions.

  • General processing steps: The producer-consumer pipeline streams photon data, bins timestamps into histograms, and supports live SPAD input, photon files, and pre-binned datasets.Modules coordinate asynchronous production and consumption around reusable processing stages.
  • Photon binner: High-photon-count streaming uses DMA through pinned host memory because transfer costs become a larger part of frame time.At low photon counts, host-pinned-memory indirection can outweigh the transfer benefit.
  • f–k migration: The f–k implementation alternates between two working FFT buffers, with optional 2× zero-padding that reduces wrap-around artifacts but increases memory use and runtime.An unpadded variant provides a faster alternative, while padding improves artifact suppression.
  • Phasor-fields optimizations: The radial representation is approximate on the finite square relay-wall grid because enforcing radial symmetry discards angular structure and ignores amplitude falloff.The offline construction also uses discrete quadrature and a maximum-radius cutoff.
  • Phasor-fields optimizations: The phasor-fields pipeline precomputes compact radial kernels offline from an analytic ring Fourier transform and performs radius lookup during propagation.It stores Fourier-domain phasor data, radial kernels, a pixel-to-radius map, and working buffers instead of rebuilding dense kernels at runtime.
  • Noise removal: Three denoising strategies combine consecutive reconstructions, including temporal averaging and a mixture-of-experts method that uses phasor-fields as a confidence gate.Short temporal windows reduce flicker and incoherent background speckle, while longer windows mainly add smoothing and runtime overhead.

5. Results and evaluation

The evaluation measures complete streaming and offline reconstruction paths, isolating implementation choices across performance, memory, scalability, and reconstruction quality. The optimized pipelines substantially improve throughput and memory use while exposing trade-offs from padding, precision, and kernel design.

  • Real-time performance: 42× faster streaming reconstruction uses 0.27 GB rather than 3.34 GB for the reference pipeline.The optimized phasor-fields variant reaches 448.5 FPS, compared with 10.7 FPS for the reference.
  • Real-time performance: 919.9 FPS is reached by unpadded f–k migration, while padded f–k migration still reaches 81.4 FPS.Removing padding improves speed and memory use but may introduce FFT wrap-around artifacts.
  • Implementation choices and ablations: Shared-memory preloading does not substantially improve Stolt interpolation because synchronization cancels most savings from reduced global reads.The optimized kernel already performs a simple two-sample interpolation along the contiguous third dimension.
  • Implementation choices and ablations: CUDA graph replay contributes 1.3% for f–k and 0.3% for phasor-fields because fused frames launch only about fifteen operations.Its benefit is most relevant to repeated streaming reconstruction, where the same sequence runs every frame.
  • Memory scalability: The radial phasor-fields representation stores propagation data per radius, reducing memory from 12.7 GB for PttR to 0.22 GB on the office scene.Dense kernels scale with retained frequencies and spatial pixels, whereas the radial representation avoids that dense spatial storage.
  • Memory scalability: The phasor-fields implementation is the only method completing the largest padded configuration, requiring 4.5 GB of memory.The padded f–k transform exceeds cuFFT’s accepted size, while the mixture-of-experts configuration runs out of memory.
  • Equivalence with reference implementations: Mean absolute difference remains 0.016 for phasor-fields and 0.008 for f–k over the dynamic sequence.FP16 and FP32 phasor-fields curves overlap, so the residual difference from the reference is attributed to the radial approximation.
  • Offline reconstruction: 98.3% lower reconstruction time for f–k and 96.1% for phasor-fields accompany a 52.5% average peak-memory reduction offline.Spatial resolution is more expensive than temporal-bin count because the dominant FFT stages grow over spatial dimensions.

6. Conclusions and future work

The CUDA pipelines accelerate streaming and offline wave-based NLOS reconstruction while reducing memory use. Remaining boundaries include camera-transfer evaluation, photon movement costs, and reliance on Fourier-domain operators.

  • 6. Conclusions and future work: 42× the frame rate is achieved in 8.2% of the memory versus the reference streaming phasor-fields pipeline.The implementation reaches 448.5 FPS in 0.27 GB, compared with 10.7 FPS in 3.34 GB.
  • 6. Conclusions and future work: 7.7× and 14.0× faster reconstruction is reported than Physics to the Rescue on the Z and office scenes, respectively.The full multi-plane volumes use on average 2.5% of that baseline’s peak video memory.
  • 6. Conclusions and future work: 98.3% and 96.1% average reductions in offline reconstruction time are reported for f–k migration and phasor-fields, respectively.On the largest office dataset, the reductions are 99.4% and 96.1%.
  • 6. Conclusions and future work: No single optimization dominates all setups, and FP16 helps stored phasor-fields propagation data but is not a universal replacement for FP32.For f–k, cuFFT planning constraints and conversion overhead can outweigh the smaller element size.
  • 6. Conclusions and future work: Streaming experiments use disk-recorded photons rather than a live SPAD interface, leaving camera-side transfer constraints for future evaluation.Photon binning and data movement remain important costs at high photon counts.

A. Integration in a SPAD array

The experiments assess processing capacity and image quality using recorded photons as a proxy for future live SPAD-array operation. The pipeline sustains high frame rates at very large photon counts while preserving recognizable hidden structure at lower counts.

  • A. Integration in a SPAD array: The recorded-photon experiments stress processing without bandwidth or buffering limits from a specific acquisition device.Each dynamic-dataset frame contains approximately 1.2 ⋅10^6 photon records.
  • A. Integration in a SPAD array: Above 60 FPS is sustained up to approximately 47M photons per frame, and above 24 FPS up to approximately 96M photons per frame.These counts exceed those in the dynamic dataset used for the main streamed comparison.
  • A. Integration in a SPAD array: Both f–k and phasor-fields preserve the main hidden shape at reduced photon counts, with phasor-fields more robust to noise in this example.Figure 20 shows progressively fewer photon records from the teaser dataset.
  • A. Integration in a SPAD array: 178M photons captured over 180 minutes are processed at slightly below 16 FPS for the teaser dataset.The dynamic dataset is processed at approximately 400 FPS in this benchmark.
  • A. Integration in a SPAD array: 125M photon events per second is the estimated USB 3.0 payload bound, implying an ideal upper bound of about 104 frames per second for the dynamic sequence.Each photon record occupies 4 bytes and the link provides at most about 500 MB s^-1 of payload.

B. Further implementation details

The CUDA f–k implementation changes the reference dataflow by fusing operations and retaining compact working data. The pseudocode highlights these allocation and postprocessing differences.

  • B. Further implementation details: The reference f–k pipeline allocates separate arrays for scaling, padding, Fourier shifts, and Stolt remapping.The CUDA path keeps the input volume and uses two complex working buffers.
  • B. Further implementation details: Maximum magnitude over depth replaces squared amplitude after the inverse FFT, improving final-image tone mapping in these experiments.The change is presented as a dataflow difference between the implementations.
  • B. Further implementation details: Algorithm 2 is the CUDA-based f–k migration pipeline, contrasting with the prior-work pipeline represented by Algorithm 1.The algorithms are used primarily to highlight their dataflow difference.
  • B. Further implementation details: Time-bin scaling is fused with padding in the CUDA f–k path.This fusion appears in the scale_pad_fftshift operation.

B.2. Phasor-fields implementation

The reference phasor-fields implementation processes transforms and weighted frequency slices sequentially while using a dense propagation kernel. Its pseudocode represents convolution across the frequency stack.

  • B.2. Phasor-fields implementation: The reference implementation handles initial Fourier transforms sequentially even though they could be executed as a group.The transformed slices are stacked in the transformed-volume representation.
  • B.2. Phasor-fields implementation: Each convolved-volume slice is weighted sequentially by w_c in the reference pipeline.The Fourier-transformed matrices are not padded, reducing memory consumption at the expense of an introduced trade-off.
  • B.2. Phasor-fields implementation: The dense propagation kernel K is convolved with the whole frequency stack in the reference phasor-fields algorithm.P holds the sampled phasor field and K the dense propagation kernel.

B.2.1. Our improved ring-and-radius construction

The paper replaces dense phasor-fields propagation kernels with symmetry-aware radial representations constructed offline, then performs runtime radius lookup, interpolation, weighting, and accumulation. Geometry-derived sampling and analytic ring transforms support this compact representation while avoiding dense kernel reconstruction during execution.

  • Ring-and-radius representation: The symmetry-aware variant stores propagation data by radial Fourier bin, temporal frequency, and depth, replacing dense Cartesian kernels with a pixel-to-radius lookup.At runtime, each Cartesian FFT pixel retrieves its continuous radius and accumulates interpolated radial values over temporal frequencies.
  • Offline construction: The analytic Fourier transform of a circular ring reduces to a zeroth-order Hankel term depending only on spatial radius, with no series truncation.The remaining approximations arise from finite trapezoidal quadrature, radial cutoff, and enforcing symmetry on the discrete rectangular grid.
  • Geometry-derived sampling: Sampling counts and spacings are derived from capture geometry, padded FFT extents, wavelength limits, and Nyquist criteria for spatial and radial-frequency domains.The construction samples radii uniformly, uses trapezoidal integration weights, and similarly samples the radial spectral domain.
  • Geometry-derived sampling: 128 rings and 256 stored radii are used for the 190 × 190 non-confocal capture, with ρmax = 70.34 m−1 and ρcutoff = 47.02 m−1.A confocal capture of the same geometry doubles the ring count through σs = 2.
  • Runtime execution: The runtime loop applies cutoff tapering, linearly interpolates neighboring radial bins, and accumulates frequency-weighted products without generating dense 2D kernel slices.A smoothstep taper spans the final Δtaper = 8 bins below the cutoff to reduce boundary artifacts.
  • Offline simplification: The offline implementation also collapses hierarchical Cartesian-to-ring averaging into one kernel, removing sparse-matrix construction and repeated CPU reductions from the critical path.Because cosine and sine phasor components are the real and imaginary parts of one complex phasor, linear transforms can process their complex combination once.

C. Additional results and benchmarks

The appendix adds configuration breakdowns, per-frame image-space error maps, an MoE pooling-weight sweep, and expanded offline benchmark tables to support the main findings.

  • Additional experiments: Four additional experiments examine dynamic-capture configurations, per-frame error maps, MoE pooling weights, and complete offline benchmark tables.The experiments vary algorithm, precision, and padding where applicable.

C.1. Dynamic reconstruction across configurations

The dynamic-capture comparison shows a speed-memory frontier: unpadded f–k migration is fastest, while the proposed phasor-fields method offers sharper letters than f–k with much lower memory than the reference.

  • Speed-memory frontier: Unpadded f–k migration gives the highest frame rate but is most exposed to padding artifacts.Frame rates are computed from mean frame times across algorithm, precision, and padding configurations.
  • Speed-memory frontier: Our phasor-fields produces sharper letters than f–k migration while using far less GPU memory than Nam et al. (2021)’s reference implementation.The comparison includes both precisions and padded or unpadded configurations where supported.
  • Speed-memory frontier: The proposed phasor-fields method retains most of the visual benefit in the low-memory corner, whereas the reference implementation is an order of magnitude away on both axes.MoE is characterized as a visual-combination option rather than the best point on the performance frontier.
  • Precision effects: FP16 makes f–k migration slower and slightly larger than FP32 in one configuration, so half precision is not useful for that branch.This behavior contrasts with the other configurations evaluated in the breakdown.

C.2. Additional image-space quality

The office-scene error maps compare reconstructions against references using FLIP and absolute-error maps, showing that precision does not explain the residual difference for phasor-fields.

  • Image-space comparisons: The FP16 and FP32 rows are indistinguishable, while phasor-fields reaches 37.0 dB PSNR and f–k reaches 57.0 dB PSNR against their respective references.The reported residual difference is attributed to the radial kernel representation rather than FP16 precision.
  • Image-space comparisons: Figure 23 compares each reconstruction with its reference using FLIP and absolute-error maps across rows of the office scene.The phasor-fields reference is Nam et al. (2021)’s implementation, while f–k uses FP32 as its reference.

C.3. Pooling weight of the mixture

Pooling weight β controls a direct tradeoff between background suppression and object signal retention. The selected β=0.3 reduces background noise substantially while preserving most of the f–k object intensity.

  • β=0.5 provides the strongest background rejection but also the weakest object signal, so no single setting is optimal across metrics.The metrics change monotonically with β.
  • β controls a direct tradeoff between background suppression and object signal retention.As β increases, background noise decreases exponentially while object intensity drops linearly.
  • β=0.3 reduces background noise by three orders of magnitude relative to either standalone branch while preserving 73% of the f–k object intensity.This operating point reproduces the MoE row in Table 6 of the main paper.
  • The offline experiments report reconstruction timings across spatial and temporal resolutions, with throughput measured in millions of voxels per second.The largest tested Zaragoza datasets are 256×256×4096, though experiments limit the temporal dimension to 2048 bins because 4096 bins create substantial memory pressure.
  • Optimized implementations retain high throughput across confocal, simulated, and exhaustive inputs, despite setup overhead at small sizes and memory-traffic limits at large sizes.The observed speed-up is attributed to dataflow and layout changes rather than a dataset-specific shortcut.
Loading 2608.28183v1…