Source-linked AI summary
Real-time 3D reconstruction from single-photon lidar data using plug-and-play point cloud denoisers
Julián Tachella, Yoann Altmann, Nicolas Mellado, Aongus McCarthy, Rachael Tobin, Gerald S. Buller, Jean-Yves Tourneret, Stephen McLaughlin
TL;DR
Single-photon lidar reconstruction is difficult to analyse quickly in outdoor scenes with weak returns, background illumination, and multiple surfaces per pixel. The paper combines statistical observation models with plug-and-play point-cloud denoisers and massively parallel updates. It reports reliable reconstruction at 320 metres and 50 frames per second, while improving execution speed by up to 105 times.
Problem
Outdoor single-photon lidar reconstruction is challenging because of low signal returns, high ambient illumination, and an unknown number of surfaces per pixel.
Method
The method combines a Poisson-based observation model with alternating gradient updates and plug-and-play point-cloud denoisers for depth, intensity, and background.
Results
The method achieved similar or better reconstruction quality while improving execution speed by up to 105 times, including reliable reconstruction at 320 metres and 50 frames per second.
Takeaways & Limitations
The framework supports real-time reconstruction of long-range daylight scenes with multiple surfaces per pixel and can be extended to other 3D reconstruction settings.
Takeaways & Limitations
Maximum-likelihood estimation is sensitive to data quality, so additional regularisation is required.
Abstract
from arXiv · showhide
Single-photon lidar has emerged as a prime candidate technology for depth imaging through challenging environments. Until now, a major limitation has been the significant amount of time required for the analysis of the recorded data. Here we show a new computational framework for real-time three-dimensional (3D) scene reconstruction from single-photon data. By combining statistical models with highly scalable computational tools from the computer graphics community, we demonstrate 3D reconstruction of complex outdoor scenes with processing times of the order of 20 ms, where the lidar data was acquired in broad daylight from distances up to 320 metres. The proposed method can handle an unknown number of surfaces in each pixel, allowing for target detection and imaging through cluttered scenes. This enables robust, real-time target reconstruction of complex moving scenes, paving the way for single-photon lidar at video rates for practical 3D imaging applications.
Introduction
Single-photon lidar enables depth imaging in challenging environments, but outdoor reconstruction is difficult because of weak returns, ambient illumination, and potentially multiple surfaces per pixel. The proposed framework combines statistical observation models with scalable point-cloud denoising to provide fast, robust reconstruction without explicit surface priors.
- Introduction: High ambient illumination and low signal returns make robust outdoor 3D reconstruction challenging, while existing methods are too slow or insufficiently robust for dynamic scenes.
- Introduction: Single-surface methods simplify reconstruction but fail for cluttered or long-range scenes where pixels can contain multiple surfaces.
- Introduction: ManiPoP improves accuracy at similar computational cost by modelling 2D surfaces in 3D using structured point clouds.
- Introduction: The proposed method models target surfaces as 2D manifolds using plug-and-play point-cloud denoisers instead of explicit prior distributions.
- Introduction: A massively parallel denoiser lets the framework incorporate observation-model information while processing dozens of frames per second in the multiple-surface setting.
Results
The reconstruction algorithm alternates data-fidelity updates for depth, intensity, and background with denoising steps that impose point-cloud or image structure. On a 40-metre head scene, it achieved high detection accuracy while reducing processing time from seconds to milliseconds.
- Observation model: The lidar data cube records Poisson-distributed photon counts across pixels and histogram bins, with signal and background represented in the observation model.
- Observation model: Maximum-likelihood estimation uses the observation model, but its sensitivity to data quality motivates additional regularisation.
- Reconstruction algorithm: Each block update takes a gradient step on the data-fidelity term followed by a denoiser acting as a proximal operator.
- Reconstruction algorithm: Depth denoising fits smooth surfaces with APSS, while intensity denoising uses local manifold neighbours and removes points below an intensity threshold.
- Large raster-scan scene results: 13 ms processing time yielded 96.6% of 3D points within 4 cm, compared with 83.46% for cross-correlation and 201 s for ManiPoP.
3D Dynamic scenes results
The dynamic-scene experiment tested reconstruction through a camouflage net at approximately 320 metres, where pixels commonly contained multiple returns. The comparison shows that the proposed method preserves structured point clouds without the single-surface correlations imposed by one competing method.
- 3D Dynamic scenes results: The 320-metre scene contained two people walking between a camouflage net and backplane, with most pixels showing two surfaces and some contours showing three.
- 3D Dynamic scenes results: Figure 3 compares cross-correlation, Rapp and Goyal, ManiPoP, and the proposed method using point colours to indicate returned target photons.
- 3D Dynamic scenes results: Cross-correlation produces noisy estimates without regularisation, whereas Rapp and Goyal, ManiPoP, and the proposed method produce structured point clouds.
- 3D Dynamic scenes results: Rapp and Goyal correlates the head and backplane borders because it assumes one surface per pixel, unlike ManiPoP and the proposed method.
Discussion
The proposed framework combines observation-model information with denoisers to reconstruct distributed, multi-surface scenes in real time. It achieves reliable long-range reconstruction at video rates and supports extensions to other 3D sensing settings.
- The algorithm reconstructs distributed scenes with few photons or spurious detections and permits an unknown number of surfaces per pixel.It reports similar or better reconstruction quality than existing methods while improving execution speed by up to a factor of 105.
- 320 m and 50 frames per second: reliable real-time reconstruction of multiple-surface scenes was demonstrated in daylight.The framework is also described as compatible with general-purpose GPUs and modern embedded systems.
- The framework combines prior information from the observation model with point-cloud and image denoisers as modular building blocks.These variants allow control over the trade-off between reconstruction quality and computing speed.
- The framework can be extended to other 3D reconstruction settings, including sonar and multispectral lidar.
Methods
The GPU implementation exploits parallelism in initialization, gradient updates, and denoising to reduce sensitivity to image size. Execution time remains approximately constant below 150 × 150 pixels but grows with active histogram bins and larger images.
- Implementation: GPU processing parallelizes initialization and gradient steps per pixel, while point-cloud and intensity denoising operate in parallel on world-coordinate pixels.GPU shared memory gathers information from neighboring points during denoising.
- Execution time: Below 150 × 150 pixels, execution time is approximately constant because pixels are processed completely in parallel.The measurement uses the mannequin head dataset and evaluates execution time per frame.
- Execution time: The number of active bins per pixel produces an approximately linear increase in execution time because per-pixel computations are not parallelized.Active bins are histogram bins containing one or more photons.
- Execution time: Larger images increase execution time when a single GPU cannot process all pixels simultaneously and memory constraints arise.
- Imaging setup: The imaging setup uses a pulsed 1550 nm fibre laser, a 32 × 32 InGaAs/InP SPAD array, and a 500 mm lens with approximately 0.5 arc degrees field of view.High-performance spectral filters reduce detected background light during broad-daylight measurements.
- Imaging setup: The optical system is bistatic, with separate transmit and receive apertures, to avoid potential back-reflection damage to the sensitive focal-plane array.The configuration introduces parallax requiring slight realignment of the system.
- Availability: The real-time executable requires an NVIDIA GPU with compute capability 5.0 or higher, and the lidar data are publicly available.