Source-linked AI summary
Semi-Dense 3D Reconstruction with a Stereo Event Camera
Yi Zhou, Guillermo Gallego, Henri Rebecq, Laurent Kneip, Hongdong Li, Davide Scaramuzza
TL;DR
The paper addresses 3D reconstruction from a moving stereo event-camera rig, where event data require specialized processing for semi-dense depth estimation. It minimizes an energy based on cross-camera spatio-temporal consistency and fuses estimates probabilistically, achieving 1%–5% relative error while outperforming state-of-the-art stereo methods using the same event representation.
Problem
The paper addresses non-instantaneous 3D reconstruction with a moving stereo event-camera pair for applications such as stereo SLAM.
Method
The method jointly estimates inverse depth by forward-projecting hypotheses across stereo time-surfaces and fuses multiple reconstructions probabilistically.
Results
1%–5% relative error was achieved, and the method outperformed state-of-the-art stereo methods using the same spatio-temporal event representation.
Takeaways & Limitations
The approach produces increasingly dense and accurate semi-dense reconstructions by integrating observations over time and supports both sparse and cluttered scenes.
Takeaways & Limitations
The reconstruction assumes known camera calibration and pose at each observation, and does not use event polarity.
Abstract
from arXiv · showhide
Event cameras are bio-inspired sensors that offer several advantages, such as low latency, high-speed and high dynamic range, to tackle challenging scenarios in computer vision. This paper presents a solution to the problem of 3D reconstruction from data captured by a stereo event-camera rig moving in a static scene, such as in the context of stereo Simultaneous Localization and Mapping. The proposed method consists of the optimization of an energy function designed to exploit small-baseline spatio-temporal consistency of events triggered across both stereo image planes. To improve the density of the reconstruction and to reduce the uncertainty of the estimation, a probabilistic depth-fusion strategy is also developed. The resulting method has no special requirements on either the motion of the stereo event-camera rig or on prior knowledge about the scene. Experiments demonstrate our method can deal with both texture-rich scenes as well as sparse scenes, outperforming state-of-the-art stereo methods based on event data image representations.
1 Introduction
Event cameras provide asynchronous, high-speed sensing for difficult vision scenarios, but require specialized algorithms. This paper addresses non-instantaneous stereo reconstruction by combining cross-camera temporal coherence with pose-based temporal integration for semi-dense depth.
- Event cameras asynchronously report pixel-wise intensity changes, reducing power and bandwidth requirements while capturing high-speed motion with microsecond temporal resolution.Unlike standard cameras, they do not acquire full frames or operate at a fixed frame rate.
- Specialized processing is needed because standard-camera algorithms do not directly exploit event cameras’ temporally asynchronous and spatially sparse data.Combining event cameras with standard or depth sensors simplifies estimation but limits use of the event sensor’s capabilities.
- 1.1 Related work on Event-based Depth Estimation: Prior instantaneous stereo methods use synchronized event cameras over very short intervals, matching events temporally before triangulating 3D points.Other approaches use constrained non-simultaneous rotating-camera setups or reconstruct intensity images for conventional stereo.
- 1.1 Related work on Event-based Depth Estimation: Single-camera methods integrate events over longer intervals for semi-dense reconstruction and require relative camera-scene pose, targeting visual odometry or SLAM rather than instantaneous depth.
- 1.1 Related work on Event-based Depth Estimation: This paper introduces non-instantaneous stereo reconstruction using temporal coherence across cameras, forward projection, pose-based temporal integration, and continuous depth estimates.The approach is designed for both sparse and cluttered scenes and is evaluated on synthetic and real event data.
2 3D Reconstruction by Event Time-Surface Maps Energy Minimization
The method estimates inverse depth by minimizing stereo temporal inconsistency between time-surface maps, using forward projection across multiple stereo observations. It replaces explicit event matching and triangulation with patch-based temporal comparisons and nonlinear optimization.
- 2.1 Event Time-Surface Maps: Stereo temporal consistency is measured over spatio-temporal neighborhoods because latency and jitter prevent strict pixel-level timestamp agreement.The method compares patches rather than individual event timestamps.
- 2.1 Event Time-Surface Maps: Event timestamps are represented by time-surface maps that apply exponential decay to the most recent event at each pixel.The decay rate δ is set to a small constant, such as 30 ms in the experiments.
- 2.2 Problem Formulation: For each reference-view event, a global energy minimizes temporal residuals between projected left- and right-camera patches across neighboring stereo observations.The inverse depth map is estimated in the reference view from multiple nearby stereo observations.
- Evaluation of the Objective: A real stereo event-camera sequence illustrates the objective’s energy profile around a selected event, using 25-pixel patches and a minimum near the ground-truth inverse depth.The figure compares individual neighboring-pixel costs with their patch-averaged cost.
- 2.2 Problem Formulation: An inverse-depth hypothesis is forward-projected into both stereo image planes, jointly establishing candidate matches and triangulated 3D locations.This avoids first solving event matching and then triangulating, the classical two-step approach described as error-prone over short intervals.
- Optimization: The temporal objective is optimized with Gauss–Newton nonlinear least squares after a coarse search localizes the basin of convergence.Inverse depths for reference-view events are estimated independently, making the computation parallelizable.
3 Semi-Dense Reconstruction
The method fuses multiple sparse inverse-depth reconstructions into a selected reference view, using uncertainty-aware compatibility tests to produce denser and less uncertain semi-dense maps.
- Depth Fusion: Multiple reference-view reconstructions are fused incrementally to improve density and reduce inverse-depth uncertainty.The final reconstruction approaches semi-dense coverage by reconstructing pixels lying along edges.
- Inverse Depth Estimation: Inverse-depth estimation uses stereo event observations, relative poses, and camera extrinsics to optimize each reference-view pixel independently.The independent estimates make the computation parallelizable.
- Uncertainty Estimation: Temporal residual uncertainty is estimated by fitting a Gaussian distribution to residuals sampled using ground-truth depth.The empirical residual distribution and Gaussian fit are illustrated in Fig. 4.
- Depth Fusion: Inverse-depth hypotheses are assigned to neighboring pixels after reprojection and fused when their compatibility passes a χ2 test at 95%.If hypotheses are incompatible, the distribution with the smallest variance is retained.
4 Experiments
Experiments evaluate the stereo event-camera reconstruction on synthetic, hand-held, and drone-collected sequences using calibrated DAVIS rigs. The method outperforms the compared instantaneous stereo methods and becomes denser and more accurate as reconstructions are fused.
- Observation Construction: Each stereo observation is formed by collecting events over a 10 ms window and refreshing time-surface maps in both cameras.Observations are generated when a pose update is available, and one is selected as the reference view.
- Datasets: The evaluation covers six sequences: one synthetic, three hand-held office scenes, and two drone-collected indoor scenes.The drone sequences include varied structures and relatively high-speed motion.
- Quantitative Evaluation: The proposed method outperforms FCVF and SGM on all sequences in mean depth error, median depth error, and relative error.The competing methods work on pairs of time-surface images and perform less well when observations are sparse or temporal consistency is weak.
- Qualitative Evaluation: Fusion progressively produces denser and more accurate reconstructions as neighboring reference-view estimates increase from 4 to 8 to 16.Remaining highly uncertain estimates generally correspond to noise or low-contrast patterns.
5 Conclusion
The paper presents stereo event-camera 3D reconstruction through spatio-temporal consistency and reports relative errors between 1% and 5%. It also identifies full stereo visual odometry as future work.
- Contribution: The proposed method reconstructs 3D structure from a pair of temporally synchronized stereo event cameras.The authors describe it as the first method addressing this problem for stereo SLAM applications with event cameras.
- Results: The energy minimization exploits spatio-temporal consistency across cameras and achieves relative errors between 1% and 5%.It outperforms state-of-the-art stereo methods using the same spatio-temporal event representation.
- Future Work: Future work is to combine the reconstruction strategy with a stereo-camera pose tracker for full stereo visual odometry.The proposed direction follows a parallel tracking and mapping framework.
6 Appendices (Supplementary Material)
The appendix derives the warping, derivatives, and first-order inverse-depth uncertainty used in the objective function. It proceeds from back-projecting a reference event into 3D, transforming it between stereo observations, and propagating residual sensitivity.
- Back-projection: Reference-view events are back-projected into a 3D point using the inverse depth ρ before stereo warping.This 3D point is the basis for computing corresponding locations in the involved observations.
- Coordinate transformation: The 3D point is transformed into the left-camera coordinate system of an observation from the stereo reference view.The calculation uses the left camera projection matrix P1 and its zero last column.
- Event warping: The resulting camera coordinates are used to obtain the warping results for the stereo observations.The objective function warps each reference-view event location to each involved stereo observation.
- Derivative calculation: Derivatives with respect to inverse depth d are calculated for the warped observations and residual terms.The derivation also considers the temporal residual as a simplifying case.
- Uncertainty propagation: The overall inverse-depth uncertainty is approximated to first order from the residual Jacobians.The uncertainty expression combines the squared Jacobian contributions across residual terms.