Source-linked AI summary
Event-based, 6-DOF Camera Tracking from Photometric Depth Maps
Guillermo Gallego, Jon E. A. Lund, Elias Mueggler, Henri Rebecq, Tobi Delbruck, Davide Scaramuzza
TL;DR
Standard cameras struggle with motion blur and limited dynamic range, motivating low-latency event-camera tracking from an existing photometric depth map. The paper introduces an asynchronous Bayesian filter that updates 6-DOF pose per event and evaluates it indoors and outdoors. It reports accurate tracking across varied depths and all six motion degrees, including high-speed motion where the frame-based method loses tracking.
Problem
Standard-camera visual odometry struggles with high-speed motion and high-dynamic-range scenes, while the paper targets 6-DOF event-camera tracking from an existing photometric depth map.
Method
A Bayesian filter uses a noise- and outlier-aware mixture-model likelihood and updates the event camera’s 6-DOF pose for every incoming event at microsecond time resolution.
Results
The method accurately tracks event-camera pose indoors and outdoors, across significant depth variations and motions exciting all 6 DOFs; during high-speed motion, it continues after the frame-based method loses tracking.
Takeaways & Limitations
Event-based tracking provides comparable accuracy to standard cameras while supporting arbitrary high-speed motion in natural scenes.
Takeaways & Limitations
The algorithm requires an existing photometric depth map, which this work obtains using an Intel Realsense R200 RGB-D camera.
Abstract
from arXiv · showhide
Event cameras are bio-inspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. These cameras do not suffer from motion blur and have a very high dynamic range, which enables them to provide reliable visual information during high-speed motions or in scenes characterized by high dynamic range. These features, along with a very low power consumption, make event cameras an ideal complement to standard cameras for VR/AR and video game applications. With these applications in mind, this paper tackles the problem of accurate, low-latency tracking of an event camera from an existing photometric depth map (i.e., intensity plus depth information) built via classic dense reconstruction pipelines. Our approach tracks the 6-DOF pose of the event camera upon the arrival of each event, thus virtually eliminating latency. We successfully evaluate the method in both indoor and outdoor scenes and show that---because of the technological advantages of the event camera---our pipeline works in scenes characterized by high-speed motion, which are still unaccessible to standard cameras.
1 INTRODUCTION
The paper addresses 6-DOF event-camera tracking from a previously built photometric depth map, targeting AR/VR and video-game use under high-speed motion. It combines asynchronous event processing with a probabilistic filter and evaluates tracking against standard-camera limitations.
- Traditional visual odometry struggles with high-speed motion and high-dynamic-range scenes because standard cameras suffer motion blur and limited dynamic range.
- The paper tracks an event camera’s 6-DOF motion from an RGB-D map previously built using standard cameras or RGB-D sensors.
- The proposed system targets event-based pose tracking during very fast motions that remain inaccessible to standard cameras because of motion blur.
- The method updates pose estimates for every event at microsecond time resolution and uses a mixture-model likelihood for event generation, noise, and outliers.
- The resulting filter supports arbitrary 6-DOF, high-speed motion in natural scenes while using a computationally efficient closed-form update.
2 RELATED WORK ON EVENT-BASED EGO-MOTION ESTIMATION
Earlier event-based ego-motion systems were restricted by motion dimensionality, scene type, external sensors, or computational demands. This paper presents a broader approach for full 6-DOF motion in natural scenes without those restrictions.
- Early DVS pose tracking handled only slow planar 3-DOF motion in artificial black-and-white line-pattern scenes.
- A prior 3D extension used an external RGB-D sensor, whose depth measurements become outdated during very fast motion.
- Another system combined a DVS with a standard grayscale camera but remained limited to slow planar motion because of motion blur.
- Existing alternatives included high-speed 6-DOF tracking restricted to artificial line maps and orientation-only systems that omitted translation and depth.
- Unlike previous approaches, the proposed method handles full 6-DOF motion without external sensors, arbitrary speeds, or restrictions to specific textures or artificial scenes.
- Event-based SLAM trackers trade off computational cost or scene representation, with one requiring a GPU and another using edge-map alignment with semi-dense depth.
3 EVENT CAMERAS
Event cameras asynchronously report per-pixel brightness changes rather than fixed-rate intensity frames. Their event-triggering mechanism provides microsecond timestamps, polarity information, and high dynamic range.
- Unlike frame cameras, event cameras transmit pixel-level intensity changes asynchronously when they occur.
- An event is generated when the temporal log-intensity contrast at a pixel exceeds a nominal threshold.
- The elapsed time since the previous event at the same pixel is represented by Δt, and contrast-increase and contrast-decrease thresholds may differ.
- Each event carries pixel coordinates, a timestamp, and the polarity of the brightness change.
- Events provide microsecond-resolution timestamps and very low latency, while the DVS offers 140 dB dynamic range compared with 60 dB for conventional cameras.
4 PROBABILISTIC APPROACH
The paper formulates event-camera pose tracking as Bayesian estimation over camera motion, sensor parameters, and inlier/outlier behavior. It combines depth-based event transfer, robust likelihood modeling, and tractable posterior approximation for efficient 6-DOF updates.
- State and Bayesian formulation: The state includes the event camera’s kinematic description together with sensor and inlier/outlier parameters.The filter can also estimate sensor parameters such as the event-triggering threshold Cth automatically.
- State and Bayesian formulation: Each incoming event updates the 6-DOF pose through Bayesian prediction and correction, using the event likelihood conditioned on the current state.The state is recursively estimated from observations, with a motion-model prediction followed by a likelihood-based correction.
- Measurement model: Known scene depth transfers an event location to corresponding points in a reference image, where the predicted contrast is computed from log-intensity differences at two event-related times.The transfer uses the camera-to-reference transformation, scene depth, and perspective projection; the relevant poses include the current pose and trajectory poses for the previous same-pixel event.
- Posterior approximation and filtering: The method assumes linearization is adequate and uses slowly varying zero-mean diffusion, with covariance growth controlled by incoming events and a maximum allowed uncertainty.The information from a new event is assumed to contribute mainly to the most recent pose when pose changes are small and many pixels generate events.
- Measurement model: The sensor likelihood combines a contrast-residual measurement function with a normal-uniform mixture to model both valid events and outliers.The mixture assigns inlier probability πm to normally distributed measurements around zero and treats the remaining probability as outliers.
- Posterior approximation and filtering: The posterior is approximated within the exponential family by minimizing KL divergence, producing tractable sequential updates without maintaining multiple particle-filter hypotheses.For Gaussian pose distributions, the resulting update has the form of an Extended Kalman Filter weighted by each event’s inlier probability.
5 EXPERIMENTAL RESULTS
Experiments evaluate event-based 6-DOF pose tracking indoors and outdoors, including fast motion, depth variation, weak texture, occlusions, and moving objects. The method achieves low trajectory errors and continues tracking during motion blur, while processing 31,000 events per second in the reported implementation.
- Experimental setup: The evaluation covers indoor and outdoor sequences with fast motion and excitations in all six degrees of freedom.The experiments use an existing photometric depth map and include motion-capture and standard-camera comparisons.
- Indoor accuracy: 1.63 cm and 2.21° mean RMS errors were measured for indoor rocks sequences at a 60 cm mean scene depth.Mean position and orientation errors were 1.38 cm and 1.89°, with standard deviations of 0.84 cm and 1.15°.
- Comparison with frame-based tracking: 2.71% versus 1.8% position error and 2.21° versus 1.04° orientation error show event-based accuracy was only slightly worse than the frame-based method.The comparison uses the same rocks sequences and reports event-based results against the frame-based baseline.
- Tracking during high-speed motions: The event-based method continues pose estimation during high-speed motion after the frame-based method loses tracking at approximately 8.66 s because of motion blur.The frame-based method tracks only within the shaded region, whereas the event-based trajectories remain accurate during the magnified high-speed interval.
- Depth variation and outdoor scenes: Across additional indoor and outdoor sequences, mean RMS errors ranged from 2.5% to 4.0% of mean scene depth and from 1.4° to 2.9° in orientation.The method also tracks scenes with large depth variation, limited texture, occlusions, and moving objects.
- Computational effort: 32 µs per event yielded a processing rate of 31,000 events per second, below data rates exceeding one million events per second during high-speed motion.The authors note that the implementation is not fully optimized and could be improved through optimization, caching, and parallelization.
6 CONCLUSION
The conclusion presents a Bayesian event-based approach for 6-DOF pose tracking from an existing photometric depth map and evaluates it against motion-capture ground truth and frame-based tracking. Experiments report accurate tracking indoors and outdoors, including depth variation and six-degree-of-freedom motion.
- Conclusion: The approach tracks an arbitrarily moving event camera from an existing photometric depth map using Bayesian filtering.Its sensor model incorporates event generation, noise, and outliers, while posterior approximation yields an EKF-like filter.
- Conclusion: Experiments show accurate event-camera pose tracking indoors and outdoors, under significant depth variation and motions exciting all six degrees of freedom.The evaluation compares the method with motion-capture ground truth and a state-of-the-art frame-based pose-tracking pipeline.
APPENDIX A REWRITING THE LIKELIHOOD FUNCTION
The appendix rewrites the likelihood function using exponential-family notation. It identifies the distribution components and collects sufficient statistics so the likelihood becomes a sum of two exponential families.
- Exponential-family form: An exponential-family distribution is parameterized by natural parameters, sufficient statistics, a log-normalizer, and a base measure.The appendix denotes these quantities by η, T(x), A(η), and h(x), respectively.
- Likelihood rewriting: The likelihood is rewritten using Einstein summation and collected sufficient statistics.This reformulation supports the subsequent exponential-family representation.
- Likelihood rewriting: The likelihood is expressed as a sum of two exponential families with j = 1, 2 and h(s) = 1.This is the appendix’s final convenient form for the likelihood.