PaperlayerView exact revision on arXiv

Source-linked AI summary

Event-based Stereo Visual Odometry

Yi Zhou, Guillermo Gallego, Shaojie Shen

arXiv:2007.15548v2cs.CVcs.RO

TL;DR

The paper addresses stereo visual odometry from event streams for natural scenes and general 6-DoF motion. It introduces a real-time tracking-and-mapping system based on spatio-temporal consistency, probabilistic depth fusion, and time-surface registration, and reports operation on a standard CPU.

  • Problem

    Stereo visual odometry with event cameras must estimate rig motion and scene structure from asynchronous events in natural scenes with arbitrary 6-DoF motion.

  • Method

    The system interleaves mapping and tracking, using spatio-temporal stereo consistency, probabilistic inverse-depth fusion, and 3D−2D registration on time surfaces.

  • Results

    Experiments on public datasets and self-recorded data demonstrate the system in natural scenes with general 6-DoF motion while running in real time on a standard CPU.

  • Takeaways & Limitations

    The work provides a complete stereo event-based visual odometry system and releases its software, stereo-rig design, and datasets under an open-source licence.

Abstract

from arXiv · show

Event-based cameras are bio-inspired vision sensors whose pixels work independently from each other and respond asynchronously to brightness changes, with microsecond resolution. Their advantages make it possible to tackle challenging scenarios in robotics, such as high-speed and high dynamic range scenes. We present a solution to the problem of visual odometry from the data acquired by a stereo event-based camera rig. Our system follows a parallel tracking-and-mapping approach, where novel solutions to each subproblem (3D reconstruction and camera pose estimation) are developed with two objectives in mind: being principled and efficient, for real-time operation with commodity hardware. To this end, we seek to maximize the spatio-temporal consistency of stereo event-based data while using a simple and efficient representation. Specifically, the mapping module builds a semi-dense 3D map of the scene by fusing depth estimates from multiple local viewpoints (obtained by spatio-temporal consistency) in a probabilistic fashion. The tracking module recovers the pose of the stereo rig by solving a registration problem that naturally arises due to the chosen map and event data representation. Experiments on publicly available datasets and on our own recordings demonstrate the versatility of the proposed method in natural scenes with general 6-DoF motion. The system successfully leverages the advantages of event-based cameras to perform visual odometry in challenging illumination conditions, such as low-light and high dynamic range, while running in real-time on a standard CPU. We release the software and dataset under an open source licence to foster research in the emerging topic of event-based SLAM.

MULTIMEDIA MATERIAL

Event cameras asynchronously encode intensity changes, offering microsecond resolution and high-speed sensing, but require specialized algorithms for robot perception. This paper addresses stereo event-based visual odometry with an interleaved tracking-and-mapping system for arbitrary 6-DoF motion.

  • MULTIMEDIA MATERIAL: Event cameras report pixel-wise intensity changes asynchronously, reducing temporal redundancy and enabling microsecond-resolution capture of high-speed motion.They do not produce grayscale images or operate at a fixed rate like traditional cameras.
  • MULTIMEDIA MATERIAL: Combining event cameras with depth or standard cameras can introduce bottlenecks because the additional sensors limit system speed and dynamic range.
  • MULTIMEDIA MATERIAL: The proposed system performs stereo visual odometry in natural scenes with arbitrary 6-DoF motion, estimating stereo-rig ego-motion and a 3D scene map in real time.
  • MULTIMEDIA MATERIAL: Its mapping module measures spatio-temporal consistency across stereo event streams and fuses inverse-depth estimates probabilistically to improve 3D-structure density and accuracy.
  • MULTIMEDIA MATERIAL: The tracking module uses 3D−2D registration that leverages the distance-field nature of a compact event representation.
  • MULTIMEDIA MATERIAL: Experiments report computational efficiency on public and self-recorded datasets, with real-time operation on a standard CPU and open-sourced software and datasets.

B. Event-based Camera Pose Estimation

Event-based pose estimation has progressed from constrained motion and specialized scenes toward arbitrary 6-DoF motion, while methods trade latency against stability. The proposed system uses time surfaces for efficient stereo mapping and tracking from raw events.

  • B. Event-based Camera Pose Estimation: Prior event-camera localization studied pure rotation, planar motion, and specialized artificial patterns before addressing arbitrary 6-DoF motion.
  • B. Event-based Camera Pose Estimation: Probabilistic filters provide microsecond event-by-event updates, whereas frame-based non-linear optimization trades latency for more stable and accurate results.
  • B. Event-based Camera Pose Estimation: Existing monocular 6-DoF methods estimate intensity, depth, and pose jointly or use geometric mapping with image alignment, with the former requiring GPU hardware for real-time operation.
  • B. Event-based Camera Pose Estimation: The paper presents the first published stereo event-based VO algorithm, evaluating it against an ICP method underlying an undisclosed industrial demonstrator.
  • B. Event-based Camera Pose Estimation: The system takes raw events from calibrated cameras and interleaves event preprocessing, mapping, and tracking to estimate rig pose and semi-dense scene structure.
  • B. Event-based Camera Pose Estimation: Time surfaces store recent event times on a 2D grid, compactly representing moving edges for pixel-wise stereo comparisons and motion estimation.

IV. MAPPING: STEREO DEPTH ESTIMATION BY SPATIO-TEMPORAL CONSISTENCY AND FUSION

The mapping module estimates event inverse depths by minimizing stereo spatio-temporal inconsistency, then fuses estimates into a denser reconstruction over time.

  • Mapping module: The mapping module computes individual event depths and fuses them into an accurate, populated depth map.Fusion refers estimates to a common time, reduces structural uncertainty, and produces depth for most edge pixels.
  • Spatio-temporal consistency: Stereo matching uses time-surface consistency across space-time neighborhoods because pixel-level temporal coincidence is unreliable.Delays, jitter, and pixel mismatch motivate minimizing a neighborhood-based temporal consistency criterion.
  • Design for real time: The method replaces longer temporal-span objectives with one stereo time-surface map and adds probabilistic fusion for longer-span depth estimation.The real-time design also replaces brute-force initialization with block matching and uses signed per-pixel residuals.
  • Inverse-depth estimation: Inverse depth is estimated by optimizing an objective over a bounded search interval along the event’s viewing ray.Events are represented before the stereo observation, with inverse depth parameterized on the left-camera viewing ray.

1) Problem Statement:

For each event, the method estimates inverse depth by comparing motion-compensated stereo neighborhoods in time surfaces rather than intensity images or explicit two-step matching.

  • Problem formulation: The objective sums squared temporal differences between corresponding pixels in left and right neighborhoods.Known calibration and interpolated camera poses determine the projected points and compensate each event using its timestamped pose.
  • Spatio-temporal consistency: The method uses only event time surfaces, replacing the photometric error used in frame-based stereo.A real sequence example shows the objective profile attaining optimal depth for a generic event.
  • Problem formulation: An inverse-depth hypothesis defines a 3D point whose stereo projections establish a candidate event match.The best candidate minimizes the temporal residual objective over corresponding neighborhoods.
  • Non-linear solver: Nonlinear least squares optimization applies Gauss–Newton updates using the residual Jacobian, with analytical derivatives accelerating computation.The residual vector contains one component per patch pixel.

3) Initialization of the Non-Linear Solver:

The nonlinear depth solver is initialized efficiently with stereo block matching, then refines each event’s inverse depth through iterative residual and Jacobian updates.

  • Initialization: The initial inverse-depth guess comes from ZNCC block matching along epipolar lines on an integer-pixel disparity grid.This avoids exhaustive inverse-depth grid search in the canonical stereo configuration.
  • Initialization: The initialization approximates corresponding patch centers using disparity while omitting compensation for motion between event and observation times.The resulting temporal-consistency violation is refined during subsequent nonlinear optimization.
  • Iteration: For each event, the algorithm iteratively computes residuals and analytical Jacobians, updates inverse depth, and returns the converged estimate.Inputs include the event, stereo time surfaces, incremental rig motion, and fixed camera extrinsics.
  • Depth fusion: Fused inverse-depth estimates produce a semi-dense map at the current time, with depth values for most edge pixels.Fusion reduces uncertainty and improves reconstruction density across estimates with different timestamps.

1) Probabilistic Model of Estimated Inverse Depth:

The method models inverse-depth uncertainty probabilistically by fitting Student’s t distributions to temporal residuals and propagating that model through the estimator.

  • Probabilistic model: Inverse depth is represented by a probability distribution rather than only an average estimate.The estimator’s spread is studied empirically using events, ground-truth depth, and poses from two datasets.
  • Residual model: A Student’s t distribution fits the empirical temporal-residual histograms well, with residuals approximately centered at zero.The fitted model uses mean, scale, and degrees of freedom parameters.
  • Uncertainty propagation: Affine transformation preserves the multivariate Student’s t family, enabling the residual model to approximate the distribution of inverse-depth updates.The transformation uses the residual Jacobian and its normalized components.
  • Robust estimation: The heavy-tailed model supports robust estimation by re-weighting squared residuals and solving with IRLS.The weighting accounts for noise and outliers and replaces Gauss–Newton in the depth-estimation algorithm.

2) Inverse Depth Filters:

The fusion module propagates inverse-depth estimates to a common observation time and combines them probabilistically into a semi-dense map. It handles pixel correspondence and conflicting hypotheses through spatial interpolation, compatibility checks, and robust updates.

  • Temporal propagation and fusion: Inverse-depth estimates from multiple stereo pairs are first propagated from each event time to the current stereo-observation time, then fused with prior estimates.The system fuses estimates from 20 stereo observations to construct the current inverse-depth map.
  • Spatial assignment: A propagated estimate at a non-integer image location influences the four nearest integer pixels.The target coordinates are generally non-integer, so the estimate is distributed spatially among neighboring pixels.
  • Distribution updates: If a target pixel has no prior distribution, the propagated Student’s t distribution initializes it; otherwise, the method checks compatibility before updating.Initialization uses the propagated distribution directly, while existing estimates require a compatibility decision.
  • Distribution updates: Compatible inverse-depth hypotheses are fused into one distribution, whereas incompatible hypotheses retain the estimate with the smallest variance.This rule preserves the more certain hypothesis when the two estimates disagree.

4) Summary:

The mapping module creates a semi-dense inverse-depth map by estimating and fusing probabilistic depths, while tracking registers that map to time-surface negatives. The design reuses event representations and selects a single time surface for efficiency.

  • Mapping: Mapping estimates inverse depth from stereo observations and events, propagates estimates across timestamps, and fuses them into a semi-dense map.The fused map is intended to provide sufficient signal-to-noise ratio for subsequent tracking.
  • Mapping: Depth is not estimated for every event because doing so is expensive and conflicts with the real-time, limited-resource objective.All events still contribute to time surfaces used by both mapping and tracking.
  • Mapping: The fusion window of 20 stereo observations was chosen empirically for good depth-map density, while adaptive selection based on statistical criteria remains future work.The proposed alternatives include apparent point density and reduction in fused-depth uncertainty.
  • Tracking: Tracking estimates the stereo-rig pose from events and a local map using time-surface negatives, aligning their dark regions with the warped inverse-depth support.The method uses an image-like registration formulation and only the left time surface because adding the right one doubles cost without significantly improving accuracy.
  • Tracking: The time-surface negative acts as an anisotropic distance field, with small values marking current edges and ramps encoding distance toward them.This transformation changes registration into a minimization problem.

B. Tracking Problem Statement

Tracking is formulated as a 3D–2D registration problem: a semi-dense inverse-depth map is warped into the current left-camera frame and aligned with minima of the time-surface negative. The implementation uses compositional Lucas–Kanade optimization with robustness and stochastic efficiency measures.

  • Registration objective: The tracker seeks the pose that aligns the warped support of a reference semi-dense map with minima in the current left-camera time-surface negative.The reference support consists of pixels with valid inverse depth.
  • Warping function: The warp back-projects reference pixels into 3D, transforms them using candidate motion, and perspective-projects them into the current left image.The motion is represented by six parameters: three Cayley orientation parameters and three translation parameters.
  • Optimization: The registration objective is reformulated with forward compositional Lucas–Kanade optimization, which iteratively refines incremental pose parameters.The compositional update minimizes the objective with respect to an incremental pose at each iteration.
  • Optimization: Compositional optimization permits precomputing parts of the Jacobian, making it more efficient than the additive formulation.Its local objective is reported as smooth, differentiable, and uniquely minimized near the ground-truth pose.
  • Robust and efficient tracking: A Huber norm with IRLS improves robustness to noise and outliers in the inverse-depth map, while stochastic LM evaluates batches of typically 300 points.Because the initial pose is usually close to the optimum, five iterations are often sufficient for convergence.

VI. EXPERIMENTS

Experiments evaluate mapping and full stereo visual odometry on public and synthetic data using multiple stereo-event-camera rigs. The proposed probabilistic mapping improves depth reconstruction, while the complete system is assessed under varied motion, illumination, and computational conditions.

  • Experimental design: The evaluation separates mapping-only tests with ground-truth poses from full-system tests that estimate trajectories using events alone.The full-system experiments also examine low-light, HDR, and computational performance.
  • Mapping optimization: The Student’s t distribution-based objective produces depth-error standard deviations 2–3 times smaller than standard least squares on synthetic planar scenes.The corresponding reconstructions contain more accurate and compact planar structures.
  • Sequence-dependent behavior: GTS density is considerably lower on upenn than rpg sequences, associated with larger depth range and predominantly translational drone motion.Relaxing the event-matching time threshold could increase density but would produce less accurate and noisier depth estimates.
  • Stereo reconstruction comparison: Figure 12 compares inverse-depth maps from GTS, SGM, CopNet, and the proposed method, with colors encoding depth ranges from close to far.Intensity frames are shown only for visualization; the methods use event-derived data rather than intensity information.
  • Stereo reconstruction comparison: The proposed method outperforms GTS, SGM, and CopNet in mean, median, and relative depth error on the sequences with LiDAR ground truth.CopNet provides satisfactory density but lower depth accuracy, while SGM produces the densest maps through regularization.

D. Full System Evaluation

The full system is evaluated against an event-based SGM+ICP baseline and ORB-SLAM2 using trajectory errors on six sequences. The proposed system outperforms the event-based baseline across all sequences, while its relative performance to ORB-SLAM2 depends on the dataset and recording conditions.

  • The evaluation measures full-system ego-motion with relative pose error and absolute trajectory error on six sequences with ground-truth trajectories.
  • SGM+ICP combines SGM dense depth estimation with ICP relative-pose estimation between successive point clouds.
  • The proposed method outperforms the event-based SGM+ICP baseline on all evaluated sequences.
  • The system is slightly less accurate than ORB-SLAM2 on the rpg dataset but performs better on the upenn indoor flying dataset.
  • Flickering induced by the motion-capture system deteriorates the proposed method on rpg sequences but does not affect ORB-SLAM2's grayscale frames.
  • Upenn accuracy is lower partly because dominant translation leaves structures parallel to the stereo baseline unreconstructed.

E. Experiments in Low Light and HDR Environments

Experiments examine the system in low-light and high-dynamic-range scenes, while also reporting trajectory plots, reconstruction behavior, and computational performance. The implementation uses threaded mapping and tracking components for real-time operation.

  • In dark-room sequences, including one illuminated by a directional lamp, the system is tested under low-light and high-dynamic-range conditions.
  • Figure 13 compares two tracking methods with ground truth using translational degrees of freedom and rotational error in SO(3) across multiple sequences.
  • Figure 14 visualizes sequential inverse-depth maps, a 3D reconstruction, and the estimated trajectory for the hkust lab sequence.
  • The mapping node uses four threads, refines 500 inverse-depth estimates in 15 ms, performs 60000 fusion operations in 20 ms, and typically runs at 20 Hz.
  • Using more events increases reconstruction density but also increases computation time, motivating the real-time choice of 1000 events.
  • The tracking node uses two threads, solves pose estimation in approximately 10 ms, and can run at up to 100 Hz.

G. Discussion: Missing Edges in Reconstructions

The discussion identifies reconstruction gaps for structures parallel to the stereo baseline and examines motion-dependent spatio-temporal consistency. Uncertainty filtering improves reliability at the cost of density, while completeness remains open for future work.

  • Edges parallel to the stereo baseline are difficult to recover regardless of motion, and estimates for such structures are typically unreliable across stereo methods.
  • Thresholding the depth-uncertainty map rejects unreliable estimates and produces a more reliable but sparser depth map.
  • Completeness of reconstructions affected by missing baseline-parallel structures is left as future work.
  • Time surfaces are motion dependent, so the proposed spatio-temporal consistency criterion may fail to hold perfectly for specific stereo-rig motions.
  • The reported t-distribution scale values are s_trans Y = 14.995 and s_rot Z = 21.838, with residuals similar to those in the upenn flying1 sequence.
  • Despite motion dependence, the authors observed no significant temporal inconsistency that broke the system in their real-data experiments.
Loading 2007.15548v2…