PaperlayerView exact revision on arXiv

Source-linked AI summary

Continuous-Time Visual-Inertial Odometry for Event Cameras

Elias Mueggler, Guillermo Gallego, Henri Rebecq, Davide Scaramuzza

arXiv:1702.07389v2cs.ROcs.CV

TL;DR

Event cameras generate high-rate asynchronous measurements that challenge discrete-time visual-inertial odometry. The paper uses cubic splines to represent the camera-IMU trajectory continuously, directly fusing events and inertial data. On real scenes, it reports improved trajectory accuracy and accurate recovery of scale and orientation, while noting limitations from its smoothness prior and fixed control-pose spacing.

  • Problem

    Discrete-time pose models do not adequately represent asynchronous event timestamps and can require very large numbers of poses for high-rate event streams.

  • Method

    The method represents the event-camera trajectory as a smooth cubic-spline curve in rigid-body motion and directly fuses event and inertial measurements.

  • Results

    The method provides improved accuracy over event-based visual odometry and recovers absolute scale and gravity direction with approximately 5% accuracy.

  • Takeaways & Limitations

    Continuous-time event-inertial fusion yields a compact, smooth trajectory representation that accommodates asynchronous measurements and supports line-based and point-based maps.

  • Takeaways & Limitations

    Cumulative B-splines impose a smooth-trajectory prior and are unsuitable for discontinuities; fixed control-pose spacing is not optimal for abrupt speed changes.

Abstract

from arXiv · show

Event cameras are bio-inspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. They offer significant advantages over standard cameras, namely a very high dynamic range, no motion blur, and a latency in the order of microseconds. However, due to the fundamentally different structure of the sensor's output, new algorithms that exploit the high temporal resolution and the asynchronous nature of the sensor are required. Recent work has shown that a continuous-time representation of the event camera pose can deal with the high temporal resolution and asynchronous nature of this sensor in a principled way. In this paper, we leverage such a continuous-time representation to perform visual-inertial odometry with an event camera. This representation allows direct integration of the asynchronous events with micro-second accuracy and the inertial measurements at high frequency. The event camera trajectory is approximated by a smooth curve in the space of rigid-body motions using cubic splines. This formulation significantly reduces the number of variables in trajectory estimation problems. We evaluate our method on real data from several scenes and compare the results against ground truth from a motion-capture system. We show that our method provides improved accuracy over the result of a state-of-the-art visual odometry method for event cameras. We also show that both the map orientation and scale can be recovered accurately by fusing events and inertial data. To the best of our knowledge, this is the first work on visual-inertial fusion with event cameras using a continuous-time framework.

I. INTRODUCTION

Event cameras provide asynchronous, microsecond-resolution brightness-change measurements, but conventional discrete-time pose estimation cannot represent their timing consistently or scale to their measurement rate. The paper addresses these issues by combining event and inertial data through a continuous-time trajectory model.

  • Event-camera sensing: Event cameras transmit asynchronous brightness-change events at microsecond resolution rather than intensity images.Each event records space-time coordinates and the sign of the brightness change.
  • Motivation: Discrete-time visual odometry is unsuitable because individual events lack sufficient information, mixed event timestamps yield no single-time pose, and event rates can overwhelm estimation.Traditional formulations would require preserving temporal information with a very large number of poses.
  • Continuous-time formulation: A continuous-time pose trajectory assigns a well-defined pose to every event timestamp while using far fewer parameters than events.The smooth parametric representation addresses timestamp correspondence and reduces state-space size and computational cost.
  • Contributions: The paper extends prior event-camera continuous-time estimation to visual-inertial odometry with a combined event-camera and IMU sensor.Fusing inertial data is reported to improve trajectory accuracy and recover absolute scale and gravity alignment.
  • Contributions: The approach is extended from line-based maps to natural scenes with point-based maps and is evaluated using real-data experiments.The experiments include full-smoothing camera-trajectory estimation and refinement of event-based visual-odometry poses.

III. RELATED WORK: EGO-MOTION ESTIMATION WITH EVENT CAMERAS

Earlier event-camera ego-motion systems used restricted scene assumptions, auxiliary sensors, probabilistic filters, or discrete groups of events. This paper instead represents camera motion continuously to fuse asynchronous events and inertial measurements without timestamp approximation.

  • Earlier event-camera systems: Early DVS localization and SLAM systems were limited to planar motion or scenes, including black-and-white line-pattern environments.The cited particle-filter approach was restricted to planar motions and planar scenes parallel to the motion plane.
  • Auxiliary sensing: Several systems attached conventional cameras or RGB-D sensors to event cameras to simplify pose estimation or provide depth.Other approaches used intensity images for feature detection and event-based tracking.
  • Event-based ego-motion: Event-only methods demonstrated 6-DOF localization, high-speed rotational tracking, probabilistic filtering, and natural-scene tracking with depth and brightness information.Reported systems include line-map localization, rotational-motion estimation, parallel filtering, and geometric tracking-and-mapping.
  • Event-IMU methods: Previous methods generally produced discrete pose estimates from individual events or event groups, while newer event-IMU systems fed tracked features into standard geometric VIO algorithms.These approaches differ from the paper’s direct continuous-time treatment of the measurements.
  • Paper’s approach: The paper estimates a continuous trajectory from control poses and basis functions, enabling timestamp-accurate fusion of events and inertial data with a compact parameterization.Pose and linear velocity are both derived from one trajectory representation, avoiding separate state estimates for these quantities.

A. Camera Pose Transformations

The paper represents camera poses as rigid-body motions in SE(3) and models the event-camera trajectory with smooth cubic B-splines. This provides local, differentiable pose interpolation at arbitrary timestamps using a compact set of control poses.

  • A. Camera Pose Transformations: Camera poses are represented as rigid-body motions in SE(3), with rotation in SO(3) and translation in R^3.
  • A. Camera Pose Transformations: A local-charts representation expresses each trajectory segment as an anchor pose followed by a small incremental motion.
  • B. Cubic Spline Camera Trajectories in SE(3): B-splines provide smooth trajectories with local support, analytical derivatives and integrals, and pose interpolation at arbitrary times.
  • B. Cubic Spline Camera Trajectories in SE(3): The cubic spline trajectory is parameterized by control poses at uniformly spaced times, with each interval depending on four neighboring poses.
  • B. Cubic Spline Camera Trajectories in SE(3): The cumulative formulation uses one absolute control pose and three incremental poses encoded by twists to compute the interpolated pose.

C. Generative Model for Visual and Inertial Observations

The generative model predicts visual and inertial observations from a continuous camera trajectory and a known geometric scene map. Cubic-spline derivatives supply the velocity and acceleration needed to model IMU measurements.

  • C. Generative Model for Visual and Inertial Observations: A continuous trajectory provides camera velocity and acceleration at any time for comparison with IMU measurements.
  • C. Generative Model for Visual and Inertial Observations: The IMU model predicts angular velocity and linear acceleration from the spline trajectory, while accounting for sensor biases and gravity.
  • C. Generative Model for Visual and Inertial Observations: The scene map is assumed known and consists of either 3D points or 3D line segments, with experiments using both primitive types.
  • C. Generative Model for Visual and Inertial Observations: For point maps, each 3D point represents a scene edge and is projected into event-camera image coordinates through a perspective projection matrix.
  • C. Generative Model for Visual and Inertial Observations: For line maps, projected line segments are formed from the projections of their 3D endpoints.

VI. CAMERA TRAJECTORY OPTIMIZATION

The trajectory-estimation problem is formulated probabilistically as maximum-likelihood inference from event and inertial measurements. Conditional independence factorizes the likelihood into visual, gyroscope, and accelerometer terms that yield a least-squares objective.

  • VI. CAMERA TRAJECTORY OPTIMIZATION: Trajectory estimation seeks the posterior density of the event-camera trajectory given the map and all visual-inertial measurements.
  • VI. CAMERA TRAJECTORY OPTIMIZATION: With no state prior, the optimal trajectory maximizes the likelihood of the event, angular-velocity, and acceleration measurements.
  • VI. CAMERA TRAJECTORY OPTIMIZATION: Assuming conditional measurement independence, the likelihood factorizes into event, gyroscope, and accelerometer likelihoods.
  • VI. CAMERA TRAJECTORY OPTIMIZATION: The visual term models event-location errors with independent measurements and a zero-mean Gaussian image-coordinate error assumption.
  • VI. CAMERA TRAJECTORY OPTIMIZATION: Predicted event locations are projected map points or points on projected line segments, and visual residuals use corresponding image-plane distances.
  • VI. CAMERA TRAJECTORY OPTIMIZATION: The inertial likelihood terms use angular-velocity and acceleration predictions derived from the modeled trajectory.

B. Parametric Trajectory Optimization

The paper converts trajectory optimization from an infinite-dimensional curve problem into finite-dimensional nonlinear least squares by using B-spline control poses. It jointly estimates trajectory controls and selected inertial and map parameters.

  • B. Parametric Trajectory Optimization: B-spline basis functions replace an arbitrary SE(3) curve with a finite set of control poses, making optimization finite dimensional.
  • B. Parametric Trajectory Optimization: The resulting objective is a nonlinear least-squares problem solvable with Gauss-Newton or Levenberg-Marquardt methods.
  • B. Parametric Trajectory Optimization: In addition to control poses, the optimization estimates gyroscope and accelerometer biases, map scale, and map orientation relative to gravity.
  • B. Parametric Trajectory Optimization: The map orientation is represented by roll and pitch angles because monocular maps lack absolute scale and orientation information.
  • B. Parametric Trajectory Optimization: Visual predictions evaluate each event at its timestamp by interpolating the pose, projecting the map, and measuring distance to the corresponding primitive.

VII. EXPERIMENTS

The experiments evaluate event-camera visual-inertial odometry with line-based and point-based maps, using inertial fusion and motion-capture ground truth to assess trajectory accuracy.

  • VII. EXPERIMENTS: The method is evaluated with both line-based and point-based map representations.These representations test line-to-point distance and point-to-point reprojection visual error terms.
  • VII. EXPERIMENTS: Trajectory accuracy is quantified against motion-capture ground truth.The experiments use ground-truth measurements from a motion-capture system.
  • VII. EXPERIMENTS: The DAVIS provides events, synchronized 1 kHz inertial measurements, and global-shutter images used for calibration, initialization, and visualization.Its spatial resolution is 240 × 180 pixels, and inertial measurements are time-synchronized with events.
  • VII. EXPERIMENTS: The line-based method tracks map lines through event-line associations and estimates pose using intersections of the tracked lines.Events not close to map lines are ignored during optimization.

1) Tracking Method:

The line-based experiment compares event tracking, vision-only spline optimization, and event-plus-IMU spline optimization against motion-capture ground truth. Fusing inertial measurements substantially improves trajectory accuracy.

  • 1) Tracking Method:: The experiment moves a DAVIS sensor by hand above a square pattern inside a motion-capture system.Position error uses Euclidean distance, while orientation error uses the geodesic distance in SO(3).
  • 1) Tracking Method:: Three algorithms are compared: event-based tracking, spline optimization without IMU, and spline optimization with IMU.The comparisons are made against motion-capture ground truth.
  • 1) Tracking Method:: 0.5 % of average scene depth is the mean position error and 0.37° is the mean orientation error for spline optimization with events and IMU.These are the smallest mean, standard deviation, and maximum errors among the compared methods.
  • 1) Tracking Method:: Errors are up to five times smaller than event-based tracking, while adding IMU measurements reduces errors by approximately a factor of 4.Vision-only spline optimization improves over event-based tracking by approximately a factor of 1.5.
  • 1) Tracking Method:: Control poses were placed every 0.1 s, yielding approximately 5000 events and 100 inertial observations per control pose.The control poses were initialized by fitting a spline through initial tracker poses.
  • 1) Tracking Method:: The square pattern used for scale estimation has a side length of 10 cm, with initial lengths ranging from 0.1 cm to 10 m.The initial values span two orders of magnitude in both directions.

B. Camera Trajectory Estimation in Point-based Maps

The point-based experiments test continuous-time trajectory estimation on natural desk, boxes, and dynamic scenes using maps and event correspondences from EVO. Trajectories and errors are evaluated against motion-capture ground truth.

  • B. Camera Trajectory Estimation in Point-based Maps: The proposed continuous-time trajectory estimation is evaluated on natural scenes without requiring strong artificial gradients.The datasets are desk, boxes, and dynamic from the Event-Camera Dataset.
  • B. Camera Trajectory Estimation in Point-based Maps: EVO supplies a map and asynchronous discrete 6-DOF event-camera poses for the point-based experiments.The map points are used to establish event correspondences for subsequent optimization.
  • B. Camera Trajectory Estimation in Point-based Maps: A map point and event are associated when their projections occupy the same pixel.Events without an associated map point are treated as noise and ignored in optimization.
  • B. Camera Trajectory Estimation in Point-based Maps: Figures 6–8 show 3D maps, event-camera trajectories, and position and orientation errors for the three datasets.The estimated trajectories are compared against motion-capture ground truth, with statistics reported in Tables II–IV.

2) Experiments:

The point-based experiments compare EVO with spline trajectory optimization using events, IMU, and absolute-scale estimation across three datasets. Inertial fusion recovers scale and gravity alignment with dataset-dependent accuracy.

  • 2) Experiments:: Four methods are compared against motion-capture ground truth: EVO, vision-only spline optimization, event-plus-IMU spline optimization, and joint trajectory-scale optimization.The first three trajectories are aligned to ground truth using rotation, translation, and uniform scaling.
  • 2) Experiments:: Continuous-time knots are spaced every 0.2 s for desk and every 0.15 s for boxes and dynamic.These settings produce approximately 104 events and 150–200 inertial measurements per control pose.
  • 2) Experiments:: Absolute-scale error is 4.1 % for desk, 6.5 % for boxes, and 2.8 % for dynamic.The scale is estimated jointly with trajectory and orientation parameters.
  • 2) Experiments:: Gravity-direction error is 3.83° for desk, 20.18° for boxes, and 3.34° for dynamic.The higher boxes error is attributed to dominant translation and insufficient rotational motion.
  • 2) Experiments:: Fusing inertial data with EVO enables absolute-scale recovery and map alignment with gravity, which monocular EVO cannot obtain alone.EVO is monocular and therefore cannot estimate absolute scale from visual observations alone.

3) Absolute Map Scale and Gravity Alignment:

Fusing event and inertial measurements enables accurate camera-motion estimation while recovering map scale and gravity alignment. The continuous-time spline representation supports compact full-history optimization but the current implementation is not real-time.

  • The current implementation runs offline and is around 3 to 13 times slower than real time, although sliding-window processing and other optimizations could improve speed.
  • Event-inertial fusion significantly increases estimated camera-motion accuracy over event-only approaches, by a factor of four in the experiments.
  • The compact B-spline representation assimilates several hundred thousand events and inertial measurements using few control poses while maintaining a smooth trajectory.
  • The method recovers absolute scale and gravity direction for both line-based and point-based maps, with approximately 5% accuracy.
  • Mean position error is less than 1% of average scene depth and mean orientation error is less than 1° for both map representations.
  • Cumulative B-splines impose a smooth-motion prior and are unsuitable for discontinuities such as bumps or crashes.

IX. CONCLUSION

The paper presents continuous-time visual-inertial odometry for event cameras, combining asynchronous events and high-frequency inertial measurements in a compact, smooth trajectory model. Experiments report sub-1% position error relative to scene depth and sub-1° orientation error.

  • The method uses a continuous-time framework to fuse asynchronous event measurements with high-frequency inertial data.
  • The camera trajectory is approximated by a smooth curve in rigid-body motion space using cubic splines.
  • The trajectory is optimized with an image-plane geometric error and direct inertial-measurement inclusion, both given probabilistic justifications.
  • Mean position error is less than 1% of average scene depth, while mean orientation error is less than 1° across the experiments.
Loading 1702.07389v2…