Real-time Event-camera Stereo Visual Odometry via Keytime Gaussian Process Regression

I. INTRODUCTION

• Naively applying continuous-time GP estimation with a WNOA prior requires a state for every event feature and does not run in real time.Continuous-time methods can preserve individual asynchronous measurement times during estimation.Source: §I · p. 1
• Sparse keytimes reduce the estimation state while WNOA interpolation evaluates each event feature at its individual timestamp.MC-RANSAC rejects outliers using event features at their individual timestamps before state estimation.Source: §I · p. 1
• The pipeline outperforms ES-PTAM on all but one tested sequence, at 22 Hz on MVSEC and 6 Hz on DSEC.The evaluation also reports similar accuracy to the non-real-time GPCT pipeline while reducing state size.Source: §I · p. 1

A. Discrete-time Approaches

• Discrete-time approaches group events during state estimation, reducing temporal resolution and potentially reintroducing motion blur and estimation inaccuracies.Other discrete pipelines may preserve event timing in the front end but group events in the back end.Source: §I · p. 1, A. Discrete-time Approaches · p. 2

B. Continuous-time Approaches

• GPCT preserves asynchronous measurements with MC-RANSAC but places a state at every measurement time, limiting real-time performance.The proposed keytime formulation is compared with GPCT as a way to reduce state size.Source: B. Continuous-time Approaches · p. 3
• The proposed stereo event-camera VO uses GP/WNOA keytime interpolation and MC-RANSAC to decouple state size from measurement count and achieve real-time performance.It reports reduced optimization times and uses a producer-consumer architecture.Source: B. Continuous-time Approaches · p. 3

III. METHODOLOGY

• The pipeline applies MC-RANSAC to timestamped event features, selects sparse keytimes, and minimizes per-measurement reprojection residuals using WNOA interpolation.The resulting continuous camera trajectory can be queried by interpolation both online and a posteriori.Source: §III · p. 3

A. Notation

• The notation maps between R6 coordinates and SE(3) using the matrix exponential and logarithm, and represents pose as an SE(3) transform.The state includes linear and angular velocity components, and the stated constant body-centric velocity over a time interval yields the transform between its endpoint frames.Source: A. Notation · p. 3

B. Outlier Rejection

• MC-RANSAC estimates constant velocity from individually timestamped event features, unlike frame-based RANSAC’s simultaneous-measurement assumption.The closed-form Euclidean-residual variant is approximate but faster than the iterative measurement-space method.Source: B. Outlier Rejection · p. 3

1) Fast Model Calculation:

• A randomly selected landmark subset yields an initial velocity through a closed-form quadratic approximation of the squared Euclidean residual.The residual can be linearized with a first-order exponential-map approximation when the transform is small, as often occurs over short durations.Source: 1) Fast Model Calculation · pp. 3–4

2) Inlier and Outlier Classification:

• A measurement pair is an inlier when its normalized nonlinear reprojection residual falls below the user-specified threshold αthresh.The initial velocity comes from a randomly sampled subset of landmark measurements.Source: 2) Inlier and Outlier Classification · p. 4

3) Inlier Refinement:

• Promising inlier sets are refined by repeating model calculation and classification over the entire inlier set, with the largest set and its velocity used to initialize estimation.The number of random-sampling iterations is user-selected.Source: 3) Inlier Refinement · p. 4

C. Continous-time Trajectory Optimization via Keytime Interpolation

• The trajectory uses a GP on local pose perturbations and their derivatives between keytimes, with acceleration driven by zero-mean white Gaussian noise.The WNOA formulation maintains a linear, time-invariant SDE; QC is the diagonal power spectral density matrix.Source: C. Continous-time Trajectory… · p. 4

• The problem contains M landmarks observed at K asynchronous measurement times and N keytimes, with N ≪ M < K.The landmark states are homogeneous coordinates.Source: 1) Problem Definition · p. 4
• Keytime states are estimated by minimizing a joint cost comprising measurement and prior terms.The prior penalizes deviation from WNOA kinematics, while the measurement term penalizes deviation from observations.Source: 1) Problem Definition · p. 4
• The measurement cost uses landmark reprojection residuals evaluated with the interpolated pose at each observation time.The residual is weighted by its measurement covariance and uses a nonlinear landmark transformation model.Source: 1) Problem Definition · p. 4

• Each measurement pose T(tk) is interpolated from adjacent keytime states xn and xn+1 when tn ≤ tk < tn+1.The interpolation matrices use local states related to global pose and velocity states; the WNOA covariance and state-transition matrix define the interpolation.Source: 2) Keytime Interpolation · p. 5

• The final trajectory is obtained by iteratively minimizing the joint cost using analytical measurement Jacobians and numerical optimization such as Gauss–Newton.The optimization uses nominal operating points and state perturbations to form a quadratic approximation.Source: 3) Optimization and Keytime Jacobians · p. 5
• The approximate Hessian and gradient define the linear system whose solution supplies the perturbation minimizing the quadratic approximation.The SE(3) inverse Jacobian is approximated by a second-order Taylor expansion.Source: 3) Optimization and Keytime Jacobians · p. 5
• Measurement Jacobians span two adjacent keytime block columns, preserving an approximate Hessian structure suitable for sparse linear solvers while reducing state size.The chain rule maps changes in adjacent keytime states through interpolation to changes in the landmark transformation model.Source: 3) Optimization and Keytime Jacobians · pp. 5–6

• MVSEC and DSEC have sensor resolutions of 346 × 260 and 640 × 480, respectively.Source: §IV · p. 6
• The GPCT comparison on MVSEC indoor1 and indoor3 evaluates how keytimes affect continuous-time trajectory accuracy.Source: §IV · p. 6
• The ES-PTAM comparison uses publicly available results to evaluate the keytime estimator as a real-time pipeline.Source: §IV · p. 6
• Experiments ran on an Intel Core i7-9750H CPU at 2.60 GHz with 16 GB of RAM.Source: §IV · p. 6

A. Implementation Details and Parameters

• The pipeline uses custom WNOA and keytime-interpolated measurement factors with incremental batch optimization via iSAM2.Source: A. Implementation Details and Parameters · p. 6
• Event clusters contain 10,500 events for MVSEC and 150,000 for DSEC, with each new keytime placed at the cluster’s last event.Source: A. Implementation Details and Parameters · p. 6
• Features are matched against the preceding five clusters, and DSEC feature extraction uses initial half-resolution downsampling.Source: A. Implementation Details and Parameters · p. 6
• MC-RANSAC uses an inlier threshold of 0.1, target confidence of 0.99, and a 5,000-iteration cap, with early stopping as the estimated success probability rises.Source: A. Implementation Details and Parameters · p. 6

B. Temporal Alignment and Evaluation Metrics

• Experiment 1 queries both continuous-time trajectories at high-frequency ground-truth timestamps; Experiment 2 uses ES-PTAM timestamps except for MVSEC indoor3.Source: B. Temporal Alignment and Evaluation… · p. 6
• Global pose error measures accuracy relative to the initial pose, while relative pose error measures drift between sequential estimates.Source: B. Temporal Alignment and Evaluation… · p. 6

C. Experiment 1: Estimation Accuracy vs. GPCT

• The keytime estimator has insignificant global and relative error differences from GPCT despite using a smaller state than GPCT’s one-state-per-measurement formulation.Source: C. Experiment 1: Estimation Accuracy… · p. 6

D. Experiment 2: Real-Time Performance vs. ES-PTAM

• The keytime estimator outperforms ES-PTAM in global error on all but one tested sequence and in relative error on all tested sequences, while averaging 22 Hz on MVSEC and 6 Hz on DSEC.Source: Table II · p. 7, Table III · p. 7, D. Experiment 2: Real-Time Performance… · p. 7

V. CONCLUSION

• Sparse keytimes decouple state size from measurement count, while WNOA interpolation evaluates reprojection errors at each measurement timestamp.Source: §V · p. 7
• A producer-consumer architecture supports real-time performance on both datasets, whereas ES-PTAM reports real-time performance only on MVSEC.Source: §V · p. 7
• Future work will compare different event representations and features in the pipeline.Source: §V · p. 7