Source-linked AI summary
Direct Sparse Odometry
Jakob Engel, Vladlen Koltun, Daniel Cremers
TL;DR
The paper addresses how to perform direct visual odometry while retaining consistent joint optimization of geometry and motion without the costly geometry priors used by dense methods. It proposes a sparse direct formulation with photometric calibration and reports superior accuracy and robustness across several real-world datasets, while noting greater sensitivity to geometric noise and calibration.
Problem
Geometry priors used by dense direct methods introduce correlations that make statistically consistent joint optimization in real time infeasible.
Method
The method jointly optimizes photometric error over sampled image neighborhoods with camera poses, intrinsics, inverse depths, and photometric calibration parameters.
Results
Evaluations on three datasets comprising several hours of video show superior tracking accuracy and robustness relative to state-of-the-art direct and indirect methods.
Takeaways & Limitations
Sparse direct odometry combines broad image sampling with efficient joint optimization and can run in real time on a laptop computer.
Takeaways & Limitations
The direct model is more severely affected by geometric noise and may fail for δg > 1.5, while accurate intrinsic calibration can alleviate sensitivity.
Abstract
from arXiv · showhide
We propose a novel direct sparse visual odometry formulation. It combines a fully direct probabilistic model (minimizing a photometric error) with consistent, joint optimization of all model parameters, including geometry -- represented as inverse depth in a reference frame -- and camera motion. This is achieved in real time by omitting the smoothness prior used in other direct methods and instead sampling pixels evenly throughout the images. Since our method does not depend on keypoint detectors or descriptors, it can naturally sample pixels from across all image regions that have intensity gradient, including edges or smooth intensity variations on mostly white walls. The proposed model integrates a full photometric calibration, accounting for exposure time, lens vignetting, and non-linear response functions. We thoroughly evaluate our method on three different datasets comprising several hours of video. The experiments show that the presented approach significantly outperforms state-of-the-art direct and indirect methods in a variety of real-world settings, both in terms of tracking accuracy and robustness.
1. Introduction
The paper introduces sparse and direct visual odometry, combining photometric optimization with sparse geometry to avoid the computational cost of geometry priors while retaining broad image sampling. It jointly optimizes model parameters and incorporates photometric calibration for real-time, accurate, and robust operation.
- Direct and sparse formulations: Sparse methods reconstruct selected independent points, whereas dense methods reconstruct all pixels or a connected, well-constrained subset.The paper distinguishes sparse, semi-dense, and dense formulations by the extent and connectedness of the reconstructed image region.
- Direct and sparse formulations: Sparse + Direct visual odometry optimizes a photometric error directly on images without incorporating a geometric prior.Unlike indirect approaches, it does not first generate keypoint matches or other intermediate measurements.
- Motivation: Sampling across edges and weak intensity variations makes the approach applicable to sparsely textured environments.Direct modeling also benefits from a precise sensor model incorporating effects such as exposure, gamma correction, and lens attenuation.
- Motivation: Adding a geometry prior introduces correlations between geometry parameters, making statistically consistent joint optimization in real time infeasible.The resulting systems are larger and harder to solve, motivating the sparse formulation.
- Contribution: The proposed method jointly optimizes camera poses, camera intrinsics, and inverse-depth geometry within a fully direct probabilistic model.This contrasts with hybrid approaches that revert to an indirect formulation for joint model optimization.
- Evaluation: Evaluations on three datasets comprising several hours of video show higher robustness and accuracy than state-of-the-art direct and indirect approaches, with real-time laptop execution.Reduced settings reach 5× real-time speed while still outperforming state-of-the-art indirect methods.
2. Direct Sparse Model
The direct sparse model jointly optimizes photometric error over camera motion, inverse-depth geometry, intrinsics, and brightness parameters, using calibrated image formation and windowed optimization. It omits geometry smoothness priors while retaining tractable factor-graph structure through marginalization.
- Model formulation: The model continuously optimizes photometric error over camera poses, intrinsics, inverse-depth values, and affine brightness parameters.It represents 3D points by inverse depth in a reference frame and performs the photometric equivalent of windowed sparse bundle adjustment.
- Photometric calibration: Photometric calibration models the response function, lens attenuation, and exposure time before evaluating image residuals.Frames are photometrically corrected using the calibrated image-formation model; unknown exposure times can instead be handled by affine brightness parameters.
- Photometric error: Each point residual is a robust, gradient-weighted SSD over a small pixel neighborhood linking a host frame, target frame, inverse depth, and camera parameters.The selected neighborhood uses eight pixels, balancing computational cost, motion-blur robustness, and information content.
- Optimization structure: The factor graph adds dependence on the host-frame pose, but Schur-complement marginalization preserves the system’s sparsity pattern.This structure reduces the computational impact of the additional variable dependencies.
- Consistency: The formulation allows repeated use of pixel values when selected observations overlap, especially in scenes with little texture.The authors state that this has negligible practical effect and can be avoided by removing or down-weighting overlapping observations.
- Windowed optimization: Windowed optimization accumulates a quadratic marginalization prior in a fixed tangent space and performs subsequent updates across poses and Euclidean parameters.The optimized state includes camera poses, affine brightness parameters, inverse depths, and camera intrinsics.
3. Visual Odometry Front-End
The front end tracks frames, creates and marginalizes keyframes, and manages sparse points for real-time joint optimization. It uses fixed-budget, spatially distributed sampling while retaining coverage of weakly textured and edge regions.
- The front end selects points and frames, determines point visibility, removes outliers, detects occlusions, initializes parameters, and decides when to marginalize.
- Keyframe Marginalization: The system keeps up to 7 active keyframes, always retaining the latest two and marginalizing frames based on visibility and a 3D-distance score.The distance score is designed to maintain well-distributed active keyframes, with more frames near the newest one.
- Initial Frame Tracking: New frames are tracked against the newest keyframe using direct multi-scale image alignment initialized by a constant motion model.Active points are projected and dilated to create a semi-dense depth map for initialization.
- Initial Frame Tracking: Tracking recovery retries alignment from up to 27 small rotations when a frame’s final RMSE exceeds twice that of the preceding frame.The recovery procedure runs only at the coarsest pyramid level and is rarely invoked.
- Keyframe Creation: New keyframes are selected using field-of-view change, translation-induced occlusions, and exposure-time changes, combined through a weighted threshold.The system initially takes around 5–10 keyframes per second and later marginalizes redundant ones.
- Point Management: A fixed budget of 2,000 active points is distributed across space and frames, with candidate activation replacing marginalized points to preserve uniform image coverage.Candidate selection prioritizes gradient-rich regions, while later passes also cover weak intensity variations more sparsely; sampling more data quickly reaches diminishing returns.
4. Results
DSO is evaluated across three datasets and multiple design choices, showing strong accuracy, robustness, real-time performance, and useful reconstructions. Its advantages are largest with photometric noise and well-calibrated geometry, while geometric noise and limited keyframe settings expose important boundaries.
- Quantitative Comparison: DSO outperforms ORB-SLAM in accuracy and robustness on TUM-monoVO and ICL NUIM, but ORB-SLAM is more accurate on EuRoC MAV while less robust.The EuRoC result is attributed to unavailable photometric calibration and loop-like trajectories benefiting ORB-SLAM's local mapping.
- Quantitative Comparison: 5× real-time execution with reduced settings still achieves very good accuracy and robustness across all three datasets.The reduced configuration uses Np=800 points, Nf=6 active frames, 424×320 resolution, and at most four Gauss-Newton iterations after keyframe creation.
- Photometric Calibration: Removing vignette and response calibration slightly decreases accuracy and robustness, while naïve brightness constancy performs worst.Known exposure times have little effect on accuracy in this evaluation.
- Parameter Choices: DSO's benefit from additional points quickly flattens after Np=500, while increasing active frames beyond Nf=7 adds little benefit and increases runtime quadratically.The evaluation also limits each point's observations to active frames during marginalization.
- Data Selection: Using only FAST corners significantly decreases performance compared with selecting gradient-rich points more broadly.The method can use edge points that indirect approaches may reject as outliers.
- Noise Study: DSO is more robust to photometric noise but deteriorates quickly under geometric noise, especially from rolling shutter or inaccurate calibration.For δg > 1.5, optimization may fail entirely, whereas photometric-error optimization better overcomes introduced distortions.
- Noise Study: On well-calibrated data the direct approach outperforms the indirect approach, whereas strong geometric noise makes the indirect model superior.The authors associate direct performance with dedicated machine-vision cameras and indirect advantages with commodity cameras prioritizing human-viewing quality.
- Qualitative Results: DSO reconstructs gradient-rich areas, including edges, and tracks through very low-texture scenes where indirect approaches fail.The active point count directly controls reconstruction density, and the shown reconstructions accumulate odometry without loop closures.
5. Conclusion
The paper presents a real-time direct sparse formulation that combines efficient joint optimization with broad point usage and full photometric calibration. Experiments show gains in accuracy and robustness under suitable conditions, while revealing distinct sensitivities of direct and indirect approaches and increased non-convexity as a limitation.
- The proposed direct sparse formulation combines direct methods’ ability to use points beyond corners with sparse methods’ efficient joint optimization.It omits the geometric prior and evaluates photometric error over small pixel neighborhoods to constrain the problem.
- Full photometric calibration completes the camera model beyond its geometric component.The calibration is incorporated as part of the formulation rather than relying only on the traditional geometric camera model.
- The monocular visual odometry implementation maintains real-time performance by incrementally marginalizing old states and uses a front-end for data selection and initialization.The front-end initializes optimization of the highly non-convex energy function.
- The parameter study finds that more data alone does not improve tracking accuracy, whereas using all points and adding photometric calibration improve accuracy, robustness, or performance.More data makes the 3D models denser; calibration helps particularly relative to brightness constancy.
- Indirect methods are more robust to geometric noise, while direct methods are more robust to photometric noise and achieve superior accuracy on well-calibrated data.The comparison specifically discusses errors from poor intrinsic calibration and rolling shutter as geometric noise sources.
- The formulation’s increased non-convexity, caused by including image values in the error function, is likely to restrict its use to video processing.Its structure can nevertheless be integrated with bundle adjustment or incremental smoothing and mapping.