Source-linked AI summary
Stereo DSO: Large-Scale Direct Sparse Visual Odometry with Stereo Cameras
Rui Wang, Martin Schwörer, Daniel Cremers
TL;DR
Stereo DSO addresses the need for accurate, robust, real-time visual odometry across challenging real-world settings. It integrates temporal multi-view stereo and static stereo in a marginalization framework, and evaluations report strong tracking performance, including more accurate results than Stereo ORB-SLAM2 with loop closure and global bundle adjustment. Its current scope excludes loop closing and remains limited by extreme brightness changes, moving objects, and strong rotational motion.
Problem
Real-time visual odometry seeks high accuracy, robustness to lighting and motion changes, and dense environmental reconstruction, while monocular direct methods suffer from photometric, scale, and motion-related shortcomings.
Method
Stereo DSO integrates temporal multi-view stereo and static stereo within a marginalization framework using the Schur complement for real-time stereo-camera tracking and mapping.
Results
Stereo DSO is reported as the most accurate and robust method evaluated, outperforming Stereo ORB-SLAM2 with loop closing and global bundle adjustment on the KITTI testing set.
Takeaways & Limitations
Stereo cameras resolve the scale drift affecting monocular methods, while Stereo DSO provides accurate pure visual odometry in challenging real-world scenarios.
Takeaways & Limitations
The current method lacks loop closing and database-based map maintenance, while extreme brightness changes combined with strong rotational motion can cause failure.
Abstract
from arXiv · showhide
We propose Stereo Direct Sparse Odometry (Stereo DSO) as a novel method for highly accurate real-time visual odometry estimation of large-scale environments from stereo cameras. It jointly optimizes for all the model parameters within the active window, including the intrinsic/extrinsic camera parameters of all keyframes and the depth values of all selected pixels. In particular, we propose a novel approach to integrate constraints from static stereo into the bundle adjustment pipeline of temporal multi-view stereo. Real-time optimization is realized by sampling pixels uniformly from image regions with sufficient intensity gradient. Fixed-baseline stereo resolves scale drift. It also reduces the sensitivities to large optical flow and to rolling shutter effect which are known shortcomings of direct image alignment methods. Quantitative evaluation demonstrates that the proposed Stereo DSO outperforms existing state-of-the-art visual odometry methods both in terms of tracking accuracy and robustness. Moreover, our method delivers a more precise metric 3D reconstruction than previous dense/semi-dense direct approaches while providing a higher reconstruction density than feature-based methods.
1. Introduction
Stereo DSO targets accurate, robust, real-time visual odometry and mapping from stereo cameras, addressing scale drift and sensitivities of prior direct methods. It combines static and temporal stereo constraints within real-time bundle adjustment and is evaluated against established stereo methods on KITTI and Cityscapes.
- Motivation: Camera-based odometry offers high resolution, sensing range, and sensing rate with lower weight, size, and hardware cost than active laser or lidar systems.
- Motivation: Real-time visual odometry seeks accuracy, robustness to lighting, scene changes, and fast motion, plus sufficient reconstruction density.
- Related work: Monocular DSO suffers without photometric calibration, cannot recover metric scale, exhibits scale drift, and is sensitive to fast motion and rolling shutter distortions.
- Contribution: Stereo DSO uses additional stereo information to provide faster-converging scale estimation and reduce sensitivity to missing photometric calibration and rolling shutter effects.
- Contribution: The method integrates temporal multi-view stereo and static stereo into a marginalized bundle-adjustment procedure using the Schur complement while retaining real-time capability.
- Evaluation: Systematic evaluations on KITTI and Cityscapes report superiority over Stereo ORB-SLAM and Stereo LSD-SLAM, particularly on the KITTI testing set and over longer ranges.
2. Direct Sparse VO with Stereo Cameras
Stereo DSO combines static stereo and temporal multi-view stereo in a direct sparse VO pipeline, using stereo depth, photometric alignment, sparse gradient-based sampling, and joint windowed optimization.
- Stereo and multi-view integration: Stereo DSO combines static stereo with temporal multi-view stereo to exploit their complementary scale, initialization, depth-range, and degeneracy properties.The known stereo baseline provides absolute scale and initial depth, while temporal multi-view stereo resolves static stereo’s limited accurate depth range.
- Frame management: New stereo frames are tracked against the newest keyframe, depths are refined, and keyframes are created when scene or illumination changes sufficiently.Tracking uses coarse-to-fine direct alignment and Gauss-Newton optimization; keyframe decisions use optical flow and relative brightness change.
- Direct image alignment: Direct alignment projects points between frames and minimizes robust photometric intensity residuals, with gradient-based weighting and affine brightness correction factors.The projection uses inverse depth and the inter-frame transformation; the Huber norm and pixel weights make the residual robust and down-weight high gradients.
- Direct image alignment: Real-time processing samples a fixed number of points uniformly across image regions with sufficient gradients, using local neighborhoods to compute photometric errors.Stereo images also verify selected points and assist depth initialization.
- Stereo and multi-view integration: The system initializes candidate depths from static stereo matching rather than random monocular depths, improving tracking accuracy.Static stereo supplies a semi-dense initial depth map and better candidate-point depth estimates.
- Windowed optimization: The windowed energy jointly represents point inverse depths, keyframe poses, affine brightness factors, camera intrinsics, and temporal and static-stereo constraints.Each factor links a point with the relevant keyframes, while static-stereo constraints introduce scale information and geometric priors.
3. Evaluation
Stereo DSO is evaluated on KITTI and Cityscapes for accuracy, robustness, tracking, and reconstruction. It outperforms compared methods across KITTI settings, while Cityscapes exposes evaluation and extreme-illumination limitations.
- KITTI Visual Odometry Benchmark: Stereo coupling factors λ = 1 and 2 significantly reduce translational and rotational errors, whereas λ > 3 makes performance worse through sensitivity to incorrect static-stereo matches.The test uses KITTI sequence 06.
- KITTI Visual Odometry Benchmark: On KITTI training data, Stereo DSO is almost always more accurate than Stereo LSD-SLAM, while its rotational errors consistently beat ORB-SLAM2.Its translational errors versus ORB-SLAM2 are mixed, which the authors suggest may reflect the dataset’s relatively low frame rate.
- KITTI Visual Odometry Benchmark: On the KITTI testing set, Stereo DSO achieves the best translational and rotational results across distance intervals and driving speeds, despite using pure VO without loop closure.The competing LSD-SLAM and ORB-SLAM2 results use SLAM with loop closure; ORB-SLAM2 also uses global bundle adjustment.
- KITTI Visual Odometry Benchmark: Compared with monocular VO methods on KITTI sequences 00 and 06, Stereo DSO avoids the scale drift that those methods cannot properly handle.The monocular trajectories are qualitatively compared after similarity alignment to ground truth.
- Cityscapes Dataset: Cityscapes provides long, high-frame-rate stereo sequences with HDR imagery, but inaccurate GPS and vehicle-odometry information makes precise VO evaluation difficult.The dataset also contains moving objects and severe uncalibrated brightness changes; the method fails under an extreme brightness change combined with strong car rotation.
- Cityscapes Dataset: On Cityscapes, Stereo DSO produces precise and much denser 3D reconstructions than feature-based methods, as illustrated on the Frankfurt sequence.The reconstructions are sparser than those from previous dense approaches.
4. Conclusion
Stereo DSO is presented as an accurate, robust real-time method for stereo-camera tracking and mapping, with evaluations on KITTI and Cityscapes. The authors identify full SLAM extension and explicit dynamic-object handling as future work.
- KITTI and Cityscapes evaluations demonstrate accurate and robust tracking in challenging real-world scenarios.
- On the KITTI testing set, Stereo DSO outperformed Stereo ORB-SLAM2 despite the latter using loop closing and global bundle adjustment.
- The planned extension to full SLAM adds loop closing and a database for map maintenance, while explicit dynamic-object handling is also proposed.
Supplementary Material
The supplementary document is identified by the paper title and lists its authors and institutional affiliation.
- The supplementary document concerns Stereo DSO: Large-Scale Direct Sparse Visual Odometry with Stereo Cameras.
- The listed authors are Rui Wang, Martin Schwörer, and Daniel Cremers.
- The authors are affiliated with the Technical University of Munich.
Abstract
The supplementary material examines static-stereo constraint weighting and provides additional trajectory comparisons and qualitative results for KITTI and Cityscapes.
- The document first studies how weighting static-stereo constraints affects tracking accuracy.
- It provides full KITTI training-set trajectory estimates compared with state-of-the-art monocular visual odometry methods.
- Additional Cityscapes Frankfurt results qualitatively demonstrate the method’s tracking accuracy.
- A supplementary video is provided to show the method’s performance.
1. Effect of Stereo Coupling Factor
The supplementary analysis reports estimated KITTI Sequence 06 trajectories for stereo coupling factors from 0 through 3.
- Estimated trajectories on KITTI Sequence 06 are shown for coupling factors λ ranging from 0 to 3.
2. Full Results on KITTI
On KITTI, Stereo DSO’s stereo trajectory estimates avoid the scale drift seen in monocular methods, while stronger static-stereo weighting further reduces translational drift. The method remains affected by unmodeled image-domain effects such as non-Lambertian reflectance and illumination changes.
- Trajectory comparisons: Stereo DSO resolves the main scale-drift problem observed in monocular ORB-SLAM and DSO by using stereo cameras.Monocular trajectories are aligned with 7DoF similarity transformations, whereas Stereo DSO uses a 6DoF rigid-body transformation.
- Trajectory comparisons: Monocular DSO appears to exhibit larger scale drift than monocular ORB-SLAM.The passage attributes this belief to direct methods’ sensitivity to low frame rate, large optical flow, and other image-domain effects.
- Static stereo constraints: Increasing the static-stereo coupling factor λ significantly reduces translational drift on KITTI Sequence 06.The figure varies λ from 0 to 3.
- Remaining effects: Unmodeled non-Lambertian reflectance and insufficiently corrected illumination changes remain sources of error in the image domain.These effects are identified as contributing to direct-method sensitivity.
3. More Results on Cityscapes
Cityscapes Frankfurt results are evaluated on shorter stereo subsections because the method cannot currently process the full sequence under moving objects and sudden strong brightness changes. The figures compare estimated trajectories with GPS-derived ground truth, whose inaccuracies are visible in some sections.
- Sequence setup: The Frankfurt sequence is divided into sections of 5000–6000 frames with coverage comparable to KITTI sequences.The full sequence contains around 107,000 frames.
- Sequence setup: Without specific handling of moving objects and sudden strong brightness changes, Stereo DSO cannot run on the entire Frankfurt sequence.This limitation motivates evaluating several shorter sections instead.
- Ground truth: Ground-truth poses are calculated from provided GPS coordinates using the Mercator projection, but some GPS-derived poses are inaccurate.The passage notes such inaccuracies in examples including Figure 4g.
- Trajectory results: Figure 4 shows estimated camera trajectories for the Cityscapes Frankfurt stereo subsequences.The sub-captions identify the corresponding frame indices in the full sequence.