Source-linked AI summary
LIMO: Lidar-Monocular Visual Odometry
Johannes Graeter, Alexander Wilczynski, Martin Lauer
TL;DR
Precise vehicle motion estimation needs methods that combine camera tracking with reliable scale, but camera–LIDAR visual odometry has received limited attention. LIMO extracts LIDAR depth for monocular feature tracks and integrates it with robust keyframe Bundle Adjustment, achieving strong KITTI benchmark performance.
Problem
Visual odometry methods largely focus on binocular imagery or pure LIDAR, leaving camera–LIDAR visual localization comparatively unattended despite the need for precise vehicle motion estimates.
Method
LIMO extracts feature depths from LIDAR using local-plane fitting and special ground-plane handling, then fuses them with monocular tracking through keyframe and landmark selection in Bundle Adjustment.
Results
LIMO achieves a mean translation error of 0.93% and rotation error of 0.0026 deg/m on KITTI, ranking 132nd overall and second among published LIDAR-camera methods.
Takeaways & Limitations
Combining LIDAR depth with camera feature tracking provides a visual odometry system that operates online without loop closure and lowers trajectory drift.
Abstract
from arXiv · showhide
Higher level functionality in autonomous driving depends strongly on a precise motion estimate of the vehicle. Powerful algorithms have been developed. However, their great majority focuses on either binocular imagery or pure LIDAR measurements. The promising combination of camera and LIDAR for visual localization has mostly been unattended. In this work we fill this gap, by proposing a depth extraction algorithm from LIDAR measurements for camera feature tracks and estimating motion by robustified keyframe based Bundle Adjustment. Semantic labeling is used for outlier rejection and weighting of vegetation landmarks. The capability of this sensor combination is demonstrated on the competitive KITTI dataset, achieving a placement among the top 15. The code is released to the community.
I. INTRODUCTION AND RELATED WORK
Visual odometry commonly relies on stereo cameras or LIDAR alone, while combining monocular cameras with LIDAR remains relatively unexplored. LIMO combines camera feature tracking with LIDAR depth and keyframe Bundle Adjustment to reduce drift without loop closure.
- Bundle Adjustment jointly estimates camera poses and landmarks but requires complexity reduction for online visual odometry.Windowed optimization reduces cost but increases drift, motivating selective keyframe and landmark use.
- Stereo methods provide scale but depend strongly on precise extrinsic calibration, where small errors can produce large depth errors.LIDAR depth measurements have small errors independent of extrinsic calibration error, although LIDAR-camera calibration remains limited to a few pixels.
- LIMO traces KITTI sequence 02 ground truth with very high accuracy and low drift without loop closing.The figure contrasts LIMO's keyframe Bundle Adjustment trajectory with frame-to-frame motion estimation and shows rejected landmarks.
- LIMO extracts LIDAR depth for image features, rejects outliers using local-plane assumptions, treats ground-plane points specially, and fuses depth with monocular VSLAM.The framework uses keyframe and landmark selection for online operation and does not employ loop closure.
- LIMO ranks 11th in rotation error and among the top methods reported for translation error on the KITTI benchmark.The cited result passage states that LIMO outperforms ORB-SLAM2 and Stereo LSD-SLAM.
II. BLOCK A AND B: FEATURE EXTRACTION AND PREPROCESSING
The pipeline tracks image features, associates them with LIDAR measurements, and obtains one-shot depth estimates without depending on a prior motion estimate. These measurements are inserted into the visual odometry pipeline through a sequence of neighborhood selection, segmentation, plane fitting, intersection, and validation steps.
- Feature extraction uses viso2 tracking with non-maximum suppression, flow-based outlier rejection, and subpixel refinement.The implementation computes 2000 feature correspondences in 30−40ms.
- LIDAR depth is extracted for detected image features using a one-shot approach that avoids dependence on a motion estimate.Compared with accumulated point clouds, this provides less data for feature depth estimation.
- For each image feature, projected LIDAR points are selected, foreground points are segmented, and a local plane is fitted.Ground-plane features use a special fitting algorithm.
- The fitted plane is intersected with the feature's line of sight to obtain depth, followed by a test that determines whether the estimate is accepted.
B. Selecting the Neighborhood
Neighborhood selection and depth fitting must account for local geometry and feature placement. The method uses planar neighborhoods and foreground segmentation for ordinary surfaces, while handling ground-plane features separately because histogram segmentation is unsuitable there.
- B. Selecting the Neighborhood: The image-plane region of interest must contain LIDAR points spanning a plane rather than points lying only on a line.This choice avoids singularities in plane estimation.
- C. Foreground Segmentation: Feature points often lie on edges or corners, so fitting a plane to an unsegmented neighborhood can produce incorrect depth.
- C. Foreground Segmentation: A depth histogram separates foreground points before plane fitting, using the nearest significant depth bin with bin width h = 0.3m.This improves local-surface estimation when the feature lies on a foreground edge.
- D. Plane Fit: Three points spanning the maximum-area triangle are selected from the segmented neighborhood to stabilize plane estimation, and small triangles are rejected.
- D. Plane Fit: Ground-plane feature depth cannot use the histogram method because LIDAR has lower vertical than horizontal resolution, so a separate local-plane approach is used.Ground-plane features are especially valuable in highway scenarios.
E. Special Case: Points on Ground Plane
Ground-plane feature points require a depth-estimation approach different from local plane fitting because LIDAR has lower vertical than horizontal resolution. The resulting frame-to-frame estimate uses depth-based reprojection, epipolar constraints, and robust losses to initialize Bundle Adjustment.
- Ground-plane depth cannot be estimated reliably by the local-plane method because LIDAR has lower vertical than horizontal resolution.The paper treats ground-plane features as valuable on rural roads and highways, motivating a separate approach.
- Frame-to-frame motion initializes Bundle Adjustment using integrated motion estimates and reconstructed landmarks.The prior is typically smaller to solve, while accurate motion improves landmark selection from reconstructed points.
- Epipolar error stabilizes estimation when valid-depth features are too sparse, especially in low-structure scenes with large optical flow.The additional constraints also improve rotation accuracy.
- Cauchy loss functions reduce outlier influence in the frame-to-frame optimization.The authors report the Cauchy function as the best loss choice, using a fixed outlier threshold.
V. BLOCK D: BACKEND
The backend improves accuracy and robustness by jointly optimizing data accumulated across frames in a keyframe-based Bundle Adjustment framework. Its main components are keyframe selection, landmark selection, and robustification.
- A keyframe-based Bundle Adjustment framework jointly optimizes data collected through many frames to reduce errors from sparse depth measurements.The backend is structurally and software-wise separate from prior estimation.
- The backend comprises keyframe selection, landmark selection, and robustification measures.
A. Why is Selection Important?
Windowed Bundle Adjustment limits computational cost but increases drift because it minimizes error locally. Keyframe selection balances stability, information retention, and online efficiency.
- Full Bundle Adjustment jointly solves for all landmarks and poses but is too complex for online Visual Odometry.Windowed optimization reduces cost by removing previous poses and landmarks, at the expense of larger drift.
- Keyframe selection is the first major step for reducing backend complexity.
- Rejected frames are frames that contain valuable information but are excluded from keyframe optimization.
- Required frames preserve stable optimization by retaining crucial measurements, particularly when landmark viewing angles change strongly during turns.Dense environments and turns shorten feature tracks, requiring a high keyframe density.
- Frames are omitted as keyframes when mean optical flow is below a fixed threshold, avoiding unstable scale estimation when the vehicle stands still.Such cases can occur at intersections with moving-vehicle occlusions and little usable depth or optical flow.
C. Landmark Selection
Landmark selection seeks precise, compact, outlier-free, and spatially distributed landmarks. The method triangulates points after motion estimation, bins them by metric depth, sparsifies local clusters, and applies bin-specific selection plus semantic weighting.
- The desired landmark set is well observable, small, free of outliers, and evenly distributed in 3D and image space.
- Late landmark selection matches many feature points first, then triangulates them and filters the resulting sparse point cloud before Bundle Adjustment.
- Near points support translation estimation but are difficult to measure because their optical flow is large.
- Middle points support both rotation and translation estimation and help recover from local minima.
- Far points support rotation estimation and are easier to track, yielding many measurements for each landmark.Near, middle, and far categories are assigned in metric space.
- Voxel filtering with median filtering sparsifies local clusters such as trees and bushes so they do not dominate Bundle Adjustment.A fixed number of landmarks is selected from each depth bin using strategies matched to feature proficiency.
- Semantic information weights vegetation landmarks because vegetation provides trackable structure but can move and violate static-scene assumptions.
D. Landmark Depth Insertion
LIMO inserts LIDAR-derived landmark depths into Bundle Adjustment to make scale observable and constrain optimization. In sparse-depth highway scenarios, it also regularizes scale using the oldest motion in the optimization window.
- Without landmark depth estimates, Bundle Adjustment cannot optimize scale because landmark depth and vehicle translation can change together without changing cost.
- An additional cost functor ξi,j penalizes deviation between estimated landmark depth and measured LIDAR depth.
- Valid depth constraints apply only to landmark-pose combinations for which the method extracts a depth estimate.
- In highway scenarios, where only a dozen potentially error-prone depth estimates may be available, additional stabilization is required.
- The method adds a cost functor ν that penalizes deviations from the oldest motion's translation length, regularizing scale and smoothing the estimate.
E. Robustification and Problem Formulation
The optimization is robustified through semantic and cheirality-based rejection, robust loss functions, and iterative removal of large residuals. These measures reduce outlier influence and improve convergence efficiency.
- Semantic labels and cheirality provide preliminary outlier rejection, but moving shadows and unclassified moving objects can remain.
- Cauchy loss functions reduce the influence of large residuals in both depth and reprojection-error cost functions.
- The optimization combines reprojection error, depth-related costs, and weights w0,w1,w2 that scale the cost functions to the same order of magnitude.
- Iterative residual rejection improves convergence speed, while robust losses alone would still repeatedly evaluate outlier parameters.
- At predefined optimization steps, the algorithm removes the rl% highest-residual depth and reprojection terms and parameters without residuals.
VI. RESULTS AND EVALUATION
LIMO is evaluated on KITTI using separate frame-to-frame and complete-pipeline assessments across urban, rural, and highway conditions. The backend is especially valuable for long tracks, sparse-depth scenes, and highway sequences.
- KITTI provides rural and highway sequences with grayscale and color images, LIDAR point clouds, and calibration for evaluating both motion-estimation stages.
- High-speed errors arise from difficult feature extraction, while low-speed errors arise from conservative standstill-detection tuning.
- Semantic images are binarized and eroded before majority voting assigns each tracked feature a label from its local 3 × 3 neighborhood.
- Highway sequences are challenging because open space limits valid depth estimates, while high speed increases optical flow and motion blur.
- The backend reduces drift on longer tracks, compensates depth errors from violated local-plane assumptions, and is particularly important on highways with few valid depth estimates.
- Liviodo ranks 302 with 1.22% mean translation error and 0.0042 deg m rotation error, whereas LIMO ranks 132 with 0.93% and 0.0026 deg m.
- LIMO's rotation-error benefit from Bundle Adjustment is nearly 40%, while its translation-error benefit is modest.
- LIMO runs at 5Hz on 4 CPU cores, compared with Liviodo at 10Hz on 2 cores.
VII. CONCLUSIONS
The paper combines LIDAR depth extraction for camera features with keyframe-based Bundle Adjustment to address camera–LIDAR measurement association. On KITTI, LIMO ranks 13th and benefits from reduced vegetation-landmark weighting.
- LIMO estimates camera-feature depth from LIDAR by fitting local planes and treats ground-plane measurements separately for highway scenarios.
- The method embeds depth extraction in an online Bundle Adjustment visual-odometry framework with keyframe and landmark selection.
- LIMO ranks 13th on KITTI and outperforms ORB-SLAM2 and Stereo LSD-SLAM.
- Reducing the weight of vegetation landmarks relative to infrastructure points improves accuracy, motivating future dynamic semantic weighting.