PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Odometry Revisited: What Should Be Learnt?

Huangying Zhan, Chamara Saroj Weerasekera, Jiawang Bian, Ian Reid

arXiv:1909.09803v4cs.CV

TL;DR

Monocular VO must combine reliable geometric estimation with learned predictions while addressing scale drift and scale ambiguity. The paper integrates CNN-based depth and optical flow with epipolar geometry and PnP in DF-VO, which outperforms pure deep and geometry-based methods and mitigates scale drift with scale-consistent depth.

  • Problem

    Monocular geometry-based VO suffers from scale drift and scale ambiguity, while end-to-end deep VO remains less reliable and accurate than geometry-based methods.

  • Method

    DF-VO uses CNN-predicted single-view depth and two-view optical flow as intermediate outputs for epipolar-geometry and PnP pose estimation.

  • Results

    DF-VO outperforms pure deep learning methods by a large margin in all reported metrics and is evaluated against geometry-based methods on KITTI.

  • Takeaways & Limitations

    Integrating learned depth and correspondence predictions with multi-view geometry provides a robust monocular VO design that mitigates scale drift.

Abstract

from arXiv · show

In this work we present a monocular visual odometry (VO) algorithm which leverages geometry-based methods and deep learning. Most existing VO/SLAM systems with superior performance are based on geometry and have to be carefully designed for different application scenarios. Moreover, most monocular systems suffer from scale-drift issue.Some recent deep learning works learn VO in an end-to-end manner but the performance of these deep systems is still not comparable to geometry-based methods. In this work, we revisit the basics of VO and explore the right way for integrating deep learning with epipolar geometry and Perspective-n-Point (PnP) method. Specifically, we train two convolutional neural networks (CNNs) for estimating single-view depths and two-view optical flows as intermediate outputs. With the deep predictions, we design a simple but robust frame-to-frame VO algorithm (DF-VO) which outperforms pure deep learning-based and geometry-based methods. More importantly, our system does not suffer from the scale-drift issue being aided by a scale consistent single-view depth CNN. Extensive experiments on KITTI dataset shows the robustness of our system and a detailed ablation study shows the effect of different factors in our system.

I. INTRODUCTION

Monocular VO combines geometry with learned depth and correspondence predictions to address scale drift while retaining robust frame-to-frame pose estimation. DF-VO integrates these components and evaluates them against both geometry- and learning-based alternatives.

  • Motivation: Monocular VO estimates robot motion between frames, while more complete SLAM additionally builds trajectories and maps, closes loops, and relocalizes after tracking loss.
  • Limitations of Existing Methods: Geometry-based VO is accurate under sufficient illumination, texture, frame overlap, and mostly static scenes, but monocular systems suffer scale drift and scale ambiguity.Scale drift typically requires scale-consistent maps, global bundle adjustment, or additional scene assumptions.
  • Limitations of Existing Methods: End-to-end deep VO can learn real-world scale and operate in challenging conditions, but remains less reliable and accurate than geometry-based methods in favorable conditions.
  • Proposed Approach: DF-VO combines single-view depth and optical-flow predictions with epipolar geometry and PnP to form a simple frame-to-frame monocular VO algorithm.The pipeline uses flow-derived 2D-2D matches, depth-derived 3D-2D correspondences, and flow consistency to support pose estimation.
  • Evaluation: The work compares DF-VO with deep and geometry-based methods and uses ablation studies to assess the effects of system components and training choices.

III. DF-VO: DEPTH AND FLOW FOR VISUAL ODOMETRY

The method revisits classical pose estimation and integrates epipolar geometry and PnP with learned CNN outputs. This section frames the learned predictions as inputs to those geometric estimators.

  • The method combines traditional epipolar-geometry and PnP-based pose estimation with learned CNN outputs.
  • Learned predictions are integrated after revisiting the corresponding classical pose-estimation methods.
  • Algorithm 1 specifies the integration of learned CNN outputs with the geometric pose-estimation pipeline.

A. Traditional Visual Odometry

Traditional visual odometry estimates relative camera pose from image correspondences using epipolar geometry or PnP, but each approach has distinct requirements and limitations.

  • Epipolar Geometry: Epipolar geometry recovers relative pose [R, t] from 2D-2D pixel correspondences by solving the essential matrix.Correspondences can be formed by feature matching or optical flow.
  • Epipolar Geometry: Essential-matrix translation is recovered up-to-scale, creating scale ambiguity in monocular visual odometry.
  • Epipolar Geometry: Pure rotation makes recovering R unsolvable, while small camera translation produces an unstable essential-matrix solution.
  • PnP: PnP estimates camera pose from 3D-2D correspondences by minimizing reprojection error.These correspondences require estimating 3D scene structure and matching 3D points to 2D pixels.

B. Deep Predictions for Correspondence

The method uses dense CNN optical flow to generate and filter frame correspondences, expanding tracking beyond sparse salient features.

  • Deep Correspondence Estimation: A two-view optical-flow CNN produces dense 2D-2D correspondences between consecutive images.Forward and backward flow estimates are used to assess correspondence consistency.
  • Deep Correspondence Estimation: Selecting the best-N matches with the least flow inconsistency filters optical-flow outliers for relative-pose estimation.
  • Deep Correspondence Estimation: Dense optical flow allows any pixel to serve as a tracking candidate, while CNN features gather visual information beyond traditional local regions.

C. Deep Predictions for Visual Odometry

Deep predictions are integrated with epipolar geometry and PnP to address scale ambiguity and unstable pose estimation. The pipeline falls back to PnP when essential-matrix conditions are not satisfied and uses RANSAC for robustness.

  • Deep Predictions for Visual Odometry: CNN depth and optical-flow predictions establish both 2D-2D and 3D-2D correspondences for essential-matrix or PnP pose estimation.Current single-view depth methods remain insufficiently accurate for recovering the 3D structure needed for accurate pose estimation.
  • Deep Predictions for Visual Odometry: Accurate, generalizable CNN optical flow supports essential-matrix motion recovery, while deep predictions help address its known issues.
  • Scale Recovery: CNN depths provide a scale reference by comparing predicted depths with triangulated up-to-scale depths after essential-matrix estimation.
  • Stability Checks: An average flow threshold δf = 5 avoids small-motion cases, and cheirality checks identify stable essential-matrix solutions.The cheirality test requires triangulated points to lie in front of both cameras and counts satisfying points to assess stability.
  • Fallback and Robustness: When these conditions fail, the pipeline estimates camera motion with PnP and wraps essential-matrix estimation, scale estimation, and PnP in RANSAC loops.

IV. CNN TRAINING FRAMEWORK

The framework trains CNNs to predict single-view depth and two-view optical flow for integration with geometry-based visual odometry. It addresses scale inconsistency in monocular training through stereo supervision or temporal geometry, while using photometric, smoothness, and consistency objectives.

  • The framework presents CNN training procedures for learning depths and optical flows as intermediate predictions.
  • Single-view depths are predicted with a fully convolutional encoder-decoder network using skip-connections.
  • Self-supervised depth training jointly optimizes depth and pose networks using per-pixel photometric, smoothness, and consistency objectives.Photometric reconstruction uses neighboring views and, when available, a stereo pair.
  • Monocular-video training can produce scale ambiguity and scale inconsistency because image snippets do not guarantee a consistent scale across snippets.
  • Stereo training aligns deep predictions with real-world scale through the known stereo baseline, while temporal geometry provides another route to address scale inconsistency.Only monocular images are required during inference even when stereo sequences are used for training.
  • The depth-consistency error uses a minimum over source views to reduce the effects of occlusions and out-of-view scenes.

B. Optical Flow Network

The optical flow network uses LiteFlowNet as a fast, lightweight, accurate, and generalizable backbone. Flow predictions are regularized with edge-aware smoothness and forward-backward consistency constraints to establish image correspondences.

  • LiteFlowNet is selected as the optical-flow backbone because it is fast, lightweight, accurate, and generalizes well.Its architecture uses two-stream feature extraction followed by cascaded flow inference and regularization.
  • The flow transformation h2(.) establishes correspondences between views from optical flow rather than the depth-based reprojection used in Eqn. 5.
  • The network regularizes optical flow with edge-aware smoothness and forward-backward consistency losses.

V. EXPERIMENTS AND EVALUATIONS

The evaluation analyzes the visual odometry system primarily on the KITTI dataset, compares it with prior VO methods, and examines factors affecting performance. Indoor results are reported in supplementary video because of space limitations.

  • The experiments mainly evaluate the VO system on KITTI and provide indoor results in supplementary video because of space limitations.
  • The evaluation compares the proposed system with prior visual-odometry methods and includes a detailed analysis of factors affecting VO performance.
  • KITTI training uses sequences 00–08, comprising 36,671 training pairs, to train the depth network and finetune the flow network.

A. Implementation Details

The networks are trained in PyTorch with Adam for 20 epochs on 640x192 KITTI images. Training uses a learning-rate schedule and separate loss weightings for depth and optical-flow experiments.

  • All self-supervised experiments use Adam optimization for 20 epochs with KITTI images resized to 640x192.
  • The learning rate is 10^-4 for the first 15 epochs and 10^-5 for the remaining epochs.
  • Depth and optical-flow experiments use separate loss-weighting settings for their respective objective terms.The depth weights are [λds, λdc] = [10^-3, 5], while flow weights are [λfs, λfc] = [10^-1, 5 × 10^-3].

B. Visual Odometry Evaluation

The evaluation compares DF-VO with deep-learning and geometry-based methods on KITTI, then examines how algorithmic components, depth training, flow estimation, matches, and resolution affect performance.

  • Visual Odometry Evaluation: DF-VO is compared against pure deep-learning methods and geometry-based systems using quantitative and qualitative KITTI evaluation.The comparison includes DSO, VISO2, ORB-SLAM2, and CNN-SVO.
  • Visual Odometry Evaluation: Deep-learning methods outperform pure geometry-based methods on Seq.01, where most geometry-based systems fail because close features are not trackable.This suggests deep models may complement geometry methods in such failure conditions.
  • Visual Odometry Evaluation: ORB-SLAM2 has lower rotation drift but higher translation drift on long sequences because of scale drift, which loop closing can sometimes resolve.The evaluation uses 6DoF optimization with respect to ATE for fair comparison.
  • Ablation Study: Using only PnP performs worse than the full algorithm because single-view depths are insufficiently accurate for reliable 3D-2D matches.The full method therefore relies more on 2D-2D matches.
  • Ablation Study: Scale-consistent monocular depth performs on par with stereo-trained depth, although monocular models without scale consistency retain scale ambiguity.The stereo-trained model does not exhibit this ambiguity.
  • Ablation Study: Self-supervised finetuning improves LiteFlowNet’s adaptation to driving sequences, while forward-backward flow consistency selects more reliable 2D-2D matches.Increasing inference resolution also produces more accurate correspondences that improve relative pose estimation.

VI. CONCLUSION

The paper presents a robust monocular VO system that combines deep predictions with classic geometry. Scale-consistent depth supports scale recovery, while deep models can complement standard VO/SLAM when geometry-based tracking fails.

  • VI. CONCLUSION: DF-VO uses deep optical flow and single-view depth predictions to establish 2D-2D and 3D-2D correspondences for camera pose estimation.The system combines these predictions with classic geometry methods rather than learning complete VO end to end.
  • VI. CONCLUSION: Scale-consistent depth models can recover scale and mitigate scale drift in monocular VO/SLAM systems.The paper reports self-supervised training with monocular and stereo sequences as alternative training schemes.
  • VI. CONCLUSION: Deep VO models can complement standard VO/SLAM systems when geometry-based tracking fails.The proposed map-to-frame tracking system is identified as future work for further improving tracking performance.
Loading 1909.09803v4…