PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepVIO: Self-supervised Deep Learning of Monocular Visual Inertial Odometry using 3D Geometric Constraints

Liming Han, Yimin Lin, Guoguang Du, Shiguo Lian

arXiv:1906.11435v2cs.ROcs.CV

TL;DR

DeepVIO addresses limitations of monocular self-supervision, including scale ambiguity and inaccurate optical flow in dynamic or textureless scenes. It trains a monocular visual-inertial network with stereo-derived 3D geometric constraints and reports better accuracy and data adaptability than state-of-the-art learning-based methods on KITTI and EuRoC.

  • Problem

    Monocular self-supervision can suffer from scale ambiguity and inaccurate flow in dynamic or textureless scenes, while supervised methods need substantial ground-truth data.

    Source: §I · p. 1

  • Method

    DeepVIO trains a monocular VIO network using stereo-derived 3D optical flow and 6-DoF pose as geometric supervisory constraints.

    Source: §I · p. 2

  • Results

    DeepVIO reports better accuracy and data adaptability than state-of-the-art learning-based VO and VIO systems on KITTI and EuRoC.

    Source: §I · p. 2

  • Takeaways & Limitations

    The system estimates VIO at real-world scale from monocular camera and IMU sequences using stereo-based supervision.

    Source: §I · p. 2

    Drift remains because the system lacks place recognition and relocalization pipelines.

    Source: §V · p. 8

Abstract

from arXiv · show

This paper presents an self-supervised deep learning network for monocular visual inertial odometry (named DeepVIO). DeepVIO provides absolute trajectory estimation by directly merging 2D optical flow feature (OFF) and Inertial Measurement Unit (IMU) data. Specifically, it firstly estimates the depth and dense 3D point cloud of each scene by using stereo sequences, and then obtains 3D geometric constraints including 3D optical flow and 6-DoF pose as supervisory signals. Note that such 3D optical flow shows robustness and accuracy to dynamic objects and textureless environments. In DeepVIO training, 2D optical flow network is constrained by the projection of its corresponding 3D optical flow, and LSTM-style IMU preintegration network and the fusion network are learned by minimizing the loss functions from ego-motion constraints. Furthermore, we employ an IMU status update scheme to improve IMU pose estimation through updating the additional gyroscope and accelerometer bias. The experimental results on KITTI and EuRoC datasets show that DeepVIO outperforms state-of-the-art learning based methods in terms of accuracy and data adaptability. Compared to the traditional methods, DeepVIO reduces the impacts of inaccurate Camera-IMU calibrations, unsynchronized and missing data.

I. INTRODUCTION

Existing visual-inertial systems can require manual failure analysis and environment-specific tuning, while monocular self-supervision faces scale ambiguity and unreliable flow in difficult scenes. DeepVIO uses stereo-derived geometric constraints to train a monocular VIO system and updates IMU bias from fusion feedback.

A. Traditional Methods

  • Traditional VIO systems are described as requiring manual failure analysis and parameter tuning for specific environments.The introduction also notes challenges for camera-based methods in non-textured scenes, image blur, and extreme lighting.Source: §I · p. 1

I. INTRODUCTION

  • Monocular photometric supervision can have scale ambiguity, and 2D optical flow can be inaccurate in dynamic or textureless scenes.The introduction also notes that supervised methods require substantial ground-truth data and may generalize poorly to unseen scenes without fine-tuning.Source: §I · pp. 1–2
  • Stereo-derived 3D optical flow and 6-DoF pose provide geometric constraints for self-supervised monocular VIO, while fusion feedback updates additional IMU bias.The paper says these constraints penalize inconsistencies in estimated 2D optical flow, IMU pose, and VIO trajectory.Source: §I · p. 2
  • DeepVIO reports better accuracy and data adaptability than state-of-the-art learning-based VO and VIO systems on KITTI and EuRoC.The paper also describes its system as estimating VIO at real-world scale from monocular camera and IMU sequences.Source: §I · p. 2

B. Supervised or Semi-supervised Learning Methods

Supervised learning-based VIO can perform well but depends on ground-truth data, while unsupervised monocular approaches face scale ambiguity. DeepVIO uses stereo sequences to supervise a monocular image-and-IMU network through geometric constraints.

B. Supervised or Semi-supervised Learning Methods

  • Supervised learning-based motion estimation can achieve state-of-the-art results, but training requires substantial ground-truth data that is expensive to gather.The supplied discussion also says such models may generalize poorly to unseen scenes without sufficient ground-truth fine-tuning.Source: B. Supervised or Semi-supervised… · p. 2

C. Unsupervised Learning Methods

  • Earlier unsupervised VIO [12] requires an RGB-D or Lidar depth sensor for absolute scale recovery, whereas DeepVIO uses a stereo camera.Stereo training is also presented as a way to avoid depth-speed ambiguity in monocular 3D reconstruction.Source: C. Unsupervised Learning Methods · p. 2

III. METHOD

  • CNN-Flow extracts optical flow features from monocular frames, LSTM-IMU estimates relative pose from IMU data and bias status, and FC-Fusion combines these features to estimate camera trajectory.The fusion and IMU pose outputs provide status corrections as feedback.Source: §III · pp. 2–3
  • Stereo-derived dense depth and point clouds yield 3D optical flow and relative pose, which supervise flow, IMU-pose, and fusion-pose losses.The relative pose and 3D flow are calculated from successive point clouds using Iterative Closest Point; the synthetic 2D flow is made by projecting the 3D flow into its view.Source: §III · p. 3

A. Stereo Network as Supervision

Stereo sequences provide geometric supervision by yielding depth-based point clouds, 3D optical flow, and relative 6-DoF pose. The point-cloud pipeline uses stereo depth and ICP to produce the pose constraint.

A. Stereo Network as Supervision

  • Stereo sequences generate the 3D optical-flow and Stereo-se3 constraints used to supervise DeepVIO.The paper uses stereo-derived geometry to provide supervision.Source: A. Stereo Network as Supervision · p. 3

1) Depth and Point Cloud:

  • PSMNet supplies dense disparity, which is converted into depth and then a point cloud with absolute scales.The stereo disparity map uses matched image coordinates; the depth calculation uses baseline and focal length.Source: 1) Depth and Point Cloud · p. 3

2) Stereo-se3 with ICP:

  • Points outside a selected near-to-far depth range are excluded before adjacent-frame point clouds are matched with ICP.The paper notes that sky-region depth estimates can have large errors and are eliminated in this case.Source: 2) Stereo-se3 with ICP · p. 3
  • ICP estimates rotation and translation from matched 3D points, producing Stereo-se3 as a six-dimensional pose vector.The six-dimensional representation is denoted [ω, υ].Source: 2) Stereo-se3 with ICP · p. 4

3) 3D Optical Flow:

The method derives real-world 3D optical flow from corresponding points in adjacent point clouds and projects it into image views. Its projection supplies dense synthetic 2D flow, while the paper contrasts its geometric representation with limitations of image-based flow.

3) 3D Optical Flow:

  • Corresponding points in adjacent point clouds yield 3D optical flow, which is reprojected into the left and right image views.The flow vector is defined in real-world XYZ coordinates.Source: 3) 3D Optical Flow · p. 4
  • The authors argue that stereo-derived 3D flow represents real-world motion more accurately than image-based flow, which can struggle in textureless regions.They attribute image-based flow's limitation to lost depth information and resulting camera-object motion ambiguity.Source: Figure 4 · p. 4, 3) 3D Optical Flow · p. 4
  • Projecting each pixel's 3D flow into the left image produces dense synthetic 2D optical flow with a corresponding value at each pixel.The paper contrasts this dense synthetic flow with sparse KITTI optical flow derived from Velodyne.Source: 3) 3D Optical Flow · p. 4

B. CNN-Flow Network

CNN-Flow processes consecutive monocular images to produce an optical-flow feature for visual input to FC-Fusion. During training, its flow output is supervised by synthetic 2D flow projected from 3D flow.

B. CNN-Flow Network

  • CNN-Flow takes two consecutive images and outputs an intermediate optical-flow feature with dimension N×6×20.N depends on image size; the input images are merged stereo images in the described network setup.Source: B. CNN-Flow Network · p. 4

4) Synthetic 2D Optical Flow:

  • Synthetic 2D optical flow supervises CNN-Flow during training, and the resulting OFF is passed to FC-Fusion as visual input.The flow map is refined by upconvolutional layers consisting of unpooling and convolution.Source: 4) Synthetic 2D Optical Flow · p. 4

C. LSTM-IMU Network

The LSTM-IMU network estimates relative pose from inertial measurements while accounting for IMU state. A feedback-based status update uses the difference between its motion and FC-Fusion's motion to update that state.

C. LSTM-IMU Network

  • The paper introduces IMU-state-aware learning because noise and changing IMU data can make LSTM training difficult to converge.The proposed approach draws on the continuous state updates used in traditional VIO systems.Source: C. LSTM-IMU Network · p. 4

1) Preintegrated Network:

  • A two-layer LSTM with six hidden states per layer takes IMU data and state and outputs a six-dimensional IMU-se3 pose.Its input dimension is N×12, where N is the number of IMU frames between image frames; Stereo-se3 constrains IMU-se3.Source: 1) Preintegrated Network · p. 4

2) Status Update Module:

  • The status-update module feeds back the motion difference between IMU preintegration and FC-Fusion to update the IMU state continuously.The defined rotation and translation errors compare the LSTM-IMU and FC-Fusion motions; the update is described as similar to traditional tightly coupled VIO.Source: 2) Status Update Module · pp. 4–5

D. FC-Fusion Network

FC-Fusion combines optical-flow features with IMU-derived motion to estimate a six-dimensional VIO motion vector. Its training includes a pixelwise optical-flow loss against synthetic flow.

D. FC-Fusion Network

  • FC-Fusion combines OFF and IMU-se3 through five fully connected layers to produce a six-dimensional VIO-se3 estimate.The camera trajectory is calculated by integrating VIO-se3, which is also constrained by Stereo-se3.Source: D. FC-Fusion Network · p. 5
  • FC-Fusion’s output, VIO-se3, represents the motion estimate used to calculate the camera trajectory by integration.Stereo-se3 also constrains VIO-se3.Source: D. FC-Fusion Network · p. 5

1) CNN-Flow Loss:

  • The CNN-Flow loss measures pixelwise differences between the network output and synthetic 2D optical flow.The loss sums endpoint error, the standard optical-flow estimation error.Source: 1) CNN-Flow Loss · p. 5

2) IMU Loss:

The IMU and fusion losses compare estimated camera motion with Stereo-se3 motion, with scale factors balancing rotation and translation. The experimental section evaluates DeepVIO on indoor and outdoor datasets.

2) IMU Loss:

  • The IMU loss compares LSTM-IMU estimates of camera rotation and translation with Stereo-se3 motion.A scale factor balances the translation and quaternion elements.Source: 2) IMU Loss · p. 5

3) Fusion Loss:

  • The FC-Fusion loss compares its rotation and translation estimates with Stereo-se3 motion.A scale factor balances the translation and quaternion elements.Source: 3) Fusion Loss · p. 5

IV. EXPERIMENTAL RESULTS

  • The experimental section evaluates DeepVIO against state-of-the-art algorithms on indoor and outdoor datasets.It introduces a detailed analysis following the evaluation setup.Source: §IV · p. 5

A. Dataset

DeepVIO is evaluated on KITTI and EuRoC, with results reporting trajectory accuracy against learning-based and traditional methods. On KITTI, its training variant with IMU status updates converges faster and has lower errors than the compared variants.

A. Dataset

  • KITTI training uses sequences 00–08, with 09–10 for testing, while EuRoC evaluation reports results on MH04 and MH05.KITTI sequence 03 is omitted because its IMU data is unavailable; 5% of sequences 00–08 are used for validation.Source: B. Network Training · pp. 5–6

B. Network Training

  • VIO-U, trained with IMU status updates, converges faster and has lower KITTI training errors than VO-F and VIO-N.VO-F uses only the LSTM-IMU output vector OFF; VIO-N omits the IMU status update module.Source: B. Network Training · p. 6, Figure 5 · p. 6

C. Comparison of the Trajectories

  • On KITTI test, the evaluated method outperforms existing methods in trajectory accuracy, assessed by average translational error trel (%) and rotational error rrel (°/100 m) over lengths 100 m–800 m, and generalizes to unseen sequences 09 and 10.Source: C. Comparison of the Trajectories · p. 6
  • On EuRoC, DeepVIO outperforms DeepVO but is slightly worse than VINS and ORB-SLAM-M.The paper attributes the latter comparisons to VINS’s high-quality IMU data and ORB-SLAM-M’s local bundle adjustment optimization.Source: Table II · p. 6, C. Comparison of the Trajectories · p. 6

D. Optical Flow Error

The 3D-flow-derived supervision is evaluated against Flownet2 and ground-truth flow, including dynamic and textureless regions. A separate KITTI sequence-10 comparison reports lower errors for VIO-U than VIO-N, particularly during rapid camera rotation.

D. Optical Flow Error

  • Synthetic 2D flow is denser than flow from raw lidar, and Flownet2 suffers from aperture ambiguity in textureless regions.The figure compares synthetic flow and Flownet2 flow with KITTI ground truth.Source: Figure 7 · p. 7, D. Optical Flow Error · p. 7
  • On KITTI, 3D optical flow has lower error than Flownet2 without dynamic targets, and dynamic-target effects are eliminated when motion is calculated between point clouds.Dynamic regions can be removed while the remaining flow trains DeepVIO; the comparison covers 50 non-dynamic and 20 dynamic-target pairs.Source: Table III · p. 7, D. Optical Flow Error · p. 7

E. Comparison of DeepVIO Network

  • VIO-U has lower location error than VIO-N, especially during rapid camera rotation, while VO-F has the largest error.The authors attribute VIO-N’s worse error than VIO-U to changing IMU status and report improved location accuracy from the IMU update module.Source: Figure 8 · p. 7, E. Comparison of DeepVIO Network · p. 7

F. Comparison with Traditional VIO Methods

DeepVIO outperforms state-of-the-art learning-based VO and VIO systems in pose accuracy and generalizes to unseen scenarios. Compared with traditional VIO, it retains acceptable performance under calibration errors, synchronization issues, and missing data, though drift remains without place recognition and relocalization.

F. Comparison with Traditional VIO Methods

  • Across calibration, synchronization, and dropped-frame tests, DeepVIO is more accurate than traditional VINS, with vision contributing more when IMU data are missing and IMU compensating when camera frames are dropped.The tests include added 20 ms IMU timestamp offsets and dropped IMU or camera frames.Source: Table IV · p. 7, F. Comparison with Traditional VIO… · p. 7

V. CONCLUSIONS

  • DeepVIO uses stereo sequences in a self-supervised end-to-end strategy to estimate an absolute visual-inertial trajectory.The conclusion identifies 3D optical flow and 6-DoF pose as geometric constraints used to optimize the networks.Source: §V · pp. 7–8
  • DeepVIO outperforms state-of-the-art learning-based VO and VIO systems in pose accuracy and generalizes well to unseen scenarios.The system estimates absolute trajectories from stereo sequences through a self-supervised end-to-end strategy.Source: §V · p. 8
  • Drift remains because the system lacks place recognition and relocalization pipelines.Loop-closure detection and global relocalization are planned extensions.Source: §V · p. 8
Loading 1906.11435v2…