PaperlayerView exact revision on arXiv

Source-linked AI summary

SelfVIO: Self-Supervised Deep Monocular Visual-Inertial Odometry and Depth Estimation

Yasin Almalioglu, Mehmet Turan, Alp Eren Sari, Muhamad Risqi U. Saputra, Pedro P. B. de Gusmão, Andrew Markham, Niki Trigoni

arXiv:1911.09968v2cs.CVcs.LGeess.IV

TL;DR

Limited labeled data and imperfections in ground-truth depth constrain supervised VIO and depth learning. SelfVIO combines adversarial training with visual-inertial fusion to jointly estimate motion and depth, and the authors report performance surpassing state-of-the-art VO, VIO, and VSLAM approaches.

  • Problem

    Dense VIO and depth estimation require labeled data that can be impractical or expensive to acquire, and available depth measurements may be imperfect or misaligned.

    Source: §I · p. 2

  • Method

    SelfVIO uses adversarial training and attentive visual-inertial fusion to estimate 6-DoF motion and scene depth from unlabeled monocular RGB sequences and raw IMU measurements.

    Source: §I · p. 2

  • Results

    SelfVIO is reported to surpass state-of-the-art depth and motion estimation performance of VO, VIO, and VSLAM approaches using monocular source-target image pairs.

    Source: §VI · p. 15

  • Takeaways & Limitations

    The authors report that even without IMU data, SelfVIO with RGB data outperforms deep monocular approaches in the same domain.

    Source: §VI · p. 16

    EuRoC depth results are qualitative because the dataset frames have no ground-truth depth maps for quantitative analysis.

    Source: A. Monocular Depth Estimation · p. 13

Abstract

from arXiv · show

In the last decade, numerous supervised deep learning approaches requiring large amounts of labeled data have been proposed for visual-inertial odometry (VIO) and depth map estimation. To overcome the data limitation, self-supervised learning has emerged as a promising alternative, exploiting constraints such as geometric and photometric consistency in the scene. In this study, we introduce a novel self-supervised deep learning-based VIO and depth map recovery approach (SelfVIO) using adversarial training and self-adaptive visual-inertial sensor fusion. SelfVIO learns to jointly estimate 6 degrees-of-freedom (6-DoF) ego-motion and a depth map of the scene from unlabeled monocular RGB image sequences and inertial measurement unit (IMU) readings. The proposed approach is able to perform VIO without the need for IMU intrinsic parameters and/or the extrinsic calibration between the IMU and the camera. estimation and single-view depth recovery network. We provide comprehensive quantitative and qualitative evaluations of the proposed framework comparing its performance with state-of-the-art VIO, VO, and visual simultaneous localization and mapping (VSLAM) approaches on the KITTI, EuRoC and Cityscapes datasets. Detailed comparisons prove that SelfVIO outperforms state-of-the-art VIO approaches in terms of pose estimation and depth recovery, making it a promising approach among existing methods in the literature.

I. INTRODUCTION

The introduction contrasts the appeal of visual odometry with challenges in monocular VO and the limited availability and imperfections of labeled data. It proposes self-supervised joint VIO and depth reconstruction using adversarial training and attentive visual-inertial fusion.

I. INTRODUCTION

  • Ground-truth data for dense VIO and depth learning can be expensive to acquire, while depth sensors and timestamps can introduce noise, artifacts, and alignment errors.The introduction notes that these problems may degrade or even cause failure in models trained on such data.Source: §I · p. 2
  • Unsupervised monocular methods use image reconstruction and geometric constraints, but ambiguities remain and photometric or temporal consistency can yield overly smoothed depth maps.The introduction also notes that optical-flow supervision has an aperture problem in locally structure-poor image regions.Source: §I · p. 2
  • The authors present the method as the first self-supervised deep joint monocular VIO and depth reconstruction approach, requiring no strict camera–IMU temporal or spatial calibration.They report evaluations on KITTI, EuRoC, and Cityscapes.Source: §I · p. 2
  • SelfVIO proposes adversarial training and attentive sensor fusion to combine motion features from raw IMU measurements and RGB images using convolutional and recurrent modules.The system is described as learning from unlabeled image sequences and raw IMU readings.Source: §I · p. 2

II. RELATED WORK

  • The related-work section surveys traditional and learning-based methods focused on VIO.Source: §II · p. 2

A. Traditional Methods

Traditional VIO combines visual and inertial sensing through filtering or nonlinear optimization, while learning-based methods seek odometry and depth without extensive ground-truth data. Prior approaches leave gaps in temporal modeling, training-data requirements, or sensor inputs that SelfVIO is designed to address.

A. Traditional Methods

  • Filtering and optimization provide complementary VIO strategies, with MSCKF-based methods generally more robust but less accurate than optimization-based approaches.Loosely coupled systems fuse vision and inertial measurements as separate inputs; tightly coupled systems treat VIO as one integrated estimation problem.Source: A. Traditional Methods · p. 3
  • OKVIS uses nonlinear batch optimization on keyframes, while VINS-Mono adds pose-graph consistency, IMU pre-integration, online extrinsic calibration, and loop detection.VINS-Mono also includes automatic initialization and failure detection.Source: A. Traditional Methods · p. 3

B. Learning-Based Methods

  • Many single-view depth methods rely on dense, pixel-aligned ground-truth depth during training, while VINet requires ground-truth pose differences.Unsupervised stereo methods avoid those labels but use known camera baselines as transforms between images.Source: B. Learning-Based Methods · p. 3
  • Joint unsupervised RGB-frame methods estimate ego-motion and depth, but these VO approaches do not fully use temporal information and can produce inaccurate, discontinuous estimates.UnDeepVO estimates trajectory on an absolute scale, but requires stereo training pairs; VIOLearner uses RGB-D input, whose depth supplies external supervision.Source: B. Learning-Based Methods · p. 4
  • Unsupervised depth estimators can produce overly smooth maps, motivating GANVO’s adversarial approach and SelfVIO’s use of GANs, recurrent temporal modeling, and attention-based feature fusion.The cited prior approaches use CNNs for pose and direct feature concatenation for sensor fusion.Source: B. Learning-Based Methods · p. 5

III. SELF-SUPERVISED MONOCULAR VIO AND DEPTH ESTIMATION ARCHITECTURE

SelfVIO jointly predicts relative camera motion and target-frame depth from unlabeled monocular RGB sequences and raw IMU measurements. Its architecture combines a depth generator, visual and inertial odometry branches, and reconstruction-based adversarial training.

III. SELF-SUPERVISED MONOCULAR VIO AND DEPTH ESTIMATION ARCHITECTURE

  • SelfVIO processes target and neighboring RGB views alongside raw IMU measurements, adaptively fuses visual and inertial pose features, and predicts target depth for view reconstruction.The reconstruction and real target are also used in adversarial training.Source: §III · p. 5

A. Depth Estimation

  • The depth generator translates the target RGB frame into a single-view depth map using an encoder, generator, and discriminator.Skip connections between layer i and layer n−i carry shared low-level information, such as prominent edge locations, across the network.Source: A. Depth Estimation · p. 5

B. Visual Odometry

  • The visual odometry branch extracts features from concatenated target and source views and regresses relative 6-DoF pose values for the source views.Its convolutional layers use separate translation and rotation branches with shared weights and dropout for regularization.Source: B. Visual Odometry · pp. 5–6

C. Inertial Odometry

  • The inertial branch processes raw linear acceleration and angular velocity samples between t−1 and t+1 in parallel branches to estimate relative motion.The network receives padded inputs of consistent size and no IMU or camera timestamps.Source: C. Inertial Odometry · p. 6

D. Self-Adaptive Visual-Inertial Fusion

SelfVIO uses attention to weight visual and inertial features before recurrent pose regression. Its spatial transformer uses estimated depth and relative motion to warp nearby source views into a reconstructed target.

D. Self-Adaptive Visual-Inertial Fusion

  • Attention-based fusion learns modality-specific filters from concatenated visual and inertial features, then weights each modality’s features by their relative importance.This design responds to the potential unreliability of direct feature concatenation when modalities differ in reliability.Source: D. Self-Adaptive Visual-Inertial Fusion · p. 6
  • A two-layer bidirectional LSTM models sequence dynamics from the fused features, and a fully connected layer regresses the 6-DoF pose values.For three input views, the pose output has 6 × (N−1) channels representing motion over the t−1 to t+1 window.Source: D. Self-Adaptive Visual-Inertial Fusion · p. 6

E. Spatial Transformer

  • The spatial transformer projects source views into the target coordinate frame using depth maps and relative camera motion, then bilinearly interpolates projected pixel intensities.Differentiable inverse warping between nearby frames provides positional constraints for the self-supervised learning scheme.Source: E. Spatial Transformer · pp. 6–7

F. View Discriminator

The view discriminator evaluates reconstructed target views, using local patches to model high-frequency image structure. The adversarial objective trains the reconstructed view to resemble a real target image.

F. View Discriminator

  • PatchGAN classifies each M×M image patch as real or fake and averages convolutional responses to model local high-frequency structure.Its Markov random field interpretation assumes independence between pixels separated by more than a patch diameter.Source: F. View Discriminator · p. 7
  • The discriminator distinguishes reconstructed colored views from real target images, while the generator aims to make the reconstruction indistinguishable from the original.The spatial transformer maps the generator’s output into the target view’s color space before discrimination.Source: F. View Discriminator · p. 7

G. The Adversarial Training

This section describes SelfVIO’s adversarial training objective and the experimental setup for evaluation on KITTI and EuRoC. It also presents qualitative depth results and the conditions in which reconstruction degrades.

G. The Adversarial Training

  • SelfVIO’s final training objective combines Lg and Ld with a balance factor β chosen experimentally from their end-of-training expected values.The network modifications include removing fully connected hidden layers, using batch normalization, replacing pooling with strided convolutions, and using LeakyReLU and ReLU activations in the discriminator and generator, respectively.Source: G. The Adversarial Training · p. 7

1) KITTI:

  • KITTI training uses sequences 00–08, with 09–10 held out for testing and 5% of the training sequences reserved for validation.The setup provides 18,422 training images, 2,791 testing images, and 969 validation images; each target uses 20 × 6 IMU samples spanning 100 ms before and after its timestamp.Source: 1) KITTI · p. 7

2) EuRoC:

  • EuRoC evaluation tests MH02(E), MH04(D), V103(D), and V202(D), while the remaining sequences are used for training.The dataset contains 11 sequences recorded in three indoor environments; its sequences vary in difficulty, including faster, loopier motion and poor illumination.Source: 2) EuRoC · p. 7
  • SelfVIO’s depth predictions preserve sharp detail in challenging KITTI scenes with low texture, shading, or uneven road lines, in both near and distant regions.The figure’s ground-truth visualization is made by interpolating sparse LiDAR points and projecting them onto the camera plane using KITTI calibration.Source: Figure 3 · p. 8
  • Depth reconstruction by SelfVIO, SfM-Learner, and CC degrades in vast open rural scenes and when large objects occlude the camera view.Source: Figure 4 · p. 8

3) Cityscapes:

This section describes the Cityscapes data and SelfVIO’s training and runtime setup. It also illustrates how visual and inertial feature weights vary with motion and visual conditions.

3) Cityscapes:

  • Cityscapes provides 22,973 stereo images from 50 German cities and precomputed disparity maps; the input images are cropped to retain the top 80%.The crop removes highly reflective car hoods.Source: 3) Cityscapes · p. 9

B. Network Training

  • Visual features dominate during straight motion, while inertial features gain importance during turns and occlusion or when salient visual signals are lacking.The figure shows feature weights and mean percentage activation for KITTI and EuRoC examples.Source: Figure 5 · p. 10
  • SelfVIO runs at 81 ms per frame on a Titan V GPU, including depth generation, visual odometry, and IMU processing with sensor fusion.The reported component times are 33 ms, 27 ms, and 21 ms, respectively.Source: B. Network Training · p. 10
  • SelfVIO is trained for 100,000 iterations with batches of 16, using three consecutive images as input and the central frame as the depth-estimation target.Training uses random scaling, cropping, and horizontal flips, with validation error checked every 1,000 iterations.Source: B. Network Training · p. 10

C. Evaluation

The evaluation compares SelfVIO with monocular VO, VIO, and VSLAM methods, and includes ablations of inertial input and IMU processing. KITTI ATE results report that SelfVIO outperforms all compared methods.

C. Evaluation

  • The comparisons use monocular versions of competing methods, align monocular trajectories with 6-DoF Umeyama alignment, and evaluate methods at 256 × 832 image resolution.SfMLearner’s scale is estimated from ground truth for each estimate.Source: C. Evaluation · p. 10
  • Separate networks are trained for KITTI and EuRoC, while Cityscapes is used to assess cross-dataset generalization; SelfVIO implicitly estimates camera–IMU extrinsics and IMU intrinsics from raw data.Source: C. Evaluation · p. 10
  • SelfVIO outperforms all compared methods in absolute trajectory error on the KITTI odometry dataset.Source: Table V · p. 11

1) Ablation Studies:

  • The no-IMU ablation replaces inertial odometry with vision-only odometry for the initial warp, while the LSTM ablation processes IMU input with a bidirectional recurrent network at greater computational cost.Source: 1) Ablation Studies · p. 11

2) Spatial Misalignments:

The section tests robustness to camera–IMU rotational miscalibration and defines the trajectory metrics used for evaluation. It also specifies the scope of the EuRoC trajectory comparisons.

2) Spatial Misalignments:

  • Camera–IMU rotation offsets drawn from a von Mises–Fisher distribution are applied during testing, but not during training, to evaluate robustness to spatial miscalibration.The miscalibration comparison covers relative rotational and translational errors across temporal and translational offsets on EuRoC.Source: 2) Spatial Misalignments · p. 11, Table VII · p. 12

3) Evaluation Metrics:

  • The KITTI relative metrics are average translational RMSE drift (%) and rotational RMSE drift (°/100m) over trajectory segments of 100–800 m.The section also reports KITTI errors at 100, 200, 300, 400, and 500 m and five-frame-snippet RMSE.Source: 3) Evaluation Metrics · pp. 11–12
  • The EuRoC table reports absolute translation RMSE in meters across all trials, and depth estimation is assessed with error and accuracy metrics from prior work.Source: 3) Evaluation Metrics · p. 11, Table VI · p. 12

V. RESULTS AND DISCUSSION

SelfVIO reports state-of-the-art single-view depth prediction and sharper, more detailed qualitative reconstructions than compared methods. Its evaluation also tests fine-tuning from Cityscapes to KITTI and shows qualitative depth recovery on EuRoC, where ground-truth depth is unavailable.

A. Monocular Depth Estimation

  • SelfVIO obtains state-of-the-art single-view depth prediction on the Eigen KITTI split, with depth capped at 80 meters and median scaling to address scale ambiguity.The predicted depth map is scaled by median(Dg)/median(Dp).Source: A. Monocular Depth Estimation · p. 12
  • SelfVIO produces sharper depth maps, captures small objects, and performs better in low-textured areas than GeoNet and Competitive Collaboration.The authors attribute less blurry results and greater detail sensitivity to adversarial feature-space loss; SelfVIO also handles inconsistencies without an additional masking network.Source: A. Monocular Depth Estimation · p. 12
  • Even with poor rural road signs and large occluders, SelfVIO performs slightly better than the compared methods.Source: A. Monocular Depth Estimation · p. 13
  • Cityscapes pretraining followed by KITTI fine-tuning tests adaptation; fine-tuning on test-related data offers an advantage, and inertial odometry brings larger gains even with KITTI-only training.EuRoC depth results are qualitative because the dataset lacks ground-truth depth; the examples indicate recovery across diverse 6-DoF motions.Source: A. Monocular Depth Estimation · p. 13

B. Motion Estimation

SelfVIO is evaluated in vision-only and visual-inertial modes across KITTI and EuRoC. The reported results include strong comparisons with VO and VIO methods, robustness under sensor offsets, and complementary roles for visual and inertial features.

1) Visual Odometry:

  • SelfVIO without IMU outperforms the listed VO approaches, indicating its results are not due solely to using inertial data.The comparison includes methods tested on data also used for training; the authors note that these unsupervised methods do not use ground-truth pose or depth maps.Source: 1) Visual Odometry · p. 13
  • On KITTI sequences 09 and 10, SelfVIO significantly outperforms SFMLearner; it also surpasses VIOLearner and VINet on longer sequence-10 trajectories.The reported longer trajectory lengths are 100, 200, 300, 400, and 500 m; the LSTM variant still performs better than VIOLearner and VINet.Source: 1) Visual Odometry · p. 13, 2) Visual-Inertial Odometry · pp. 13–14

2) Visual-Inertial Odometry:

  • SelfVIO outperforms OKVIS and ROVIO on KITTI sequences 09 and 10, despite their synchronization requirements and the dataset's limited acceleration.The authors attribute those methods' poor performance most likely to loose synchronization and note that SelfVIO compensates without explicitly estimating temporal offsets.Source: 2) Visual-Inertial Odometry · p. 14
  • On KITTI sequences 09 and 10, SelfVIO surpasses the five-frame-snippet RMSE performance of SFMLearner, Mahjourian et al., and VIOLearner.Source: 2) Visual-Inertial Odometry · p. 14
  • On EuRoC, SelfVIO produces the most accurate trajectories on many sequences, including without explicit loop closing.The paper evaluates trajectory-segment errors at 7, 14, 21, 28, and 35 m across the sequences.Source: 2) Visual-Inertial Odometry · p. 14, Figure 10 · p. 15
  • On difficult EuRoC sequences, brightness inconsistency can disrupt feature matching and cause filter-based approaches to diverge; on easy sequences, OKVIS and VINS-Mono are slightly better, while SelfVIO is similar to several other methods.Source: 2) Visual-Inertial Odometry · p. 15
  • SelfVIO has low errors for orientation offsets below 10 degrees and the smallest relative translation and rotation errors under tested temporal and translational offsets.The comparison reports that OKVIS fails on MH02–05 at a 90 ms time offset; neither OKVIS nor VINS-Mono gives reasonable estimates at 120 ms.Source: 2) Visual-Inertial Odometry · p. 15
  • Visual and inertial features are complementary: inertial features contribute more during fast rotation, while visual features are more active during large translations.Source: 2) Visual-Inertial Odometry · p. 15

VI. CONCLUSION

SelfVIO uses monocular source-target image pairs for depth and motion estimation. The authors report that it surpasses traditional and learning-based VO, VIO, and VSLAM approaches, and that its RGB-only version outperforms deep monocular approaches in the same domain.

VI. CONCLUSION

  • SelfVIO surpasses state-of-the-art depth and motion estimation approaches, including traditional and learning-based VO, VIO, and VSLAM methods.Source: §VI · p. 15
  • The authors attribute this performance to adversarial training and visual-inertial sensor fusion in an end-to-end trainable architecture.Source: §VI · p. 16
  • Even without IMU data, SelfVIO using RGB data outperforms deep monocular approaches in the same domain.The authors identify a stereo version using disparity maps as future work.Source: §VI · p. 16

Related papers

Loading 1911.09968v2…