Source-linked AI summary
GANVO: Unsupervised Deep Monocular Visual Odometry and Depth Estimation with Generative Adversarial Networks
Yasin Almalioglu, Muhamad Risqi U. Saputra, Pedro P. B. de Gusmao, Andrew Markham, Niki Trigoni
TL;DR
The paper addresses limited labelled data and the comparatively limited use of unsupervised learning for VO and depth recovery. It proposes GANVO, which estimates 6-DoF pose and monocular depth from unlabelled RGB sequences using view reconstruction and adversarial learning, and reports better pose and depth results than existing traditional and unsupervised methods on KITTI and Cityscapes.
Problem
Supervised VO methods require labelled data, while unsupervised localization and mapping from unlabelled sequences have received comparatively less attention.
Method
GANVO jointly predicts 6-DoF camera motion and monocular depth from unlabelled RGB sequences using CNN-RNN pose regression, GAN-based depth generation, and view reconstruction.
Results
Evaluations on KITTI and Cityscapes report better pose estimation and depth recovery than existing traditional and unsupervised deep VO methods.
Takeaways & Limitations
The proposed framework demonstrates the reported effectiveness of adversarial learning for monocular pose estimation and depth-map reconstruction.
Abstract
from arXiv · showhide
In the last decade, supervised deep learning approaches have been extensively employed in visual odometry (VO) applications, which is not feasible in environments where labelled data is not abundant. On the other hand, unsupervised deep learning approaches for localization and mapping in unknown environments from unlabelled data have received comparatively less attention in VO research. In this study, we propose a generative unsupervised learning framework that predicts 6-DoF pose camera motion and monocular depth map of the scene from unlabelled RGB image sequences, using deep convolutional Generative Adversarial Networks (GANs). We create a supervisory signal by warping view sequences and assigning the re-projection minimization to the objective loss function that is adopted in multi-view pose estimation and single-view depth generation network. Detailed quantitative and qualitative evaluations of the proposed framework on the KITTI and Cityscapes datasets show that the proposed method outperforms both existing traditional and unsupervised deep VO methods providing better results for both pose estimation and depth recovery.
I. INTRODUCTION
The paper introduces GANVO, an unsupervised framework for jointly estimating 6-DoF camera motion and monocular depth from unlabelled image sequences. It combines pose regression, generative depth estimation, and view reconstruction to avoid depth groundtruth and strict parameter tuning.
- Architecture: The architecture uses depth generation, multi-view pose estimation, view reconstruction, and target discrimination to establish supervision from unlabelled sequences.Relative translation and rotation are estimated between consecutive frames, while depth is represented as a disparity map.
- Motivation and proposed approach: GANVO jointly estimates 6-DoF camera pose and monocular depth from raw unlabelled image sequences in an end-to-end unsupervised framework.The pose regressor uses CNN-RNN modules, while the depth generator uses a convolutional GAN.
- Contributions: The method is presented as the first monocular VO approach combining adversarial and recurrent unsupervised learning for joint pose and depth estimation.
- Contributions: GANVO generates depth images without depth groundtruth information through an adversarial technique.
- Contributions: Evaluations on KITTI and Cityscapes report successful pose estimation and depth reconstruction without strict parameter tuning.
II. RELATED WORK
Prior VO methods face correspondence failures in challenging scenes, while unsupervised deep methods can produce overly smooth depth and miss temporal information. GANVO addresses these issues with GAN-based depth estimation and CNN-RNN pose regression.
- Limitations of traditional VO: Traditional VO depends on accurate image correspondence, which is often violated by low texture, complex structure, and occlusions.
- Unsupervised deep VO: Unsupervised deep VO jointly trains motion, depth, surface normal, and optical-flow components to obtain an implicit supervisory signal.
- Depth estimation: Existing unsupervised depth estimators tend to generate overly smooth images, motivating GANs for sharper and more accurate depth maps.
- Pose estimation: CNN-only pose estimators analyze momentary information, so GANVO employs CNN-RNN architecture to capture temporal relations across frames.
III. ARCHITECTURE OVERVIEW
GANVO reconstructs a target view from estimated depth, relative pose, and source-image colors, using reconstruction and adversarial discrimination to train without labels. The pose module estimates source-to-target motion while the depth module predicts the target view’s map.
- Multi-view pose estimation: The pose module regresses relative 6-DoF poses of source views with respect to a target view from stacked raw RGB sequences.
- Depth generation: The depth generation module predicts a depth map for the target view in parallel with pose estimation.
- View reconstruction: The view reconstruction module synthesizes the target image using predicted depth, estimated camera pose, and nearby source-image colors.
- Adversarial training: The GAN generator learns depth maps whose reconstructed target views cannot be distinguished from original images by the discriminator.
IV. UNSUPERVISED DEPTH AND POSE ESTIMATION WITH GENERATIVE ADVERSARIAL NETWORK
The architecture jointly estimates monocular depth and relative 6-DoF pose from image sequences using adversarial depth generation and recurrent temporal modeling.
- Depth Generator: The depth network uses an encoder E, generator G, and discriminator D to generate a single-view depth map for the target frame.E maps the target image to a feature vector, G maps that vector into depth-image space, and D produces an image likelihood.
- Pose Regressor: The pose estimator combines convolutional feature extraction with recurrent modeling of temporal relations across input frame sequences.The recurrent component follows the convolutional feature extractor and captures sequential information.
- Pose Regressor: The discriminator evaluates reconstructed RGB images against original target images after depth and pose predictions are used for view reconstruction.Its output represents a likelihood associated with the target image.
- Pose Regressor: The LSTM modules output 6∗(N −1) channels representing translation and rotation parameters for relative 6-DoF pose.Here, N denotes the input sequence length.
C. View Reconstruction
View reconstruction creates the target image by projecting source views using estimated depth and camera motion, with bilinear interpolation providing intensities at non-discrete coordinates.
- View Reconstruction: Three consecutive frames provide one target view and two source views for reconstructing the target image.The sequence is < It−1,It,It+1 >, with It as target and It−1, It+1 as source views.
- View Reconstruction: The reconstruction projects each source view into the target coordinate frame using estimated depth, a 4×4 camera transformation matrix, and camera intrinsics.The projected source image is denoted ˆIs, and pixel coordinates are mapped from the target view to the source view.
- View Reconstruction: Because projected coordinates are non-discrete, the method uses bilinear interpolation over four neighboring pixels.The neighboring-pixel weights represent proximity and sum to 1.
D. View Discriminator
The view discriminator distinguishes reconstructed target images from real images sampled from the target data distribution within the adversarial training scheme.
- View Discriminator: The reconstruction module synthesizes an image from generated depth and estimated pose, while D discriminates it from a real target image.The discriminator compares reconstructed and real images in RGB space.
- View Discriminator: The adversarial objective samples real images from pdata and latent feature encodings z for generated images.The optimization objective includes the discriminator's real-image and generated-image terms.
E. The Adversarial Training
Adversarial training modifies the GAN architecture to support deeper networks and combines generator and discriminator losses through a balance factor.
- The Adversarial Training: The modified GAN removes fully connected hidden layers, uses batch normalization, replaces pooling with strided convolutions, and applies LeakyReLU in D.In G, pooling is replaced by fractional-strided convolutions and ReLU is used except at the output layer.
- The Adversarial Training: The final optimization objective combines generator loss Lg and discriminator loss Ld as Lfinal = Lg +βLd.The balance factor β controls the relative contribution of the discriminator loss.
- The Adversarial Training: β is experimentally selected as the ratio between the expected values of Lg and Ld at the end of training.This selection defines the loss balance used for weight optimization.
V. EXPERIMENTS AND RESULTS
The architecture is trained on sequential RGB inputs resized to 128×416, using three consecutive images during training.
- Training inputs are sequential images sized 128×416, while test images are not restricted to a specific size.Three consecutive images are stacked to form each training input.
A. Pose estimation benchmark
GANVO is evaluated on the KITTI odometry split using held-out sequences and compared with unsupervised deep methods and monocular ORB SLAM.
- KITTI sequences 00–08 are used for training and 09–10 for testing without pose or depth ground truth during training.The dataset contains 11 driving sequences with IMU/GPS ground-truth odometry.
- GANVO predicts 6-DoF relative motion between sequences and is evaluated with Absolute Trajectory Error over five consecutive input frames.An optimized scaling factor resolves scale ambiguity in the evaluation.
- GANVO outperforms the compared unsupervised and traditional baselines without loop closure, bundle adjustment, or re-localization.The comparison includes unsupervised deep methods and monocular ORB SLAM.
B. Single-view depth evaluation
The depth evaluation uses KITTI benchmark data, qualitative comparisons, and robustness experiments to assess GANVO against existing methods across challenging scenes and settings.
- Dataset and protocol: The KITTI depth benchmark uses 40,000 frames for training and 4,000 frames for validation.Training uses three-frame inputs with the central frame as the target view.
- Dataset and protocol: Predicted depths are median-scaled to LiDAR ground truth before error and accuracy evaluation.The scale factor is ŝ = median(Dg)/median(Dp).
- Qualitative depth results: GANVO produces sharper depth maps and preserves small objects and details in low-textured, shaded, and uneven-road scenes.The qualitative comparison includes SfM-Learner and GeoNet, with LiDAR ground truth interpolated for visualization.
- Failure cases: GANVO slightly outperforms existing methods even in failure cases involving vast rural scenes and large objects occluding the view.All compared methods struggle in these settings.
- Quantitative results: GANVO significantly outperforms existing unsupervised and supervised methods on the KITTI monocular depth benchmark.The evaluation reports both error and accuracy, including a Cityscapes-to-KITTI adaptation benchmark.
- Robustness analysis: GANVO is robust to different weight initializations, dataset splits, and optimization parameters, showing lower mean and variance of normalized loss.The robustness analysis compares GANVO with SfM-Learner and GeoNet across multiple hyperparameter experiments.
VI. CONCLUSIONS
The study presents an unsupervised generative method for monocular pose and depth estimation and reports stronger pose and depth results than competing baselines.
- GANVO estimates camera pose and depth maps from monocular video using unsupervised generative deep learning.The conclusion emphasizes adversarial learning for both tasks.
- GANVO outperforms competing unsupervised and traditional baselines for pose estimation.
- GANVO produces more detailed, sharper, and more accurate scene depth maps than the competing methods.