Source-linked AI summary
TartanVO: A Generalizable Learning-based VO
Wenshan Wang, Yaoyu Hu, Sebastian Scherer
TL;DR
Learning-based visual odometry has struggled with robust real-world performance and generalization across datasets, cameras, and settings. TartanVO combines diverse synthetic training with an up-to-scale loss and camera-intrinsics encoding, and experiments show generalization to real-world datasets without finetuning. The paper concludes that this approach can outperform dedicated learning models and geometry-based methods in challenging evaluations.
Problem
Robust learning-based VO remains difficult because insufficient data diversity, monocular scale ambiguity, and differing camera intrinsics hinder generalization across environments and datasets.
Method
TartanVO trains on diverse synthetic TartanAir data and combines an up-to-scale loss with an intrinsics layer, using matching and pose sub-modules.
Results
TartanVO generalizes to unseen real-world datasets without finetuning and achieves performance even better than dedicated learning models trained directly on those datasets.
Takeaways & Limitations
Diverse synthetic training together with explicit scale and camera-intrinsics handling supports a single VO model's cross-dataset and cross-camera generalization.
Takeaways & Limitations
The approach relies on the assumption that broad diversity in purely synthetic training data can transfer effectively to real-world scenes.
Abstract
from arXiv · showhide
We present the first learning-based visual odometry (VO) model, which generalizes to multiple datasets and real-world scenarios and outperforms geometry-based methods in challenging scenes. We achieve this by leveraging the SLAM dataset TartanAir, which provides a large amount of diverse synthetic data in challenging environments. Furthermore, to make our VO model generalize across datasets, we propose an up-to-scale loss function and incorporate the camera intrinsic parameters into the model. Experiments show that a single model, TartanVO, trained only on synthetic data, without any finetuning, can be generalized to real-world datasets such as KITTI and EuRoC, demonstrating significant advantages over the geometry-based methods on challenging trajectories. Our code is available at https://github.com/castacks/tartanvo.
1 Introduction
Learning-based VO has not matched geometry-based methods in robust real-world performance, partly because existing models lack diverse training data and cross-dataset handling. TartanVO addresses these issues with diverse synthetic training, an up-to-scale loss, and an intrinsics layer.
- Robust and reliable visual odometry remains challenging for real-world applications despite progress in geometric and learning-based methods.
- Learning-based VO has not yet achieved the robust performance seen in other visual tasks, while geometric methods remain insufficiently robust in many real-life situations.
- Existing VO models often lack sufficient diversity in scenes and motion patterns, limiting their ability to generalize across environments.
- TartanVO targets these limitations by using diverse data, an up-to-scale loss for monocular scale ambiguity, and an intrinsics layer for cross-camera generalization.
- The authors present TartanVO as a learning-based VO model designed to generalize across datasets and achieve competitive performance on varied real-world data without finetuning.
2 Related Work
Prior learning-based VO work uses supervised or unsupervised end-to-end designs, often with auxiliary depth or optical-flow outputs, but commonly faces scale and generalization limitations. TartanVO follows the optical-flow intermediate-representation design while targeting broader cross-dataset and cross-camera generalization.
- Recent end-to-end VO models increasingly use unsupervised learning, while supervised models trained on labeled odometry data generally achieve better performance.
- Auxiliary depth and optical-flow predictions provide temporal matching or consistency signals for estimating camera motion.
- Optical flow serves as an intermediate 2D matching representation that lets camera-motion estimators process flow instead of raw images.
- Monocular VO has scale ambiguity, which most supervised learning models historically neglected and geometry-based methods address using additional information.
- Many learning-based VO models are trained and tested on the same dataset, while adaptation to new environments or cameras requires additional training.
- Figure 1 depicts a two-stage architecture in which a matching network estimates optical flow before a pose network predicts camera motion.
3 Approach
TartanVO addresses monocular VO generalization by learning dense optical-flow matching and camera pose from diverse synthetic data, while explicitly handling scale and camera-intrinsics variation. Its design combines an up-to-scale translation loss, an intrinsics layer, and random crop-resize augmentation to support cross-dataset use.
- Model architecture: The model estimates optical flow from consecutive RGB images, then predicts camera motion from the dense matching result through separate matching and pose modules.This modular design processes optical flow as an intermediate representation between images and camera motion.
- Training data: TartanAir supplies more than 400,000 frames spanning diverse scenes and rich 6DoF motion patterns for training and generalization.Its environments include indoor, outdoor, urban, nature, and sci-fi scenes, with multimodal labels including optical flow and camera pose.
- Training objective: The end-to-end objective jointly minimizes optical-flow loss and camera-motion loss, weighted by a hyper-parameter λ.The matching module is supervised against optical-flow labels, while the pose module is supervised against camera motion.
- Up-to-scale loss function: The up-to-scale loss normalizes translation or compares its direction while retaining the original rotation loss, addressing monocular scale ambiguity.The authors report that the proposed loss is crucial for generalization, with cosine-similarity and normalized-distance formulations showing similar preliminary performance.
- Intrinsics layer: The intrinsics layer gives the pose module camera-dependent 2D position information by combining optical flow with intrinsic parameters K = {fx, fy, ox, oy}.This explicitly couples image positions and matching estimates, avoiding reliance on warping images to match training-camera intrinsics.
- Intrinsics augmentation: Random cropping and resizing generates varied camera intrinsics from TartanAir’s single camera setting, covering cameras with FoV between 40° and 90°.The image and intrinsics layer can be cropped and resized together without recomputing the intrinsics layer.
4 Experimental Results
Experiments evaluate TartanVO’s training setup and generalization across synthetic, real-world, and challenging trajectories. Results show that more diverse data, up-to-scale supervision, and camera-intrinsics handling improve generalization without dataset-specific finetuning.
- Experimental setup: 40 ms inference on an NVIDIA GTX 1080 GPU enables the model to process optical flow and estimate pose efficiently.The network combines PWC-Net matching with a modified ResNet50 pose network operating at one-quarter image resolution.
- Data quantity: Generalization improves constantly as training data increases from 20,000 to 100,000 and 400,000 frames.Testing uses three unseen TartanAir environments, with more than 400,000 training frames and about 40,000 testing frames available overall.
- Up-to-scale loss: The up-to-scale loss decreases the training–testing translation-loss gap, which remains substantial without that loss even with abundant data.Translation error, rather than rotation error, is identified as the main contributor to the gap.
- Camera intrinsics: The intrinsics layer is critical when training includes varied camera intrinsics through RCR augmentation.Training with RCR and the intrinsics layer produces lower testing loss than training with only one intrinsic configuration.
- Real-world evaluation: On KITTI, TartanVO achieves competitive results without finetuning, backend optimization, or more than two consecutive input images.Other compared learning-based methods are trained on KITTI, while several baselines use multiple frames or trajectory optimization.
- Real-world evaluation: TartanVO performs best on EuRoC trajectories VR1-03 and VR2-03 and shows more robust performance than ORB-SLAM on challenging TartanAir cases.The difficult settings include aggressive motion, changing illumination, dynamic objects, fog, rain, limited features, and large motion.
5 Conclusions
TartanVO is presented as a generalizable learning-based visual odometry model whose diverse-data training, up-to-scale loss, and intrinsics layer support cross-dataset performance without finetuning.
- TartanVO demonstrates the effectiveness of diverse data for model generalization.
- The up-to-scale loss reduces the gap between training and testing losses, further increasing generalization capability.
- An intrinsics layer helps TartanVO handle different cameras and generalize to unseen datasets.
- Extensive experiments show performance even better than dedicated learning models trained directly on those datasets.
A Additional experimental details
This section provides additional experimental details, covering the network structure, training parameters, qualitative results, and quantitative results.
- The supplementary experiments section expands on the paper’s experimental setup and findings.
- Additional details cover the network structure and training parameters.
- The section also presents qualitative and quantitative results.
A.1 Network Structure
The network uses separate matching and pose sub-modules, with the pose network’s architecture specified through residual blocks and translation and rotation heads. Supplementary evaluation compares TartanVO with ORB-SLAM on held-out TartanAir trajectories using ATE.
- Network components: The network consists of a matching network Mθ and a pose network Pφ.PWC-Net serves as the matching network, and its optical-flow output is fed into the pose network.
- Matching network: PWC-Net processes two consecutive 640 x 448 images and outputs optical flow at 160 x 112 resolution.The stated image dimensions satisfy PWC-Net’s requirement that image sizes be multiples of 64.
- Pose network: The pose-network architecture uses residual blocks followed by separate fully connected translation and rotation heads.Downsampling occurs through Conv1 and at the beginning of each residual block before feature reshaping.
- Evaluation: Supplementary evaluation compares ORB-SLAM and TartanVO on held-out TartanAir trajectories using the ATE metric.ORB-SLAM is run five times, with the best result reported.
A.2 Testing Results on TartanAir
Testing on TartanAir covers 16 challenging trajectories, comparing TartanVO with monocular ORB-SLAM under a repeated-trial evaluation protocol. ORB-SLAM frequently loses tracking in these cases, failing on 9 of 16 trajectories.
- Test set: TartanAir testing uses 16 challenging trajectories, with eight reported in the main experiment section and eight provided in supplementary results.The complete set is visualized in Figure 8.
- Evaluation protocol: ORB-SLAM is repeatedly run for five trials because of randomness, and the best result is reported.A trial counts as a failure when ORB-SLAM tracks less than 80% of the trajectory.
- Tracking outcome: 9/16 testing trajectories are failures for monocular ORB-SLAM in the challenging visualization cases.Figure 8 shows ground truth, TartanVO estimates, and ORB-SLAM estimates as black dashed, orange, and blue lines, respectively.
- Trajectory visualization: Figure 8 compares estimated trajectories from TartanVO and ORB-SLAM against ground truth across all 16 testing trajectories.The figure uses orange for TartanVO, blue for ORB-SLAM, and a black dashed line for ground truth.