Source-linked AI summary
TartanAir: A Dataset to Push the Limits of Visual SLAM
Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafei Hu, Ashish Kapoor, Sebastian Scherer
TL;DR
Visual SLAM needs benchmarks that better reflect changing conditions, dynamic objects, and diverse motion than established datasets provide. TartanAir addresses this with large-scale photo-realistic simulation, multimodal ground truth, and an automatic data-collection pipeline. Evaluations show that state-of-the-art algorithms struggle and frequently lose tracking on challenging sequences.
Problem
Existing SLAM benchmarks cover limited scenarios and motion patterns, while real-world conditions include changing light, low illumination, dynamic objects, and other inconsistencies.
Method
TartanAir combines diverse photo-realistic simulation environments with multimodal ground truth and an automatic pipeline for mapping, trajectory sampling, data processing, and verification.
Results
State-of-the-art SLAM algorithms struggle to track camera pose and frequently get lost on challenging sequences, with performance dropping as motion complexity increases.
Takeaways & Limitations
TartanAir is intended to complement existing benchmarks, reduce overfitting to limited datasets, and support development of visual SLAM algorithms for real-world applications.
Abstract
from arXiv · showhide
We present a challenging dataset, the TartanAir, for robot navigation tasks and more. The data is collected in photo-realistic simulation environments with the presence of moving objects, changing light and various weather conditions. By collecting data in simulations, we are able to obtain multi-modal sensor data and precise ground truth labels such as the stereo RGB image, depth image, segmentation, optical flow, camera poses, and LiDAR point cloud. We set up large numbers of environments with various styles and scenes, covering challenging viewpoints and diverse motion patterns that are difficult to achieve by using physical data collection platforms. In order to enable data collection at such a large scale, we develop an automatic pipeline, including mapping, trajectory sampling, data processing, and data verification. We evaluate the impact of various factors on visual SLAM algorithms using our data. The results of state-of-the-art algorithms reveal that the visual SLAM problem is far from solved. Methods that show good performance on established datasets such as KITTI do not perform well in more difficult scenarios. Although we use the simulation, our goal is to push the limits of Visual SLAM algorithms in the real world by providing a challenging benchmark for testing new methods, while also using a large diverse training data for learning-based methods. Our dataset is available at \url{http://theairlab.org/tartanair-dataset}.
I. INTRODUCTION
Visual SLAM benchmarks often cover limited scenarios and motion patterns, while real-world environments introduce changing light, low illumination, dynamic objects, and texture-less scenes. TartanAir addresses these gaps with diverse photo-realistic simulation environments, an automatic collection pipeline, and evaluations showing state-of-the-art methods struggle on challenging sequences.
- Real-world Visual SLAM must handle changing light, low illumination, dynamic objects, and texture-less scenes.
- Existing benchmarks standardize evaluation but can encourage benchmark over-fitting and cover limited scenarios and motion patterns.
- Driving-focused datasets provide adverse-condition data but restrict motion patterns through ground-vehicle dynamics.
- TartanAir uses diverse photo-realistic simulation environments to address limitations of physical collection and reduce the sim-to-real gap through increased diversity.
- The dataset emphasizes changing light, low illumination, adverse weather, and dynamic objects across diverse scene categories.
- State-of-the-art SLAM algorithms struggle to track camera pose and frequently get lost on challenging sequences.
- The paper contributes a diverse multimodal dataset, a fully automatic collection and verification pipeline, and evaluations of popular SLAM algorithms.
II. DATASET FEATURES
TartanAir is designed as a large, diverse synthetic dataset with realistic rendering, multimodal sensor data, precise ground truth, and varied motion patterns. Its scale spans 30 environments, 1037 sequences, over 1 million frames, and 4TB of data.
- The dataset targets large size, high diversity, realistic lighting, multimodal ground truth, diverse motion patterns, and challenging scenarios.
- Unreal Engine and AirSim provide photo-realistic scenes with complex geometry, high-fidelity textures, dynamic lighting, and object motions.
- 30 environments yield 4TB of data, 1037 long motion sequences, and more than 1 million labeled frames.
- TartanAir provides synchronized stereo RGB, depth, segmentation, and camera-pose data, plus generated occupancy maps, optical flow, stereo disparity, and simulated LiDAR.
C. Diversity of motion patterns
TartanAir addresses the limited motion patterns of established SLAM datasets by generating and measuring more diverse 3D camera motions. Its evaluation indicates that increasing motion complexity substantially challenges SLAM performance.
- Motivation: Existing SLAM datasets such as KITTI and RobotCar mainly contain forward motion with small turns, limiting motion-pattern evaluation and learning-based generalization.The paper identifies both insufficient evaluation difficulty and bias toward car-like motion patterns as consequences of this regularity.
- Evaluation: As motion-pattern complexity increases, SLAM performance drops significantly, while constrained datasets receive low diversity scores.The paper attributes low scores for TUM and EuRoC to MAV dynamics and human habits.
- Motion generation: TartanAir randomizes motion distributions and combinations in 3D space, then compares translation and rotation patterns using PCA.Translation and rotation sequences are decomposed into principal components, and frames are projected into the resulting vector spaces for visualization.
- Motion comparison: KITTI shows one dominant axis in translation and rotation, whereas TartanAir achieves more diverse motion patterns.Simulation can also impose constraints such as fixed roll and pitch to mimic ground-robot motion.
- Metric: The proposed motion-diversity metric uses σ, where smaller values indicate one-dimensional dominance and values approaching 1 indicate greater diversity.The metric is reported for SLAM datasets in Table II.
D. Challenging Scenes
TartanAir includes challenging scene conditions spanning illumination, weather, and dynamic objects. Experiments show that these conditions heavily affect SLAM algorithms.
- Scene conditions: TartanAir includes dynamic lighting, low illumination, adverse weather, and dynamic objects across its challenging environments.The scenarios include changing light, weather and seasonal effects, and moving entities.
- Lighting: Its lighting scenarios include strong changes, shadows, over-exposure, reflections, and light sources ranging from lamps and neon to sunlight and moonlight.AirSim cameras can use auto-exposure to add camera-like responses to dynamic lighting.
- Weather: Outdoor environments vary by time of day and season and include rain, snow, and fog.
- Dynamic objects: Dynamic objects include humans, vehicles, machinery, and vegetation, with configurable levels of motion.
- Evaluation: SLAM algorithms are heavily affected when the challenging scene settings are enabled.The experiment compares SLAM with and without these settings.
III. METHOD
The dataset uses a fully automatic pipeline to scale collection across 30 diverse environments. It combines environment mapping, trajectory generation, multimodal data collection, ground-truth processing, and verification.
- The fully automatic pipeline scales data collection to 30 diverse environments with minimal human intervention.
- The pipeline samples collision-free trajectories, collects multimodal sensor data, generates additional ground-truth labels, and verifies data correctness.
- Incremental mapping reconstructs occupancy grid maps for obstacle avoidance and path planning using depth images and camera poses.A frontier-based algorithm selects mapping locations, while RRT* plans collision-free trajectories; mapping ends when no frontiers remain.
B. Trajectory Sampling
Trajectory sampling builds feasible paths through mapped free space, randomizes camera motion, and records multimodal data and derived geometric labels. These outputs include optical flow, disparity, masks, and simulated LiDAR measurements.
- Trajectory Sampling: Trajectory sampling randomly connects free-space nodes with RRT* paths to construct a graph of feasible environment trajectories.Loop trajectories are then sampled from the graph and smoothed while avoiding obstacles.
- Trajectory Sampling: Randomized incremental distances and angles along trajectories create diverse virtual-camera motions for data collection.The resulting poses are sent to a simulated camera through the AirSim interface.
- Data Processing: Optical flow is computed by warping depth images between reference and test camera poses, with occlusion and out-of-FOV masks.The reference and test images, flow, and color mapping are visualized in Fig. 6; masked flow variants appear in Fig. 7.
- Data Processing: Stereo disparity is calculated from depth and camera intrinsics, with corresponding occlusion and out-of-FOV masks.Masked and unmasked disparity visualizations are shown in Fig. 8.
- Data Processing: Simulated LiDAR points are extracted by sampling depth from four 90°-FOV virtual cameras and interpolating distances over nearby depth pixels.The procedure produces a point cloud that simulates a 32-line LiDAR.
D. Data Verification
Data verification checks synchronization, occlusion, and collisions to ensure the collected simulated sensor data are consistent. Photometric consistency provides a quantitative synchronization check.
- Synchronization is verified by projecting reference-camera pixels into the test image using optical flow and measuring mean RGB photometric error.
- The maximum mean photometric error is less than 5 on a 0–255 scale across approximately 700 consecutive image pairs.
- Images with large occluded areas are detected using optical-flow masks.
- Depth images are used to verify collisions with the environment.
IV. EXPERIMENTAL EVALUATION
The evaluation applies three visual SLAM systems to six representative environments selected for interesting features. The experiments compare ORB-SLAM monocular, ORB-SLAM stereo, and DSO monocular.
- ORB-SLAM monocular, ORB-SLAM stereo, and DSO monocular are evaluated on six representative environments with interesting features.
A. Testing environments
TartanAir tests visual SLAM in environments combining adverse illumination, weather, dynamic elements, and limited visual features. Evaluation uses ATE, RPE, and success rate, with sequence-length control and success-rate caveats.
- Testing environments: Testing environments include rain with lens flare, blinking lights, low illumination, falling leaves, sparse ground features, and moving fish or bubbles.These conditions are represented across outdoor, indoor, transition, and underwater scenes.
- Metrics: ATE, RPE, and SR evaluate trajectory accuracy, relative pose, and the fraction of sequences completed without being lost.Monocular methods receive scale correction before ATE and RPE calculation.
- Metrics: SR is more informative on challenging datasets because ATE and RPE exclude failed trajectories and can reward algorithms that abandon difficult sequences.Less robust methods may therefore report higher ATE or RPE despite failing more often.
- Metrics: Sequences are cut to 200 frames because longer trajectories are often harder to complete and can affect SR.
- Evaluation: Table III compares SLAM methods across the selected environments using the three reported metrics.
C. Evaluation results
Evaluation varies motion complexity and isolates challenging scene factors. SLAM success generally declines with harder motion, while weather, dynamic objects, and low illumination produce distinct accuracy and robustness effects.
- Motion difficulty: The easy setting fixes pitch and roll, whereas medium and hard settings use 6 DoF motion with increasing translation and rotation speeds.The difficulty definitions specify motion complexity, maximum translation, and maximum inter-frame rotation.
- Baseline evaluation: SR drops remarkably as motion difficulty increases, and monocular methods have low SR even with easy motion in challenging scenes.
- Baseline evaluation: ORB-S is more robust than ORB-M and DSO in all easy cases and most medium cases, but performs slightly worse than ORB-M in all hard cases.
- Baseline evaluation: DSO performs best in Japanese-alley, reflecting the advantage of direct methods in the low-feature, low-illumination environment.
- Controlled experiment: In 9 out of 10 controlled tests, challenging features reduce SR or accuracy relative to matched trajectories without those features.The experiment uses five environments, medium motion, and repeated runs of ORB-M and ORB-S.
- Controlled experiment: Dynamic objects have limited SR effect but significantly reduce ATE in 3 out of 4 trajectories, with large moving machinery affecting accuracy more than small, widespread leaves.
- Controlled experiment: Rain and storm reduce SR by more than 50% for ORB-M and by 22% with storm for ORB-S.ORB-S is more robust to adverse weather than ORB-M.
- Controlled experiment: Low illumination harms SLAM, but synthetic dark scenes omit camera noise and motion blur that commonly occur in real-world night scenes.The authors identify image-noise augmentation as future work.
V. CONCLUSION
The authors present TartanAir as a challenging visual SLAM dataset intended to complement existing benchmarks and support progress toward algorithms that work well in practice.
- Conclusion: TartanAir is intended to reduce overfitting to datasets with limited training or testing examples and push visual SLAM toward real-world applications.