Source-linked AI summary
Virtual KITTI 2
Yohann Cabon, Naila Murray, Martin Humenberger
TL;DR
The paper addresses the need for varied, fully annotated driving data by introducing Virtual KITTI 2, an upgraded synthetic dataset with improved rendering, stereo imagery, and multiple modalities. It evaluates the dataset across autonomous-driving tasks and reports that the original multi-object-tracking conclusion remains valid, while performance varies across conditions and tasks.
Problem
Varied, fully annotated data are critical for training and testing driving models, but synthetic data are supplements rather than complete replacements for real-world data.
Method
The paper upgrades Virtual KITTI with Unity HDRP/Lit rendering, stereo cameras, post-processing, expanded flow modalities, and metadata, then evaluates it with several driving algorithms and tasks.
Results
The original multi-object-tracking conclusion remains valid: suitable DP-MCF parameters yield high and similar MOTA metrics for every real–virtual pair, while condition changes affect tasks differently.
Takeaways & Limitations
Virtual KITTI 2 supports evaluation across multiple driving problems and conditions, including real-like, weather-modified, and camera-angle-modified settings.
Abstract
from arXiv · showhide
This paper introduces an updated version of the well-known Virtual KITTI dataset which consists of 5 sequence clones from the KITTI tracking benchmark. In addition, the dataset provides different variants of these sequences such as modified weather conditions (e.g. fog, rain) or modified camera configurations (e.g. rotated by 15 degrees). For each sequence, we provide multiple sets of images containing RGB, depth, class segmentation, instance segmentation, flow, and scene flow data. Camera parameters and poses as well as vehicle locations are available as well. In order to showcase some of the dataset's capabilities, we ran multiple relevant experiments using state-of-the-art algorithms from the field of autonomous driving. The dataset is available for download at https://europe.naverlabs.com/Research/Computer-Vision/Proxy-Virtual-Worlds.
1 Introduction
Virtual KITTI 2 extends synthetic driving data for training and evaluation by providing a more photo-realistic, better-featured dataset and broader experiments across autonomous-driving tasks.
- Synthetic datasets provide a cost-effective supplement to real-world data for training, testing, prototyping, and sometimes improving model performance.They cannot completely replace real-world data but can exhibit good transferability.
- The original Virtual KITTI recreated KITTI tracking videos in a game engine and produced synthetic data comparable to real data for some applications.
- Virtual KITTI 2 uses Unity lighting and post-processing improvements to reduce the visual gap between Virtual KITTI and KITTI images.The paper also reruns prior experiments and adds stereo matching, monocular depth, camera pose estimation, and semantic segmentation experiments.
2 Changelog
Virtual KITTI 2 upgrades rendering and expands the dataset with stereo imagery, additional flow modalities, and metadata while preserving the original scene geometry and trajectories.
- The original Virtual KITTI used older Unity versions, standard shaders, and no post-processing for rendered images.
- Virtual KITTI 2 upgrades scenes to Unity 2018.4 LTS and HDRP/Lit materials, without altering geometry or trajectories.A stereo camera was also added.
- The updated modality-generation code supports multiple cameras, RGB post-processing, custom anti-aliasing, the game engine’s post-processing package, and Virtual KITTI optical flow.
- The dataset retains 5 sequence clones and adds stereo views, forward and backward optical flow, and forward and backward scene flow.Each camera provides RGB, class segmentation, instance segmentation, and depth, while camera parameters, vehicle color, poses, and bounding boxes are available.
3 Related datasets
Virtual KITTI addresses a broad union of road-scene appearance and geometry problems while uniquely reproducing real camera sequences for domain-adaptation research.
- Recent road-scene datasets target specific needs including semantic segmentation, stereo matching, diverse modalities, and visual localization.Examples include Cityscapes, SYNTHIA, DrivingStereo, Playing for Benchmarks, and Virtual Gallery.
- Virtual KITTI can be applied to the union of problems addressed by the cited datasets.
- Because it reproduces camera sequences captured in real environments, Virtual KITTI enables domain adaptation across synthetic and real data, camera angles, filming conditions, and environments.
4 Experiments
The experiments evaluate Virtual KITTI 2 across tracking, stereo matching, depth and pose estimation, and semantic segmentation. Results generally support similarity between virtual and real data while exposing sensitivity to weather and camera variations.
- 4.1 Multi-Object Tracking: Virtual KITTI 2 preserves the original finding that suitable DP-MCF parameters produce high and similar MOTA for real and virtual video pairs.
- 4.1 Multi-Object Tracking: Lighting-condition changes in Virtual KITTI 2 have a smaller impact on tracking results, although differing implementations make variation comparisons difficult.
- 4.1 Multi-Object Tracking: All three tracking models converged very rapidly, and improved RGB appeared to close the real–virtual performance gap by 50%.
- 4.2 GANet: GANet stereo matching found that the Virtual KITTI 2 clone performed similarly to the corresponding real sequence under the evaluated conditions.
- 4.3 SfmLearner: SfmLearner experiments evaluated monocular depth and camera pose estimation on Virtual KITTI 1, Virtual KITTI 2, and real KITTI sequences using depth and pose error metrics.
- 4.4 Semantic segmentation: Adapnet++ RGB segmentation remained relatively stable for 15° and 30° camera-angle changes but degraded significantly with fog and rain.
- 4.4 Semantic segmentation: Depth-based segmentation was generally poorer than RGB-based segmentation, while meteorological and lighting variations produced identical depth inputs and identical performance.