Source-linked AI summary
Augmented Reality Meets Computer Vision : Efficient Data Generation for Urban Driving Scenes
Hassan Abu Alhaija, Siva Karthik Mustikovela, Lars Mescheder, Andreas Geiger, Carsten Rother
TL;DR
Large labeled datasets are costly to create, while complete synthetic worlds require substantial effort and may not reproduce real backgrounds. The paper augments real driving images with photo-realistic cars, and reports better generalization than purely synthetic or limited-real training.
Problem
Training high-capacity vision models needs huge labeled datasets, but pixel-level annotation is labor-intensive and complete realistic 3D worlds are difficult to create.
Method
The paper augments real images with photo-realistic car instances using 3D models, scene locations, and environment maps.
Results
Models trained on augmented data generalize better than models trained on purely synthetic data or smaller amounts of manually annotated real data.
Takeaways & Limitations
Combining real backgrounds with flexible synthetic foreground configurations produces highly realistic training datasets for car instance segmentation and object detection.
Takeaways & Limitations
Complete realistic 3D-world creation remains constrained by the time and effort required to model and populate environments.
Abstract
from arXiv · showhide
The success of deep learning in computer vision is based on availability of large annotated datasets. To lower the need for hand labeled images, virtually rendered 3D worlds have recently gained popularity. Creating realistic 3D content is challenging on its own and requires significant human effort. In this work, we propose an alternative paradigm which combines real and synthetic data for learning semantic instance segmentation and object detection models. Exploiting the fact that not all aspects of the scene are equally important for this task, we propose to augment real-world imagery with virtual objects of the target category. Capturing real-world images at large scale is easy and cheap, and directly provides real background appearances without the need for creating complex 3D models of the environment. We present an efficient procedure to augment real images with virtual objects. This allows us to create realistic composite images which exhibit both realistic background appearance and a large number of complex object arrangements. In contrast to modeling complete 3D environments, our augmentation approach requires only a few user interactions in combination with 3D shapes of the target object. Through extensive experimentation, we conclude the right set of parameters to produce augmented data which can maximally enhance the performance of instance segmentation models. Further, we demonstrate the utility of our approach on training standard deep models for semantic instance segmentation and object detection of cars in outdoor driving scenes. We test the models trained on our augmented data on the KITTI 2015 dataset, which we have annotated with pixel-accurate ground truth, and on Cityscapes dataset. Our experiments demonstrate that models trained on augmented imagery generalize better than those trained on synthetic data or models trained on limited amount of annotated real data.
1. Introduction
The paper addresses the need for large, diverse labeled datasets by augmenting real driving images with realistically rendered cars. This preserves real backgrounds while enabling flexible object configurations and improves model generalization.
- Motivation: Deep neural networks require huge labeled datasets, but pixel-level annotation can require up to one hour per image.Existing real-world datasets therefore contain only hundreds or thousands of annotated examples.
- Motivation: Synthetic data reduces labeling demands but realistic 3D environments require substantial modeling effort and may lack fine scene details.The paper highlights approximate modeling of details such as tree leaves and the effort required to create realistic 3D content.
- Approach: The proposed approach augments real images with photo-realistic car instances, retaining background realism while generating arbitrary foreground configurations.It uses real images with side information such as camera calibration and environment maps.
- Approach: Rendered cars provide pose, shape, and appearance variation for training car instance segmentation and bounding-box detection models.The augmented objects are intended to remain visually close to the realism of the input images.
- Results: Models trained on augmented data generalize better than models trained on purely synthetic data or smaller amounts of manually annotated real data.The comparison covers both the paper’s augmented-data setting and the stated synthetic and limited-real baselines.
- Evaluation: The experiments include newly labeled KITTI-15 and KITTI-360 instance-segmentation datasets containing 400 images in total.These datasets provide the paper’s pixel-level evaluation annotations.
2. Related Work
Prior work uses synthetic data for diverse vision tasks, including autonomous driving, but often requires labor-intensive environment modeling. This paper instead targets high-fidelity mixed-reality augmentation of complex scenes.
- Synthetic data: Synthetic datasets have supported tasks including pedestrian detection, human depth estimation, pose estimation, action recognition, and indoor scene understanding.The related work surveys synthetic data across multiple computer-vision applications.
- Autonomous driving: Autonomous-driving datasets such as SYNTHIA and Virtual KITTI provide urban scenes with dense annotations, while other work renders objects on real images for viewpoint estimation.These approaches supply synthetic variation and automatically generated labels.
- Limitations of prior work: Many prior synthetic-data methods require labor-intensive 3D models of complete environments.The paper positions its method as exploiting synergies between real backgrounds and synthetic objects instead.
- Mixed reality: Only a few studies train deep models with mixed reality, including approaches using estimated rendering parameters, random backgrounds, or single rendered objects.Examples span text detection, pedestrian detection, and optical-flow learning.
- Novelty: The proposed approach differs by targeting high-fidelity augmentations of complex multi-object scenes at high resolution.It incorporates scene geometry, environment maps, and image-capture artifacts rather than relying on simplistic object placement or random backgrounds.
3. Data Augmentation Pipeline
The augmentation pipeline renders 3D cars into real scenes using scene geometry and environment lighting, then post-processes composites to better match real-image statistics. Placement strategies are evaluated empirically.
- Pipeline components: The pipeline requires detailed 3D car models, plausible 3D locations and poses, and an environment map for matching scene lighting and reflections.These components jointly support photo-realistic rendering over real scenes.
- Car models: The system uses 28 high-quality car models spanning 7 categories and randomizes car color to increase data variety.The categories include SUVs, sedans, hatchbacks, station wagons, mini-vans, and vans.
- Object placement: Four placement strategies are explored: manual annotations, automatic road segmentation, road-plane estimation, and unconstrained random sampling.The strategies are intended to match plausible distributions of positions, poses, and occlusions.
- Object placement: Manual car-location annotations perform slightly better than automatic road segmentation and on par with road-plane estimation.Manual labeling is used in the experiments unless otherwise stated.
- Rendering and compositing: Cars are rendered with 360-degree KITTI-360 environment panoramas, overlaid on real images, and post-processed to compensate for missing image-formation artifacts.The targeted artifacts include motion blur, lens blur, and chromatic aberrations.
4. Evaluation
The evaluation studies how augmented driving imagery affects segmentation and detection across datasets, varying augmentation scale, realism, placement, and data composition. Augmented data improves generalization while retaining realistic backgrounds and reducing dependence on fully modeled virtual environments.
- Augmentation pipeline: The augmentation pipeline combines high-quality 3D car models, plausible locations and poses, and environment maps, with road segmentation and bird’s-eye annotations supporting placement.Real backgrounds preserve scene appearance while synthetic cars add varied poses, colors, models, and occlusions.
- Evaluation setup: The evaluation covers car instance segmentation and bounding-box detection using real, synthetic, augmented, and combined training data.Models are tested on KITTI-15 and Cityscapes, with Cityscapes providing a more challenging domain shift.
- Real, synthetic, and augmented data: 49.7% performance is achieved with augmented data in one step, compared with 48.2% after VKITTI pretraining and real-data fine-tuning; VKITTI+Aug reaches 51.3%.These results are reported for the comparison in Figure 6 after training and evaluation on the paper’s test sets.
- Dataset size and variability: Augmented data reaches good performance with few real images and improves as dataset size increases, outperforming full real data by a large margin.With a fixed 4000-image augmented dataset, performance highlights the importance of real-background diversity; beyond 4000 images, additional data does not improve performance.
- Realism and rendering quality: Real background imagery is important, while environment-map choice affects instance segmentation performance only minimally.The evaluation also compares black, Flickr, Virtual KITTI, and real backgrounds, and examines post-processing and car-placement strategies.
5. Conclusion
The paper concludes that augmented reality can efficiently enlarge existing data distributions with realistic training images. Its future direction is to extend the approach and improve low-level realism using additional labels or generative modeling.
- Conclusion: Augmented reality efficiently enlarges existing data distributions with realistic images suitable for training deep neural networks.The proposed paradigm preserves realistic input backgrounds while adding rendered objects for data expansion.
- Future work: Future work will extend the method to other datasets and tasks and improve realism using depth, optical flow, or generative adversarial training.These additions are proposed for further fine-tuning low-level image statistics toward real-world imagery.