Source-linked AI summary
Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization
Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, Stan Birchfield
TL;DR
The paper addresses the cost of collecting and annotating varied real training data and the expense of high-fidelity synthetic worlds. It uses domain randomization to train object detectors on varied synthetic scenes, then evaluates real-data fine-tuning. Synthetic domain-randomized data achieves competitive real-world performance, and fine-tuning improves on real-data-only training.
Problem
Collecting and manually annotating sufficiently varied real images is expensive, while high-fidelity synthetic data requires costly detailed environment modeling.
Method
Domain randomization varies synthetic objects, textures, lighting, viewpoints, backgrounds, and distractors, producing automatically labeled training images for real-world object detection.
Results
Synthetic domain-randomized data alone achieves competitive real-world performance, while fine-tuning on real images outperforms training on real KITTI data alone.
Takeaways & Limitations
Domain randomization provides a way to use inexpensive, non-photorealistic synthetic data for training deep networks and improving results with real-data fine-tuning.
Takeaways & Limitations
At high recall, domain randomization consistently has lower precision than Virtual KITTI, likely because simplified scenes omit variations such as parked-car context.
Abstract
from arXiv · showhide
We present a system for training deep neural networks for object detection using synthetic images. To handle the variability in real-world data, the system relies upon the technique of domain randomization, in which the parameters of the simulator$-$such as lighting, pose, object textures, etc.$-$are randomized in non-realistic ways to force the neural network to learn the essential features of the object of interest. We explore the importance of these parameters, showing that it is possible to produce a network with compelling performance using only non-artistically-generated synthetic data. With additional fine-tuning on real data, the network yields better performance than using real data alone. This result opens up the possibility of using inexpensive synthetic data for training neural networks while avoiding the need to collect large amounts of hand-annotated real-world data or to generate high-fidelity synthetic worlds$-$both of which remain bottlenecks for many applications. The approach is evaluated on bounding box detection of cars on the KITTI dataset.
NVIDIA
The passage contains contact information for the authors.
- The contact information lists an NVIDIA email address.
- The address includes four abbreviated names before the NVIDIA domain.
- The passage does not provide substantive research content.
1. Introduction
The paper addresses the cost and difficulty of obtaining varied, manually labeled data by extending domain randomization to real-world object detection. It reports competitive synthetic-only performance and further gains from fine-tuning with real data.
- Manual collection and annotation of sufficiently varied training data is time-consuming, expensive, and sometimes requires difficult-to-specify labels.Examples include 3D poses and pixelwise segmentation, which can take substantial time to label manually.
- High-fidelity synthetic datasets provide automatic labels but often require artists to model detailed environments, undermining synthetic data’s low-cost advantage.
- Domain randomization perturbs simulated environments non-photorealistically to encourage networks to focus on essential image features.
- The paper extends domain randomization to detecting real-world objects and studies synthetic-only performance, real-data augmentation, parameter effects, and comparisons with expensive synthetic datasets.
- The contributions include applying domain randomization to complex-background detection, introducing flying distractors, and investigating parameter importance.
- Synthetic domain-randomized data alone achieves competitive real-world performance, while fine-tuning on real data outperforms training on real KITTI data alone.
2. Previous Work
Previous work used synthetic data for geometric tasks, robotics, and object detection, with approaches ranging from photorealistic rendering to domain randomization and composited real images.
- Many synthetic datasets target optical flow, scene flow, stereo disparity, and camera pose estimation.
- Prior object-detection work used synthetic compositing or photorealistic images, while this paper emphasizes domain-randomized images.
- Earlier findings suggest synthetic data can rival or sometimes beat real data, with additional benefit from fine-tuning on real images.
- Domain randomization was previously applied to estimating object position, indoor quadcopter control, optical flow, and scene flow.
- Other related work studies robotic control, high-fidelity rendering, procedural synthetic data, and classifiers trained from 3D CAD models.
- Pasting real object images onto backgrounds offers an alternative, but accurate object segmentation can be time-consuming.
3. Domain Randomization
The domain-randomization pipeline generates labeled training images by varying scene content, appearance, lighting, and viewpoint. Its deliberately crude images are faster to create and encourage attention to task-relevant structure.
- 3D objects are placed at random positions and orientations, while geometric flying distractors are added to help networks ignore irrelevant scene objects.
- Random textures, lights, camera viewpoints, backgrounds, object configurations, and ground-plane visibility create diverse synthetic scenes.
- The rendered images are paired with automatically generated ground-truth labels such as bounding boxes for neural-network training.
- Figure 1 depicts synthetic cars over random backgrounds, with distractors, lighting, viewpoints, textures, and ground truth.
- The Unreal Engine pipeline outputs 1200 × 400 annotated images at 30 Hz.
- Compared with high-fidelity Virtual KITTI, the crude images are orders of magnitude faster to create and require less expertise.
4. Evaluation
The evaluation compares domain-randomized synthetic training with Virtual KITTI on real KITTI car detection, then examines fine-tuning and randomized-component effects. DR generates greater variety and transfers effectively despite visual dissimilarity from test images.
- Object detection: The study evaluates object detectors trained on DR or Virtual KITTI synthetic images using real-world KITTI testing.The comparison includes three detector architectures and AP@0.5 evaluation.
- Dataset construction: 100K DR images provide substantially more generated training data than the 2.5K-image Virtual KITTI dataset.DR images use randomly selected car models and textures, with up to 14 cars per image.
- Object detection: DR outperforms Virtual KITTI for two detectors, while Faster R-CNN performs better with Virtual KITTI.The result holds despite Virtual KITTI being closely correlated with the test set and DR images being randomly generated.
- Object detection: DR-trained Faster R-CNN detects most cars in real KITTI images without seeing real images during detector training.The network had only ImageNet pretraining for its early layers.
- Precision-recall behavior: DR achieves higher precision than Virtual KITTI across most recall values, but lower precision at high recall.The authors associate the high-recall weakness with distribution mismatch, including omitted image context for parked cars.
- Fine-tuning: Fine-tuning on all 6000 real images gives DR an AP@0.5 of 98.5, 1.6% above Virtual KITTI and 2.1% above real-only training.DR surpasses Virtual KITTI as more real images are added.
5. Conclusion
Domain randomization bridges the reality gap by using inexpensive, non-photorealistic synthetic data to train deep networks for complex tasks. Fine-tuning on real images further improves performance, while future work should expand object and scene variation.
- Domain randomization trains object detectors on synthetic data by intentionally abandoning photorealism and perturbing images to emphasize relevant features.The approach uses randomized synthetic images to encourage networks to focus on task-relevant structure rather than realistic appearance.
- Synthetic domain-randomized data alone achieves performance comparable to more labor-intensive and expensive datasets on complex tasks such as object detection.The conclusion presents this as evidence that domain randomization can bridge the reality gap without requiring photorealistic synthetic worlds.
- Fine-tuning with real images enables domain randomization to outperform more photorealistic datasets and improve on training with real data alone.
- Future directions include adding object models, incorporating scene structure, applying domain randomization to heavily textured objects, and studying mixtures of synthetic and real data.