Source-linked AI summary
Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data
Aayush Prakash, Shaad Boochoon, Mark Brophy, David Acuna, Eric Cameracci, Gavriel State, Omer Shapira, Stan Birchfield
TL;DR
The paper addresses the cost of labeled vision data and the difficulty of detecting small, occluded, or truncated objects when context is needed. It introduces structured domain randomization, which generates synthetic data using scene structure, and reports superior KITTI detection results against synthetic and cross-domain real-data baselines, with additional gains when combined with real KITTI data.
Problem
Labeled vision data is costly, while challenging KITTI objects are small, occluded, or truncated and require surrounding scene context.
Method
SDR generates synthetic training images by sampling structured scene components and placing objects according to contextual scene structure rather than context-independent random placement.
Results
SDR outperforms VKITTI, Sim 200k, DR, and BDD100K on KITTI Easy, Moderate, and Hard vehicle detection, while SDR combined with real KITTI data outperforms real KITTI data alone.
Takeaways & Limitations
Context-aware synthetic data supports competitive real-data vehicle detection and can improve training when labeled KITTI data is also available.
Abstract
from arXiv · showhide
We present structured domain randomization (SDR), a variant of domain randomization (DR) that takes into account the structure and context of the scene. In contrast to DR, which places objects and distractors randomly according to a uniform probability distribution, SDR places objects and distractors randomly according to probability distributions that arise from the specific problem at hand. In this manner, SDR-generated imagery enables the neural network to take the context around an object into consideration during detection. We demonstrate the power of SDR for the problem of 2D bounding box car detection, achieving competitive results on real data after training only on synthetic data. On the KITTI easy, moderate, and hard tasks, we show that SDR outperforms other approaches to generating synthetic data (VKITTI, Sim 200k, or DR), as well as real data collected in a different domain (BDD100K). Moreover, synthetic SDR data combined with real KITTI data outperforms real KITTI data alone.
I. INTRODUCTION
The paper addresses the difficulty of detecting small, occluded, and truncated objects using synthetic training data by adding scene structure and context to domain randomization. SDR places objects according to contextual structure and achieves strong KITTI detection results against synthetic and cross-domain real-data baselines.
- Large labeled datasets are costly to create, especially for tasks with difficult annotations such as instance segmentation, optical flow, depth estimation, and even 2D detection.
- Domain Randomization reduces dependence on artistic environment design or prior real data, but earlier results were limited to larger KITTI Easy objects.
- KITTI Moderate and Hard include small, occluded, and significantly truncated objects that occupy few pixels and require surrounding scene context.
- SDR adds scene structure and context to DR, training object detectors using only SDR-generated synthetic data.
- SDR achieves state-of-the-art KITTI 2D detection across Easy, Moderate, and Hard, outperforming VKITTI, Sim 200k, and real BDD100K data from another domain.
II. RELATED WORK
Prior work uses synthetic data for many vision tasks and for object detection through realistic scene replicas, game-based simulation, compositing, and domain randomization. The paper positions SDR as addressing DR's data, feature-learning, and context limitations for small-vehicle detection.
- Synthetic training data has benefited vision tasks including optical flow, scene flow, classification, stereo, segmentation, pose, keypoint extraction, and 3D reconstruction.
- VKITTI trains detectors on synthetic replicas of KITTI videos, while GTA-based Sim 200k uses synthetic GTA V data and reports benefits from photorealism.
- Compositing approaches improve object-background integration or robustness to object boundaries through noise, blurring, blending, and staged network training.
- Domain Randomization varies textures, colors, lighting, camera position, and object placement rather than prioritizing realistic rendering.
- Earlier DR work required substantial data, made feature learning difficult, and lacked context for detecting small vehicles; SDR targets these limitations.
III. STRUCTURED DOMAIN RANDOMIZATION (SDR)
SDR procedurally generates synthetic images by sampling global scene parameters, constructing context splines, and placing objects conditionally on those splines. It preserves scene structure while randomizing appearance and geometry, unlike context-independent DR.
- SDR generates synthetic images with three component types: global parameters, context splines, and objects placed along the splines.
- In the SDR dependency structure, a scenario determines global parameters, which govern context splines; objects are placed on the splines and the image is rendered from all components.
- The generator samples scenarios such as rural, suburban, and urban roads, then determines spline shape, lighting, time of day, and other global parameters.
- Context splines represent lanes, medians, sidewalks, gutters, and side stretches, sharing shape while receiving randomized materials and modeled imperfections.
- Objects are associated with suitable splines: vehicles use lanes, pedestrians and cyclists use sidewalks, and buildings, houses, and signs use side stretches.
- Unlike DR's context-independent placement, SDR respects context-boundary geometry while randomizing position, texture, lighting, saturation, and scene geometry.
- SDR uses an Unreal Engine 4 scene generator and exporter to produce rendered scenes, depth, segmentation masks, bounding boxes, truncation, and occlusion labels.
IV. EVALUATION
The evaluation tests SDR for KITTI vehicle 2D bounding-box detection against synthetic-data methods and real data from the same or another domain, and also examines initialization and ablations.
- SDR is evaluated on KITTI vehicle 2D bounding-box detection against other synthetic-data approaches and real data from the same and another domain.
- The evaluation also studies SDR as a network initialization strategy and includes an ablation study.
A. Comparative Study
The comparative study evaluates Faster-RCNN trained on several synthetic datasets for vehicle detection on real KITTI images. SDR performs strongly in difficult detection settings, including occlusion and limited synthetic-data regimes.
- Experimental setup: Faster-RCNN was trained on VKITTI, Sim 200k, DR, and SDR datasets, with DR and SDR each containing 25k images.VKITTI contains 21k images, while Sim 200k contains 200k images; all experiments used standard data augmentation.
- Comparative performance: SDR outperforms other synthetic datasets on KITTI Easy, Moderate, and Hard vehicle-detection criteria.Performance is evaluated using AP at 0.7 IoU on 7,500 real KITTI images.
- Comparative performance: More than 2x better results than DR are reported for SDR, especially because context helps detect smaller objects in Moderate and Hard categories.DR performs well on larger Easy objects but poorly on smaller objects requiring contextual information.
- Qualitative results: SDR-trained networks detect complicated and severely occluded vehicles despite never seeing a real KITTI image during training.Figure 4 distinguishes detections with green boxes from ground truth with black boxes.
- Dataset size: 43.7 AP is achieved with 1,000 SDR images, while SDR performance saturates around 10k images and DR around 50k images.These results concern KITTI Hard vehicle detection evaluated at 0.7 IoU.
B. Domain Gap
The study compares synthetic and real training data across domains using a 1,500-image KITTI evaluation subset. It finds that domain differences affect transfer performance, while SDR exceeds real data from another domain.
- Cross-domain comparison: Synthetic data remains difficult to outperform real KITTI data when training and test images share the same distribution.The authors distinguish this same-domain comparison from transfer across different real-world datasets.
- Domain gaps: The results indicate both a synthetic-to-real reality gap and substantial domain gaps among real-world datasets.Poor performance from BDD100K-trained models on KITTI is cited as evidence of cross-domain mismatch.
C. SDR as an Initialization Strategy
The paper also evaluates SDR as pretraining for settings with limited labeled KITTI data. SDR initialization consistently improves performance over training on the available real KITTI images alone.
- Initialization and fine-tuning: Training solely with SDR yields 52.5 AP on the 1,500-image real KITTI evaluation subset.This value is reported before fine-tuning with additional real KITTI images.
- Initialization and fine-tuning: SDR initialization is especially beneficial when only a small labeled real dataset is available.Figure 5 reports that the improvement is most pronounced with few labeled real images.
- Initialization and fine-tuning: SDR plus real KITTI data performs better than KITTI-only training across the evaluated amounts of labeled real data.The experiment fine-tunes SDR-pretrained networks using subsets of the remaining 6,000 KITTI images.
- Initialization and fine-tuning: SDR plus real KITTI data also outperforms DR plus real KITTI data, particularly for smaller labeled datasets.This comparison supports SDR as a stronger initialization strategy than DR in low-label settings.
D. Ablation Study
The ablation study evaluates how individual SDR components affect detection, finding context, scene variety, appearance variation, lighting, and pose diversity all contribute to performance. Random saturation has the largest effect among the tested parameters.
- SDR ablates context, scene randomization, high contrast, random saturation, random light, and multiple pose.The study uses the same Faster-RCNN network, ResNet V1 feature extractor, and validation set across experiments.
- Random 2D backgrounds replace structured 3D scene elements, showing the importance of context.The structured scene normally includes roads, sidewalks, trees, and other 3D objects.
- 46.0 AP, 47.7 AP, and 51.9 AP result from rural, suburban, and urban-only scenes, respectively, revealing the importance of scene variety.The scene-randomization ablation uses only one scenario type at a time.
- Removing random saturation has the largest effect, suggesting that the texture gap between real and synthetic data is important.The study also tests fixed contrast, single-time-of-day lighting, and lane-constrained vehicle poses.
- Fixed vehicle poses degrade parked-vehicle and side-street detection, while fixed daylight removes lighting variation.The pose ablation keeps vehicles within lanes, and the lighting ablation uses broad daylight only.
V. CONCLUSION
The conclusion presents SDR as a context-aware extension of domain randomization for vehicle detection. It reports stronger KITTI performance than other synthetic and cross-domain real data, and improved results when SDR pretraining is combined with real data.
- SDR imposes scene structure on domain randomization to provide contextual information during training.For vehicle detection, it places vehicles on roads so the network can learn their relationship to the scene.
- SDR improves vehicle-detection performance significantly over standard domain randomization.
- SDR achieves state-of-the-art vehicle-detection results on KITTI compared with other synthetic data and real data from a different domain.
- Pretraining on SDR improves results from real data.The conclusion also identifies multiple object classes and additional vision tasks as future research directions.