Source-linked AI summary
Unity Perception: Generate Synthetic Data for Computer Vision
Steve Borkman, Adam Crespi, Saurav Dhakad, Sujoy Ganguly, Jonathan Hogins, You-Cyuan Jhang, Mohsen Kamalzadeh, Bowen Li, Steven Leal, Pete Parisi, Cesar Romero, Wesley Smith, Alex Thaman, Samuel Warren, Nupur Yadav
TL;DR
Real-world computer-vision datasets can require costly annotation and review, while rare scenarios and privacy concerns complicate collection. Unity Perception provides customizable synthetic-data tools and randomized environments; in a grocery-detection evaluation, adding synthetic data improved performance, with the largest synthetic-plus-real combination performing best.
Problem
Complex vision tasks make annotation and review costly, while rare scenarios and privacy concerns complicate real-world data collection.
Source: §1 · p. 1Method
Unity Perception extends Unity with tools for annotated synthetic datasets and configurable randomization, which SynthDet used to generate grocery images for object detection.
Source: §1 · p. 2, §3 · p. 2, §4.2 · p. 6Results
Synthetic data improved performance at every tested real-data size, and 400K synthetic plus 760 real images performed best on the real-world test set.
Source: §4.2 · p. 7Takeaways & Limitations
Reusable randomized environments support faster dataset and model iteration without repeated real-world collection, cleanup, review, and auditing.
Source: §1 · p. 1
Abstract
from arXiv · showhide
We introduce the Unity Perception package which aims to simplify and accelerate the process of generating synthetic datasets for computer vision tasks by offering an easy-to-use and highly customizable toolset. This open-source package extends the Unity Editor and engine components to generate perfectly annotated examples for several common computer vision tasks. Additionally, it offers an extensible Randomization framework that lets the user quickly construct and configure randomized simulation parameters in order to introduce variation into the generated datasets. We provide an overview of the provided tools and how they work, and demonstrate the value of the generated synthetic datasets by training a 2D object detection model. The model trained with mostly synthetic data outperforms the model trained using only real data.
1. Introduction
Complex vision tasks make data collection, annotation, review, and auditing costly, while rare events and privacy concerns further complicate real-world datasets. Unity Perception proposes controllable synthetic data generation and reusable randomized environments as a faster alternative, packaged with tools for annotated datasets.
1. Introduction
- Complex detection and segmentation tasks require increasingly complex datasets and labels, raising annotation costs and challenges involving cost, bias, and privacy.Annotating objects and pixels, then reviewing and auditing labels, adds effort; rare scenarios are difficult to collect and privacy rules restrict some training data.Source: §1 · p. 1
- Virtual environments can generate controllable examples with scene-derived annotations, including rare cases, while avoiding reliance on private data.Rendering uses compute-time rather than human-time and provides scene information that can bypass manual annotation and review.Source: §1 · p. 1
- Reusable environments allow randomization changes at simulation time, supporting faster dataset and model iteration without collection, cleanup, review, and audit steps.Figure 1 contrasts the human-intensive real-data workflow with synthetic generation from 3D assets and randomized parameters.Source: §1 · p. 1, Figure 1 · p. 2
- Unity Perception extends Unity with customizable tools for generating annotated computer-vision datasets, an extensible randomization framework, and Python support for consuming and parsing datasets.The package targets several common vision tasks and is open source.Source: §1 · p. 2
2. Related Work
Prior datasets and domain-specific synthetic environments have advanced computer vision, but they do not cover every desired scenario or provide a broadly reusable route to new datasets. Unity Perception is presented as a building block for extensible robotics and computer-vision projects.
2. Related Work
- Existing datasets such as PASCAL VOC [16], NYU-Depth V2 [17], MS COCO [18], and SUN RGBD [19] support complex vision research but cannot cover every scenario or offer a path for others to create datasets.Some earlier synthetic-data environments achieved task-specific results, but were often unavailable publicly, limiting reproducibility and extension [22, 23, 24].Source: §2 · p. 2
- SYNTHIA [25], CARLA [26], AI2-THOR [27], Habitat [28], and iGibson [29] support reproducibility but couple simulation to particular domains and tasks.Isaac Sim [30] targets broader robotics simulation, while BlenderProc [31] and NVISII [32] provide APIs for some vision tasks without focusing on physics or simulation.Source: §2 · p. 2
- Unity Perception is positioned as a building block for robotics and computer-vision projects, and SynthDet uses it for grocery detection with synthetic data and mixed-data model evaluation.The paper identifies SynthDet as the first project to use the package.Source: §2 · p. 2
3. The Unity Perception Package
Unity Perception combines ground-truth labeling, configurable randomization, dataset schemas, and analysis tools in an extensible synthetic-data workflow. Its components support local or cloud generation and inspection of generated datasets.
3. The Unity Perception Package
- The package extends Unity with ground-truth dataset generation, domain randomization, extensibility for new labeling methods, and cloud generation through Unity Simulation.It supports 2D and 3D object detection, semantic and instance segmentation, and keypoints; cloud runs can generate millions of annotated images without powerful local resources.Source: §3 · p. 2
3.1. Ground truth generation
- Labelers derive frame-level ground truth from semantic labels on 3D assets and map readable labels to canonical class IDs used by target models.Users can extend Labelers in C# to support additional tasks.Source: §3.1 · p. 3
3.2. Randomization tools
- Scenarios coordinate ordered Randomizers across user-defined iterations, while customizable samplers and schedules control when parameters vary.Users can extend Randomizers and Scenarios, choose provided distributions such as normal or uniform, or draw custom distribution curves; deterministic sampling and ordering support reproducible cloud runs.Source: §3.2 · p. 4
3.3. Schema
- The lightweight JSON schema links simulated sensor outputs with Labeler outputs and their associated metrics through identifiers.Capture and annotation identifiers enable lookup of extra data in metrics files, while annotation definitions describe how to decode annotation records.Source: §3.3 · p. 4, Figure 3 · p. 4
3.4. Dataset Insights
- DatasetInsights provides Python modules to parse, load, transform, visualize, and analyze Perception datasets.Its extensible statistics include object counts, visible pixels, and frame-by-frame ground-truth visualizations, helping users inspect datasets and debug issues before training.Source: §3.4 · p. 5
4. Example Project - SynthDet
SynthDet generated randomized grocery imagery and evaluated object detectors trained with synthetic and real data. On real-world testing, adding synthetic data improved performance across real-data amounts, with the best result from the largest synthetic-plus-real training combination.
4.1. Randomizations
- SynthDet randomized grocery-object selection, placement, density, scale, and rotation, with textured background and foreground occluders that also varied in texture, hue, and rotation.Lighting intensity and color, occasional background overexposure, contrast, saturation, and blur were varied as well.Source: §4.1 · pp. 5–6
4. Example Project - SynthDet
- SynthDet generated 400,000 synthetic images with 2D boxes for 63 grocery items and collected a 1,267-image real dataset of the same targets.The real dataset was split into 760 training, 253 validation, and 254 test images, with smaller training subsets used for limited-data fine-tuning.Source: §4.2 · p. 6
4.2. Results
- Faster R-CNN models were trained using real-only, synthetic-only, and synthetic-pretrained then real-data fine-tuning strategies.The study used a ResNet50 backbone and repeated training with different random initializations.Source: §4.2 · pp. 6–7
- More synthetic data improved performance at each real-data size, and 400K synthetic plus 760 real images performed best on the real-world test set.The study also reports fewer false positives and false negatives and better localization for synthetic plus 760 real data, while complex lighting remained difficult.Source: §4.2 · p. 7, Table 3 · p. 7
5. Future Work
Future work targets lowering barriers to computer-vision research through broader sensor and ground-truth support, simulation-ready assets, improved Linux rendering, and distributed simulation control.
5. Future Work
- These directions are framed as ways to lower entry barriers to computer-vision research and potentially enable new research avenues.The figure compares prediction quality and notes improvements with synthetic data for complex object orientations, configurations, and lighting conditions.Source: §5 · p. 7, Figure 7 · p. 8
- An extensible sensor framework is intended to support radar and lidar environments, including robotics and autonomous vehicles, and to simulate specific real-world sensors.The authors also identify continued improvement of existing Labelers and support for additional ground-truth types, such as frame sequences for object tracking.Source: §5 · p. 7
- Simulation-ready asset standards and validation tools are proposed to address the difficulty of using game or film assets in new domains and tasks.The proposed checks include texture scales, semantically meaningful pivots, and semantic labels mappable to a global taxonomy.Source: §5 · p. 7
- The authors plan to improve Linux compatibility and headless operation for Unity’s High Definition Render Pipeline, which supports advanced global illumination.The stated goal is to match the headless capability already available on Windows.Source: §5 · p. 7
- External control of Perception simulations and distributed compute graphs are proposed to reduce the complexity of scaling experiments across simulation nodes.The planned support includes workloads such as automated domain randomization.Source: §5 · pp. 7–8
Appendix A. Perception package diagrams
The appendix diagrams abstract the Perception package’s ground-truth generation and randomization systems.
Appendix A. Perception package diagrams
- Figures 8, 9, and 10 show high-level component diagrams and a sequence diagram for the Perception package’s ground-truth generation and randomization systems; their detail is abstracted to make them easier to digest.Source: Appendix A · p. 10
Appendix B. SynthDet model performance
The appendix presents SynthDet detection performance measurements alongside diagrams of the package’s ground-truth and randomization systems.
Appendix B. SynthDet model performance
- Camera Labelers generate task-specific ground truth, while LabelConfig maps string labels to object classes and determines which scene labels are tracked.The package includes Labelers for five tasks and allows users to add others by extending CameraLabeler.Source: Figure 8 · p. 11
- The randomization framework uses Scenarios to run Iterations, triggering Randomizers whose Parameters use Samplers to generate typed values.Scenario, Randomizer, and Parameter are extensible, and users can implement ISampler when built-in distributions are insufficient.Source: Figure 9 · p. 11, Figure 10 · p. 12
- Table 4 reports mAP, mAPIoU50, and mARmax=100 on the UnityGrocreies-Real test set, with means and standard deviations over five training runs.It covers combinations of real and synthetic data; the supplied passages do not provide the table’s cell values.Source: Appendix B · p. 10, Table 4 · p. 13