PaperlayerView exact revision on arXiv

Source-linked AI summary

From Unity Simulation to Diffusion-Based Augmentation: Quantifying Dataset Balance for Robust Object Detection

Mohamed Benkedadra, Aissa Saoudi, Maxime Gloesener, Sidi Ahmed Mahmoudi, Matei Mancas

arXiv:2609.38010v1cs.CVcs.AI

TL;DR

Costly and hazardous labeled-data collection constrains object detection in safety-critical settings. The paper benchmarks Unity simulation and diffusion-based augmentation under controlled conditions, finding that limited synthetic or generative inclusion can improve generalization while excessive inclusion induces domain drift.

  • Problem

    In safety-critical settings, labeled-data collection is costly, hazardous, and ethically constrained, and simulation and diffusion generation had not been systematically compared or jointly studied for real-world object detection.

    Source: §1 · p. 1

  • Method

    The study benchmarks Unity simulation and diffusion-based augmentation under controlled data-volume and model settings to examine their separate and combined effects.

    Source: §1 · p. 2

  • Results

    Limited synthetic or generative proportions can enhance generalization, but neither purely synthetic nor purely generative augmentation alone achieves optimal transferability.

    Source: §6 · p. 7, §7 · p. 8

  • Takeaways & Limitations

    Small, well-calibrated proportions of synthetic or generative data can improve generalization, whereas excessive inclusion induces domain drift.

    Source: §7 · p. 8

    The reported gains require out-of-domain evaluation to establish whether they extend beyond intra-domain realism.

    Source: §6 · p. 8

Abstract

from arXiv · show

Modern computer vision models achieve high accuracy when trained on large-scale annotated datasets. In critical domains such as construction safety monitoring, data collection is costly, hazardous, and ethically constrained. This paper presents a systematic study comparing two complementary data generation paradigms, (1) Unity Simulation-based rendering and (2) Controllable Diffusion-based generation (CIA), for object detection under real data-scarce conditions. A unified experimental framework enables controlled dataset mixing across real, simulated, and generative sources, while maintaining identical model and training settings. Quantitative evaluation using Precision, Recall, mAP, and custom $\Delta$-metrics, reveals that neither simulation nor generative augmentation alone achieves optimal transferability. Unity-only training yields an mAP@0.5 drop of $-50\%$ relative to real data, while CIA-only training shows a milder $-16.5\%$ degradation. Hybrid compositions significantly improve performance, with the 90\% real + 10\% Unity configuration achieving the best overall mAP@0.5 of $62.68\%$ ($+7.64\%$ over baseline), and the 90\% real + 10\% CIA configuration maximizing precision at $74.45\%$. Results demonstrate that limited synthetic inclusion enhances generalization, while excessive substitution induces domain drift.

1 Introduction

Safety-critical object detection is constrained by costly and hazardous labeled-data collection, while simulation and diffusion-based generation have not been systematically compared or jointly studied in real-world detection. This work benchmarks both under controlled settings and examines their separate and combined effects across dataset mixtures.

1 Introduction

  • Costly, hazardous, and ethically constrained data collection limits object detection in safety-critical settings, where scenes and lighting vary.The paper frames these systems as data-scarce rather than model-capacity limited.Source: §1 · p. 1
  • Simulation renders controllable, annotated scenes, while diffusion-based generation synthesizes visual variations conditioned on structure, pose, or semantics.The introduction presents these as complementary data-generation paradigms.Source: §1 · p. 1
  • Prior work had not systematically compared these paradigms under controlled conditions or studied them jointly in real-world object detection.The paper identifies this as the gap its study addresses.Source: §1 · p. 1
  • The proposed pipeline combines Unity rendering and CIA-generated augmentations in a fused dataset intended to balance realism and diversity.The paper also describes its pipeline as open-source and designed for broader industrial and academic adoption.Source: Figure 1 · p. 2, §1 · p. 2
  • The study benchmarks Unity simulation and diffusion-based CIA for object detection under controlled data volumes and model settings.It also examines the separate and combined effects of real, simulated, and generated data across dataset mixtures.Source: §1 · p. 2

2 Related Work

Prior work uses simulation and domain randomization to generate controllable training scenes, while diffusion-based methods enable conditioned image synthesis. The paper situates its comparison within continuing challenges around photorealism, generative bias, and representative datasets.

2 Related Work

  • Synthetic rendering offers controllable, cost-efficient scenes and can support rare or dangerous scenarios, with Unity Perception providing automatic ground-truth annotations.The cited Unity toolkit can generate boxes, instance masks, depth maps, and keypoints.Source: §2 · p. 2
  • Domain randomization varies visual factors to encourage reliance on structural cues, but randomized rendering often misses complex real-world photometric and material properties.The paper notes that models trained solely on such data typically underperform on real imagery [1].Source: §2 · p. 2
  • Diffusion models and ControlNet support image generation conditioned on edges, poses, or semantic maps, while CIA adds structural conditioning and quality filtering.The paper also notes that pretrained diffusion models can introduce semantic bias from their training corpora [11] [13] [24].Source: §2 · p. 2
  • The paper aligns its study with data-centric AI work emphasizing dataset quality, balance, and representativeness over model complexity.It empirically examines how synthetic and generative data affect generalization in real-world evaluation.Source: §2 · p. 2

3 Methodology

The methodology compares real, Unity-rendered, and diffusion-generated data through controlled dataset mixtures and a consistent training and evaluation protocol. It quantifies how data origin and mixture composition relate to detection performance on real-world data.

3.1 Data Baseline and Augmentation Strategies

  • CIA generates variants from real images using structural controls and text prompts, then transfers each source image’s annotation to its generated counterpart.The described process also applies image-quality checks and integrates the variants into a dataset intended to maintain semantic alignment.Source: §3.1 · p. 3
  • Unity simulation uses randomized 3D scenes to explore geometric and photometric variation that may be difficult or unsafe to capture in the real world.The pipeline uses Unity Perception; synthetic images can still diverge from real visual statistics because photometric realism is limited.Source: §3.1 · p. 3
  • Mixtures replace portions of the real dataset with Unity and CIA samples, using ratios that sum to one to compare individual and joint effects at identical training volumes.The real, Unity, and CIA datasets are described as having equal cardinality.Source: §3.1 · p. 3

3.2 Model Training and Evaluation Metrics

  • Models are evaluated on held-out real data using Precision, Recall, and mAP, while Δsim→real and Δgen→real characterize simulation and generative performance gaps.The simulation gap concerns visual discrepancies between Unity-rendered and real images; the generative metric captures visual realism and semantic priors from pretrained generative models.Source: §3.2 · pp. 3–4
  • Identical model training and evaluation conditions, combined with controlled mixture ratios, provide a reproducible protocol for isolating data-origin effects.The methodology frames this as measuring trade-offs between realism, controllability, and semantic bias.Source: §3.2 · pp. 3–4

4 Experimental Setup

The experiments use equal-sized real, Unity-rendered, and CIA-generated datasets, train YOLOv11n1 under shared settings, and evaluate on held-out real images. The tested compositions include pure-source and hybrid mixtures.

Dataset

  • The real MOCS subset contains 3,000 labeled construction-site images, and evaluation uses a separate test set of 300 real images.The dataset includes varied outdoor conditions, including differences in illumination, occlusion, and weather.Source: §4 · p. 4
  • Unity scenes were rendered with randomized environmental parameters and automatically exported bounding boxes, instance masks, and class annotations, matching the real dataset’s size.The simulated environment includes human avatars, cranes, and vehicles.Source: §4 · p. 4
  • CIA data used Canny edge extraction, and the three datasets were equal in size; Figure 2 shows real, CIA-generated, and Unity-rendered examples.The paper identifies Canny as the extractor-generator couple that produced the most improvement among those introduced in the original paper.Source: §4 · p. 4, Figure 2 · p. 4

Training and evaluation

  • YOLOv11n1 models were trained for 20 epochs with batch size 64 and 640×640 images, using Adam and identical standard augmentations across runs.The experiments varied real, synthetic, and generative proportions and tested models on a held-out real validation set.Source: §4 · p. 4

5 Results

Results favor limited hybrid substitution over pure synthetic sources: 90% real + 10% Unity leads in mAP@0.5 gain, while 90% Real-CIA peaks in precision.

5 Results

  • Over 50% mAP@0.5 drop relative to the real baseline occurs with simulation-only training, while pure diffusion remains below real-only training.The reported results describe pure diffusion as performing moderately better than simulation-only.Source: §5 · p. 5
  • 0.7445 precision at 90% Real-CIA exceeds the 0.7266 real-only baseline, while Real-Unity mixtures peak in recall, mAP@0.5, mAP@0.5:0.95, and Fitness.The two hybrid regimes show a precision-oriented versus recall-oriented split.Source: §5 · p. 5, Table 1 · p. 5
  • Across dataset-composition experiments involving Real, Unity-simulated, and CIA-generated data, generalization improves by up to +0.076 mAP@0.5 relative to the baseline on a held-out real test set.Source: §5 · p. 5, Table 1 · p. 5

(b) 100% CIA

The results distinguish CIA's photometric realism from Unity's geometric contribution, while showing that limited synthetic shares improve transferability more reliably than larger substitutions.

(b) 100% CIA

  • Figure 3's shared-scale raw counts expose FN→background as the dominant error mode and show its reduction with small Unity/CIA mixes.The 100% Real baseline has false negatives assigned to background; 100% Unity instead collapses toward background predictions.Source: Figure 3 · p. 6, (b) 100% CIA · p. 6
  • 100% CIA retains photometric realism but shows weaker diagonality on structurally and spatially complex classes.CIA mixing suppresses off-diagonal confusions between visually similar categories, while Unity mixing strengthens the main diagonal by increasing true positives.Source: (b) 100% CIA · p. 6
  • 10% substitution yields positive Δsim→real and Δgen→real gains, while higher proportions cause overfitting.Δgen→real decays more slowly than Δsim→real, indicating that CIA data maintains transferability under higher substitution ratios.Source: (b) 100% CIA · pp. 6–7, Figure 4 · p. 7

6 Discussion

The discussion finds that controlled mixtures outperform single-source augmentation, but Unity and CIA contribute differently: simulation broadens geometry while diffusion better preserves photometric realism. Their benefits depend on proportion, and the paper identifies unresolved domain-shift questions for future work.

6 Discussion

  • At 10% substitution, CIA yields the highest precision and Unity the highest recall; neither alone achieves optimal transferability.The precision–recall asymmetry is supported by the confusion matrices.Source: §6 · p. 7
  • Unity-only training drops over 50% in mAP@0.5:0.95 versus the real baseline, while limited simulation expands geometric coverage and reduces overfitting.The discussion attributes the drop to simplified photometric properties and says a 10% simulated fraction improves recall and overall mAP.Source: §6 · p. 7
  • CIA-only training degrades less than Unity-only training, but its photometric realism is paired with comparatively limited recall.The discussion links CIA's higher precision to realistic illumination and texture reproduction.Source: §6 · p. 7
  • A 90% Real + 10% Unity mixture performs best, whereas tri-source compositions can underperform dual-source mixtures as conflicting cues cause representational interference.The discussion suggests progressive mixing from simulation-heavy training toward CIA and real data for photometric alignment.Source: §6 · p. 7
  • Across substitution ratios, generative data maintains positive deltas longer than simulated data, whose faster degradation reflects a different decay pattern.The discussion characterizes simulation as higher-variance diversity and CIA as lower-variance, lower-bias perturbation; together, their utility lies in controlled proportions.Source: §6 · pp. 7–8
  • The discussion identifies out-of-domain evaluation, feature-level analysis, adaptive data selection, and domain-adaptation objectives as future ways to investigate or mitigate residual drift.Out-of-domain tests would assess whether gains extend beyond intra-domain realism.Source: §6 · p. 8

7 Conclusion

The study offers a principled framework for quantifying simulated and generative data contributions to real-world object detection. It concludes that small, calibrated synthetic proportions can enhance generalization, whereas excessive inclusion induces domain drift.

7 Conclusion

  • The study provides a principled framework for quantifying simulated and generative data contributions to real-world object detection.It uses quantitative Δ metrics to evaluate how synthetic and generative sources influence robustness and generalization.Source: §7 · p. 8
  • Small, well-calibrated proportions of synthetic or generative data can significantly enhance generalization, while excessive inclusion induces domain drift.The conclusion states that optimal combinations maximize performance without compromising realism.Source: §7 · p. 8
  • Unity offers geometric variability, CIA offers photometric fidelity, and real data provides semantic grounding; future work will explore adaptive mixing and cross-domain fine-tuning.Source: §7 · p. 8

Related papers

Loading 2609.38010v1…