Source-linked AI summary
Curating Synthetic Data for Task-Specific Visual Perception
Saptarshi Neil Sinha, Paul Julius Kühn, Michael Weinmann
TL;DR
Industrial surface defect detection faces scarce, small and strongly imbalanced datasets because defects are rare and annotation requires expert knowledge. The paper examines procedural rendering, physical-effect simulation, and generative AI, including strategies for combining synthetic and real data. Across the applications, procedural and physical methods reproduce modeled variations, while generative AI can improve target-domain appearance but risks annotation errors and hallucinations.
Problem
Industrial surface defect detection faces scarce, small and strongly imbalanced datasets because defects are rare and annotation requires expert knowledge.
Source: §2 · p. 5Method
The paper examines procedural rendering, physical-effect simulation, and generative AI, including strategies for combining synthetic and real data.
Source: §1 · pp. 3–4, §2 · pp. 5–7Results
Across the applications, procedural and physical methods reproduce modeled variations, while generative AI can improve target-domain appearance but risks annotation errors and hallucinations.
Source: §3 · p. 7Takeaways & Limitations
Hybrid curation can pair model-based reliable ground truth with generative target-sensor appearance, while curated synthetic data complement rather than replace real observations.
Source: §1 · p. 4, §3 · p. 7Physically-based simulation’s achievable diversity is bounded by its model’s expressiveness, so effects not described by the model are absent from the data.
Source: §1 · p. 4
Abstract
from arXiv · showhide
Synthetic data are most valuable where general-purpose datasets cannot provide the domain-specific priors a task requires, and where manual annotation is expensive, imprecise, or infeasible. In this article we argue that the central question for specialized vision systems is not how to generate more data, but which data to generate. We therefore discuss curated synthetic data, whose scene content, appearance variations, sensing characteristics, and annotations are deliberately designed around a given task. We examine three complementary curation paradigms. Procedural rendering offers explicit control over scene parameters and the annotations follow by construction. Physically-based simulation encodes the mechanism behind an observed effect and yields exactly aligned supervision pairs. Generative AI learns sensor-specific appearance from small real seed sets and attains plausible realism, though it remains prone to hallucination and to inaccurate annotation. These paradigms are illustrated with examples from industrial surface defect detection, restoration of degraded digitized autochrome plates, and 6DoF pose estimation from RGB and event data. Using these examples, we analyze different data regimes and training strategies that combine synthetic and real data across these paradigms. We conclude that curated synthetic data are best understood as a complement to real observations, and that hybrid pipelines combining controllable supervision with learned appearance are the most promising direction for reliable sim-to-real transfer.
1 Curation of synthetic data
Curated synthetic data embeds task-specific priors through procedural rendering, physical-effect simulation, or generative AI, trading controllable annotations against target-domain appearance. The section also highlights a hybrid strategy that combines model-based supervision with learned appearance.
Trade-offs and hybrid curation
- At 50% real training data, mixed curated renderings exceed the full real-only baseline; synthetic-weight fine-tuning leads overall by +0.016 YOLO26 and +0.008 LW-DETR.Real-only training collapses below a 50% real share, while mixed training sustains high performance with only 10% real images.Source: Figure 3 · p. 3
Procedural rendering
- Procedural rendering samples parameterized scene programs to provide exact masks, boxes, depth, and metric 6DoF poses, but requires explicit assets and models only specified variations.Rendering budget can make compute the dominant constraint; BlenderProc2 scales poorly to large sets, while real-time simulators such as SPEAR can partly mitigate throughput limits.Source: §1 · pp. 2–3
Physically-based simulation
- Physically-based simulation applies an analytical effect model to intact observations, producing perfectly aligned clean-and-affected pairs with exact affected regions.The achievable diversity is bounded by the model’s expressiveness, and effects it does not describe are not captured.Source: §1 · pp. 3–4
Simulation utilizing generative AI
- Generative AI adapts to a small annotated real seed set to learn target sensor statistics and plausible appearance, using conditioning and filtering rather than scene parameters.Annotation accuracy is the main limitation: promptable segmentation works reliably for compact, well-contrasted structures but not thin, low-contrast ones.Source: §1 · p. 4
Trade-offs and hybrid curation
- Model-based paradigms provide exact annotations only for modeled characteristics, whereas adapted generative simulation matches target appearance at the cost of approximate labels and potential hallucinations.The proposed hybrid pairs reliable model-based ground truth with a generative stage that aligns appearance with the target sensor.Source: Figure 2 · p. 3, §1 · p. 4
2 Applications
The applications show how procedural rendering, generative AI, and physical-process simulation address distinct data and annotation needs across defect detection, image restoration, and pose estimation. Their results illustrate both the value of task-specific synthetic supervision and the limits of synthetic data when real performance is already high.
Generative AI for defect detection
- Beyond a 50% real share, BSData AP50 costs ≤0.02, while union gains +0.007 YOLOv26, +0.027 LW-DETR, and +0.017 YOLOX-S; near-saturated MSD union drops −0.005.The figure reports AP50 on real test splits, with solid curves varying real share in a fixed-size set and dashed curves adding synthetic images to the full real set.Source: Figure 4 · p. 4
Procedural rendering
- ScratchSim combines CAD geometry, procedural scratch maps, a physically motivated material model, and randomized surroundings to generate labels from sampled scratch parameters.Synthetic renderings prevent real-only training from collapsing in scarce-data conditions and surpass the full real-only baseline at half the annotation budget; synthetic-only training remains below that baseline.Source: §2 · p. 5
Generative AI for defect detection
- SynSur adapts a text-to-image model to a small annotated seed set, then uses mask-guided inpainting and filtering to produce 420 annotated images in roughly eight GPU hours.Domain adaptation is essential; prompt engineering alone produces domain-inconsistent defects, and thin low-contrast scratches remain difficult to label reliably.Source: §2 · pp. 5–6
Restoration of degraded autochromes
- Physical simulation of autochrome greening yields aligned clean–degraded pairs at scale, while generative AI produces more plausible defects with weaker control over their spatial extent.Restorations keep intervention confined to affected regions, but conservators must refine and colour-grade the output; expert analysis found roughly one-quarter the time was needed for larger defective areas.Source: §2 · p. 6
6DoF pose estimation
- For strawberry harvesting, procedural scenes produce RGB, depth, masks, and pose labels; a YOLOX 6D Pose model trained only on synthetic data achieves virtually identical ADD-S accuracy on two devices.Inference takes 24.51 ms on an NVIDIA RTX 3090 and 49.59 ms on a Jetson Orin Nano, with the gap dominated by non-maximum suppression.Source: Figure 8 · p. 6, Figure 7 · p. 6, §2 · p. 7
Synthetic event data
- In synthetic event data, polarity is the most important cue, and time-surface encoding more than triples the ADD(-S) 0.5d score versus a plain event histogram.Generating YCB-Ev SD required millions of rendered frames and about one week on six GPUs; validation on real event recordings remains future work.Source: §2 · p. 7
3 Conclusion
Curated synthetic data are most effective when designed around a task’s failure modes, annotation needs, objects, and sensors. They complement real observations, while combining controllable simulation with generative appearance is a promising direction for sim-to-real transfer.
Conclusion
- Task-specific curation helps where real data are scarce, labels are difficult to obtain, or sensing modalities are challenging, but gains shrink once the real task is already solved.Procedural rendering and physical simulation reproduce only modeled variations; adapted generative AI can look more plausible but may introduce annotation errors.Source: §3 · p. 7
- A promising future direction is combining procedural rendering and physical simulation for reliable geometry and supervision with generative AI for realistic textures, lighting, and sensor characteristics.The stated aim is reliable sim-to-real transfer that preserves accurate annotations alongside sensor-captured appearance.Source: §3 · p. 7
- Synthetic data are most effective when curated for the specific failure modes, annotation requirements, objects, and sensors relevant to a task.Source: §3 · p. 7