Source-linked AI summary
Synscapes: A Photorealistic Synthetic Dataset for Street Scene Parsing
Magnus Wrenninge, Jonas Unger
TL;DR
Computer-vision simulation and synthetic-data evaluation need sensor data that matches real-world inputs, while existing evidence on realism, domain shift, and model behavior remains limited. Synscapes addresses this with procedurally generated, photorealistically rendered, richly annotated images and controlled metadata. Across semantic segmentation and object detection evaluations, it performs significantly better than the other datasets and supports fine-grained analysis of existing models.
Problem
Synthetic perception simulation requires sensor data quantitatively and qualitatively equivalent to real inputs, while realism and domain-shift effects need investigation across uses of synthetic data.
Method
Synscapes uses parameterized procedural scenario generation, physically based rendering, unique images, and detailed annotations and metadata to evaluate validation, training, and model behavior.
Results
Synscapes performs significantly better than the other datasets for both semantic segmentation and object detection across validation, training, and in-domain evaluation methods.
Takeaways & Limitations
The results provide strong evidence that realism matters in synthetic data and that richly annotated synthetic images can reveal correlations and biases in organically trained networks.
Takeaways & Limitations
Domain shift remains a limiting factor, especially for training, and virtual worlds must provide enough variation for meaningful learning at scale.
Abstract
from arXiv · showhide
We introduce Synscapes -- a synthetic dataset for street scene parsing created using photorealistic rendering techniques, and show state-of-the-art results for training and validation as well as new types of analysis. We study the behavior of networks trained on real data when performing inference on synthetic data: a key factor in determining the equivalence of simulation environments. We also compare the behavior of networks trained on synthetic data and evaluated on real-world data. Additionally, by analyzing pre-trained, existing segmentation and detection models, we illustrate how uncorrelated images along with a detailed set of annotations open up new avenues for analysis of computer vision systems, providing fine-grain information about how a model's performance changes according to factors such as distance, occlusion and relative object orientation.
1. Introduction
Synthetic data spans lightweight augmentation to fully artificial image generation, including Synscapes and earlier street-scene datasets. Its value includes not only low-cost production but also arbitrary scale, distributions, and annotation detail.
- Synthetic data ranges from augmentation-based variations to images created entirely by artificial means.
- Computer-vision synthetic datasets generally use computer graphics, commonly through game engines or offline physically based rendering.
- Low production cost is a commonly cited benefit of synthetic data, but it can obscure control over scale, probability distributions, and annotation detail.
2. Previous synthetic street scene datasets
Earlier synthetic street-scene datasets differ in their construction, realism, complexity, and relationship to real-world benchmarks. They include Unity-based asset recombination, GTA V environments, and high-level recreations of KITTI.
- Virtual KITTI recreates parts of KITTI with matching building and actor placement and field of view, but has low complexity and realism.
- Synthia uses Unity and off-the-shelf assets recombined into relevant street scenarios.
- Richter et al. used GTA V to create a more complex environment than Synthia, while retaining simplistic geometry and non-photorealistic real-time rendering.
- Playing for Benchmarks extended GTA V-based data with more images and a wider range of annotations.
3. Synscapes overview
Synscapes uses procedural, parameterized scene generation and physically based rendering to produce varied street-scene images with extensive annotations and metadata. Its design supports controlled analysis across scenario and object-level factors.
- 3. Synscapes overview: Synscapes generates an entirely unique scene for each of 25,000 images using an end-to-end realism approach.The approach covers illumination, geometry, materials, camera optics, sensor effects, and processing.
- 3. Synscapes overview: The procedural engine parameterizes 3D-world and image-synthesis factors, sampling scenarios that control objects, road properties, time, and weather.Each scenario is an instantiation of a 3D world defined by a point in the high-dimensional parameter space.
- 3. Synscapes overview: Unbiased path tracing models physically based light transport, surface interactions, and camera-optics scattering for realistic image synthesis.
- 3.2. Annotation images: The dataset provides class, instance, depth, scene, camera, and actor-level metadata alongside RGB images.It includes Cityscapes’ 19 semantic-segmentation training classes and 25,000 images at 1440×720 resolution, with an additional 2048×1024 version.
- 3.4. Distribution of metadata parameters: Independent, decorrelated scenario parameters and unique scene variations support slicing or binning images and instances by weather, object density, occlusion, heading, and distance.
4. Synthetic data in testing and validation
The paper evaluates synthetic datasets as testing and validation environments for semantic segmentation and object detection, emphasizing sensor realism and domain shift. Synscapes shows stronger alignment with real-world models and higher detection performance than GTA in the reported comparison.
- Real-to-synthetic inference is complicated by domain shift, which can make synthetic data score lower than the real-world data used to train models.
- Semantic segmentation: Cityscapes-trained FRRN and DeepLab models achieve their best overall performance on Synscapes among the evaluated synthetic datasets.The consistent ordering across architectures suggests that relative performance reflects dataset differences rather than network architecture alone.
- The evaluation uses Faster R-CNN with ResNet101 for KITTI car and pedestrian detection, alongside FRRN and DeepLab for semantic segmentation.Segmentation models use Cityscapes-pretrained weights and are evaluated on each synthetic dataset and Cityscapes for reference.
- Object detection: Synscapes achieves higher object-detection performance than GTA, with mAP = 0.206 versus mAP = 0.061.Both test sets contain 1,000 images; the paper attributes the difference most likely to more accurate sensor simulation and better feature variation.
5. Synthetic data for training
The paper tests synthetic datasets as sources of training data by training segmentation and detection models under shared settings and evaluating them on real-world data. Synscapes generally provides the strongest transfer results and substantially higher self-validation performance, while domain shift and scene variation remain central constraints.
- Domain shift and insufficient virtual-world variation limit the usefulness of synthetic data as training material.
- The study trains multiple state-of-the-art segmentation and detection architectures with shared hyperparameters to compare dataset strengths and weaknesses on real-world data.
- Semantic segmentation: Synscapes produces the highest overall and per-class Cityscapes validation performance for both FRRN and DeepLab, except for one class.
- Semantic segmentation: Starting from a 76.56% baseline, fine tuning with Synscapes provides more than twice the gain obtained with Richter and Synthia.
- Semantic segmentation: DeepLab self-validation reaches 87% on Synscapes, compared with 63% for Richter and 57% for Synthia.Class-score standard deviation is also lower for Synscapes: σ = 8.25 versus 17.51 and 24.55.
- Object detection: FastBox evaluation also reports a significant performance improvement from using synthetic data.
6. Analysis using Synscapes
Synscapes enables controlled, metadata-rich analysis of how segmentation and detection performance varies with scene and instance factors. The analyses reveal class-specific dependencies on orientation, occlusion, depth, and scenario parameters.
- Analysis framework: Procedural generation and decorrelated metadata enable balanced slicing by scene and instance parameters for algorithm analysis.Images can be binned along one parameter while retaining broad variation across other dimensions.
- Effect of object orientation: Car, Motorcycle, and Bicycle segmentation performs worse for forward/backward orientations than left/right, while Person and Rider are largely direction-invariant.Truck, Bus, and Train also differ between oncoming and same-side instances.
- Effect of object occlusion: Person and Rider segmentation scores highest when unoccluded, whereas Car and Bus score highest under partial occlusion.The authors associate these patterns with differing exemplar balances in Cityscapes, where unoccluded vehicles are relatively uncommon.
- Scenario parameters: Motion blur from ego speed and time of day, represented by sun height, have the strongest effects among the analyzed scenario parameters.Motion blur particularly affects classes with strong vertical features, while reduced contrast near the horizon makes features harder to distinguish; curb height correlates with Sidewalk performance.
- Effect of object orientation: Pedestrian detection is consistent across directions but drops sharply around 50 meters, unlike the segmentation model’s more linear degradation with distance.Car detection also depends on direction, with side views becoming somewhat more reliable beyond 80 meters.
- Effect of object occlusion: For detection, 80% occluded persons at 10 meters are as difficult as unoccluded persons beyond 50 meters, while occlusion matters less beyond 40 meters.Cars are more evenly affected by occlusion, and unoccluded cars are detected more consistently than partially occluded ones.
7. Conclusions and future work
The paper evaluates realism across synthetic-data validation, training, and analysis settings, comparing Synscapes with prior datasets. Synscapes performs better across the evaluated segmentation and detection uses, while future work should separate the effects of specific realism choices.
- Conclusions: The study compares Synscapes with Synthia and a GTA-based dataset to examine the role of realism in synthetic data.Synscapes uses higher-fidelity geometry, textures, and image synthesis, while the GTA-based dataset offers more varied models and environments.
- Conclusions: Synscapes performs significantly better than the other datasets across semantic segmentation and object detection evaluation methods.The consistent advantage provides strong evidence for the importance of realism in synthetic data.
- Conclusions: Richly annotated, unique synthetic images act as a testing proxy for studying Cityscapes- and KITTI-trained networks and discovering correlations and biases.The analysis uses Synscapes annotations and metadata to examine model behavior across controlled factors.
- Future work: Future work should examine how individual realism and sensor-simulation choices contribute, including geometry, textures, materials, illumination, and optics.The paper notes that metrics for separating these contributions have not yet been established.
A.1. Camera metadata
Camera metadata records the dataset’s fixed camera configuration for completeness.
- Camera metadata: Camera intrinsics and extrinsics remain constant throughout Synscapes and are stored under the camera key in each metadata file.
A.2. Scenario metadata
Scenario metadata exposes the variables used to configure each generated image, including geometry, appearance, weather, lighting, motion, and actor counts.
- Scenario metadata: The scene key contains variables driving each image’s scenario configuration.
- Scenario metadata: Geometry fields include altitude variation, wall height, curb height, and fence height or presence.
- Scenario metadata: Ego speed is measured in meters per second and implicitly indicates the amount of overall motion blur.
- Scenario metadata: Scene-structure fields include median presence and the number of visible actors for each class.
- Scenario metadata: Parking angle records whether cars are parked parallel, at 45 degrees, or at 90 degrees, while parking presence indicates whether a parking lane exists.
- Scenario metadata: Intersection distance and sidewalk width are recorded as distances from the ego vehicle or in meters, respectively.
- Scenario metadata: Sky contrast is the natural logarithm of the ratio between the 99th-percentile pixel value and the mean pixel value, with values around 2.0 indicating overcast conditions and around 6.0 sunny midday.
- Scenario metadata: Sun height is normalized from 0.0 at the horizon to 1.0 at zenith, although simulated values never reach full zenith.
A.3. Instance metadata
Synscapes provides per-instance metadata for non-static classes, including 2D and 3D geometry, class identity, occlusion, and truncation.
- Each non-static instance receives an instance id linked to detailed metadata.The covered classes are person, rider, car, truck, bus, train, motorcycle, and bicycle.
- 2D bounding boxes specify normalized horizontal and vertical image coordinates, with depth extents provided in meters.The fields are [xmin, xmax], [ymin, ymax], and [zmin, zmax].
- Oriented 3D bounding boxes encode object position and orientation relative to the ego vehicle reference frame.The origin is at the rear lower right corner; vectors point forward, left, and up, and their lengths define metric extents.
- Class metadata identifies each instance and can also be inferred jointly from class and instance images.
- Occlusion is recorded as the ratio of visible pixels to the fully unoccluded instance area, generally accurate within 1%.
- Truncation records the portion of an instance’s surface area outside the image view, with accuracy similarly within 1%.