PaperlayerView exact revision on arXiv

Source-linked AI summary

STPLS3D: A Large-Scale Synthetic and Real Aerial Photogrammetry 3D Point Cloud Dataset

Meida Chen, Qingyong Hu, Zifan Yu, Hugues Thomas, Andrew Feng, Yu Hou, Kyle McCullough, Fengbo Ren, Lucio Soibelman

arXiv:2203.09065v3cs.CV

TL;DR

Large-scale 3D dataset creation is challenging because collection, sanitization, and annotation are difficult, while datasets may have imbalanced classes or low-quality samples. STPLS3D builds a procedurally generated aerial-photogrammetry dataset alongside real surveys, using automatic labels and reconstruction steps modeled on real data collection. Experiments report that synthetic data can strengthen existing models when incorporated into training, while the authors note contextual and appearance gaps between synthetic and real data.

  • Problem

    Producing large-scale 3D datasets is challenging because collection and annotation are demanding, and datasets may have imbalanced classes or low-quality samples.

    Source: §1 · p. 2

  • Method

    STPLS3D procedurally creates synthetic aerial-photogrammetry point clouds from geospatial data and automatically generates semantic and instance annotations.

    Source: §1 · pp. 2–3

  • Results

    All evaluated baselines achieved their best mIoU when trained on real and synthetic data, with KPConv improving by nearly 8% in mIoU score.

    Source: §5.1 · p. 8

  • Takeaways & Limitations

    The authors conclude that synthetic data can serve as data augmentation that strengthens existing models’ learning capacity and generalization ability.

    Source: §7 · p. 10

    Synthetic environments lack sufficient high-level object-context priors and have a visible 2D appearance domain gap from real images.

    Source: §6 · p. 10

Abstract

from arXiv · show

Although various 3D datasets with different functions and scales have been proposed recently, it remains challenging for individuals to complete the whole pipeline of large-scale data collection, sanitization, and annotation. Moreover, the created datasets usually suffer from extremely imbalanced class distribution or partial low-quality data samples. Motivated by this, we explore the procedurally synthetic 3D data generation paradigm to equip individuals with the full capability of creating large-scale annotated photogrammetry point clouds. Specifically, we introduce a synthetic aerial photogrammetry point clouds generation pipeline that takes full advantage of open geospatial data sources and off-the-shelf commercial packages. Unlike generating synthetic data in virtual games, where the simulated data usually have limited gaming environments created by artists, the proposed pipeline simulates the reconstruction process of the real environment by following the same UAV flight pattern on different synthetic terrain shapes and building densities, which ensure similar quality, noise pattern, and diversity with real data. In addition, the precise semantic and instance annotations can be generated fully automatically, avoiding the expensive and time-consuming manual annotation. Based on the proposed pipeline, we present a richly-annotated synthetic 3D aerial photogrammetry point cloud dataset, termed STPLS3D, with more than 16 $km^2$ of landscapes and up to 18 fine-grained semantic categories. For verification purposes, we also provide a parallel dataset collected from four areas in the real environment. Extensive experiments conducted on our datasets demonstrate the effectiveness and quality of the proposed synthetic dataset.

1 Introduction

Large-scale 3D dataset creation is difficult because annotation is costly, surveys can yield small or low-quality clouds, and class distributions can be imbalanced. STPLS3D addresses these challenges with an automatic synthetic aerial-photogrammetry pipeline and a dataset combining real and synthetic point clouds.

1 Introduction

  • Large-scale 3D dataset production is difficult because annotation is labor-intensive, surveys can yield small or low-quality clouds, and class distributions may be imbalanced.The cited data-quality issues include non-uniform density, holes, and outliers.Source: §1 · p. 2
  • The pipeline uses open geospatial sources, varied procedural buildings, and deliberately placed underrepresented objects to produce controllable synthetic environments.It follows real-world UAV flight patterns and photogrammetry reconstruction to create clouds with similar quality and noise to real data.Source: §1 · p. 2
  • Experiments report visible improvement on real data when synthetic data is added to training, even without domain adaptation.The synthetic data spans more than 16 km2, while the parallel real dataset was collected from four areas.Source: §1 · p. 3
  • More than 17 km2 of city landscape is covered by STPLS3D’s richly annotated real and synthetic point-cloud subsets.The dataset includes 62 synthetic point clouds, up to 18 semantic classes, and 14 instance classes; the real subset covers 1.27 km2.Source: §1 · p. 3

2 Related works

Prior work includes real-world 3D datasets collected with several sensor types and synthetic datasets generated in virtual environments. The paper positions STPLS3D as large-scale aerial-scene synthesis using photogrammetry, against limitations in existing synthetic data and quality issues in some released datasets.

3D Real-World Datasets

  • Existing real-world outdoor datasets use terrestrial, mobile, or aerial laser scanners, while some large scene datasets use drone or fixed-wing UAV photogrammetry.Earlier datasets also include indoor RGB-D collections and recent UAV systems combining multiple sensors.Source: §2 · p. 4
  • Released real-world datasets can contain missing vertical-surface points, large holes, non-uniform density, or insufficient and incorrect annotations.The passage attributes these drawbacks to survey configuration and photogrammetry software.Source: §2 · p. 4

3D Synthetic Datasets

  • Earlier synthetic point clouds focused on individual objects, while later outdoor-scene datasets used virtual gaming environments.These gaming environments were manually created and limited in spatial scale.Source: §2 · p. 4
  • Gaming-based synthetic datasets differ from real environments in geometry, noise, and sampling, creating visible domain gaps.The paper contrasts these limitations with its aerial-view photogrammetry synthesis using procedural virtual environments.Source: §2 · p. 4

3 Synthetic Data Generation Pipeline

The pipeline seeks to reproduce real aerial photogrammetry by building realistic virtual environments, rendering aerial images, reconstructing point clouds, and transferring automatic semantic and instance labels. Its design targets similar quality to real clouds while providing rich annotations.

3 Synthetic Data Generation Pipeline

  • The pipeline aims to minimize the synthetic-real domain gap by using realistic virtual assets and reconstructing clouds with quality similar to real ones.Direct surface sampling or ray casting produces point clouds without the real-world quality and noise characteristics described in the paper.Source: §3 · p. 4, §3.2 · pp. 5–6

3.1 Procedural 3D Environment Generation

  • GIS building footprints, road networks, and surface models ground city-like virtual environments, with placement rules for roads, objects, and clustered trees.Distance constraints help avoid object intersections and keep vehicles, signs, and poles near roads.Source: §3.1 · p. 5
  • Procedural building generation varies building types, heights, façades, and architectural elements from OSM footprints using CGA shape-grammar tools.Building types and heights are randomly assigned to broaden variation.Source: §3.1 · p. 5

3.2 2D Image Rendering and 3D Reconstructions

  • Rather than directly sampling surfaces or ray-casting final clouds, the pipeline renders virtual aerial images with UE4 and AirSim, including weather effects, then reconstructs them with ContextCapture.The reconstruction software is kept consistent with that used for real-world photos.Source: §3.2 · pp. 5–6

3.3 Semantic and Instance Annotation

  • Semantic and instance labels are transferred from rendered images through a ray-cast proxy cloud and nearest-neighbor matching to photogrammetry points.Ground-point connectivity constrains transfer to reduce inconsistent projections from simulated wind; some boundary mislabels may remain.Source: §3.3 · p. 6
  • Automatically generated instance labels support applications such as tree segmentation and counting, where manual segmentation of many overlapping trees is difficult or infeasible.The paper also identifies forest management as a potential use.Source: §3.3 · p. 6

4 Datasets

The dataset pairs four manually annotated real-world survey areas with synthetic clouds generated using the same UAV flight pattern. Its synthetic data covers a larger area and can be generated with substantially less time and staffing than the real-world collection described.

4 Datasets

  • Four real sites were surveyed with crosshatch flights, 75%–85% image overlap, and 25–70 m altitudes, then reconstructed and manually labeled with six semantic classes.The sites are USC Park Campus, Wrigley Marine Science Center, Orange County Convention Center, and a residential area.Source: §4 · p. 6
  • The pipeline generated 62 synthetic point clouds across varied environments using the same UAV flight pattern as the real surveys.The dataset includes three synthetic versions with different focuses; examples of these and real data appear in Figure 3.Source: §4 · pp. 6–7, Figure 3 · p. 7

4.1 Comparison

  • More than 16 km2 of synthetic data was generated by one person on one desktop PC within a month, compared with four months of team effort for 1.27 km2 of real data.The real collection effort included permits, planning, repeated flights, processing, and annotation.Source: §4.1 · p. 7

5 Experiments

Experiments show synthetic data improves semantic segmentation on both WMSC and FDc, with real-plus-synthetic training performing best. HAIS outperforms PointGroup on the SyntheticV3 instance benchmark, where outdoor-scene differences remain challenging.

5.1 Evaluation of 3D Semantic segmentation

  • The semantic benchmark compared five methods using mIoU and oAcc across real-only, synthetic-only, and combined training; instance tests tuned voxel size and cluster radius and used weighted loss.All semantic experiments were tested on the real-world WMSC test set; instance metrics were mAP, mAP50, and mAP25.Source: §5.1 · p. 7, §5.2 · p. 9
  • MinkowskiNet reached 46.52% mIoU with real-only WMSC training; all baselines improved with synthetic-only training and peaked with combined training, while KPConv gained nearly 8%.The authors attribute small-object gains being limited or worse to discrepancies between synthetic and real data, including limited object geometry and context.Source: §5.1 · pp. 7–8, Table 2 · p. 8
  • Synthetic augmentation improved ground and building results, but gains for small objects were limited or sometimes negative, which the authors link to domain discrepancy.They identify limited geometric variation and missing comprehensive object-context relationships as issues to address.Source: §5.1 · p. 8
  • Nearly 13%: PointTransformer's FDc mIoU improved when synthetic data augmented training; KPConv led across training settings, and all baselines performed best with real and synthetic data.FDc differs from STPLS3D real data in vegetation, image overlap and quality, and vehicle types.Source: Table 3 · p. 8, §5.1 · p. 9

5.2 Evaluation of 3D Instance Segmentation

  • 40.4 mAP, 51.9 mAP50, and 57.3 mAP25: HAIS outperformed PointGroup on the SyntheticV3 instance-segmentation benchmark.Performance remained far below indoor-dataset results; the authors point to outdoor object-scale differences and unclear boundaries in photogrammetry reconstructions as challenges.Source: §5.2 · p. 9, Table 4 · p. 9
  • STPLS3D is intended to support future development of instance-segmentation methods suited to large-scale outdoor scenes.The authors frame outdoor-scene challenges as motivation for more general and effective methods.Source: §5.2 · p. 10

6 Discussions and Limitations

STPLS3D is intended to support computer-vision tasks beyond semantic and instance segmentation, with point clouds and associated data products to be released. Its synthetic environments lack sufficient high-level object-context priors and retain a visible appearance gap from real-world data.

6 Discussions and Limitations

  • The planned release includes point clouds, source images, annotation masks, camera parameters, depth maps, and meshes, supporting tasks beyond semantic and instance segmentation.The paper names neural rendering, style transfer, scene reconstruction, and object detection as examples.Source: §6 · p. 10
  • Synthetic environments lack high-level object-context priors, and rendered image appearance differs visibly from real-world images; domain adaptation is left for future work.Source: §6 · p. 10

7 Conclusion

The paper presents STPLS3D, a large-scale aerial photogrammetry dataset with real and synthetic point clouds, and a fully automatic pipeline for generating annotated synthetic data. Experiments demonstrate the synthetic data’s quality and suggest it can strengthen model learning and generalization when used for training augmentation.

7 Conclusion

  • STPLS3D combines real and synthetic aerial photogrammetry point clouds with a fully automatic pipeline for generating richly annotated synthetic data.The conclusion describes the dataset and pipeline as the paper’s central contributions.Source: §7 · p. 10
  • Experiments demonstrate the quality and function of the generated synthetic datasets.Source: §7 · p. 10
  • The authors report that synthetic-data augmentation could strengthen existing models’ learning capacity and generalization ability.They describe synthetic data as easy to acquire and free of annotation, and identify domain adaptation and pretraining as possible future directions.Source: §7 · p. 10

B Data collection & generation details

STPLS3D combines real aerial surveys with synthetic point clouds generated across varied landscapes and scene designs. The synthetic photogrammetry pipeline produces point distributions closer to real photogrammetry than ray casting, while parameterized placement rules support diverse layouts.

B Data collection & generation details

  • Four real sites were surveyed with a crosshatch UAV flight pattern, 75%–85% image overlap, and flight altitudes of 25–70 m.The sites span USC, WMSC, OCCC, and a residential area; the selected locations include varied building and terrain types.Source: B Data collection & generation details · p. 18
  • The synthetic dataset contains 62 point clouds across approximately 16 km², with varied landscapes, 60–120 m flight altitudes, and 75%–85% image overlap.Sunlight directions were randomized, and three dataset versions were generated with different focuses.Source: B Data collection & generation details · p. 19
  • SyntheticV1 adds diversity in large objects and terrain details, while SyntheticV2 expands terrain, small-object variety, fine-grained labels, and instance annotations.V1 includes sculpted ground features and forests; V2 adds vegetation height classes, vehicle types, and road-related object placement.Source: B Data collection & generation details · p. 19
  • SyntheticV3 emphasizes large building footprints and material variation, adding grass and dirt ground labels.It assigns materials by building component and material category; class distributions are shown in Figure 5, and point-cloud examples in Figure 6.Source: B Data collection & generation details · p. 19, Figure 5 · p. 20, Figure 6 · p. 20

Comparison of Data Quality

  • Synthetic photogrammetry produces hollow tree-crown shells and point distributions more similar to real photogrammetry than ray-casted clouds.The comparison uses sectional views and volume-density histograms.Source: Comparison of Data Quality · p. 21, Figure 7 · p. 21

D Object placement principles for creating the synthetic environment layouts

  • Parameterized rules place vehicles on roads, city furniture alongside roads, and clusters of small objects near buildings and fences.Randomized shapes, spacing, rotations, and scales create varied layouts with a degree of realism.Source: D Object placement principles for… · p. 22

E Evaluation of the synthetic V1, V2, and V3

The evaluation compares synthetic-subset training combinations on real-world test data using three baselines. All baselines perform best when trained on all three synthetic subsets, and combinations outperform individual subsets overall.

E Evaluation of the synthetic V1, V2, and V3

F Evaluation on Synthetic V3 with 18 labels

On SyntheticV3, KPConv achieved the highest reported segmentation performance, while the baselines struggled with small objects and underrepresented categories. The section also reports an instance-segmentation benchmark with reduced semantic classes, benchmark input settings, category definitions, class mapping, and an annotated FDc visualization.

F Evaluation on Synthetic V3 with 18 labels

  • KPConv achieved 70.35% mIoU on SyntheticV3, ahead of SCF-Net and RandLA-Net, while all three struggled with small objects and underrepresented medium vegetation.The authors attribute small-object difficulty likely to geometry and texture details lost during photogrammetric reconstruction, and note that imbalanced classes remain challenging.Source: Table 6 · p. 23, F Evaluation on Synthetic V3 with 18… · p. 23

G Instance Segmentation with Reduced Semantic Classes

H Data-Related Hyperparameters of Benchmark Methods

  • Benchmark inputs used 0.3 m grid downsampling, with model-specific sample sizes, radii, or point counts.The settings trade off data scale, resolution, and computing resources.Source: H Data-Related Hyperparameters of… · p. 24

I Definition of Semantic Categories

  • Synthetic categories include fine-grained vegetation heights and object types, whereas real-world labels group vegetation as tree and combine several man-made objects.The definitions also distinguish categories such as roads, fences, vehicles, and ground materials across synthetic versions and real data.Source: I Definition of Semantic Categories · pp. 24–25

J Class mapping between synthetic and real data

  • For real-world evaluation, roads, dirt, and grass map to ground; vegetation classes merge, vehicle classes combine, street signs map to light poles, and other non-fence objects map to man-made structures.Source: J Class mapping between synthetic and… · p. 25

K Visualization of the FDc dataset

Related papers

Loading 2203.09065v3…