PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Semantic Segmentation of Urban-Scale 3D Point Clouds: A Dataset, Benchmarks and Challenges

Qingyong Hu, Bo Yang, Sheikh Khalid, Wen Xiao, Niki Trigoni, Andrew Markham

arXiv:2009.03137v3cs.CVcs.AIcs.RO

TL;DR

Precisely segmenting diverse urban 3D objects supports applications including robotics, autonomous driving, and urban modeling. The paper introduces SensatUrban, evaluates existing methods, and identifies challenges in urban-scale semantic learning, including poor performance on several categories and limited cross-city generalization.

  • Problem

    Precisely segmenting diverse objects in 3D scenes is important for competent interaction with the physical world and applications such as robotics, autonomous driving, and urban 3D modeling.

  • Method

    The paper constructs SensatUrban, an urban-scale UAV photogrammetric dataset spanning three UK cities, and studies how existing segmentation pipelines generalize to its large, dense point clouds.

  • Results

    Existing methods achieve unsatisfactory overall segmentation, with KPConv obtaining the highest mIoU scores while categories such as bridge, rail, street, footpath, and bike remain difficult.

  • Takeaways & Limitations

    SensatUrban provides a benchmark for urban-scale semantic learning and highlights challenges involving preprocessing, appearance, class imbalance, and cross-city learning.

  • Takeaways & Limitations

    Cross-city generalization remains an open problem, particularly for minor or morphologically variable categories and unseen urban regions.

Abstract

from arXiv · show

An essential prerequisite for unleashing the potential of supervised deep learning algorithms in the area of 3D scene understanding is the availability of large-scale and richly annotated datasets. However, publicly available datasets are either in relative small spatial scales or have limited semantic annotations due to the expensive cost of data acquisition and data annotation, which severely limits the development of fine-grained semantic understanding in the context of 3D point clouds. In this paper, we present an urban-scale photogrammetric point cloud dataset with nearly three billion richly annotated points, which is three times the number of labeled points than the existing largest photogrammetric point cloud dataset. Our dataset consists of large areas from three UK cities, covering about 7.6 km^2 of the city landscape. In the dataset, each 3D point is labeled as one of 13 semantic classes. We extensively evaluate the performance of state-of-the-art algorithms on our dataset and provide a comprehensive analysis of the results. In particular, we identify several key challenges towards urban-scale point cloud understanding. The dataset is available at https://github.com/QingyongHu/SensatUrban.

1. Introduction

Urban-scale 3D semantic segmentation is needed for applications ranging from robotic interaction and autonomous driving to smart-city planning, but existing datasets and methods leave major scale, imbalance, appearance, and generalization questions. SensatUrban addresses this gap with a large photogrammetric dataset and benchmarks focused on these challenges.

  • Motivation: 3D semantic segmentation supports object-level robotic grasping, scene-level navigation, autonomous driving, and large-scale urban 3D modeling.The paper links these applications to competent machine interaction with the physical world and smart-city planning.
  • Open challenges: Existing datasets are typically smaller than 200 m, whereas urban-scale data spans extremely wide areas collected by aerial platforms.This creates a preprocessing challenge for feeding massive point sets into neural networks.
  • Open challenges: Urban-scale point clouds contain extreme class imbalance because buildings and ground dominate while sparse classes remain under-represented.The paper identifies overcoming this imbalance in neural networks as a central open question.
  • Open challenges: Color-rich aerial point clouds raise the question of whether appearance improves semantic learning without causing networks to overfit geometry-relevant features.The paper later investigates the necessity of colorful appearance for selected categories.
  • Contribution: SensatUrban covers more than 7.6 km^2 across Birmingham, Cambridge, and York, with nearly four billion points and 13 semantic categories manually labeled in Birmingham and Cambridge.The point clouds are generated from high-quality UAV aerial images.
  • Contribution: The paper studies preprocessing, color, class imbalance, and cross-city learning, while exposing challenges rather than thoroughly solving them.Its contributions are a unique urban-scale dataset and an in-depth study of generalizing existing algorithms to massive urban point clouds.

2. Related Work

Prior 3D segmentation datasets span object, indoor-scene, roadway, and photogrammetric settings, while existing learning pipelines differ in how they represent point-cloud data. SensatUrban extends photogrammetric benchmarking to larger urban areas than concurrent Campus3D.

  • Dataset landscape: Existing 3D datasets include object-level models, indoor scene scans, and outdoor roadway-level point clouds.Examples include ModelNet and ShapeNet, S3DIS and ScanNet, and SemanticKITTI and Semantic3D.
  • Photogrammetric datasets: Campus3D also releases large-scale photogrammetric point clouds, but SensatUrban is several times larger in spatial size and labeling points.Both datasets are generated from high-resolution aerial images.
  • Learning pipelines: Existing learning algorithms are grouped into voxel-based, 2D projection-based, and point-based pipelines according to 3D data representation.Voxel methods use 3D CNNs, projection methods learn after mapping points to images, and point-based methods process points directly.
  • Dataset comparison: Table 1 compares representative point-cloud segmentation datasets by spatial size, evaluated and annotated classes, and acquisition system.The acquisition-system abbreviations include MLS, TLS, and ALS.
  • Learning pipelines: Voxel-based methods require substantial computation and memory, limiting scalability to urban-scale point clouds.Projection-based methods similarly rely on 2D CNNs but may lose critical geometric information during projection.

3. The SensatUrban Dataset

SensatUrban is built from coordinated UAV imagery over three UK urban areas, reconstructed into dense colored point clouds and manually labeled using 13 semantically meaningful categories. The resulting dataset combines kilometer-scale coverage with detailed annotations and quality checks.

  • Dataset construction: The dataset is collected and processed over three large urban areas in the UK using UAV photogrammetry.The collection pipeline includes aerial capture, point-cloud reconstruction, and semantic labeling.
  • Collecting Aerial Imagery: Figure 2 depicts nine Cambridge flight plans, with different colors marking UAV paths and a circular route marking takeoff and landing.The plans are collated to cover the survey site.
  • Collecting Aerial Imagery: A fixed-wing Ebee X drone with a SODA camera captures high-resolution imagery along pre-planned grid flight paths.The camera records oblique and nadir photographs to capture vertical surfaces appropriately.
  • Point-cloud reconstruction: Captured image sequences are reconstructed into dense colored 3D point clouds using Pix4D, Structure from Motion, and dense image matching.Ground validation points measured with professional GNSS equipment assess data accuracy and quality.
  • Dataset scale: The reconstructed clouds contain 569,147,075 Birmingham points, 2,278,514,725 Cambridge points, and 904,155,619 York points.The corresponding areas are 1.2, approximately 3.2, and approximately 3.2 square kilometers.
  • Annotating Semantic Labels: The 13 semantic categories are chosen for clear meaning, social or commercial relevance, and geometric or appearance variance.Birmingham and Cambridge are labeled, while York is unlabeled for possible semi-supervised pre-training; labels are manually cross-checked.
  • Annotating Semantic Labels: The categories include ground, vegetation, building, wall, bridge, parking, rail, traffic road, street furniture, car, footpath, and bike.The supplied category list is truncated after bike, so the complete thirteenth class is not stated here.

4. Benchmarks

The benchmark evaluates seven representative 3D point-cloud segmentation methods on tiled SensatUrban data using standard accuracy and IoU metrics. Results show severe class imbalance and unsatisfactory segmentation of several important categories, with no clear method leader on those challenges.

  • Dataset split: 14 Birmingham tiles and 29 Cambridge tiles are split into training, validation, and testing subsets, with tiles approximately 400×400 square meters.Birmingham uses 10/2/2 tiles for training/validation/testing; Cambridge uses 20/5/4.
  • Class distribution: Ground, building, and vegetation comprise more than 50% of points, while bike and rail account for only 0.025%.The distribution demonstrates extreme semantic-class imbalance in the benchmark.
  • Representative baselines: Seven representative baselines span sparse convolution, tangent-plane projection, point-based, superpoint, kernel-point, and efficient architectures.The selected methods include SparseConv, TangentConv, PointNet/PointNet++, SPGraph, KPConv, and RandLA-Net.
  • Evaluation metrics: Overall Accuracy and mean Intersection-over-Union are used as the principal evaluation metrics, alongside mean class accuracy and per-class IoU.Table 2 reports OA, mAcc, mIoU, and per-class IoU scores.
  • Benchmark results: PointNet performs worst and KPConv achieves the highest mIoU, yet bridge, rail, street, and footpath remain poorly segmented.The bike category is entirely unsegmented by all methods, and challenging categories show no clear leading technique.

5. Challenges

The study evaluates how data preparation, RGB appearance, class imbalance, and cross-city transfer affect urban-scale point-cloud segmentation. Existing methods improve under suitable preprocessing and color inputs, but minority-class recognition and generalization remain difficult.

  • 5.1. Data Partition: Number-constant input sets outperform constant-volume sets for both PointNet and RandLA-Net, while grid downsampling is marginally better than random downsampling.These findings come from eight experiments using representative PointNet and RandLA-Net baselines.
  • 5.1. Data Partition: Grid downsampling with number-consistent block partition improves mIoU scores by up to 10%.The experiments compare grid or random downsampling with constant-number or constant-volume input sets.
  • 5.2. Geometry vs. Appearance: Coordinates plus RGB significantly improve PointNet, PointNet++, KPConv, and RandLA-Net segmentation accuracy over coordinates alone.Several urban classes, including bridge, footpath, and water, are difficult to distinguish using only 3D coordinates.
  • 5.3. The Impact of Imbalance Class Distribution: Advanced loss functions improve mIoU by up to 5%, while weighted cross-entropy with sqrt frequency improves RandLA-Net bike performance by more than 20%.The authors conclude that off-the-shelf losses alleviate imbalance to an extent, but performance remains hardly satisfactory.
  • 5.4. Cross-City Generalization: Cross-city testing produces mIoU gaps of up to 20% for most approaches, with rail, street, and water generalizing worst.Major categories such as ground and building do not show similarly severe performance drops.

6. Summary and Outlook

The paper concludes that SensatUrban provides a foundation for studying urban-scale 3D segmentation while exposing unresolved challenges in scalability, appearance, imbalance, and generalization. It also points to broader segmentation and real-time mapping directions.

  • 6. Summary and Outlook: SensatUrban contains two accurately labelled regions covering 4.4km^2 and an unlabelled region covering 3.2km^2.The unlabelled region is provided for self- or semi-supervised learning schemes.
  • 6. Summary and Outlook: Benchmarking identifies sampling and partitioning, RGB acquisition, class imbalance, and unseen-scenario generalization as open challenges.The outlook also names instance-level and panoptic segmentation as pressing challenges.
  • 6. Summary and Outlook: The authors identify real-time photogrammetric reconstruction and segmentation, accurate 3D maps, smart cities, intelligent transport, and digital twins as future considerations.These directions are framed around emerging urban and cyberphysical applications.

A. Details of the Data Collection

The dataset is reconstructed from aerial imagery using structure-from-motion and validated through professional GNSS RTK surveying. Reported reconstruction accuracy is at the centimetre scale.

  • A. Details of the Data Collection: Structure-from-motion reconstructs the point cloud from 2D aerial images, while orthomosaics are used only for visualization.The method recovers camera extrinsic parameters for each image.
  • A. Details of the Data Collection: GNSS RTK manual surveying validates the data, yielding horizontal and vertical RMSEs of ±50mm and ±75mm, respectively.The passage compares this positioning accuracy with LiDAR point clouds at around 5 to 10 cm.

B. Details of the Data Annotation

The authors annotate all points directly in 3D, cross-check the annotations manually, and merge initially fine-grained categories when some contain too few points. Instance labels are deferred.

  • B. Details of the Data Annotation: All points are labelled in pure 3D with no unassigned points discarded, and annotations are manually cross-checked.The authors report that adding instance annotations would be meaningful but require substantial labeling effort.
  • B. Details of the Data Annotation: An initial 31-category labeling scheme is merged into fewer similar categories because some categories contain scarce data points.The initial labels, merged labels, and detailed mapping are released with the dataset.

C. Visualization of the Dataset

The urban-scale point clouds are divided into non-overlapping tiles, with Birmingham and Cambridge tiles visualized for intuitive dataset inspection. Zoomed-in urban scenes from York are also shown.

  • The complete urban-scale point clouds are divided into several non-overlapping tiles.The tiling follows an approach similar to DALES.
  • Birmingham and Cambridge tiles are visualized to provide an intuitive understanding of the data.
  • Zoomed-in urban scenes from the York data are also presented.

D.1. Pre-training on pretext task

The paper evaluates self-supervised pretraining schemes on SensatUrban using baseline networks. Object-level pretraining improves fine-tuned performance, particularly for minority categories, while large-scale extension remains difficult.

  • Two pretraining schemes, occlusion completion and context prediction, are evaluated on SensatUrban using three baseline networks.
  • Object-level pretraining still improves fine-tuned performance on the urban-scale dataset.
  • Minority categories such as rail and bridge show performance improvements of up to nearly 10%.The paper attributes this to reduced overfitting to majority categories compared with training from scratch.
  • Existing pretraining paradigms remain limited to object-level point clouds and are non-trivial to extend to large-scale point clouds.The authors release unlabeled York point clouds to encourage further research.

E. Qualitative Results

The paper presents qualitative and quantitative views of SensatUrban and baseline predictions. Dataset visualizations cover Birmingham, Cambridge, and unlabeled York, while Figure 8 compares several methods qualitatively.

  • The paper states that detailed quantitative results are provided in Section 5.2, alongside the qualitative baseline visualizations.
  • Table 8 reports OA, mAcc, mIoU, and per-class IoU for OcCo, Jigsaw, and random initialization using PointNet, PCN, and DGCNN encoders.
  • Figures 5 and 6 show Birmingham and Cambridge dataset splits with semantic classes encoded by different colors.
  • The York split is unlabeled and is provided for possible semi-supervised or self-supervised pretraining.It covers varied urban elements including train stations, churches, stadiums, and highways.
  • Figure 8 presents qualitative test-set results for PointNet, PointNet++, RandLA-Net, and KPConv against ground-truth labels.Black dashed boxes highlight inconsistent predictions with the ground truth.
Loading 2009.03137v3…