PaperlayerView exact revision on arXiv

Source-linked AI summary

The Hessigheim 3D (H3D) Benchmark on Semantic Segmentation of High-Resolution 3D Point Clouds and Textured Meshes from UAV LiDAR and Multi-View-Stereo

Michael Kölle, Dominik Laupheimer, Stefan Schmohl, Norbert Haala, Franz Rottensteiner, Jan Dirk Wegner, Hugo Ledoux

arXiv:2102.05346v2cs.CV

TL;DR

Labeled 3D datasets are scarce for training and evaluating semantic segmentation. This paper presents the H3D benchmark of high-resolution point clouds and textured meshes; its UAV point-cloud density is about a hundred times higher than V3D’s, and the results indicate potential for testing machine-learning approaches.

  • Problem

    Labeled datasets are lacking for training and evaluating semantic segmentation approaches.

    Source: §1 · p. 2

  • Method

    The paper presents a benchmark on semantic segmentation of high-resolution 3D point clouds and textured meshes.

    Source: §4 · p. 26

  • Results

    H3D’s point density is about a hundred times higher than V3D’s, allowing an expanded class catalog; H3D(Mesh) also covers more area than H3D(PC).

    Source: §1 · pp. 3, 5

  • Takeaways & Limitations

    The results indicate that H3D has potential for testing machine-learning approaches.

    Source: §4 · p. 26

    The baseline-based conclusion is scoped to H3D’s detailed structures and expanded class catalog.

    Source: §3.3.1 · p. 23

Abstract

from arXiv · show

Automated semantic segmentation and object detection are of great importance in geospatial data analysis. However, supervised machine learning systems such as convolutional neural networks require large corpora of annotated training data. Especially in the geospatial domain, such datasets are quite scarce. Within this paper, we aim to alleviate this issue by introducing a new annotated 3D dataset that is unique in three ways: i) The dataset consists of both an Unmanned Aerial Vehicle (UAV) laser scanning point cloud and a 3D textured mesh. ii) The point cloud features a mean point density of about 800 pts/sqm and the oblique imagery used for 3D mesh texturing realizes a ground sampling distance of about 2-3 cm. This enables the identification of fine-grained structures and represents the state of the art in UAV-based mapping. iii) Both data modalities will be published for a total of three epochs allowing applications such as change detection. The dataset depicts the village of Hessigheim (Germany), henceforth referred to as H3D. It is designed to promote research in the field of 3D data analysis on one hand and to evaluate and rank existing and emerging approaches for semantic segmentation of both data modalities on the other hand. Ultimately, we hope that H3D will become a widely used benchmark dataset in company with the well-established ISPRS Vaihingen 3D Semantic Labeling Challenge benchmark (V3D). The dataset can be downloaded from https://ifpwww.ifp.uni-stuttgart.de/benchmark/hessigheim/default.aspx.

1. Introduction

Geospatial semantic segmentation is constrained by scarce labeled 3D data, while full expert annotations are valuable for evaluating segmentation models. H3D addresses this gap with a high-resolution airborne benchmark pairing LiDAR point clouds and textured meshes.

1. Introduction

  • H3D(Mesh) covers a larger area than H3D(PC).Source: §1 · p. 3
  • H3D is presented as the first ultra-high-resolution, fully annotated 3D dataset acquired with LiDAR and cameras integrated on one UAV.The paired representations are the H3D(PC) point cloud and textured H3D(Mesh).Source: §1 · p. 5
  • Combining the acquisition methods offers possibilities for high-accuracy georeferencing and semantic segmentation.Source: §1 · p. 5
  • H3D’s point density is about a hundred times higher than V3D’s, allowing an expanded class catalog.The catalog is refined from V3D’s, and the higher density allows finer details to be distinguished.Source: §1 · p. 5

2. The H3D Dataset

H3D depicts Hessigheim, Germany, which was surveyed across three epochs; this paper covers the first, March 2018, because only that epoch had been labeled. A UAV-mounted LiDAR and camera setup produces point-cloud and textured-mesh representations.

2. The H3D Dataset

  • Only the March 2018 epoch is covered in this paper because later epochs had not yet been labeled.The March 2018, November 2018, and March 2019 datasets are all intended for public release.Source: §2 · p. 6

2.1. Capturing H3D

  • The survey used a RIEGL VUX-1LR scanner and two oblique Sony Alpha 6000 cameras integrated on a RIEGL Ri-copter.Source: §2.1 · p. 6
  • At 50 m above ground, the setup achieved a laser footprint under 3 cm and camera ground sampling distance of 2–3 cm, producing H3D(PC) and H3D(Mesh).Source: §2.1 · p. 6

2.2. H3D(PC)

H3D(PC) combines dense UAV LiDAR coverage with multiple measured attributes and manually assigned class labels. Its high point density supports more comprehensive 3D scene analysis than conventional ALS datasets.

2.2. H3D(PC)

  • The H3D survey used 11 longitudinal (north–south) strips and several diagonal strips; scanner parameters were set to achieve about 5 cm point distance in both directions.Source: §2.2 · p. 6
  • About 800 pts/m² are achieved in the complete point cloud through strip overlap, compared with about 400 pts/m² for a single strip.Additional stabilization strips produce significantly higher densities in some areas.Source: §2.2 · p. 7
  • The high-resolution point cloud allows more comprehensive 3D scene analysis than conventional ALS datasets.LiDAR attributes include XYZ coordinates, echo number, number of echoes, and reflectance; up to six echoes were recorded per pulse, with second and third echoes most common.Source: §2.2 · p. 7
  • LiDAR reflectance is provided as range-corrected intensity but is not corrected for differences caused by laser-beam incidence angles.RGB values are transferred from mesh textures to LiDAR points by nearest-neighbor interpolation in 3D, a simple approximation of occlusion-aware projection.Source: §2.2 · p. 10

2.3. H3D(Mesh)

H3D’s mesh combines LiDAR and imagery for reconstruction and uses oblique images for texturing. It is distributed in tiled textured and labeled forms.

2.3. H3D(Mesh)

  • Fusing LiDAR and imagery produces a more complete mesh than imagery alone, particularly where urban canyons are difficult to reconstruct from images.The paper notes that a single received LiDAR echo can suffice for reconstructing such areas, while imagery requires visibility in at least two images.Source: §2.3 · p. 10
  • Oblique images texture the mesh and allow realistic representation of vertical faces.The mesh was generated with SURE.Source: §2.3 · p. 10
  • Each textured tile includes geometry, material properties, and a texture atlas; its labeled counterpart uses class-colored material properties without a texture atlas.Centers of Gravity are also provided for each face, with transferred labels in a point cloud.Source: §2.3 · p. 12

2.4. Class Catalog

H3D uses a shared fine-grained class catalog for its point cloud and mesh, refining the V3D-based catalog for the dataset’s higher point density and monitoring purpose.

2.4. Class Catalog

  • The H3D class catalog is based on V3D but refined to capture more detail, drawing on H3D’s higher point density and the Hessigheim project’s monitoring purpose.The monitoring purpose includes the shipping lock depicted in Figure 1.Source: §2.4 · p. 12
  • H3D adds Urban Furniture, Soil/Gravel, Vertical Surface, and Chimney to distinguish details not represented in the V3D catalog.The same fine-grained catalog is used for H3D(PC) and H3D(Mesh).Source: §2.4 · p. 12

2.5. Generating Ground Truth Data for H3D

H3D provides labeled point-cloud and mesh data for training and evaluating semantic segmentation. Point-cloud labels are manually segmented and checked; mesh labels are transferred from points, leaving some mesh areas unlabeled.

2.5. Generating Ground Truth Data for H3D

  • The dataset’s stated objective is to provide multi-temporal, multi-modal labeled data for training and evaluating machine-learning systems on semantic point-cloud segmentation.The mesh inherits the point cloud’s class catalog.Source: §2.5 · pp. 12–13
  • Point-cloud ground truth was created by manually grouping homogeneous point-cloud segments into semantic classes, followed by cross-checking between student assistants and author verification.Manual annotations remain error-prone, and label noise cannot be avoided.Source: §2.5 · pp. 12–13
  • Mesh labels are transferred from the manually annotated point cloud through geometry-driven associations between points and faces, with each face assigned the majority label of its associated points.Faces without associated points remain unlabeled and receive pseudo class label −1.Source: §2.5 · pp. 13–14
  • For the relevant LiDAR–mesh overlap, 84% of the surface carries annotations; unlabeled faces cover about 40% of the entire mesh, and 99.7% of unlabeled area lies where the mesh exceeds the labeled point cloud.Source: §2.5 · p. 14

2.6. Data Splits

H3D divides each epoch into training, validation, and test areas with identical splits across point-cloud and mesh modalities. Training and validation labels are available, while test labels are withheld.

2.6. Data Splits

  • The training, validation, and test areas are distinct, and the splits are identical across modalities and selected to match mesh tiling.Source: §2.6 · p. 14
  • Test labels are withheld from participants, while training and validation labels may be used as desired and predefined splits are recommended.Source: §2.6 · p. 14
  • The point-cloud class distribution is dominated by classes covering large areas; Tree has additional points from multiple echoes, while Vehicle and Chimney are most underrepresented.Source: §2.6 · p. 14
  • Mesh distributions have fewer ground-class instances because planar surfaces use large faces, while rough Tree surfaces are represented by more surface elements.Source: §2.6 · p. 14
  • Table 2 reports training and validation class frequencies for both modalities and mesh covered area; Figure 6 shows point-cloud training and validation distributions.Source: Table 2 · p. 15, Figure 6 · p. 16, §2.6 · p. 16

2.7. Benchmark Challenge

H3D challenges participants to classify both point clouds and textured meshes using the training and validation data, then submit test predictions for evaluation. Mesh predictions are returned as labeled center-of-gravity points that link labels to faces in the OBJ files.

2.7. Benchmark Challenge

  • The challenge covers both H3D(PC) and H3D(Mesh), with supervised approaches developed from training and validation data and applied to the test set.Source: §2.7 · p. 17
  • Participants return predicted labels to the authors for evaluation; mesh predictions are submitted as labeled center-of-gravity point clouds.Source: §2.7 · p. 17
  • For mesh evaluation, the authors match each predicted per-face label to the corresponding OBJ face, using the center-of-gravity cloud as an efficient link while keeping submitted data memory low.Source: §2.7 · p. 17

2.8. Evaluation Metrics

The benchmark evaluates point-cloud and mesh segmentation using confusion-matrix-derived class metrics and overall scores. Mesh evaluation accounts for the covered area of correctly and incorrectly classified faces.

2.8. Evaluation Metrics

  • The H3D team derives confusion matrices to compute per-class true positives, false positives, and false negatives, then calculates precision, recall, and F1-score.Source: §2.8 · p. 17
  • Overall Accuracy and macro-F1 summarize classifier performance by combining the individual class scores.Source: §2.8 · pp. 17–18
  • The measures are computed for both point clouds and meshes; mesh evaluation weights covered areas of correctly and incorrectly classified faces using participant-provided center-of-gravity clouds.Source: §2.8 · p. 18

3. Baselines

This section presents Random Forest and Sparse Convolutional Network baselines for H3D point-cloud and mesh segmentation and discusses their results and challenges. It highlights difficulties classifying detailed structures in H3D(PC) and RF’s overall advantage on H3D(Mesh), alongside class-specific exceptions and errors.

3.1. Random Forest

  • The RF baseline combines multi-scale geometric features with LiDAR echo and intensity features, mesh-face geometry and transferred LiDAR features, and color features.Mesh faces are represented by their centers of gravity; the feature set also includes face area, density, and normal orientation.Source: §3.1 · pp. 18–19

3.3.1. H3D(PC)

  • Table 3’s H3D(PC) and H3D(Mesh) baseline results show RF and SCN performed similarly, with SCN slightly better in OA and mean F1; their accuracies exceeded the V3D challenge’s best OA of 85.2% (Zhao et al. (2018)).Source: Table 3 · p. 19, §3.3.1 · p. 20

3.2. Sparse Convolutional Network

  • The SCN baseline uses a 3D U-Net with submanifold sparse convolutions on 25-cm voxel grids, using raw point attributes for LiDAR and texture RGB for mesh voxels.The network avoids the engineered features used by the RF baseline; mesh voxels are derived from face centers of gravity.Source: §3.2 · p. 20

3.3. Discussion of Baseline Results

  • The baseline analysis examines both H3D(PC) and H3D(Mesh) to clarify dataset challenges and launch the benchmark competition.Source: §3.3 · p. 20

3.3.1. H3D(PC)

  • Soil/Gravel was difficult to distinguish from other ground classes, while Shrub and Tree were confused because of similar color and surface roughness; tree extraction nevertheless succeeded well.The authors attribute successful tree extraction probably to the sensor’s multi-echo capability.Source: §3.3.1 · pp. 20, 22
  • Roofs were extracted successfully but facades were harder, and Urban Furniture labels were dispersed across classes because the category spans varied objects.Cars were also often misclassified as Urban Furniture, which the authors relate to the classes’ varied colors and special cases.Source: §3.3.1 · p. 22
  • On H3D(PC), the new Vertical Surface and Chimney classes are demanding: RF confuses façades with vertical surfaces, while SCN allocates few vertical-surface points to Façade.RF performs better on chimneys, probably because SCN’s discretization limits its precision on small structures.Source: §3.3.1 · pp. 22–23

3.3.2. H3D(Mesh)

  • Weighted mesh metrics scarcely differ from unweighted metrics, indicating that most large faces are correctly predicted.Metrics are weighted by each face’s surface area, so large faces contribute more than small faces.Source: §3.3.2 · p. 23
  • On H3D(Mesh), RF outperforms SCN in OA by 2.8 percentage points and in mF1 by 3.5 percentage points.The study attributes SCN’s slight inferiority mainly to voxelization and its small input-feature set.Source: §3.3.2 · p. 25
  • For H3D(Mesh), RF’s F1-score exceeds SCN’s by 5.4 percentage points for Urban Furniture and 9.5 for Shrub; SCN outperforms RF only on Soil/Gravel.Both classifiers struggle with Urban Furniture’s large intra-class variance, and SCN often predicts Shrub as Urban Furniture.Source: §3.3.2 · pp. 25–26
  • On H3D(Mesh), both classifiers struggle to distinguish vertical surfaces from façades; SCN also confuses Impervious Surface with Roof, while color helps separate similar ground classes.For Chimney, SCN has high recall but lower precision, whereas RF has worse recall but higher precision and a better F1-score.Source: §3.3.2 · pp. 25–26

4. Conclusion

The paper introduces H3D, a semantic-segmentation benchmark pairing high-resolution 3D point clouds with textured meshes from UAV LiDAR and Multi-View-Stereo. It presents the March 2018 data and describes two follow-on epochs.

4. Conclusion

  • H3D is a semantic-segmentation benchmark for high-resolution 3D point clouds and textured meshes from UAV LiDAR and Multi-View-Stereo.Source: §4 · p. 26
  • The March 2018 first epoch comprises both H3D(PC) and H3D(Mesh).Source: §4 · p. 26
  • Follow-on epochs in November 2018 and March 2019 were to cover an extended area and offer a more detailed class catalog.Source: §4 · p. 26
  • The authors applied different classifiers to both modalities as benchmark baselines and report that H3D has great potential for testing machine-learning approaches.Source: §4 · p. 26
  • The authors hope H3D will become a second established ISPRS benchmark alongside V3D.Source: §4 · p. 26
Loading 2102.05346v2…