Source-linked AI summary
DALES: A Large-scale Aerial LiDAR Data Set for Semantic Segmentation
Nina Varney, Vijayan K. Asari, Quinn Graehling
TL;DR
Annotated point-cloud benchmarks have focused mainly on mobile or terrestrial scanning, leaving a substantial gap in large-scale aerial LiDAR data. The paper introduces DALES, constructs and benchmarks this dataset, and reports strong performance from current methods while exposing challenges from aerial class imbalance and object scale.
Problem
Annotated semantic-segmentation datasets are limited for aerial LiDAR, despite aerial scanning’s distinct challenges and broad applications.
Method
The authors construct DALES with expert hand-labeled aerial scenes, prepare varied scene types and formats, and benchmark six state-of-the-art algorithms with multiple metrics.
Results
The benchmark shows room for improvement in current methods, especially for evaluating classes with different physical sizes and point counts.
Takeaways & Limitations
DALES provides a large publicly available aerial LiDAR resource for evaluating and advancing semantic-segmentation algorithms.
Takeaways & Limitations
Aerial class disparity is substantial: poles have about 340 points per object, and most non-ground, non-vegetation, non-building classes comprise less than 1% of the dataset.
Abstract
from arXiv · showhide
We present the Dayton Annotated LiDAR Earth Scan (DALES) data set, a new large-scale aerial LiDAR data set with over a half-billion hand-labeled points spanning 10 square kilometers of area and eight object categories. Large annotated point cloud data sets have become the standard for evaluating deep learning methods. However, most of the existing data sets focus on data collected from a mobile or terrestrial scanner with few focusing on aerial data. Point cloud data collected from an Aerial Laser Scanner (ALS) presents a new set of challenges and applications in areas such as 3D urban modeling and large-scale surveillance. DALES is the most extensive publicly available ALS data set with over 400 times the number of points and six times the resolution of other currently available annotated aerial point cloud data sets. This data set gives a critical number of expert verified hand-labeled points for the evaluation of new 3D deep learning algorithms, helping to expand the focus of current algorithms to aerial data. We describe the nature of our data, annotation workflow, and provide a benchmark of current state-of-the-art algorithm performance on the DALES data set.
1. Introduction
DALES addresses the limited availability of large annotated aerial LiDAR data by introducing a benchmark designed for semantic segmentation and evaluating current algorithms on it.
- Semantic segmentation of LiDAR scenes supports applications including wireless signal mapping, forest fire management, terrain modeling, and public utility asset management.
- Aerial LiDAR differs from ground-based scanning in sensor orientation, occlusion patterns, resolution, scene scale, applications, and collection cost.Aerial sensors use a nadir orientation and collect broad swaths with more consistent but lower resolution.
- DALES contains forty scenes of hand-labeled aerial data spanning eight categories and urban, suburban, rural, and commercial environments.The dataset is split into roughly 70% training and 30% testing and is provided in .las, .txt, and .ply formats.
- DALES is presented as the largest and densest publicly available semantic segmentation dataset for aerial LiDAR.Its objective is to expand semantic segmentation algorithm development to aerial point cloud data.
- The benchmark evaluates six state-of-the-art algorithms using mean IoU, per-class IoU, Class Consistency Index, and overall accuracy.The authors also suggest additional evaluation metrics for point cloud datasets.
2. Related Work
Existing point-cloud benchmarks largely use mobile or terrestrial scanners, while annotated aerial LiDAR resources remain scarce and comparatively small.
- Large-scale labeled datasets are established benchmarks for evaluating supervised deep learning methods across imagery and point-cloud data.
- Most listed datasets, including Sydney Urban Objects and Paris-Lille-3D, use mobile laser scanners mounted on vehicles in urban environments.These mobile-scanner datasets have lower point density than static-scanner data.
- Semantic 3D provides four billion points and eight labeled classes from high-density terrestrial laser scans of outdoor urban environments.
- The ISPRS 3D Semantic Labeling dataset is among the few annotated aerial datasets, with three Vaihingen views, nine categories, and about 1.2 million points.Its point density is 5-7 ppm.
3. DALES: The Data Set
DALES is an expert-labeled aerial LiDAR semantic-segmentation benchmark built from diverse, non-overlapping scenes and eight application-oriented object categories. Its collection and annotation workflow combine standardized formats, dense sampling, quality checks, and automated initialization with manual refinement.
- DALES provides a semantic segmentation benchmark made exclusively of aerial LiDAR data, with class labels inferred for points in unordered point sets.
- Aerial acquisition produces multiple returns because broad pulses can strike objects at different distances; DALES tracks up to four returns from one pulse.
- 40 tiles cover 10 km^2, averaging 12 million points per tile at 50 ppm resolution, with no overlap between neighboring scenes.
- The collection includes commercial, urban, rural, and suburban scenes selected to provide varied landscapes despite being limited to one municipality.
- The eight principal categories are buildings, cars, trucks, poles, power lines, fences, ground, and vegetation; an additional unknown class contains sparsely represented objects.
- Annotation combines DEM-based ground labeling, K-nearest-neighbor neighborhoods, surface-normal initialization for buildings, manual refinement, and category-wise consistency checks.
4. Benchmark Statistics
DALES exhibits strongly imbalanced object categories and geometric associations across classes. Small objects, especially poles, are sparsely sampled, while fixed bounding-box methods face a context-versus-object-density trade-off.
- Almost 50% of the data set consists of ground points, while vegetation contributes around 30%.
- Around 1,100 pole objects contain about 340 points each, making poles among the smallest and most challenging objects to detect.
- Every class except ground, vegetation, and buildings represents less than 1% of the data set, producing severe object-category disparity.
- Fixed bounding boxes create a trade-off: smaller boxes may lack scene context, whereas larger boxes may undersample small objects.
- Surface-normal histograms reveal strong associations among some categories, including buildings, trucks, and ground.
5. Evaluation
DALES evaluation combines mean IoU, per-class IoU, overall accuracy, distributional robustness, and CCI to compare six benchmark algorithms. KPConv leads performance, while results vary substantially across object categories, especially for smaller or less-represented objects.
- Evaluation metrics: Mean IoU averages performance across the eight labeled categories, excluding unknown objects.
- Evaluation metrics: Overall accuracy is reported for benchmark consistency but can obscure poor performance across imbalanced categories.
- Evaluation metrics: CCI complements mean IoU by measuring consistency across classes, while the Algorithm Performance Map compares mean IoU against CCI.Robust performance is characterized by high mean IoU and high CCI.
- Evaluation metrics: Six benchmark algorithms are evaluated using mean IoU, per-class IoU, overall accuracy, CCI, and distributional analyses.The tested methods are KPConv, ConvPoint, PointNet++, PointCNN, ShellNet, and Superpoint Graphs.
- Algorithm performance: Ground, vegetation, and buildings perform strongly across networks, whereas trucks, fences, and poles have much lower IoU.Performance varies more as point counts and object sizes decrease; truck errors are linked to similarity with ground and buildings and limited distinctive point distributions.
- Algorithm performance: 81.1% mean IoU makes KPConv the strongest method, exceeding other networks by over 10%.KPConv also receives the highest Algorithm Performance Map rating based on mean IoU and CCI.
- Evaluation resources: DALES provides a full annotated training and testing set, plus a website and leaderboard for submitting and tracking results.
6. Conclusion
The paper presents a large-scale ALS benchmark and finds that current methods still need improvement, particularly across classes differing in physical size and point count. Future work will expand the labels and coverage before releasing the data as a community challenge.
- Over half a billion points across 10 km^2 form a large-scale ALS benchmark with eight hand-labeled classes.
- Six state-of-the-art algorithms evaluated on the data show room for improvement, especially across classes with different physical sizes and point counts.
- Future labeling will add more classes and eventually cover the entire 330 km^2 area.
- The expanded data will be presented as a challenge to the earth vision community.