PaperlayerView exact revision on arXiv

Source-linked AI summary

Through the Perspective of LiDAR: A Feature-Enriched and Uncertainty-Aware Annotation Pipeline for Terrestrial Point Cloud Segmentation

Fei Zhang, Rob Chancia, Josie Clapp, Amirhossein Hassanzadeh, Dimah Dera, Richard MacKenzie, Jan van Aardt

arXiv:2510.06582v3cs.CVcs.RO

TL;DR

High-quality annotated TLS datasets are scarce, and manual labeling of full-resolution scans is a bottleneck. The paper presents a feature-enriched, uncertainty-aware segmentation pipeline; across mangrove, boreal forest, and urban benchmarks, it consistently improves accuracy.

  • Problem

    High-quality annotated TLS datasets are scarce, while full-resolution scans require manual labeling.

    Source: §1 · p. 2

  • Method

    The paper presents a feature-enriched, uncertainty-aware TLS segmentation pipeline with ensemble-based uncertainty estimation.

    Source: §5 · p. 28

  • Results

    The framework consistently improves segmentation accuracy across mangrove, boreal forest, and urban benchmarks, while compact nine-channel configurations capture nearly all discriminative power.

    Source: §5 · p. 28

  • Takeaways & Limitations

    Performance saturates after approximately 12 annotated scans, regardless of feature configuration.

    Source: §4 · p. 27

    The ensemble's increased computational cost may limit applicability in real-time or resource-constrained settings.

    Source: §4 · p. 28

Abstract

from arXiv · show

Accurate semantic segmentation of terrestrial laser scanning (TLS) point clouds is limited by costly manual annotation. We propose a semi-automated, uncertainty-aware pipeline that integrates spherical projection, feature enrichment, ensemble learning, and targeted annotation to reduce labeling effort, while sustaining high accuracy. Our approach projects 3D points to a 2D spherical grid, enriches pixels with multi-source features, and trains an ensemble of segmentation networks to produce pseudo-labels and uncertainty maps, the latter guiding annotation of ambiguous regions. The 2D outputs are back-projected to 3D, yielding densely annotated point clouds supported by a three-tier visualization suite (2D feature maps, 3D colorized point clouds, and compact virtual spheres) for rapid triage and reviewer guidance. Using this pipeline, we build Mangrove3D, a semantic segmentation TLS dataset for mangrove forests. We further evaluate data efficiency and feature importance to address two key questions: (1) how much annotated data are needed and (2) which features matter most. Results show that performance saturates after ~12 annotated scans, geometric features contribute the most, and compact nine-channel stacks capture nearly all discriminative power, with the mean Intersection over Union (mIoU) plateauing at around 0.76. Finally, we confirm the generalization of our feature-enrichment strategy through cross-dataset tests on ForestSemantic and Semantic3D. Our contributions include: (i) a robust, uncertainty-aware TLS annotation pipeline with visualization tools; (ii) the Mangrove3D dataset; and (iii) empirical guidance on data efficiency and feature importance, thus enabling scalable, high-quality segmentation of TLS point clouds for ecological monitoring and beyond. The dataset and processing scripts are publicly available at https://fz-rit.github.io/through-the-lidars-eye/.

1. Introduction

The introduction presents a semi-automated, uncertainty-aware TLS annotation pipeline and the Mangrove3D benchmark, describing components intended to reduce annotation effort in structurally complex mangrove forests. It also frames systematic evaluation of LiDAR feature channels as a way to address ad hoc feature selection and identify compact, efficient stacks for TLS segmentation.

1. Introduction

  • The pipeline combines spherical projection, feature-enriched segmentation, and uncertainty-guided human correction to reduce annotation burden and improve fidelity in challenging forest scenes.The paper also releases Mangrove3D, a TLS benchmark designed for structurally complex mangrove forests, and evaluates cross-domain effectiveness on ForestSemantic and Semantic3D.Source: §1 · p. 2
  • Mangrove3D is presented as a TLS benchmark explicitly designed for structurally complex mangrove forests, with evaluations also conducted on ForestSemantic and Semantic3D.The paper frames its aims as alleviating annotation effort, introducing a mangrove-focused TLS dataset, and evaluating feature-enriched segmentation and cross-domain robustness.Source: §1 · p. 2

1.1.1. Manual Annotation Software and Methodologies

  • Manual TLS annotation is slow and error-prone in scenes with dense occlusions, irregular branching, and intertwined roots, while existing tools require navigation of dense 3D scenes and often unavailable RGB imagery.The proposed pipeline emphasizes efficiency, uncertainty awareness, and adaptability for complex TLS datasets.Source: §1.1.1 · pp. 2–3

1.1.2. Spherical (Equirectangular) Projection for TLS Point Clouds

  • The paper adapts spherical projection and deep networks to TLS, extending single-band spherical maps to multi-channel feature stacks that aid annotators and segmentation models.The approach targets ecological, indoor, and construction TLS settings where camera imagery may be unavailable or unreliable.Source: §1.1.2 · p. 4

1.1.3. Active Learning and Self-Training for Efficient Annotation

  • The framework combines active learning and self-training on spherical TLS maps, using ensemble uncertainty to direct human review and high-confidence predictions as pseudo-labels.Its ensemble uses UNet++, DeepLabV3+, and Segformer; the pipeline retains human verification on full-resolution TLS projection maps.Source: §1.1.3 · p. 4

1.1.4. Feature fusion in semantic segmentation of point clouds

  • The feature-fusion framework operates solely on LiDAR-derived signals, combining intrinsic radiometric and geometric information without relying on external imagery.The cited intrinsic signals include intensity, range, and derived geometric attributes.Source: §1.1.4 · p. 5
  • Feature fusion lacks consensus on an optimal LiDAR-only set; redundant or correlated inputs can inflate dimensionality without consistent performance gains.Source: §1.1.4 · p. 5
  • Progressive feature-channel analysis yields a compact, robust stack that reduces redundancy, enhances segmentation efficiency, and provides practical guidance for TLS applications.The analysis starts with single feature families and progressively integrates others to assess complementarity and marginal improvements.Source: §1.1.4 · p. 5

2. Methods and Materials

This section presents Mangrove3D and a three-stage TLS annotation workflow, covering spherical projection, feature generation, human-in-the-loop labeling, 3D back-projection, refinement, and visualization. It also characterizes dataset point density and surface roughness and describes evaluation of feature configurations and annotated training-set sizes using segmentation and uncertainty metrics.

2.1. A New TLS Dataset for Mangrove Forest

  • Point density decreases with range and concentrates near the ground; ground and root surfaces have lower roughness, while stem and canopy regions are rougher and more variable.The roughness statistics use local PCA; the reported pattern is consistent with reduced point density at larger ranges.Source: Figure 2 · p. 6, §2.1 · p. 7, Table 2 · p. 7
  • Mangrove3D is presented as the first TLS semantic-segmentation benchmark for mangrove forests, capturing centimeter-scale geometry in occlusion-rich tidal ecosystems.It comprises 39 scans from seven plots, totaling 31.3 million points; five semantic classes are labeled, with a 27/3/9 train/validation/test split in the main experiment.Source: §2.1 · p. 7

2.2. Semi-automated Annotation Pipeline

  • The annotation pipeline has three stages: spherical projection, iterative uncertainty-guided active learning and self-training, then 3D label refinement and reprojection.The final stage back-projects 2D segmentation masks to 3D to produce annotated point clouds and refined 2D masks.Source: §2.2 · p. 8, Figure 4 · p. 8

2.2.1. Stage 1: Spherical Projection & Visualization

  • Stage 1 unwraps TLS scans into a 540 × 1440 spherical grid and stacks radiometric, geometric, and PCA-derived statistical features; tiling and adaptive neighborhoods support scalable computation.At this resolution, the grid provides a near one-to-one mapping between beam directions and pixels.Source: §2.2.1 · p. 10

2.2.2. Stage 2: Hybrid Annotation with Semi-Supervised and Active Learning

  • Stage 2 uses manually labeled seed scans to train an ensemble of UNet++, DeepLabV3+, and SegFormer models, combining pseudo-labels with mutual-information uncertainty for human-guided learning.The three models use distinct encoder backbones—ResNet-34, EfficientNet-B3, and MiT-B1—to promote representational diversity and reduce correlated errors.Source: §2.2.2 · p. 10, Figure 7 · p. 11
  • Each three-channel feature group uses a dedicated encoder, with bottleneck features concatenated and interpolated before decoding; the modular design supports 3 × N channel configurations.The passage reports that this design outperforms naive early fusion and allows channel configurations with minimal retraining.Source: §2.2.2 · p. 11
  • Ensemble predictions produce segmentation masks and epistemic-uncertainty maps, sending ambiguous regions to manual refinement while retaining confident predictions as pseudo-labels.The inference-refinement cycle repeats until scans are fully annotated; joint displays of masks, uncertainty, and input features focus review on ambiguous areas.Source: §2.2.2 · p. 12, Table 3 · p. 12

2.2.3. Stage 3: Back Projection and Refinement in 3D Space

  • Spherical projection can create pixel-collision ambiguities at object boundaries, so 3D refinement applies kNN majority voting and a Random Forest trained on reliable core regions.The Random Forest uses geometric and radiometric features to correct systematic boundary ambiguities.Source: §2.2.3 · p. 12
  • Compact virtual spheres re-project feature maps onto tunable spherical grids, creating density-neutral, resolution-controllable 3D thumbnails that preserve each scan’s global structure.They enable rapid inspection of coverage, spatial proportions, and segmentation quality without full-resolution point-cloud overhead.Source: §2.2.3 · pp. 12, 14
  • In before-and-after point-cloud examples, 3D label refinement reduces isolated label noise and improves boundary consistency; local corrections preserve global class balance, following back-projection and re-projection from 2D spherical annotations.Source: Figure 9 · p. 13, §2.2.3 · p. 13

2.3. Performance Evaluation and Analysis

  • The evaluation measures oAcc, mAcc, mIoU, and per-class IoU, alongside Shannon Entropy and AUPRC for uncertainty and error detection.AUPRC measures how well the uncertainty map exposes errors against a binary error mask.Source: §2.3 · p. 14
  • Feature experiments compare basic raw or preprocessed inputs with added geometric descriptors, normals, and PCA, MNF, or ICA feature trios to assess accuracy, uncertainty, and computational cost.The study notes that extra channels increase CPU and GPU memory and computation requirements.Source: §2.3 · pp. 14–15
  • The input comparison includes six-channel combinations and nine- and twelve-channel stacks, with the latter adding PCA components to the nine-channel set.The nine-channel set combines the core features with normals and geometric descriptors.Source: §2.3 · p. 15

3. Results

The results assess feature enrichment, uncertainty mapping, and transfer to ForestSemantic and Semantic3D. They report that multi-feature inputs outperform single-feature inputs on Semantic3D, with IRZ_N3_CAP leading overall metrics, and that feature maps reveal complementary information.

3.1. 2D Segmentation Result

  • Errors cluster at cluttered canopy–stem and root–ground transitions, where elevated epistemic uncertainty largely overlaps misclassified pixels.The qualitative comparison shows this uncertainty–error correspondence across representative scans.Source: §3.1 · pp. 15–16, Figure 12 · p. 16
  • AUPRC is 0.30–0.40 across scans, while the highest-uncertainty pixels achieve near-unity precision at low recall.Thus, a small fraction of uncertain regions captures many true errors for targeted inspection.Source: §3.1 · pp. 15–16, Figure 12 · p. 16

3.2. Effect of feature enrichment

  • 0.768 for IRZ_N3_CAP, three points above raw intensity and six above raw range; compact six-channel features capture nearly all available discriminative power.Larger feature stacks add computation with little return.Source: §3.2 · p. 17, Figure 13 · p. 17
  • 0.712 mIoU with surface normals and C.A.P outperforms the 0.634 mIoU XYZ baseline; Objects IoU rises from 0.516 to 0.837.Additional features beyond this six-channel setup bring no further benefit.Source: §3.2 · p. 18, Table 6 · p. 18

3.3. Data efficiency

  • Performance improves with more training samples and largely saturates around 12 scans across the representative feature configurations.The sample-size evaluation spans 4–28 train/validation scans.Source: §3.3 · p. 18, Figure 14 · p. 19
  • With fewer than 12 samples, models show elevated entropy and more variable AUPRC; beyond this threshold, AUPRC values converge across feature configurations.Source: §3.3 · p. 19

3.4.1. ForestSemantic

  • ForestSemantic visualizations show that I.R.Z clarifies class stratification, normals and C.A.P accentuate structural boundaries, and PCA yields meaningful color gradients.Together, the feature visualizations qualitatively support transfer to new TLS forest environments.Source: §3.4.1 · p. 20
  • mIoU reaches 0.511 with IRZ_N3_CAP_PCA on ForestSemantic, where geometric features dominate and fused geometric–radiometric cues perform best.Radiometric preprocessing alone yields only marginal gains.Source: §3.4.1 · p. 20, Table 7 · p. 21

3.4.2. Semantic3D

  • Feature maps convey largely complementary information, with pairwise Pearson correlations |ρ| ≤0.4, supporting their fusion.Source: §3.4.2 · p. 22
  • IRZ_N3_CAP achieved the best overall performance across global metrics on Semantic3D, with mIoU 0.516, within the range of mid-ranked official benchmark methods.The comparison covers different feature-input combinations on the official Semantic3D test set under the standard protocol.Source: Table 8 · p. 22, §3.4.2 · p. 22
  • Basic intensity, range, and inverse height distinguish facades, ground, and hardscape, while normals, geometric descriptors, and statistical transforms expose additional structure.Normal pseudo-color provides an intuitive orientation cue; anisotropy, curvature, and planarity sharpen structural edges, while PCA, MNF, and ICA composites reveal subtle material transitions.Source: §3.4.2 · pp. 21–22, Figure 19 · p. 25
  • Feature-driven 3D colorizations and virtual rings provide object-characteristic insights beyond original RGB or intensity alone.The virtual rings are back-projected from colorized feature maps with the top and bottom cropped to Zenith ∈[45°, 136°].Source: §3.4.2 · p. 22, Figure 20 · p. 26, Figure 21 · p. 26

4. Discussion

The discussion highlights compact feature selection, uncertainty-guided annotation, cross-dataset robustness, and a three-tier visualization suite. It also notes practical limits in computational cost and the evidence available on annotation effort.

High-quality 2D and 3D visualizations

  • The visualization suite combines 2D spherical feature maps, 3D colorized point clouds, and virtual spheres for segmentation, fine-grained verification, and global scan summaries.Together, the products support dataset triage, annotation, and cross-scene comparison.Source: §4 · p. 27

Data efficiency

  • Ensemble performance on Mangrove3D saturates after approximately 12 annotated scans, and uncertainty-guided selection can approach peak performance with substantially fewer labeled samples.The authors caution that additional scans may still help when they introduce substantially novel scene geometries or conditions.Source: §4 · p. 27

Generalization across TLS datasets

  • Feature enrichment improves accuracy across Mangrove3D, ForestSemantic, and Semantic3D, with feature-group rankings broadly preserved across the differing environments.The authors report that the cross-dataset results and modular design support deployment across diverse TLS environments with minimal adjustment.Source: §4 · pp. 27–28

Feature importance

  • Surface normals most consistently improve class-level IoUs, while compact nine-channel configurations capture nearly all discriminative power with less redundancy and training time than larger stacks.The reported trade-off favors thoughtful feature selection over maximal channel inclusion.Source: §4 · p. 27

Scalable and adaptable design

  • LiDAR-only features, bottleneck-stage fusion, adaptive patch sizing, and flexible input channels support a modular framework that can accommodate different backbones and descriptors.The stated design allows backbone substitution without altering the overall pipeline.Source: §4 · p. 28

Limitations and future directions

  • The ensemble’s increased computational cost may limit real-time or resource-constrained use, and the study reports no quantitative measures of annotation time or interaction effort.The authors also identify a remaining gap between uncertainty estimates and actual error distributions.Source: §4 · p. 28

5. Conclusion

The conclusion presents a feature-enriched, uncertainty-aware TLS segmentation framework with modular components and visualization products. Across mangrove, boreal forest, and urban benchmarks, it reports improved segmentation accuracy and highlights LiDAR feature design as a strong alternative to relying on multi-source sensing.

5. Conclusion

  • Across mangrove, boreal forest, and urban benchmarks, the framework consistently improves segmentation accuracy, while compact nine-channel configurations capture nearly all available discriminative power.The conclusion describes the framework as modular, combining diverse feature groups, flexible encoder backbones, and ensemble-based uncertainty estimation.Source: §5 · p. 28
  • Complementary visualization products—2D spherical feature maps, 3D colorized point clouds, and compact virtual spheres—prove valuable for annotator guidance and rapid scene assessment, enabling efficient, targeted annotation and dataset triage.Source: §5 · p. 28
  • The framework relies solely on LiDAR-derived features and is presented as a portable solution for monitoring forestry, ecology, and urban environments.The conclusion says carefully exploiting LiDAR-derived features can, in some cases, surpass early LiDAR-camera fusion in spherical projection space.Source: §5 · p. 29
  • The authors identify preprocessing, backbone efficiency, uncertainty evaluation, and mobile deployment as areas for future development.They expect these developments to further expand applicability and impact.Source: §5 · p. 29
  • The conclusion argues that rigorous LiDAR feature design and preprocessing are no less important than integrating multi-source sensing.It characterizes the results as highlighting the untapped potential of LiDAR point clouds alone.Source: §5 · p. 29

CRediT authorship contribution statement

Fei Zhang is credited with leading the methodology, implementation, validation, analysis, investigation, data curation, visualization, and original drafting. The statement also assigns experimental design, Mangrove3D data collection and annotation, review, software, and supervision across the listed contributors.

CRediT authorship contribution statement

  • Fei Zhang is credited with conceptualization, methodology, software, validation, formal analysis, investigation, data curation, visualization, original drafting, project administration, and supervision.Fei Zhang and Rob Chancia are also credited with conceptualization; Chancia is credited with experimental design, and Zhang with experiment execution.Source: CRediT authorship contribution statement · p. 29
  • Josie Clapp is credited with Mangrove3D data annotation, while Rob Chancia, Richard MacKenzie, and Jan van Aardt are credited with dataset data collection.Source: CRediT authorship contribution statement · p. 29
  • Rob Chancia and Amirhossein Hassanzadeh are credited with validation, review and editing, and software, while Jan van Aardt is credited with review and editing and supervision.Dimah Dera is credited with review and editing.Source: CRediT authorship contribution statement · p. 29

Data and Code Availability

The Mangrove3D dataset is released on Zenodo, and the accompanying processing and evaluation code is available on the project website.

Data and Code Availability

  • The Mangrove3D dataset has been released on Zenodo.The accompanying preprocessing, feature engineering, training, and evaluation code, including environment specifications and reproducibility instructions, is available on the project website.Source: Data and Code Availability · p. 30

Appendix A. Annotation Guideline for Mangrove3D Dataset

The annotation guideline for semantic segmentation masks on Mangrove3D spherical maps is accessible via the linked resource.

Appendix A. Annotation Guideline for Mangrove3D Dataset

  • The guideline for creating semantic segmentation masks on Mangrove3D spherical maps is accessible via the provided link.Source: Appendix A · p. 30

Appendix B. Supplementary Material for Stage 1

Appendix B details the feature maps and neighborhood-based geometric descriptors used in Stage 1. It also documents feature stacking, correlation analysis, and dimensionality-reduction alternatives.

Additional feature maps

  • Single-channel maps depict intensity, range, inverted height, anisotropy, curvature, and planarity; maps are normalized to [0,1] and displayed with the plasma colormap.The colormap runs from deep blue for low values to bright yellow for high values.Source: Appendix B · p. 30, Figure B.22 · p. 31

Eigenvalue-based Geometric Descriptors

  • A local covariance matrix is computed for each point from neighboring coordinates centered on their neighborhood centroid.The selected neighborhood configuration uses a 0.06 m radius and a maximum of 50 neighbors, empirically chosen to balance point-density robustness and sensitivity to local geometry.Source: Appendix B · p. 31
  • Ordered covariance eigenvalues describe variance along orthogonal axes and define neighborhood curvature, anisotropy, and planarity descriptors, capturing linearity or planarity, principal-axis dominance, and points’ alignment within a 2D plane.Source: Appendix B · pp. 31–32

Feature correlations and dimensionality reduction

  • A Pearson correlation matrix is provided to describe relationships among the feature maps, without reporting specific correlation values in the cited text.The appendix also identifies MNF as noise-whitening followed by PCA and ICA as seeking mutually independent, non-Gaussian components.Source: Appendix B · p. 32, Figure B.23 · p. 32

Stacked channels and dimensionality reduction

  • The appendix presents stacked-channel maps, including I.R.Z and C.A.P, alongside PCA, MNF, and ICA component stacks.The three-channel displays are pseudo-color images; the listed dimensionality-reduction variants use the first three components.Source: Appendix B · pp. 30, 32

Appendix C.1. Loss functions

These appendices specify the loss combination and the post-processing and uncertainty procedures around ensemble segmentation. They describe averaging model outputs, projecting labels back to points, smoothing, and relabeling suspect points.

Appendix C.1. Loss functions

  • The training objective combines Dice loss and Cross-Entropy loss with equal weights.The appendix defines pixel-level predicted probabilities and ground-truth labels in its descriptions of the loss terms.Source: Appendix C.1 · p. 33

Appendix C.2. Ensemble inference and uncertainty estimation

  • Epistemic uncertainty is estimated as mutual information between the ensemble-averaged prediction and expected model entropy, quantifying pixelwise model disagreement.The appendix describes this as a principled measure of epistemic uncertainty.Source: Appendix C.2 · p. 34

Appendix D.1. Back projection

  • Each 3D point receives the label of its corresponding pixel in the 2D spherical map, with spherical angles and grid steps defining the mapping.Source: Appendix D.1 · p. 34

Appendix D.2. Geometric smoothing

  • Geometric smoothing updates a point's label by majority vote among its k nearest neighbors.Source: Appendix D.2 · p. 34

Appendix D.3. Random-Forest relabeling

  • A balanced Random Forest is trained on multiscale features using reliable core points, then applied to suspect points with a confidence threshold of 0.8.The reliable core set comprises points whose labels match the predictions and are nonzero.Source: Appendix D.3 · p. 35

Appendix D.4. Virtual spheres

Virtual spheres re-project feature maps onto uniformly sampled spherical grids to provide compact, standardized scan representations that preserve global structure. The appendix describes their storage and quality-control uses and points to animations of the visualizations.

Appendix D.4. Virtual spheres

  • Virtual spheres re-project feature maps onto synthetic, uniformly sampled spherical grids, creating a compact abstraction that preserves global structure while reducing data volume.Their point count depends on angular resolution and field of view rather than original point-cloud density.Source: Appendix D.4 · p. 35
  • At 1° angular resolution, an average 800K-point CBL scan becomes approximately 48K points, or about 1/16 of the memory, with minimal structural loss.The representation is density-neutral, and its resolution can be tuned to balance fidelity and storage.Source: Appendix D.4 · p. 35
  • A Semantic3D scan of up to 200M points yields about 0.8M points at 0.2° angular resolution.Source: Appendix D.4 · p. 35
  • Virtual spheres support rapid checks of scan coverage, occlusion, and segmentation, and can summarize large TLS archives or support visual comparisons across sites and conditions.Source: Appendix D.4 · p. 35

Appendix E. Animations of the Colorized Point Clouds and Virtual Spheres

Loading 2510.06582v3…