PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Robust LiDAR Semantic Segmentation for Real-World Deployment: Evaluation under Coarse Labels, Adverse Conditions, and Domain Shifts

Samir Abou Haidar, Alexandre Chariot, Mehdi Darouich, Cyril Joly, Jean-Emmanuel Deschaud

arXiv:2609.02830v1cs.RO

TL;DR

LiDAR segmentation benchmarks often assess clean, fine-grained, single-domain accuracy, leaving safety semantics, corruption robustness, domain transfer, and deployment efficiency incompletely evaluated. The paper proposes a unified protocol covering coarse labels, eight LiDAR corruptions, cross-dataset evaluation without adaptation, and inference speed. Results show that benchmark rankings do not reliably predict deployment readiness, corruption robustness varies by architecture, and domain generalization remains insufficient.

  • Problem

    Existing protocols focus on clean, fine-grained, single-domain benchmarks, without jointly assessing safety-oriented semantics, sensor degradation, cross-domain variability, and computational efficiency.

  • Method

    The paper evaluates models using coarse safety-oriented labels, eight synthetic LiDAR corruptions, cross-dataset transfer without adaptation, and runtime measurements on Jetson AGX Orin.

  • Results

    Clean-benchmark rankings do not always reflect safety-relevant performance; all methods degrade under corruptions, robustness is architecture-dependent, and domain generalization remains insufficient.

  • Takeaways & Limitations

    The protocol provides a reproducible framework for assessing deployment readiness beyond conventional benchmarks and guiding more realistic evaluation practices.

Abstract

from arXiv · show

LiDAR-based semantic segmentation is a core perception module for autonomous vehicles and mobile robots. Despite the strong performance of recent state-of-the-art methods on standard benchmarks, existing evaluation protocols remain focused on clean, single-domain settings and fine-grained label taxonomies, leaving deployment readiness largely unassessed. Real-world systems must handle safety-critical label semantics, degraded sensing conditions, and cross-domain variability, yet no unified protocol currently addresses all three aspects together. In this paper, we propose a structured evaluation protocol that assesses the deployment readiness of LiDAR semantic segmentation models along three complementary dimensions: (i) coarse-label evaluation aligned with autonomous driving safety priorities, revealing how label granularity affects different methods; (ii) robustness under eight types of LiDAR corruptions designed to emulate real-world atmospheric, geometric, and sensor degradations; and (iii) domain generalization across datasets without adaptation. The evaluation includes inference speed measured on an embedded Jetson AGX Orin platform, directly reflecting deployment constraints. Our results show that fine-grained benchmark rankings do not always reflect safety-relevant performance, that all methods experience substantial degradation under corruptions with architecture-dependent robustness characteristics, and that current domain generalization remains insufficient for reliable deployment. These findings expose concrete gaps between benchmark performance and deployment readiness, and provide a reference protocol for more practically grounded evaluation of LiDAR semantic segmentation.

I. INTRODUCTION

Existing LiDAR segmentation benchmarks emphasize clean, fine-grained, in-distribution accuracy, while deployment requires safety-relevant semantics, corruption robustness, cross-domain generalization, and embedded efficiency. The proposed protocol combines these requirements into three evaluation pillars with runtime measurements.

  • Standard benchmarks use clean data and fine-grained taxonomies, but deployment also requires safety-critical semantics, sensor robustness, unseen-domain generalization, and embedded computational efficiency.
  • No existing protocol jointly evaluates semantic relevance, corruption robustness, cross-domain generalization, and computational efficiency.
  • The protocol organizes evaluation around coarse-label safety assessment, robustness to adverse sensing conditions, and domain generalization.
  • Robustness testing covers eight synthetic LiDAR corruptions spanning atmospheric effects, geometric perturbations, and sensor-induced artifacts.Corruptions are analyzed individually and by category, with Jetson AGX Orin inference speed reported to quantify robustness–efficiency trade-offs.
  • Domain generalization evaluates models trained on SemanticKITTI and nuScenes directly on ParisLuco3D without adaptation.The setting measures transfer across geographic environments, acquisition conditions, and dataset distributions.

II. RELATED WORK

Prior evaluation protocols typically isolate one dimension of LiDAR segmentation performance, such as clean accuracy, corruption robustness, coarse-label semantics, or domain transfer. This paper integrates these dimensions with computational efficiency under consistent conditions.

  • SemanticKITTI and nuScenes primarily measure clean, in-distribution accuracy with fine-grained class taxonomies, which does not fully reflect deployment challenges.
  • Robo3D evaluates weather-, geometry-, and sensor-related corruptions but omits label semantics, domain generalization, and inference speed.
  • The COLA framework addresses label granularity through safety-oriented coarse semantic groupings but does not assess corruption robustness or computational efficiency.
  • The proposed protocol integrates semantic consistency, robustness, domain generalization, and computational efficiency into one assessment framework.

III. EVALUATION PROTOCOL

The evaluation protocol assesses deployment readiness through safety-oriented label semantics, adverse sensing conditions, and cross-domain transfer, supplemented by inference-speed measurements on GPU and embedded hardware.

  • The protocol evaluates safety-oriented label semantics, robustness under adverse sensing conditions, and cross-domain transfer as complementary deployment dimensions.Computational efficiency is measured through inference speed on an NVIDIA RTX 4090 GPU and Jetson AGX Orin embedded platform.

A. Coarse-label evaluation for safety assessment

Coarse-label evaluation reframes segmentation around functional categories relevant to autonomous navigation, reducing penalties for semantically similar fine-grained confusions while preserving safety-relevant errors.

  • Fine-grained taxonomies can penalize semantically reasonable predictions, such as confusing a car with a truck despite similar avoidance or braking behavior.
  • The COLA framework groups dataset-specific labels by their functional role in autonomous navigation and compares fine- versus coarse-grained performance.
  • Seven coarse categories are used: Driveable Ground, Other Ground, Structure, Vehicle, Nature, Living Being, and Object.
  • Cross-category confusions, such as predicting a vehicle point as road, remain penalized because they represent safety-relevant failures.

B. Robustness under real-world LiDAR corruptions

Robustness is evaluated by testing models trained on clean data directly on corrupted nuScenes-C and SemanticKITTI-C, using eight degradations grouped by atmospheric, geometric, and sensor effects. The protocol also relates robustness to real-time inference constraints on Jetson AGX Orin.

  • Models are evaluated without fine-tuning on corrupted versions of nuScenes and SemanticKITTI to measure out-of-distribution robustness.
  • Eight corruption types are organized into atmospheric, geometric, and sensor-related groups.Atmospheric corruptions include fog, wet ground, and snow; geometric corruptions include motion blur and beam missing; sensor corruptions include crosstalk, incomplete echo, and cross-sensor effects.
  • Mean Corruption Error (mCE) is the primary robustness metric, averaging corruption-specific errors across eight corruption types and severities relative to MinkowskiNet.
  • Real-time operation requires under 100 ms per SemanticKITTI scan and under 50 ms per nuScenes scan on Jetson AGX Orin.These thresholds correspond to more than 10 FPS for SemanticKITTI and more than 20 FPS for nuScenes.

C. Domain generalization

Domain generalization measures whether LiDAR segmentation models trained on one dataset operate on an unseen dataset without adaptation or target-domain data. The protocol uses nuScenes and SemanticKITTI as sources and ParisLuco3D as the target, preserving native fine-grained labels.

  • Domain generalization evaluates operation in previously unseen environments without adaptation or access to target-domain data during training.
  • Models trained on nuScenes or SemanticKITTI are evaluated directly on ParisLuco3D as an unseen target domain.ParisLuco3D differs in geographic environment, acquisition conditions, and scene composition.
  • Evaluation retains each source dataset’s native fine-grained taxonomy, so per-class IoU reflects detailed segmentation under domain shift.

IV. RESULTS AND DISCUSSIONS

The study benchmarks representative methods from four LiDAR segmentation architecture families under a unified retraining protocol and reports deployment-relevant runtime measurements. This design supports comparisons across architectural paradigms while accounting for training consistency and computational cost.

  • The benchmark covers point-based, projection-based, sparse convolution-based, and fusion-based architectures.Evaluated examples include KPConv, WaffleIron, PTv3, SalsaNext, CENet, FRNet, Minkowski, Cylinder3D, SPVCNN, and HARP-NeXt.
  • The architectural diversity is intended to make benchmark conclusions applicable across different segmentation paradigms.
  • All models are reproduced and retrained from scratch under a unified training protocol on one RTX 4090 GPU.Training uses standard point-cloud augmentations and class-imbalance strategies including PolarMix and Instance CutMix.
  • Inference speed is measured on both an RTX 4090 and an embedded Jetson AGX Orin using end-to-end runtime.Reported FPS includes data preprocessing and network inference.

A. Coarse-label evaluation

Coarse-label evaluation complements fine-grained benchmarks by grouping classes according to safety relevance and exposing architecture-dependent differences in category performance and efficiency. Across datasets, coarse-label scores generally improve, but accuracy, safety-critical perception, and runtime remain differently balanced across methods.

  • A. Coarse-label evaluation: COLA aggregates fine-grained classes into broader categories that better reflect high-level autonomous-driving perception needs.Examples include distinguishing vehicles or living-beings rather than individual subclasses such as cars or trucks.
  • A. Coarse-label evaluation: All evaluated methods improve when transitioning from fine-grained mIoU_FG to coarse-level mIoU_COLA, with architecture-dependent improvement magnitudes.The pattern indicates that many fine-level errors remain within broader semantic categories.
  • A. Coarse-label evaluation: 87.8% mIoU_COLA is achieved by PTv3 on nuScenes, leading across most semantic groups.FRNet reaches 86.3%, while HARP-NeXt reaches 86.7% with 100 FPS.
  • A. Coarse-label evaluation: 87.4% mIoU_COLA is achieved by WaffleIron on SemanticKITTI, followed by Minkowski at 87.3% and SPVCNN at 86.8%.HARP-NeXt reaches 86.2% while providing 76.9 FPS and strong driveable-ground and structure performance.
  • A. Coarse-label evaluation: 66.0% fine-grained mIoU_FG for FRNet on SemanticKITTI contrasts with 83.3% coarse-label performance and weaker living-being performance at 55.6%.Minkowski and SPVCNN score 83.5% and 83.2% on living-being, respectively, while point-based methods offer stronger category-level understanding at higher computational cost.

B. Robustness evaluation against adverse conditions

The evaluation tests LiDAR segmentation under eight synthetic corruptions across atmospheric, geometric, and sensor categories, alongside embedded-device throughput. Results reveal architecture-dependent robustness and substantial deployment trade-offs.

  • Robustness evaluation: Eight synthetic corruptions are grouped into atmospheric, geometric, and sensor degradations to compare robustness systematically.The evaluation also reports inference speed on NVIDIA Jetson AGX Orin; group averages mean mIoU across corruption severities.
  • nuScenes-C: Sparse convolution methods achieve the strongest geometric robustness on nuScenes-C, with MinkowskiNet obtaining the highest geometric group average.Voxelized spatial aggregation reduces sensitivity to structural corruption.
  • nuScenes-C: HARP-NeXt achieves 77.1% clean mIoU, 71.2% atmospheric group average, mCE = 105.7%, and 14.1 FPS on Jetson AGX Orin.Its results illustrate a favorable balance among clean accuracy, corruption robustness, and efficiency.
  • Cross-dataset interpretation: Sensor-related corruptions remain particularly challenging across architectural categories, and robustness varies with dataset characteristics and sensing density.SemanticKITTI’s denser HDL-64E scans make geometric completeness and structural inconsistencies especially consequential compared with nuScenes-C.
  • SemanticKITTI-C: On SemanticKITTI-C, KPConv reaches the lowest mCE at 99.5% and the highest atmospheric group average at 55.4%, while FRNet reaches 66.0% clean mIoU and 56.1% geometric average.FRNet operates at only 2.5 FPS, whereas SalsaNext reaches 9.2 FPS but degrades substantially across corruption categories.
  • Implications: Robust deployment requires architectural advances and corruption-aware augmentation because many state-of-the-art methods overfit clean data distributions.Suggested augmentations include motion-blur simulation, beam dropping, and nonuniform subsampling.

C. Domain generalization evaluation

Domain generalization is evaluated by transferring models from nuScenes or SemanticKITTI to unseen ParisLuco3D without adaptation. Performance remains limited and uneven, with sensor configuration, scene variation, and architecture all shaping cross-domain behavior.

  • Evaluation protocol: The protocol uses single-source to single-target transfer to ParisLuco3D without adaptation or target-domain data access.Sources are nuScenes and SemanticKITTI; ParisLuco3D differs in geographic environment, acquisition conditions, and scene composition.
  • nuScenes → ParisLuco3D: The nuScenes-to-ParisLuco3D setting changes appearance and scenes without sensor shift because both datasets use Velodyne HDL-32E LiDAR.ParisLuco3D additionally includes adverse-weather sequences with rain, distorted point clouds, and partial road occlusions.
  • nuScenes → ParisLuco3D: 32.5% mIoU and 32.1% mIoU are the best nuScenes-to-ParisLuco3D scores, achieved by Minkowski and SPVCNN respectively, yet no method performs consistently well across classes.This indicates that appearance and scene variation can degrade performance even with identical acquisition settings.
  • nuScenes → ParisLuco3D: Sparse convolution methods perform best overall under nuScenes-to-ParisLuco3D shift, while projection and fusion methods remain less stable when relying on range-view features.Large structured classes score higher, whereas bicycle, traffic cone, and trailer remain poorly segmented.
  • SemanticKITTI → ParisLuco3D: The SemanticKITTI-to-ParisLuco3D setting adds sensor shift from HDL-64E to HDL-32E, altering point density and sampling patterns.The shift combines sensor, scene, and appearance differences.
  • SemanticKITTI → ParisLuco3D: 33.3% mIoU and 31.5% mIoU are the leading SemanticKITTI-to-ParisLuco3D results for Minkowski and SPVCNN, while CENet and FRNet achieve near-zero performance.HARP-NeXt reaches 18.8% mIoU and performs better than those projection-based methods under this shift.
  • Cross-domain findings: Most methods generalize better from nuScenes than SemanticKITTI, but KPConv and Cylinder3D generalize better from SemanticKITTI because they rely more on local geometric structure.The results show that transfer depends on both sensor and scene similarity and architectural design.

V. CONCLUSION

The paper presents a unified protocol for assessing LiDAR segmentation deployment readiness across safety-oriented labels, corruptions, domain generalization, and runtime. Results show that benchmark success does not ensure deployment readiness, motivating evaluation beyond conventional benchmarks.

  • The protocol evaluates deployment readiness through coarse-label safety assessment, corruption robustness, cross-domain generalization, and embedded inference runtime.Runtime is measured on Jetson AGX Orin and RTX 4090 hardware.
  • Clean-benchmark accuracy poorly predicts deployment readiness, while methods may fail on safety-critical classes, corruptions, or domain shifts.Projection-based methods generally show greater sensitivity to sensor and scene variations.
  • No single state-of-the-art method achieves a strong trade-off among accuracy, robustness, generalization, and computational efficiency.
  • The protocol provides a reproducible framework for comprehensive deployment-readiness assessment beyond conventional benchmarks.It is intended to promote realistic evaluation practices and guide future model design for autonomous-driving perception.
Loading 2609.02830v1…