PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Identify Out-of-Distribution Objects for 3D LiDAR Anomaly Segmentation

Simone Mosco, Daniel Fusaro, Alberto Pretto

arXiv:2604.23604v1cs.CVcs.RO

TL;DR

3D LiDAR anomaly segmentation remains limited by closed-set methods and scarce, mismatched datasets. LIDO models inlier feature distributions directly and introduces mixed real-synthetic datasets, achieving state-of-the-art and competitive results on real and mixed datasets, respectively.

  • Problem

    Closed-set LiDAR segmentation methods struggle to identify unknown objects, while available anomaly datasets are scarce, domain-mismatched, proprietary, or contaminated by training objects.

  • Method

    LIDO jointly performs semantic and anomaly segmentation by modeling inlier class feature distributions with prototype, contrastive, and objectosphere losses, without anomaly samples during training.

  • Results

    LIDO achieves state-of-the-art performance on the real dataset and competitive performance on the introduced mixed real-synthetic datasets.

  • Takeaways & Limitations

    The method and datasets provide an effective basis for 3D LiDAR anomaly segmentation across real and mixed environments.

  • Takeaways & Limitations

    Performance metrics, particularly average precision, remain bounded by LiDAR class imbalance and uncertainty in semantic segmentation models.

Abstract

from arXiv · show

Understanding the surrounding environment is fundamental in autonomous driving and robotic perception. Distinguishing between known classes and previously unseen objects is crucial in real-world environments, as done in Anomaly Segmentation. However, research in the 3D field remains limited, with most existing approaches applying post-processing techniques from 2D vision. To cover this lack, we propose a new efficient approach that directly operates in the feature space, modeling the feature distribution of inlier classes to constrain anomalous samples. Moreover, the only publicly available 3D LiDAR anomaly segmentation dataset contains simple scenarios, with few anomaly instances, and exhibits a severe domain gap due to its sensor resolution. To bridge this gap, we introduce a set of mixed real-synthetic datasets for 3D LiDAR anomaly segmentation, built upon established semantic segmentation benchmarks, with multiple out-of-distribution objects and diverse, complex environments. Extensive experiments demonstrate that our approach achieves state-of-the-art and competitive results on the existing real-world dataset and the newly introduced mixed datasets, respectively, validating the effectiveness of our method and the utility of the proposed datasets. Code and datasets are available at https://simom0.github.io/lido-page/.

1. Introduction

3D LiDAR anomaly segmentation addresses the need to identify previously unseen objects alongside known-class semantic segmentation. The paper proposes feature-space modeling and mixed real-synthetic datasets to address methodological and data limitations.

  • Autonomous vehicles and robots require LiDAR perception that can classify known classes while identifying previously unseen objects.
  • The LiDAR anomaly task simultaneously predicts known-class semantic labels and per-point anomaly probabilities.
  • 3D anomaly segmentation remains limited, with existing methods often adapting 2D post-processing, using ensembles, or failing to segment anomalous objects.
  • Available datasets suffer from limited scenarios, few anomalies, domain gaps, proprietary access, or anomalies that may appear during training.
  • The proposed method models inlier class prototypes directly in feature space to jointly perform semantic and anomaly segmentation without anomaly samples during training.
  • The paper introduces mixed real-synthetic datasets and reports state-of-the-art performance on real data and competitive performance on mixed datasets.

2. Related Work

Prior LiDAR anomaly-segmentation research is sparse relative to broader 3D semantic segmentation and 2D anomaly-segmentation work. LIDO and new datasets target gaps in methods, public data, sensor resolutions, and domain coverage.

  • 3D Semantic Segmentation: 3D semantic-segmentation methods process point clouds directly, through projections, voxels, or hybrids.
  • Anomaly Segmentation: 2D anomaly segmentation includes softmax post-processing, anomaly classes, ensembles, generative resynthesis, and synthetic-anomaly training.
  • LiDAR Anomaly Segmentation: LiDAR anomaly segmentation remains underexplored, with methods relying on 2D post-processing or ensembles and others performing poorly on anomaly objects.
  • LiDAR Anomaly Segmentation: Public LiDAR anomaly datasets are scarce and span mismatched resolutions, while some real-world datasets remain proprietary.
  • Proposed Direction: LIDO uses a feature-extraction backbone with semantic and distribution-modeling branches, and the paper adds three datasets at different resolutions.

3. Methodology

LIDO jointly performs semantic and anomaly segmentation by learning class prototypes and modeling feature distributions directly in feature space. Its inference scores combine prototype similarity, semantic uncertainty, and contrastive feature norms to identify anomalous points.

  • Architecture: LIDO uses a backbone with semantic and contrastive branches to produce per-point features, semantic predictions, class prototypes, and anomaly scores.The semantic branch builds inlier-class prototypes, while the contrastive branch models feature distributions for anomaly identification.
  • Semantic Head: Confidence-based prototypes aggregate pre-softmax features from true-positive points, weighted by each point’s maximum feature component.Previous-epoch prototypes guide features toward their corresponding inlier-class prototypes through a cosine embedding loss.
  • Contrastive Head: The contrastive head combines contrastive and objectosphere losses to separate class representations and push inlier features away from the hypersphere center.Unlike approaches using unlabeled or void regions, LIDO learns anomaly discrimination without anomaly features from such regions during training.
  • Inference: The semantic head predicts inlier classes by maximizing cosine similarity between point features and accumulated class prototypes.It also derives anomaly evidence from maximum cosine distance and normalized Shannon entropy; higher entropy indicates greater uncertainty about class assignment.
  • Inference: Anomaly predictions fuse semantic and contrastive scores, with the contrastive score increasing as a point’s feature norm falls below threshold r.The contrastive score equals 1 at zero feature norm and 0 when the norm exceeds r.

4. Out-of-Distribution Datasets

The proposed OoD datasets combine real LiDAR scans with synthetic anomaly objects from filtered ModelNet models. They vary sensor resolution, anomaly multiplicity, placement, and scan realism to support diverse evaluation settings.

  • Dataset Construction: The datasets derive from three autonomous-driving benchmarks with different LiDAR resolutions and complement the real-world STU anomaly-segmentation dataset.The benchmarks are nuScenes, SemanticPOSS, and SemanticKITTI.
  • Dataset Construction: Filtered ModelNet objects provide synthetic anomalies whose categories and instances are selected to avoid overlap with real-world LiDAR datasets.This supplies out-of-distribution objects for insertion into established semantic-segmentation benchmarks.
  • Insertion Protocol: Synthetic objects are inserted on planar surfaces after rotation and scaling, then projected into range-image geometry and assigned realistic intensity values.The protocol manipulates point distributions, intensity, and beam-like sensor geometry before fusing the refined object and scan points.
  • Dataset Variants: nuScenes-OoD contains 6019 validation scans from a 32-beam LiDAR sensor, SemanticPOSS-OoD contains 500 validation samples from a 40-beam sensor, and SemanticKITTI-OoD contains 4071 validation scans from a 64-beam sensor.These dataset sizes and sensor resolutions provide complementary evaluation conditions.
  • Dataset Variants: Single and multi splits contain one or multiple anomaly objects per scan, respectively, with anomalies present in about 40% and 60% of scans.The split design balances realistic anomaly frequency with a stronger evaluation setup.

5. Experiments

Experiments evaluate the approach on the STU dataset and three mixed real-synthetic OoD benchmarks, including accuracy, efficiency, semantic segmentation, and ablation analyses. Results show strong STU performance, competitive mixed-dataset results, and a lightweight runtime profile, while performance remains bounded by data sparsity, resolution, imbalance, and model uncertainty.

  • STU evaluation: The approach achieves state-of-the-art results on STU validation and test sets, surpassing other methods across all metrics.
  • STU evaluation: +9.82% AP over ensemble-based approaches is reported on STU, alongside robust FPR metrics despite the domain gap.
  • Mixed real-synthetic benchmarks: The approach significantly surpasses other methods on both SemanticPOSS-OoD splits, although sparse instances and lower scan resolution bound performance.
  • Mixed real-synthetic benchmarks: On SemanticKITTI-OoD, the method is state-of-the-art on the simple split and competitive on the multi split, where baselines are slightly ahead.
  • Mixed real-synthetic benchmarks: On nuScenes-OoD, the approach remains competitive despite lower resolution and is lighter and faster than ensemble-based alternatives.
  • Semantic segmentation: Semantic segmentation remains competitive, though additional anomaly losses slightly reduce results and lower resolution reduces effectiveness on nuScenes-OoD.
  • Efficiency: The approach achieves real-time performance below 100 ms and a balanced accuracy–computational-cost trade-off compared with ensemble models.
  • Ablation study: The combined ablation score performs best overall, with prototype, contrastive, and objectosphere losses contributing complementary improvements.

6. Conclusion

The paper introduces feature-space modeling for LiDAR anomaly segmentation and mixed real-synthetic OoD datasets. Experiments report state-of-the-art and competitive results, while future work targets cross-domain generalization and uncertainty.

  • The approach models inlier class representations in feature space to identify out-of-distribution objects.It combines prototype, contrastive, and objectosphere losses to constrain anomaly features in the embedding space.
  • The proposed mixed real-synthetic datasets use geometrically aligned synthetic anomalies to address scarcity of LiDAR anomaly-segmentation data.
  • Experiments achieve state-of-the-art results on real data and competitive results on mixed datasets.
  • Future work will investigate cross-domain tasks and uncertainty to improve generalization, reliability, and robustness of anomaly predictions.

A. Further Details on OoD Datasets

The proposed OoD datasets combine benchmark LiDAR scans with carefully selected ModelNet objects, realistic intensity modeling, and range-image-based geometric alignment. The construction varies insertion surfaces, object counts, labels, and sensor-specific projection settings across datasets.

  • The datasets are built from three autonomous-driving benchmarks with ModelNet objects selected to avoid overlap with training and evaluation data.
  • ModelNet surfaces are densely sampled, assigned temporary intensity values, and scaled to support alignment with LiDAR scan geometry.
  • Insertion surfaces differ by dataset and split, including road, parking, sidewalk, other-ground, and ground classes.
  • Multi-split scenes insert 1, 2, 3, or 4 objects with probabilities of 40%, 30%, 20%, and 10%, respectively, within a 50 m sensor-centered radius.
  • Anomaly labels use value 2 in SemanticKITTI-OoD and SemanticPOSS-OoD, but value 100 in nuScenes-OoD to avoid a class conflict.
  • Per-point intensity approximates real-world behavior using Lambertian reflectance and object-specific material reflectivity values in [0, 1].
  • Spherical projection to a range image preserves occlusion and beam-like sampling, retaining the closest point when multiple points share a cell.
  • Projection and reprojection preserve visible object points and prevent overlap with existing LiDAR instances.

B. Further Details on Experiments

The experiments compare available LiDAR anomaly-segmentation baselines and analyze dataset construction, thresholds, losses, runtime, and range-based performance. The supplied passages describe the evaluation setup and ablation targets but provide few numerical outcomes.

  • Experiments on the proposed datasets use implementations of max logit, RbA, and deep ensemble, supplemented by standard OoD methods.
  • Void Classifier identifies anomalies using confidence from a network trained with an additional unlabeled/outlier class.
  • The evaluation includes threshold ablation, semantic-segmentation loss ablation, runtime comparison, and range-based AP on STU.
  • MC Dropout averages predictions across 10 inference forward passes, while deep ensembles combine three differently seeded models.

B.2. Additional Ablation Study

Additional ablations examine threshold selection, semantic segmentation, and runtime. The results show a trade-off between semantic mIoU and anomaly segmentation, alongside real-time inference for the proposed approach.

  • Threshold r requires tuning when training on SemanticKITTI and testing on STU because sensor resolution and feature distributions create a domain gap.
  • Adding prototype loss decreases semantic mIoU, while contrastive and objectosphere losses slightly recover it by improving class-feature separation.
  • The slight semantic-segmentation reduction accompanies improved anomaly-segmentation results.
  • The proposed approach maintains real-time performance below 100 ms across datasets, including 128-beam STU scans.
  • Mask4Former3D with post-processing, especially ensembles, requires seconds for a single prediction.

B.4. Additional Results

Additional experiments show that the method performs strongly for anomaly segmentation across STU and mixed real-synthetic datasets, while semantic segmentation remains comparable to a standard baseline. Performance is strongest at shorter anomaly distances, whereas distant objects and lower-resolution scans remain challenging.

  • At shorter distance thresholds, the method achieves superior AP, but performance decreases as anomalous objects become more distant.This distance-related decline is observed across all approaches, leaving distant objects as a challenging setting.
  • The method significantly outperforms all baselines in AP on STU and achieves strong performance across the proposed mixed real-synthetic datasets.Object-level metrics are also generally better, while standard OoD techniques with comparable backbones can be comparable or inferior to the Mask4Former3D baseline.
  • Semantic segmentation performance is comparable to the standard baseline, with only a small degradation on nuScenes-OoD.The degradation is attributed to nuScenes' lower resolution and fewer points per scan, which may affect robust prototype construction.

C. Qualitative Results

Qualitative comparisons visualize anomaly segmentation on STU and three mixed real-synthetic OoD datasets. Across these datasets, the proposed approach shows strong performance that is better than or comparable to the deep ensemble model, with both successes and failures reported.

  • Qualitative Results: Figures 7–10 compare anomaly segmentation results on STU, SemanticPOSS-OoD, SemanticKITTI-OoD, and nuScenes-OoD.The visualizations include both successful and failure cases and use the deep ensemble model as the comparison method.
  • Qualitative Results: The proposed approach demonstrates better or comparable anomaly segmentation performance to the deep ensemble across the evaluated datasets.The comparison covers the STU validation set and the three mixed real-synthetic OoD datasets.
  • Qualitative Results: The reported qualitative datasets correspond to STU, SemanticPOSS-OoD, SemanticKITTI-OoD, and nuScenes-OoD.The associated figures are Figures 7–10, while Tables 15–18 provide anomaly-segmentation performance for the same dataset group.
Loading 2604.23604v1…