Source-linked AI summary
Panoptic-PolarNet: Proposal-free LiDAR Point Cloud Panoptic Segmentation
Zixiang Zhou, Yang Zhang, Hassan Foroosh
TL;DR
Efficient LiDAR panoptic segmentation remains under-explored because LiDAR data are irregularly sampled and applications require real-time processing. Panoptic-PolarNet combines semantic prediction with class-agnostic instance clustering in a proposal-free polar BEV framework. It reports state-of-the-art results on SemanticKITTI and nuScenes with negligible computational overhead, while end-to-end proposal-free training remains open for future work.
Problem
LiDAR panoptic segmentation remains an open problem because irregular point clouds, prediction conflicts, and real-time accuracy requirements complicate efficient processing.
Method
Panoptic-PolarNet jointly learns semantic segmentation and class-agnostic instance clustering with shared decoding and a proposal-free polar BEV design.
Results
Panoptic-PolarNet achieves state-of-the-art results on SemanticKITTI and nuScenes with negligible computation overhead and near-real-time latency.
Takeaways & Limitations
The framework provides a strong baseline and a migration path for semantic segmentation methods toward LiDAR panoptic segmentation.
Takeaways & Limitations
End-to-end trainable proposal-free panoptic segmentation remains an open problem for future exploration.
Abstract
from arXiv · showhide
Panoptic segmentation presents a new challenge in exploiting the merits of both detection and segmentation, with the aim of unifying instance segmentation and semantic segmentation in a single framework. However, an efficient solution for panoptic segmentation in the emerging domain of LiDAR point cloud is still an open research problem and is very much under-explored. In this paper, we present a fast and robust LiDAR point cloud panoptic segmentation framework, referred to as Panoptic-PolarNet. We learn both semantic segmentation and class-agnostic instance clustering in a single inference network using a polar Bird's Eye View (BEV) representation, enabling us to circumvent the issue of occlusion among instances in urban street scenes. To improve our network's learnability, we also propose an adapted instance augmentation technique and a novel adversarial point cloud pruning method. Our experiments show that Panoptic-PolarNet outperforms the baseline methods on SemanticKITTI and nuScenes datasets with an almost real-time inference speed. Panoptic-PolarNet achieved 54.1% PQ in the public SemanticKITTI panoptic segmentation leaderboard and leading performance for the validation set of nuScenes.
1. Introduction
LiDAR panoptic segmentation remains an open, real-time processing challenge because point clouds are irregularly sampled and semantic and instance predictions can conflict or duplicate work. Panoptic-PolarNet addresses this with proposal-free instance clustering, shared semantic-instance decoding, and polar BEV features, achieving strong accuracy and latency results.
- Motivation: Separate semantic and instance heads can create conflicting labels, redundant information, and inference inefficiency.Proposal-based methods may assign different labels to the same point, while proposal-free instance features correlate strongly with class labels.
- Motivation: 3D LiDAR panoptic segmentation remains an open problem requiring real-time processing for autonomous driving, robotics, and environment mapping.LiDAR point clouds are irregularly sampled, creating distinct challenges from conventional 3D data.
- Approach: Panoptic-PolarNet jointly learns semantic prediction and class-agnostic instance clustering on a discretized polar BEV representation.It builds on PolarNet for semantic prediction, adds a lightweight 2D instance head, and fuses predictions through majority voting.
- Approach: Shared decoding layers enable early fusion, reducing redundancy and increasing PQ despite a smaller computation load.The architecture differs from methods using separate semantic and instance decoding modules connected by attention.
- Approach: Proposal-free design avoids class-instance prediction overlap and trains the instance head without bounding-box annotations.This design addresses conflicts associated with proposal-based panoptic segmentation.
- Results: Panoptic-PolarNet outperforms strong baselines on SemanticKITTI and nuScenes with smaller, near-real-time latency.On top of PolarNet, the instance head adds 0.1M parameters and 0.027s of inference time.
2. Related works
Related work spans proposal-based and proposal-free panoptic segmentation, point-, voxel-, BEV-, and range-image representations, and emerging LiDAR panoptic methods. LiDAR-specific challenges include sparse, imbalanced 2.5D sampling and large autonomous-driving data volumes.
- Panoptic segmentation: Panoptic segmentation methods are categorized as top-down proposal-based or bottom-up proposal-free according to how they separate instances.Top-down methods first obtain object instance masks, whereas bottom-up methods learn representations or predictions for grouping instances.
- Panoptic segmentation: Recent 2D methods address semantic-instance conflicts through spatial ranking, unknown labels, confidence-based fusion, end-to-end training, or attention modules.
- LiDAR representations: LiDAR is inherently 2.5D and has sparse, imbalanced spatial sampling, motivating point-, voxel-, BEV-, range-image-, and multi-view representations.Autonomous-driving applications also create larger data volumes than conventional point-cloud settings.
- LiDAR representations: LiDAR object detection likewise includes proposal-based methods that refine bounding boxes and proposal-free methods based on voting or keypoint and center estimation.
- LiDAR panoptic segmentation: LiDAR panoptic segmentation remains relatively under-studied, while indoor approaches commonly combine semantic segmentation with discriminative embedding-based instance clustering.
3. Panoptic-PolarNet
Panoptic-PolarNet converts LiDAR points into a polar BEV representation, uses a shared encoder-decoder with semantic and class-agnostic instance heads, and fuses their predictions into panoptic segmentation. Its design exploits BEV instance separability and adds augmentation, pruning, and visibility features to improve learnability and efficiency.
- Polar BEV representation: Panoptic-PolarNet encodes variable-size LiDAR point clouds into a fixed-size 2D polar BEV representation for efficient processing.BEV enables 2D convolutional processing, while polar coordinates balance point distributions across ranges.
- Shared network: A shared encoder-decoder backbone produces semantic predictions, center heatmaps, and offset regressions for panoptic segmentation.The backbone shares decoding layers between semantic and instance processing before producing task-specific outputs.
- Prediction fusion: Voting-based fusion combines semantic foreground predictions with center-based instance grouping to assign unique instance labels to thing classes.Top heatmap centers are selected after non-maximum suppression, and foreground pixels are assigned according to offset-adjusted distances.
- Design assumption: The method assumes most thing-class objects remain separable after projection onto the XY-plane, making BEV suitable for instance clustering.The paper links this separability to reducing LiDAR instance clustering to a 2D problem.
- Instance grouping: The instance head predicts object centers and offsets, then groups BEV pixels by their nearest center using class-agnostic clustering.This bottom-up design avoids overlapping class predictions between semantic and instance branches and does not require bounding-box annotations.
- Training and features: The framework augments training with instance transformations, self-adversarial point pruning, and a polar-coordinate visibility feature.Adversarial pruning removes influential points during late training to encourage general features, while visibility is concatenated with polar BEV features.
4. Experiment
Experiments evaluate Panoptic-PolarNet on SemanticKITTI and nuScenes using panoptic quality, semantic quality, recognition quality, and inference time. The method outperforms baselines, while ablations and oracle tests analyze shared decoding, augmentation, pruning, visibility features, and semantic-prediction limits.
- SemanticKITTI results: Panoptic-PolarNet outperforms the best SemanticKITTI test baseline by 1.4% in PQ with near-real-time inference speed.The mini version achieves a comparable result with only two-thirds of the inference time.
- SemanticKITTI results: All three Panoptic-PolarNet settings outperform the best SemanticKITTI validation baseline by a large margin, while polar coordinates prevail over Cartesian coordinates on every metric.The polar representation has slower inference time than the Cartesian alternative.
- nuScenes results: Panoptic-PolarNet outperforms the nuScenes validation combining baseline by 1.1% in PQ with only half the inference time.Inference is slightly slower on nuScenes because the dataset contains more instances.
- Ablation studies: Sharing the first three decoding layers increases PQ from 51.6% to 52.3% while reducing computation relative to independent semantic and instance decoding networks.The ablation replaces an attention connection between the heads with directly shared decoding layers.
- Ablation studies: Visibility features improve PQ by 1.6%, with the largest benefits for bicyclist and motorcyclist classes surrounded mostly by visible space.Instance global and local augmentation also improve results, while self-adversarial pruning slightly improves PQ and stabilizes semantic results, especially for stuff classes.
5. Conclusion
Panoptic-PolarNet is a real-time, proposal-free LiDAR panoptic segmentation framework that builds on semantic segmentation and performs instance segmentation through center regression on a polar BEV map. It achieves state-of-the-art results on SemanticKITTI and nuScenes while adding negligible computation overhead.
- Panoptic-PolarNet is a real-time proposal-free framework for LiDAR point cloud panoptic segmentation.
- The method builds on an established semantic segmentation network and performs instance segmentation by center regression on a polar BEV map.
- Its design simplifies panoptic segmentation complexity and adds negligible computation overhead beyond semantic segmentation.
- Panoptic-PolarNet achieves state-of-the-art results on both SemanticKITTI and nuScenes datasets.
- The proposed augmentation methods are presented as generalizable to other LiDAR point cloud segmentation methods.
6. Supplementary Material
The supplementary material discusses training choices, proposal-free design, pruning behavior, class-wise performance, visualizations, and the remaining limitation of non-end-to-end training.
- Training choices: Cross-entropy and Lovasz softmax losses are combined to address class imbalance while optimizing semantic segmentation toward overall mIoU.Cross-entropy favors frequent classes, whereas Lovasz softmax directly optimizes the mIoU Jaccard index and treats classes equally.
- Training choices: Instance oversampling addresses imbalance between “thing” and “stuff” classes and between different “thing” classes.In SemanticKITTI, cars have around 107 times more points than motorcyclists.
- Proposal-free design: Proposal-free methods are favored for LiDAR because proposal-based approaches rely heavily on bounding-box annotations and may struggle to maintain competitive speed after modification.
- Training limitation: The final panoptic segmentation is generated by majority-voting fusion from an intermediate network result.
- Self-adversarial pruning: SAP tends to remove challenging or ambiguous points, including tangled vegetation, fences, and garden curbs annotated as fences.
- Evaluation: Lower class-wise performance is associated with classes having small physical shapes or few dataset instances, such as bicycles, trucks, and construction vehicles.
- Evaluation: Visualizations report accurate instance predictions across distance and point-density variation, while also showing duplicated predictions at short distances.
- Evaluation: Class-wise results are reported for the SemanticKITTI test split and nuScenes validation split.