PaperlayerView exact revision on arXiv

Source-linked AI summary

LiDAR-based Panoptic Segmentation via Dynamic Shifting Network

Fangzhou Hong, Hui Zhou, Xinge Zhu, Hongsheng Li, Ziwei Liu

arXiv:2011.11964v2cs.CV

TL;DR

LiDAR perception research has largely focused on objects or scenes, while holistic point-level panoptic segmentation remains a challenge, especially with non-uniform point distributions. DS-Net combines shared LiDAR features, adaptive dynamic shifting, and consensus-driven fusion; it outperforms state-of-the-art methods on both benchmarks and ranks first on SemanticKITTI’s public leaderboard.

  • Problem

    LiDAR-based panoptic segmentation aims to provide full-spectrum point-level predictions, but non-uniform point distributions and varying instance sizes make reliable clustering difficult.

    Source: §1 · pp. 1–2

  • Method

    DS-Net shares cylinder-convolution features across semantic and instance branches, uses dynamic shifting to cluster regressed centers, and fuses the two predictions.

    Source: §1 · p. 2

  • Results

    DS-Net outperforms all state-of-the-art methods on both benchmarks and places first on the SemanticKITTI public leaderboard.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work reports strong baseline results and extensive comparisons for LiDAR-based panoptic segmentation on large-scale datasets.

    Source: §1 · p. 2

Abstract

from arXiv · show

With the rapid advances of autonomous driving, it becomes critical to equip its sensing system with more holistic 3D perception. However, existing works focus on parsing either the objects (e.g. cars and pedestrians) or scenes (e.g. trees and buildings) from the LiDAR sensor. In this work, we address the task of LiDAR-based panoptic segmentation, which aims to parse both objects and scenes in a unified manner. As one of the first endeavors towards this new challenging task, we propose the Dynamic Shifting Network (DS-Net), which serves as an effective panoptic segmentation framework in the point cloud realm. In particular, DS-Net has three appealing properties: 1) strong backbone design. DS-Net adopts the cylinder convolution that is specifically designed for LiDAR point clouds. The extracted features are shared by the semantic branch and the instance branch which operates in a bottom-up clustering style. 2) Dynamic Shifting for complex point distributions. We observe that commonly-used clustering algorithms like BFS or DBSCAN are incapable of handling complex autonomous driving scenes with non-uniform point cloud distributions and varying instance sizes. Thus, we present an efficient learnable clustering module, dynamic shifting, which adapts kernel functions on-the-fly for different instances. 3) Consensus-driven Fusion. Finally, consensus-driven fusion is used to deal with the disagreement between semantic and instance predictions. To comprehensively evaluate the performance of LiDAR-based panoptic segmentation, we construct and curate benchmarks from two large-scale autonomous driving LiDAR datasets, SemanticKITTI and nuScenes. Extensive experiments demonstrate that our proposed DS-Net achieves superior accuracies over current state-of-the-art methods. Notably, we achieve 1st place on the public leaderboard of SemanticKITTI, outperforming 2nd place by 2.6% in terms of the PQ metric.

1. Introduction

The introduction frames LiDAR-based panoptic segmentation as holistic, point-level parsing of both objects and scenes, and presents DS-Net as an approach to the task. It highlights complex point distributions, dynamic shifting, and consensus-driven fusion as central design concerns.

1. Introduction

  • LiDAR-based panoptic segmentation requires full-spectrum point-level predictions, combining semantic labels for background stuff and instance segmentation for foreground things.The introduction positions this task as a step toward holistic perception for challenging autonomous-driving scenes.Source: §1 · pp. 1–2
  • Non-uniform LiDAR density and varying instance sizes produce noisy regressed-center distributions that challenge heuristic clustering methods.The introduction contrasts these distributions with the dense, uniform indoor point clouds for which many existing instance-segmentation methods were designed.Source: §1 · p. 2
  • Cylinder convolution extracts grid-level features in one pass and shares them between DS-Net’s semantic and instance branches.The paper describes this as its backbone design for LiDAR frames.Source: §1 · p. 2
  • Dynamic shifting adaptively weights shift candidates to move regressed centers toward cluster centers across instances with different densities and sizes.The paper reports that the module is robust to parameter settings.Source: §1 · p. 2
  • Consensus-driven fusion unifies semantic and instance predictions to resolve disagreement associated with class-agnostic instance segmentation, with negligible computation overhead.The fusion module produces the final panoptic segmentation results.Source: §1 · p. 2
  • DS-Net outperforms state-of-the-art methods on SemanticKITTI and nuScenes, placing first on the SemanticKITTI public leaderboard.The paper also reports adapting a LiDAR-based panoptic segmentation dataset from nuScenes and establishing strong baselines.Source: §1 · p. 2

2. Related Works

Related work covers point-cloud semantic and instance segmentation, including approaches tailored to LiDAR’s sparse, non-uniform distributions. Prior LiDAR instance methods use several clustering and prediction strategies, while DS-Net is framed as offering insights into the distinct panoptic task.

2. Related Works

  • Point-cloud semantic segmentation includes point-based methods and voxel-based sparse convolutions, while LiDAR methods use range images or polar and cylindrical divisions.The related-work discussion describes cylindrical division as better suited to sparse, non-uniform LiDAR point distributions while retaining efficiency.Source: §2 · p. 2
  • Indoor instance-segmentation methods include blockwise point embeddings and whole-scene sparse-convolution approaches, with heuristic post-processing and lack of perception noted as limitations.The cited whole-scene approaches include PointGroup, 3D-MPA, and OccuSeg.Source: §2 · p. 2
  • Prior LiDAR instance methods cluster XYZ coordinates after semantic segmentation, use top-down metric learning, or apply grid-level center voting.The authors state that DS-Net is not targeting LiDAR instance segmentation but may provide insights into that task.Source: §2 · p. 3

3. Our Approach

DS-Net combines cylinder-convolution features shared by semantic and instance branches with dynamic shifting for clustering regressed centers and majority-vote fusion of the two predictions.

3.1. Strong Backbone Design

  • Cylinder convolution extracts grid-level LiDAR features shared by semantic and instance branches, with center regression preparing thing points for clustering into instance IDs.The semantic branch predicts voxel-grid confidences, while the instance branch predicts offsets from points to instance centers.Source: §3.1 · pp. 3–4

3.2. Dynamic Shifting

  • Non-uniform density, sparsity, incomplete scans, and varying instance sizes produce noisy center distributions that make heuristic clustering unreliable.A fixed-radius method can over-segment distant instances or under-segment nearby ones; fixed-bandwidth Mean Shift cannot handle large and small instances simultaneously.Source: §3.2 · p. 4
  • Dynamic shifting iteratively moves regressed centers toward cluster centers using learned weights over multiple bandwidth-based shift candidates.Farthest point sampling selects seed points; after shifting, heuristic clustering assigns seed IDs and other thing points inherit the nearest seed's ID.Source: §3.2 · pp. 4–5

3.3. Consensus-driven Fusion

  • Consensus-driven fusion assigns each instance the majority semantic label among its points, unifying semantic and instance predictions.The ablation caption reports that both proposed modules contribute to DS-Net's final performance and that weighting bandwidth candidates is better than directly regressing bandwidth.Source: §3.3 · p. 6, Figure 4 · p. 6

4. Experiments

Experiments evaluate DS-Net on SemanticKITTI and nuScenes, with ablations and further analyses examining its modules, parameter robustness, learned bandwidths, and shifting behavior.

4. Experiments

  • The SemanticKITTI evaluation uses 23,201 training and 20,351 test frames, with 28 annotated classes remapped to 19 panoptic classes.The nuScenes benchmark is constructed from lidarseg point labels and detection boxes; results are compared on its validation split because test annotations are held out.Source: §4 · p. 6

4.1. Ablation Study

  • Both consensus-driven fusion and dynamic shifting contribute to performance; dynamic shifting improves PQTh by 3.2% over the backbone with fusion on the validation split.Dynamic shifting outperforms BFS, DBSCAN, HDBSCAN, and Mean Shift; among heuristic methods, Mean Shift performs best.Source: §4.1 · p. 6

4.2. Evaluation Comparisons on SemanticKITTI

  • 5.7% in PQ and 11.9% in PQTh over KPConv + PV-RCNN on the SemanticKITTI test split; DS-Net also leads the public leaderboard.The leaderboard margin over PolarNet seg is 2.6% in PQ and 3.0% in PQTh.Source: Table 2 · p. 7, §4.2 · p. 7

4.1. Ablation Study

  • Direct bandwidth regression is difficult to optimize because the best per-point bandwidth is unclear, whereas choosing among and combining candidates is easier to supervise.The supervision can determine whether a shift moves a seeding point closer to its target center, rather than specify an ideal bandwidth.Source: §4.1 · p. 7

4.3. Evaluation Comparisons on nuScenes

  • 2.4% in PQ and 3.5% in PQTh over the best baseline on nuScenes, supporting DS-Net's generalizability to extremely sparse single-frame point clouds.DS-Net outperforms the best baseline in most metrics.Source: §4.3 · p. 7, Table 3 · p. 8

4.4. Further Analysis

  • DS-Net remains stable across bandwidth-candidate settings when candidates are comparable to instance sizes, unlike heuristic clustering methods requiring extensive parameter adjustment.Further iteration-number analyses are reported in supplementary materials.Source: §4.4 · pp. 7–8, Table 4 · p. 8
  • Learned bandwidths are roughly proportional to the corresponding classwise instance sizes, consistent with dynamic shifting adapting to different instance sizes.Figure 5 relates classwise average regressed-center sizes to average learned bandwidths.Source: §4.4 · p. 8, Figure 5 · p. 8
  • In visualizations, seeding points converge around instance centers after four shifting iterations, with farther seeds tending to learn higher bandwidths.Figure 6 encodes original points in black and seeding points in spectral colors.Source: §4.4 · p. 8, Figure 6 · p. 8

5. Conclusion

The conclusion presents DS-Net as an early LiDAR panoptic segmentation approach designed for non-uniform point distributions, with strong results on two benchmarks.

5. Conclusion

  • DS-Net outperforms strong baselines on SemanticKITTI and nuScenes, with further analyses showing dynamic-shifting robustness and interpretable learned bandwidths.Its dynamic shifting adapts regressed instance centers to different densities and sizes, while consensus-driven fusion combines semantic and instance results.Source: §5 · p. 8

Appendix

The appendix gathers supplementary analyses, implementation details, per-class results, and additional visualizations.

Appendix

  • The appendix covers heuristic clustering details, dynamic-shifting differentiability and iteration analyses, implementation details, per-class results, and more visualizations.These sections are provided to support understanding of the main paper and reproducibility.Source: Appendix · p. 11

1. Details of Heuristic Clustering Algorithms

The section reviews heuristic clustering methods and explains why their assumptions limit performance on LiDAR point clouds. Mean Shift is more robust to density changes, but its bandwidth and nearest-center assignment remain problematic for differently sized instances.

1. Details of Heuristic Clustering Algorithms

  • BFS and DBSCAN rely on fixed-radius neighborhoods, which cannot adapt to LiDAR’s large density differences or reliably distinguish sparse instances from noise.DBSCAN also labels points with too few neighbors inside a radius as noise.Source: §1 · p. 11
  • HDBSCAN treats low-density points as likely noise, but distant valid LiDAR instances can also be low-density, limiting clustering quality.Its mutual-reachable-distance graph and cluster hierarchy do not remove this mismatch with LiDAR point distributions.Source: §1 · p. 11
  • Mean Shift iteratively moves sampled seeds with a kernel, then assigns points to their nearest converged cluster centers.The described flat kernel averages points within a query ball, and the bandwidth sets that ball’s radius.Source: §1 · p. 11
  • A fixed Mean Shift bandwidth can split single instances or merge neighbors, while nearest-center assignment can misassign points from differently sized instances.Small bandwidth risks over-segmentation; large bandwidth risks under-segmentation.Source: §1 · p. 11

2. Gradient Calculation of Dynamic Shifting

This section describes gradient calculation for dynamic shifting and notes that directly regressing bandwidth with a flat kernel is not differentiable. A Gaussian kernel makes that direct-regression variant differentiable.

2. Gradient Calculation of Dynamic Shifting

  • Dynamic shifting’s forward pass can be decomposed to derive gradients for one iteration, while direct bandwidth regression with a flat kernel is not differentiable.The backward calculation treats S_j as constant, so it has no gradients.Source: §2 · pp. 11–12
  • The gradient derivation uses backbone features F, function parameters p_j, shifted seeds Y, and loss c.The prose identifies ⊙ as element-wise product and states that S_j is constant.Source: §2 · p. 12
  • Using a Gaussian kernel makes the direct-regression version of dynamic shifting differentiable.The bandwidth δ is learnable in this variant.Source: §2 · p. 12

3. Further Analyses of the Number of Iterations

The analysis selects four dynamic-shifting iterations as the best setting, balancing convergence with computational cost. Learned bandwidths fall to their lower limit for most classes by the fourth iteration.

3. Further Analyses of the Number of Iterations

  • Too few iterations leave convergence insufficient, whereas too many add unnecessary time and space complexity.The iteration count is a hyperparameter considered alongside bandwidth candidates.Source: §3 · p. 12
  • 4 iterations achieves the best result among the tested settings, balancing sufficient convergence against added time and space complexity.Table 5 reports experiments on iteration counts; the prose identifies four as the best setting.Source: §3 · p. 12, Table 5 · p. 12
  • After four iterations, learned bandwidths for most classes reach 0.2, their lower limit, which the authors take as evidence that thing points have converged.The figure description also reports that bandwidths decrease with iteration count.Source: §3 · p. 12, Figure 7 · p. 12

4. Implementation Details

The implementation voxelizes each LiDAR frame in cylindrical coordinates and specifies dataset-specific instance thresholds and clustering settings. It approximates instance centers using tight-box centers rather than mass centers of incomplete point clouds.

4. Implementation Details

  • For both datasets, each input point is a four-dimensional vector of XYZ coordinates and intensity; the backbone voxelizes each single frame into 480 × 360 × 32 voxels in cylindrical coordinates.For segmentation, each instance’s ground-truth center is approximated by the center of its axis-parallel tight box.Source: §4 · p. 12
  • The minimum valid-instance point count is 50 for SemanticKITTI and 5 for nuScenes.Both datasets use 10,000 FPS-downsampled points, bandwidth candidates 0.2, 1.7, and 3.2, and four iterations; final clustering uses Mean Shift for SemanticKITTI and BFS for nuScenes.Source: §4 · p. 13

5. Per-class Evaluation Results

On SemanticKITTI, DS-Net leads the public leaderboard and outperforms the backbone with fusion across all classes in PQ, RQ, and SQ. It also exceeds the second-place method on most classes.

5. Per-class Evaluation Results

  • The paper reports that DS-Net achieved 1st place on SemanticKITTI’s public leaderboard, with a screenshot documenting the leaderboard as accessed on 2020-11-16.Source: §5 · p. 13
  • DS-Net surpasses the backbone with fusion in all classes across PQ, RQ, and SQ.The authors attribute this result to the dynamic shifting module.Source: §5 · p. 13
  • Compared with semantic segmentation plus detection baselines, DS-Net has large advantages on things classes in PQ, RQ, and SQ.Source: §5 · p. 13
  • DS-Net outperforms the second-place PolarNet seg method on most classes.Source: §5 · p. 13

6. More Visualization Results of the DS-Net

The qualitative comparison visualizes nine SemanticKITTI validation frames, showing the backbone with fusion, DS-Net, and ground truth side by side. The highlighted regions illustrate how the methods handle instances of different sizes and densities.

6. More Visualization Results of the DS-Net

  • Nine SemanticKITTI validation frames compare the backbone with fusion, DS-Net, and ground truth side by side.Regions of interest are enlarged; stuff colors follow the official definition, while things-instance colors are randomly assigned.Source: §6 · p. 13
  • In complex regions with varied instance sizes and densities, DS-Net handles instances correctly while the backbone tends to oversegment or undersegment.Source: §6 · p. 13
  • Tables 6, 7, and 8 present per-class test-set results for PQ, RQ, and SQ, respectively.Source: Table 6 · p. 14, Table 7 · p. 14, Table 8 · p. 14
  • The leaderboard screenshot records DS-Net in 1st place on SemanticKITTI as of 2020-11-16.Source: Figure 8 · p. 14

Related papers

Loading 2011.11964v2…