PaperlayerView exact revision on arXiv

Source-linked AI summary

Panoptic nuScenes: A Large-Scale Benchmark for LiDAR Panoptic Segmentation and Tracking

Whye Kit Fong, Rohit Mohan, Juana Valeria Hurtado, Lubing Zhou, Holger Caesar, Oscar Beijbom, Abhinav Valada

arXiv:2109.03805v3cs.CVcs.AIcs.LGcs.RO

TL;DR

Existing datasets provide limited urban diversity and dynamic-object coverage for learning and benchmarking LiDAR panoptic scene understanding and tracking. The paper introduces Panoptic nuScenes with point-wise annotations across three tasks, strong baselines, and the instance-centric PAT metric. Experiments report benchmark results, including a best PAT score of 67.1% on the test set and a 20.8 PQ gap favoring independently combined baselines for thing classes.

  • Problem

    Existing datasets lack urban-scene diversity, dynamic-object instances, or temporal evaluation needed for LiDAR panoptic segmentation and tracking.

  • Method

    The paper extends nuScenes with point-wise annotations for semantic segmentation, panoptic segmentation, and panoptic tracking, and proposes the PAT metric.

  • Results

    67.1% is the highest PAT score achieved by EfficientLPS with Kalman Filter on the test set; independently combined baselines exceed end-to-end methods by 20.8 PQ for thing classes.

  • Takeaways & Limitations

    Panoptic nuScenes provides a public multi-task LiDAR benchmark with an evaluation server and evidence that task-specific combinations remain stronger than unified approaches.

Abstract

from arXiv · show

Panoptic scene understanding and tracking of dynamic agents are essential for robots and automated vehicles to navigate in urban environments. As LiDARs provide accurate illumination-independent geometric depictions of the scene, performing these tasks using LiDAR point clouds provides reliable predictions. However, existing datasets lack diversity in the type of urban scenes and have a limited number of dynamic object instances which hinders both learning of these tasks as well as credible benchmarking of the developed methods. In this paper, we introduce the large-scale Panoptic nuScenes benchmark dataset that extends our popular nuScenes dataset with point-wise groundtruth annotations for semantic segmentation, panoptic segmentation, and panoptic tracking tasks. To facilitate comparison, we provide several strong baselines for each of these tasks on our proposed dataset. Moreover, we analyze the drawbacks of the existing metrics for panoptic tracking and propose the novel instance-centric PAT metric that addresses the concerns. We present exhaustive experiments that demonstrate the utility of Panoptic nuScenes compared to existing datasets and make the online evaluation server available at nuScenes.org. We believe that this extension will accelerate the research of novel methods for scene understanding of dynamic urban environments.

I. INTRODUCTION

Existing LiDAR datasets provide increasingly detailed scene representations, but often lack instance labels, temporal consistency, diversity, or sufficient moving-object coverage for panoptic tracking. Panoptic nuScenes addresses these gaps by extending nuScenes into a broad benchmark spanning semantic segmentation, panoptic segmentation, and panoptic tracking.

  • Motivation: Bounding boxes approximate cars but poorly capture object extremities and pedestrian interactions, motivating point-level scene representations.Point-level semantic labels provide finer geometric detail than fixed-size 3D bounding boxes.
  • Task gap: Panoptic segmentation adds instance labels to semantic point labels, while panoptic tracking additionally enforces temporally consistent instance identities across scans.The tracking task combines panoptic segmentation with multi-object tracking into one scene-understanding problem.
  • Benchmark: Panoptic nuScenes annotates 40,000 keyframes with 32 semantic classes and 1.1B labeled LiDAR points across 1,000 scenes from Singapore and Boston.The dataset extends nuScenes, which also includes other autonomous-driving tasks and diverse sensor coverage.
  • Contributions: The benchmark covers semantic segmentation, panoptic segmentation, and panoptic tracking, with a new PAT metric and extensive baselines for comparison.The PAT metric combines panoptic and tracking quality while penalizing track fragmentation.
  • Dataset gap: Existing datasets and benchmarks commonly lack instance annotations, contain few moving instances, or ignore the temporal dimension.The related-work review identifies a shortage of public datasets supporting both panoptic segmentation and tracking.

III. DATASET

Panoptic nuScenes extends a geographically diverse multi-city driving dataset with temporally consistent point-level semantic and instance annotations. Its class distribution and abundance of moving agents make it a challenging benchmark for panoptic tracking.

  • Dataset overview: Panoptic nuScenes contains 1,000 scenes collected across multiple cities, with both left- and right-hand driving and diverse dynamic agents.The varied scenes and many moving agents are intended to support panoptic-tracking benchmarking.
  • Annotation: The dataset uses 32 semantic classes, comprising 23 thing classes and 9 stuff classes.Each LiDAR point receives a manually annotated semantic label, with 3D boxes used to initialize thing-class labels before refinement.
  • Annotation: Instance labels are formed by combining point-level labels with 3D bounding boxes, while overlapping points are labeled as noise.An instance contains points inside a box that share the box’s segmentation type.
  • Dataset analysis: The dataset contains 1.1B LiDAR points with temporally consistent instance IDs for thing classes.Dynamic classes such as cars and adult pedestrians are frequent, while some classes are rare.
  • Dataset analysis: Panoptic nuScenes contains significantly more moving object instances and a greater variety of moving object classes than SemanticKITTI.Figure 3 compares scan-wise moving instances, using ridden bicycles and motorcycles as a proxy for moving objects.

IV. TASKS AND METRICS

Panoptic nuScenes evaluates point-level semantics, instances, and temporal association for LiDAR scene understanding. Its PAT metric combines panoptic quality with instance-level tracking quality while targeting fragmentation, ID errors, and long-term consistency.

  • Benchmark Tasks: The evaluation includes only object instances with more than 15 LiDAR points and merges similar or rare classes into 10 thing and 6 stuff classes.These task-level conventions define the evaluated label space and instance eligibility.
  • Benchmark Tasks: Semantic segmentation is evaluated primarily with mean IoU across classes, while panoptic segmentation reports separate thing and stuff panoptic-quality components.The benchmark also reports frequency-weighted IoU for semantic segmentation and PQ†_c for completeness.
  • Panoptic Tracking: Panoptic tracking predicts temporally consistent semantic categories and instance IDs, extending static panoptic segmentation with cross-frame association.The benchmark combines panoptic segmentation and multi-object tracking into one scene-understanding task.
  • Tracking Metric: PAT is the harmonic mean of panoptic quality PQ and tracking quality TQ, providing two separable components for interpretation.TQ is computed for thing classes using an instance-level association formulation.
  • Tracking Metric: TQ uses association scores based on true-positive, false-negative, and false-positive associations to evaluate instance matching across frames.Association matches require mask overlap greater than 0.5 IoU, while unmatched or inconsistently matched tracks contribute association errors.
  • Tracking Metric: TQ penalizes track fragmentation and ID switches while promoting long-term track consistency and retaining correctly segmented instances.An ID switch includes consecutive frames without a matching prediction or with inconsistent predicted IDs.

V. EXPERIMENTAL EVALUATION

The experimental evaluation covers quantitative benchmarking across semantic segmentation, panoptic segmentation, and panoptic tracking, followed by metric analysis and dataset ablations.

  • Quantitative Comparisons: Experiments compare methods on semantic segmentation, panoptic segmentation, and panoptic tracking tasks.These are the three benchmark tasks defined earlier in the paper.
  • PAT Analysis: The evaluation analyzes whether PAT captures the intended properties of panoptic tracking metrics.The paper presents a detailed analysis of PAT after reporting task benchmarks.
  • Ablation Studies: Ablation studies assess the utility of the proposed dataset, with additional details and analysis provided in the supplementary material.The ablations are presented as a separate experimental component after benchmarking and metric analysis.

A. Baseline Results

The benchmark compares standard and task-specific baselines across semantic, panoptic segmentation, and panoptic tracking tasks. Combined task-specific baselines generally outperform end-to-end panoptic segmentation approaches, while EfficientLPS with Kalman Filter leads panoptic tracking.

  • Baseline Construction: Published challenge submissions provide semantic segmentation baselines, while panoptic segmentation and tracking use both end-to-end and task-specific baseline groups.The panoptic segmentation combinations merge detection and semantic segmentation submissions; tracking combinations merge segmentation and tracking submissions.
  • Panoptic Segmentation: 1,470 independently combined panoptic segmentation baselines are generated from 70 detection and 21 semantic segmentation submissions.Higher-performing detection and semantic segmentation combinations generally achieve higher PQ scores.
  • Panoptic Segmentation: Task-specific panoptic segmentation combinations significantly outperform end-to-end approaches, indicating a need for stronger end-to-end methods.This comparison is reported as evidence that the benchmark remains challenging for unified approaches.

3) Panoptic Tracking:

Panoptic tracking combines segmentation and temporal identity preservation, and the benchmark evaluates this through many task-specific combinations and targeted tracking scenarios. PAT is presented as more informative and balanced than alternatives because it accounts for panoptic quality, tracking quality, and fragmentation-related behavior.

  • Panoptic Tracking: 924 task-specific tracking baselines are formed by merging 21 LiDAR segmentation submissions with 44 tracking submissions.Three thing classes absent from the tracking challenge contribute zero in the relevant evaluation metrics.
  • Panoptic Tracking: EfficientLPS with Kalman Filter reaches PAT scores of 64.6% on validation and 67.1% on test, the highest among the baselines.The reported explanation links training-statistics initialization of covariance matrices to filter convergence and data association accuracy.
  • Metric Evaluation: The PAT evaluation includes decoupling panoptic and tracking errors plus four challenging scenarios under perfect segmentation across seven consecutive scans.These scenarios target the metric requirements described for panoptic tracking.
  • Error-Type Differentiability: PAT can rank methods differently from LSTQ because it incorporates panoptic quality and instance identification rather than segmentation quality alone.The authors consider PAT more informative for a task requiring both stuff and thing predictions with preserved identities across frames.
  • Error-Type Differentiability: PAT correlations with AMOTA and mIoU are 0.48 and 0.57, compared with PTQ correlations of 0.23 and 0.69, indicating a more balanced metric.The passage compares PAT’s correlation pattern with PTQ and LSTQ.
  • Error-Type Differentiability: PAT and PTQ penalize track fragmentation under frame permutation, whereas LSTQ remains invariant to the permutation.The intended metric behavior is to encourage consistency in representing scene dynamics.

2) Track Fragmentation:

The tracking analysis tests whether metrics recognize incorrect identity transitions, fragmentation, and track duration. PAT handles all four reported tracking challenges, including cases where competing metrics fail or remain invariant.

  • ID-Transfer: PAT and LSTQ penalize a track ending and a different object’s track beginning, while PTQ incorrectly evaluates the case as the same track.This scenario tests association precision when identity changes occur.
  • Long-Term Tracking Consistency: PAT and LSTQ give higher scores to the longer of two-track predictions, whereas PTQ fails to account for this duration difference.The scenario evaluates whether metrics encourage long-term track consistency through association recall.
  • Prediction Processing: All three metrics penalize ignoring correctly segmented instances that have incorrect track IDs.This processing test examines susceptibility to evaluation-time removal of incorrectly tracked instances.

1) Influence of Pre-Training:

The paper evaluates transfer learning and generalization using EfficientLPS trained on Panoptic nuScenes, with results indicating that diverse-scene training improves performance in unseen environments.

  • Influence of Pre-Training: EfficientLPS models trained on Panoptic nuScenes and SemanticKITTI are evaluated for panoptic segmentation on an unseen PandaSet validation set.PandaSet lacks official panoptic annotations, so the authors generate annotations using an approach described in the paper.
  • Influence of Pre-Training: Training on a dataset with diverse scenes and objects results in better generalization.
  • Influence of Pre-Training: The improved instance segmentation capability consequently results in an overall higher panoptic quality.

S.1. ADDITIONAL DATASET DETAILS

Panoptic nuScenes contains diverse urban scenes, varied object tracks, and detailed annotations spanning semantic, panoptic, and tracking tasks.

  • Dynamic and Diverse Scenes: Panoptic nuScenes emphasizes dense urban scenes with many dynamic agents, including intersections, construction sites, varied locations, weather, and lighting.Examples include jaywalkers, lane changes, and turning situations.
  • Instance Statistics: Common classes include 152k moving adult instances and 114k moving car instances, while rarer classes include 882 police and 298 construction-vehicle instances.
  • Track Statistics: Track lengths span short, medium, and long durations, challenging methods to track objects that either persist or appear briefly.Short, medium, and long tracks persist for less than one-third, one-third to two-thirds, and more than two-thirds of a scene, respectively.
  • Annotations: Camera-view annotations include construction zones, junctions, nighttime scenes, bendy buses, and vehicle wheels tightly outlined without nearby ground points.
  • Annotations: The dataset provides semantic, panoptic segmentation, and panoptic tracking annotations, with visual examples covering all three challenges.

S.2. ADDITIONAL PANOPTIC SEGMENTATION RESULTS

Additional analyses compare combination and end-to-end panoptic segmentation baselines and examine how component segmentation and detection quality relate to panoptic performance.

  • Per-Class Results: Independently combined baselines outperform end-to-end methods across classes, with larger gains for thing classes than stuff classes.
  • Per-Class Results: 20.8 PQ is the average gap between the best independently combined and end-to-end results for thing classes, versus 4.6 PQ for stuff classes.
  • Per-Class Results: The larger thing-class gap is attributed to stronger task-specific detection methods used for thing-instance segmentation.
  • Additional Results: The supplementary analysis reports class-wise panoptic segmentation results and compares methods across the dataset.
  • Correlation Analysis: Panoptic segmentation performance is analyzed through 1,470 combinations of LiDAR semantic-segmentation and detection submissions.

S.4. ANALYSIS OF PANOPTIC TRACKING METRICS

The paper analyzes panoptic tracking metrics, contrasting their aggregation levels and sensitivity to fragmented tracks while illustrating qualitative differences between leading methods.

  • Metric Analysis: VPQ is excluded for LiDAR panoptic tracking because multi-frame 3D-IoU matching becomes difficult and introduces an image-oriented limitation.
  • Metric Differences: PAT evaluates at the instance level, whereas LSTQ evaluates at the point level, while both separately obtain panoptic and tracking evaluations.
  • Qualitative Comparisons: EfficientLPS + Kalman Filter tracks parked trucks and a light green car that 4D-PLS fails to track consistently in qualitative examples.
  • Qualitative Comparisons: PolarSeg-Panoptic segments cars and a truck missed by EfficientLPS in qualitative examples.
  • Metric Differences: PAT uses a harmonic mean and is stricter when one component value is low, whereas LSTQ uses a geometric mean.
  • Metric Differences: PAT’s Tracking Quality score penalizes fragmented tracks, while LSTQ does not penalize this error.
  • Metric Scope: PAT and LSTQ penalize different aspects of panoptic tracking, making both useful in different scenarios.

S.5. IMPACT OF DIVERSE SCENES IN THE TRAINING SET

The study examines whether training on diverse scenes and objects improves generalization, using EfficientLPS trained separately on SemanticKITTI and Panoptic nuScenes. It also compares end-to-end panoptic segmentation approaches qualitatively, where PolarSeg-Panoptic outperforms EfficientLPS in the shown examples.

  • Generalization study: EfficientLPS is trained separately on SemanticKITTI and Panoptic nuScenes using the same training protocols.The experiment evaluates generalization for an approach trained on datasets with different scene and object diversity.
  • Generalization study: PandaSet panoptic labels are computed by combining point-level semantic labels with 3D bounding boxes to assign instance labels.Points inside a box with the same segmentation type as the box form an instance; overlapping points are treated as noise.
  • Qualitative comparison: PolarSeg-Panoptic achieves the highest PQ score among the compared end-to-end panoptic segmentation approaches, followed by EfficientLPS.In the qualitative examples, PolarSeg-Panoptic detects cars and a truck that EfficientLPS misses.
Loading 2109.03805v3…