Source-linked AI summary
Identifying Unknown Instances for Autonomous Driving
Kelvin Wong, Shenlong Wang, Mengye Ren, Ming Liang, Raquel Urtasun
TL;DR
Recognizing and segmenting objects whose categories were not seen during training remains challenging for deep networks. OSIS groups point-cloud observations in a category-agnostic embedding space, and outperforms baselines across known and unknown things on two datasets.
Problem
Recognizing and segmenting objects without seeing their categories during training is challenging because deep networks cannot exploit category-specific shape and appearance information.
Source: §1 · p. 1Method
OSIS uses category-agnostic point embeddings and known-class prototypes, then clusters remaining unknown points using embeddings and 3D locations.
Source: §1 · p. 2, §3.2 · p. 5Results
OSIS outperforms baselines on known and unknown things across all metrics on both datasets and achieves state-of-the-art performance in the open-set setting.
Source: §4.2 · p. 7, §5 · p. 8Takeaways & Limitations
The method groups points into instances irrespective of whether they belong to known or unknown classes.
Source: §5 · p. 8The formulation requires points with the same instance ID to share a semantic label and ignores instance IDs for stuff points.
Source: §3.1 · p. 4
Abstract
from arXiv · showhide
In the past few years, we have seen great progress in perception algorithms, particular through the use of deep learning. However, most existing approaches focus on a few categories of interest, which represent only a small fraction of the potential categories that robots need to handle in the real-world. Thus, identifying objects from unknown classes remains a challenging yet crucial task. In this paper, we develop a novel open-set instance segmentation algorithm for point clouds which can segment objects from both known and unknown classes in a holistic way. Our method uses a deep convolutional neural network to project points into a category-agnostic embedding space in which they can be clustered into instances irrespective of their semantics. Experiments on two large-scale self-driving datasets validate the effectiveness of our proposed method.
1 Introduction
The paper addresses the challenge of recognizing and segmenting objects from classes absent during training. OSIS uses category-agnostic embeddings to group points into instances and outperforms competing methods on two self-driving datasets.
1 Introduction
- Open-set perception aims to recognize and localize known and unknown objects, unlike closed-set systems trained only on known classes.Source: §1 · p. 1
- Earlier work addressed novel-object grasping and open-set classification, but those approaches did not provide open-set instance segmentation.Source: §1 · p. 1
- Recognizing objects without category-specific training is challenging because deep networks cannot exploit category shape or appearance information during training.Source: §1 · p. 1
- Perceptual grouping motivates learning to organize visual elements into objects before deciding whether they belong to known classes.Source: §1 · p. 2
- OSIS embeds points in a category-agnostic space, filters known-class points using prototypes, and clusters the remaining points into unknown instances.Source: §1 · p. 2
- OSIS outperforms competing methods at identifying instances from known and unknown classes on two large-scale self-driving datasets.Source: §1 · p. 2
2 Related Work
Prior segmentation and open-set methods leave gaps in identifying and grouping unknown instances. OSIS builds on embedding-based grouping while adding detection anchors and prototypes suited to sparse, non-uniform LiDAR point clouds.
2 Related Work
- Standard instance and panoptic segmentation do not capture unknown instances absent from training, while open-set recognition has mainly addressed classification.Source: §2 · p. 2
- Prior open-set instance approaches combine known-class segmentation with contour detection, weak labels, video proposals, or connected components.Source: §2 · pp. 2–3
- Its two-stage pipeline associates points with known-class prototypes first, then clusters remaining unknown points using embeddings and 3D locations.Source: Figure 2 · p. 3
- Detection-box methods may leave unknown objects unrecognized because segmentation follows detection, while bottom-up grouping methods cluster pixels or points into instances.Source: §2 · p. 3
- OSIS differs from embedding-based grouping by using detection anchors for top-down guidance and predicting prototypes to address LiDAR sparsity and non-uniform density.Source: §2 · p. 3
3 Identifying the Unknowns
This section formulates open-set instance segmentation as assigning each point both an instance identity and a semantic label, including for unknown instances. It then presents a two-stage prototype-and-clustering network and its detection-plus-embedding training.
3.1 Problem Formulation
- The task maps each point feature to an instance ID and open-set semantic label, requiring instance labels for unknown points while ignoring instance IDs for stuff points.Known classes are divided into thing and stuff classes, and points sharing an instance ID must share a semantic label.Source: §3.1 · pp. 3–4
3.2 Open-Set Instance Segmentation
- The network learns a category-agnostic embedding space and combines a shared backbone, a detection head for known-thing anchors, and an embedding head for point features and prototypes.The point branch produces point embeddings, while the thing and stuff branches produce prototypes.Source: §3.2 · pp. 4–5
- Inference first associates points with known thing and stuff prototypes, then classifies uncertain points as unknown and clusters them using embeddings and 3D coordinates.The DBSCAN distance combines squared embedding distance with squared 3D location distance, and the unknown-instance labels are combined with closed-set predictions.Source: §3.2 · pp. 4–5
- The input is a BEV-rasterized LiDAR point cloud voxelized into a 3D occupancy grid, with multiple frames optionally stacked and ego-motion compensated using localization.Source: §3.2 · p. 4
3.3 Learning
- Training combines detection and embedding losses; detection uses classification and box-regression supervision, while embedding training encourages correct point-to-prototype associations.Predicted box sizes and orientations provide stronger supervision but are not used during inference.Source: §3.3 · p. 5
4 Experiments
Experiments evaluate OSIS for open-set instance segmentation on TOR4D and Rare4D, comparing it with adapted baselines and testing model design choices. The reported results show stronger performance across known and unknown things, alongside qualitative examples and ablations.
4.2 Results
- OSIS outperforms the baselines on known and unknown things across all metrics on both datasets.Table 1 reports quantitative test-set results for TOR4D and Rare4D.Source: Table 1 · p. 6, §4.2 · p. 7
4.1 Experimental Setup
- TOR4D includes four labeled classes and treats other points as unknown; Rare4D contains 289 rare objects and is used only to evaluate unknown-object identification.TOR4D's validation and test sets include 5,702 and 10,127 annotated unique unknown objects, respectively.Source: §4.1 · pp. 6–7
- Known classes are evaluated with PQ, RQ, and SQ, while unknown quality is recall-based and matches predicted instances at IoU above 0.5.Baselines were adapted to the open-set setting because prior point-cloud open-set instance-segmentation work was lacking.Source: §4.1 · p. 7
4.2 Results
- In Figure 3, OSIS correctly segments the horse as unknown; it misclassifies a construction vehicle as unknown but still segments it as an instance.The qualitative comparison covers TOR4D and Rare4D.Source: Figure 3 · p. 6, §4.2 · p. 8
4.3 Ablation Studies
- Ablations on TOR4D validation results indicate that discriminative loss, bounding-box regression, and per-prototype scalar variance each contribute to overall performance.Table 2 reports the model-component ablation.Source: §4.3 · p. 8, Table 2 · p. 8
- Instance-aware embeddings perform best among the tested clustering features, and combining them with geometry features yields further improvements.Table 3 compares clustering features; Figure 4 varies the relative weight of 3D-location and embedding distances.Source: §4.3 · p. 8, Table 3 · p. 8, Figure 4 · p. 8
5 Conclusion
OSIS achieves state-of-the-art performance in open-set point-cloud instance segmentation on two large-scale self-driving datasets. It groups points in a category-agnostic embedding space and is intended to handle both known and unknown classes.
5 Conclusion
- OSIS achieves state-of-the-art performance in the open-set setting on two large-scale self-driving datasets.Source: §5 · p. 8
- A deep convolutional neural network encodes points in a category-agnostic embedding space where they can be clustered into instances.Source: §5 · p. 8
- The method perceptually groups points into instances whether they belong to known or unknown classes.Source: §5 · p. 8
- Future work will explicitly reason about motion as a cue for segmenting moving objects better.Source: §5 · p. 8
A Additional Results
Figure 5 presents qualitative open-set instance-segmentation results on TOR4D and Rare4D.
A Additional Results
- Figure 5 presents qualitative open-set instance-segmentation results on TOR4D and Rare4D.Source: Figure 5 · p. 11