Source-linked AI summary
Segment Any Point Cloud Sequences by Distilling Vision Foundation Models
Youquan Liu, Lingdong Kong, Jun Cen, Runnan Chen, Wenwei Zhang, Liang Pan, Kai Chen, Ziwei Liu
TL;DR
Automotive point-cloud segmentation needs representations that reduce annotation dependence and transfer across changing sensor and dataset configurations. Seal distills vision foundation model semantics into point clouds using spatial and temporal consistency, and it outperforms prior methods across 11 datasets and 20 few-shot tasks.
Problem
Point-cloud segmentation relies heavily on costly annotations and often performs well only within the training distribution because sensor configurations differ across datasets.
Method
Seal performs self-supervised representation learning by transferring vision foundation model semantics through camera–LiDAR correspondence, semantic superpixels, and temporal consistency regularization.
Results
Seal shows clear superiority over previous state-of-the-art methods in linear probing and fine-tuning across 11 point-cloud datasets with diverse configurations.
Takeaways & Limitations
Seal supports off-the-shelf knowledge transfer to diverse automotive point-cloud segmentation tasks beyond the pretraining distribution.
Takeaways & Limitations
Seal assumes impeccably calibrated and synchronized LiDAR and cameras and is pretrained on a single unified point-cloud setup.
Abstract
from arXiv · showhide
Recent advancements in vision foundation models (VFMs) have opened up new possibilities for versatile and efficient visual perception. In this work, we introduce Seal, a novel framework that harnesses VFMs for segmenting diverse automotive point cloud sequences. Seal exhibits three appealing properties: i) Scalability: VFMs are directly distilled into point clouds, obviating the need for annotations in either 2D or 3D during pretraining. ii) Consistency: Spatial and temporal relationships are enforced at both the camera-to-LiDAR and point-to-segment regularization stages, facilitating cross-modal representation learning. iii) Generalizability: Seal enables knowledge transfer in an off-the-shelf manner to downstream tasks involving diverse point clouds, including those from real/synthetic, low/high-resolution, large/small-scale, and clean/corrupted datasets. Extensive experiments conducted on eleven different point cloud datasets showcase the effectiveness and superiority of Seal. Notably, Seal achieves a remarkable 45.0% mIoU on nuScenes after linear probing, surpassing random initialization by 36.9% mIoU and outperforming prior arts by 6.1% mIoU. Moreover, Seal demonstrates significant performance gains over existing methods across 20 different few-shot fine-tuning tasks on all eleven tested point cloud datasets.
1 Introduction
Seal addresses annotation and distribution challenges in automotive point-cloud segmentation by distilling semantic knowledge from 2D vision foundation models into self-supervised 3D representation learning. It combines camera–LiDAR correspondence, semantic superpixels, and temporal consistency to improve scalable and generalizable segmentation.
- Motivation: Existing point-cloud segmentation models require labor-intensive annotations and often generalize poorly across sensors with different configurations.Differences include beam number, camera angle, and emit rate.
- Method: Seal distills semantically rich knowledge from vision foundation models through camera–LiDAR correspondence and contrastive samples built from semantic superpixels.The framework uses raw point clouds and targets reduced annotation costs, spatial and temporal cues, and transfer across diverse downstream point clouds.
- Method: Semantic superpixels mitigate self-conflicts in contrastive learning, improve optimization coherence and convergence, and reduce pretraining overhead compared with prior configurations.
- Method: Temporal consistency regularization uses point-cloud geometry to reduce sensitivity to imperfect LiDAR–camera calibration and synchronization.This regularization is intended to improve resilience and reliability across automotive point-cloud sequences.
- Method: Seal is presented as the first use of 2D vision foundation models for self-supervised representation learning on large-scale 3D point clouds.
- Results: Seal outperforms previous state-of-the-art methods in linear probing and fine-tuning across 11 point-cloud datasets with diverse data configurations.
2 Related Work
Prior work spans vision foundation models, supervised and weakly supervised point-cloud segmentation, and self-supervised 3D representation learning. These lines of research motivate Seal’s focus on transferring semantic visual knowledge to diverse outdoor driving scenes.
- Vision Foundation Models: Vision foundation models such as SAM, X-Decoder, OpenSeeD, SegGPT, and SEEM advance general-purpose image segmentation and zero-shot transfer.
- Point Cloud Segmentation: Point-cloud segmentation methods operate on raw points, range views, bird’s-eye views, voxels, or multi-view fusion, but commonly depend on large annotated datasets.
- 3D Representation Learning: Most 3D self-supervised learning methods target object-centric point clouds or indoor scenes, whose scale and diversity are lower than outdoor driving data.
3 Seal: A Scalable, Consistent, and Generalizable Framework
Seal distills vision foundation model knowledge into point clouds through semantic spatial contrastive learning and temporal consistency regularization. Its design addresses semantic partitioning, cross-modal alignment, sensor synchronization, and temporal point-to-segment consistency.
- Semantic Superpixel Spatial Consistency: VFM-generated semantic superpixels reduce self-conflict by grouping semantically coherent image regions for contrastive learning.Unlike SLIC, VFMs provide semantically rich regions that improve representation learning between near and far LiDAR points.
- Semantic Superpixel Spatial Consistency: The spatial objective aligns pooled superpoint and superpixel embeddings by contrasting corresponding features against other region pairs.The embeddings are mapped into a shared space, and cosine-style similarity is scaled by a temperature term.
- Framework Overview: Seal transfers knowledge from synchronized camera images to LiDAR point clouds using VFM-generated semantic superpixels and corresponding superpoints.Point-to-pixel correspondence uses camera calibration parameters, while the framework jointly processes a LiDAR-camera pair and another LiDAR frame at a later timestamp.
- Consistency Regularization: Point-cloud geometry supplements imperfect LiDAR-camera synchronization, improving the framework’s resilience to calibration and synchronization errors.The point-to-segment regularization uses geometric information from point clouds to reduce reliance on perfectly synchronized sensors.
- Semantic Superpoint Temporal Consistency: Seal uses clustered non-ground point segments across consecutive frames to impose temporal consistency on point features.Segments are formed after RANSAC ground removal and HDBSCAN clustering, then aligned across timestamps through global-frame transformations.
- Semantic Superpoint Temporal Consistency: The temporal objective pulls points toward the corresponding next-frame segment mean while separating them from other segments, implicitly clustering same-instance points.A symmetric formulation encourages correspondence in both temporal directions.
4 Experiments
Seal is evaluated across diverse point-cloud datasets, downstream protocols, robustness settings, foundation models, and component ablations. The experiments show strong transfer performance, broad generalization, and benefits from its proposed consistency and distillation components.
- Evaluation settings: 11 datasets cover real-world, synthetic, adverse-weather, off-road, small-scale, and corrupted automotive point clouds.The evaluation includes nuScenes, SemanticKITTI, Waymo Open, ScribbleKITTI, RELLIS-3D, SemanticPOSS, SemanticSTF, SynLiDAR, Synth4D, DAPS-3D, and nuScenes-C.
- Comparative results: 44.95% mIoU under linear probing on nuScenes exceeds ST-SLidR by 4.47% mIoU and SLidR by 6.15% mIoU.Seal also achieves the best scores across all downstream few-shot fine-tuning tasks reported for nuScenes.
- Downstream generalization: Seal constantly outperforms prior arts across all downstream tasks on all eleven datasets despite differences in sensors, environments, scales, and fidelity.The evaluation spans diverse data-collection protocols and distributions.
- Robustness: Seal achieves the best robustness under almost all nuScenes-C corruption types.The robustness evaluation uses eight out-of-distribution corruptions and reports mCE, mRR, and mIoU.
- Foundation model comparisons: SEEM generally performs best among the tested VFMs, while SAM benefits more from fine-tuning with additional annotated data.All tested VFMs outperform SLIC with both frameworks, and Seal surpasses SLidR by large margins across setups.
- Ablation study: Integrating VFMs, temporal consistency, and point-to-segment regularization yields a 6.15% mIoU advantage over prior art in linear probing.VFM integration, temporal consistency, and their combination provide 4.20%, 1.65%, and 5.21% mIoU gains, while point-to-segment regularization adds around 4.55%.
5 Concluding Remark
Seal is a versatile self-supervised framework for automotive point-cloud segmentation, using spatial and temporal consistency during representation learning. Experiments across 20 downstream tasks and eleven datasets support its effectiveness, while calibration, synchronization, and pretraining diversity remain limitations.
- Seal encourages spatial and temporal consistency during representation learning to segment diverse automotive point clouds.
- 20 downstream tasks across eleven point-cloud datasets verify Seal’s effectiveness and superiority.
- Seal assumes impeccably calibrated and synchronized LiDAR and cameras, which may not always hold in real-world scenarios.
- Seal is pretrained on a single unified point-cloud setup, while aggregating datasets with different setups could further improve generalizability.
Appendix
The appendix supplies implementation details, complete quantitative results, qualitative comparisons, and information about public resources used in the study.
- Section 6 provides additional implementation details to facilitate reproduction.
- Section 7 reports the complete quantitative experimental results.
- Section 8 includes additional qualitative results for visual comparisons.
- Section 9 acknowledges the public resources used during the work.
6 Additional Implementation Detail
The appendix details experiments across eleven diverse point-cloud datasets and compares visual partition methods used to generate superpixels. Semantic superpixels improve segmentation performance and convergence relative to traditional partitioning.
- 6.1 Datasets: nuScenes supplies LiDAR and synchronized six-camera imagery for pretraining, while SemanticKITTI, Waymo Open, and other datasets support downstream evaluation.
- 6.1 Datasets: ScribbleKITTI provides weak line-scribble supervision, whereas SemanticPOSS and SemanticSTF represent small-scale or adverse-weather settings.
- 6.1 Datasets: Table 7 summarizes sensor configurations and data statistics for the eleven experimental datasets.
- 6.1 Datasets: The appendix includes synthetic, semi-synthetic, and corrupted datasets for evaluating generalizability and robustness.
- 6.2 Vision Foundation Models: SLIC generates superpixels through efficient k-means clustering, while its fixed quota is 150 superpixels per image.
- 6.2 Vision Foundation Models: Semantic superpixels alleviate over-segmentation and improve both linear probing and downstream fine-tuning.
- 6.2 Vision Foundation Models: VFM-generated superpixels provide performance gains and faster pretraining convergence.
7 Additional Quantitative Result
Additional quantitative results show that Seal consistently outperforms prior methods across semantic classes and robustness metrics, while semantic superpixels accelerate convergence relative to SLidR.
- Seal outperforms prior methods for most semantic classes across nuScenes, SemanticKITTI, Waymo Open, and Synth4D.
- Seal converges faster than SLidR when using semantic superpixels.
- Seal is superior to previous methods on both corruption error and relative robustness metrics for nuScenes-C.
8 Additional Qualitative Result
The appendix provides additional cosine-similarity visualizations, qualitative downstream results, and detailed per-class or corruption-oriented evaluations across multiple datasets and settings.
- The appendix identifies Figures 11–13 as additional cosine-similarity examples and Figures 14–16 as additional downstream fine-tuning results.
- Figures 11–13 visualize cosine similarity between a query point and features learned with SLIC and different VFMs, using violet-to-yellow colors for low-to-high scores.
- Figures 14–16 compare qualitative predictions after nuScenes pretraining and fine-tuning with 1% labeled data, marking correct predictions gray and incorrect predictions red.
9 Public Resources Used
The work acknowledges public datasets and software resources used during the study, along with institutional, national, and industry-linked funding support.
- The acknowledged public resources include nuScenes, nuScenes-devkit, SemanticKITTI, and SemanticKITTI-API, with listed licensing information.
- The study reports support from Singapore’s Ministry of Education, NTU NAP, an RIE2020 industry collaboration initiative, industry partners, and China’s National Key R&D Program.
- The authors thank Tai Wang for insightful review and discussion.