Source-linked AI summary
Sonata: Self-Supervised Learning of Reliable Point Representations
Xiaoyang Wu, Daniel DeTone, Duncan Frost, Tianwei Shen, Chris Xie, Nan Yang, Jakob Engel, Richard Newcombe, Hengshuang Zhao, Julian Straub
TL;DR
Existing point-cloud SSL methods fall short as reliable representations for simple linear probing, partly because geometric shortcuts collapse features onto low-level spatial cues. Sonata combines self-distillation with spatial obfuscation, input-feature emphasis, and scaled training to learn stronger representations. It achieves 72.5% linear-probing accuracy on ScanNet versus 21.8% for prior methods, performs well with limited data, and advances fully fine-tuned indoor and outdoor perception results.
Problem
Prior point-cloud SSL approaches do not yet provide reliable representations usable across 3D applications through simple linear probing, achieving at most 21.8% mIoU on ScanNet.
Method
Sonata combines point self-distillation with coarser-scale losses, disturbed masked-point spatial information, progressive task difficulty, and training scaled to 140k point clouds.
Results
72.5% linear-probing accuracy on ScanNet improves over the prior 21.8% result, while full fine-tuning advances state-of-the-art results across indoor and outdoor perception tasks.
Takeaways & Limitations
Sonata provides semantically meaningful zero-shot representations and strong parameter and data efficiency for 3D perception.
Takeaways & Limitations
Sonata currently separates indoor and outdoor pre-training rather than unifying the settings, leaving domain-gap challenges for future work.
Abstract
from arXiv · showhide
In this paper, we question whether we have a reliable self-supervised point cloud model that can be used for diverse 3D tasks via simple linear probing, even with limited data and minimal computation. We find that existing 3D self-supervised learning approaches fall short when evaluated on representation quality through linear probing. We hypothesize that this is due to what we term the "geometric shortcut", which causes representations to collapse to low-level spatial features. This challenge is unique to 3D and arises from the sparse nature of point cloud data. We address it through two key strategies: obscuring spatial information and enhancing the reliance on input features, ultimately composing a Sonata of 140k point clouds through self-distillation. Sonata is simple and intuitive, yet its learned representations are strong and reliable: zero-shot visualizations demonstrate semantic grouping, alongside strong spatial reasoning through nearest-neighbor relationships. Sonata demonstrates exceptional parameter and data efficiency, tripling linear probing accuracy (from 21.8% to 72.5%) on ScanNet and nearly doubling performance with only 1% of the data compared to previous approaches. Full fine-tuning further advances SOTA across both 3D indoor and outdoor perception tasks.
1. Introduction
Existing point-cloud SSL models do not yet provide reliable representations under simple linear probing because they exploit low-level geometric shortcuts. Sonata obscures spatial information, emphasizes input features, and achieves stronger zero-shot and data-efficient performance.
- Motivation: 21.8% mIoU is the maximum reported by prior point-cloud SSL methods under linear probing on ScanNet semantic segmentation.This motivates evaluating representation quality with simple linear probes rather than relying only on end-task fine-tuning.
- Problem: The geometric shortcut collapses representations onto accessible cues such as surface normals or point height, hindering reliable semantic representations.Point coordinates enter point-cloud operators directly, making these cues difficult to obscure or mask.
- Approach: Sonata addresses the shortcut by applying losses at coarser spatial scales, disturbing masked-point spatial information, and progressively increasing task difficulty while emphasizing input features.The framework combines these strategies with point self-distillation and scales training to 140k point-cloud scenes.
- Results: Sonata produces semantically meaningful zero-shot visualizations, including PCA-colored point clouds, feature clustering, and nearest-neighbor matching.These evaluations probe semantic grouping and correspondence without task-specific labels.
- Results: 72.5% mIoU on ScanNet is achieved by Sonata with linear probing, up from 21.8% for previous methods, while combined Sonata and DINOv2 features reach 76.4%.The combined representation suggests complementary 3D and image-derived information.
2. Related Work
Image SSL has established linear probing and visualization as evidence of reliable representations, whereas point-cloud SSL remains less mature and faces a 3D-specific geometric shortcut. Sonata builds on prior point-cloud methods while targeting this gap.
- Image self-supervised learning: Linear probing has become a standard image-SSL test because it assesses representation quality using only a minimal linear transformation.Image SSL also uses direct visualization to expose semantic meaning in learned representations.
- 3D geometric shortcut: Point clouds retain geometric information in point positions after input features are removed, unlike images whose information is concentrated in the input feature.Point operators directly use these positions, creating the geometric shortcut in 3D SSL.
- Positioning: Sonata extends Mask Scene Contrast by combining self-distillation, geometric-shortcut mitigation, and larger-scale training.This positions Sonata as a reliability-oriented successor to prior point-cloud SSL approaches.
3. Pilot Study and Design Principle
The pilot study attributes weak point-cloud SSL representations to a geometric shortcut: sparse coordinates expose low-level cues that models can exploit instead of learning semantics. Coarser encoder representations and decoder removal are therefore central design principles.
- Uncovering the geometric shortcut: Existing 3D SSL losses rapidly reach an ideal range early in training, suggesting that the pretext tasks do not force sufficient learning difficulty.The authors connect this lack of “struggle” to shortcut-driven representation collapse.
- Uncovering the geometric shortcut: Previous methods produce representations sensitive to local surface normals or point height, which the paper terms the geometric shortcut.Linear probing exposes the resulting semantic weakness: PC and MSC obtain 5.6% and 21.8% mIoU on ScanNet, respectively.
- Why 3D differs: Point-cloud sparsity requires coordinate information to enter operators directly, making spatial cues difficult to obscure or effectively mask.This differs from dense images, where regularly spaced pixels do not create the same coordinate-access problem.
- Hierarchical backbone: As PTv3 resolution decreases through max-pooling, encoder features become less local and rely less on fine spatial information tied to point coordinates.The encoder shows diverse multi-scale features, whereas the decoder produces more task-specific representations.
- Design principle: Removing the decoder during SSL restricts access to fine-grained spatial information while preserving an expressive multi-scale encoder representation for later probing or fine-tuning.The design addresses both geometric-shortcut risk and the structural constraints imposed by the decoder.
4. Point Self-distillation with Sonata
Sonata uses point self-distillation to align difficult local and masked views with richer global views, while coarser-scale training and progressive masking reduce geometric shortcuts. The framework scales from an initial 23k-data, 39M-parameter setup to 140k assets and a 108M model.
- Macro Framework: Sonata begins with a point self-distillation framework that aligns points expected to represent the same content across augmented views.The framework provides the foundation for designing harder pretext tasks aimed at reliable representations.
- Macro Framework: Local and masked views are aligned with unmasked global views, with task difficulty controlled by cropping and masking ratios.Local views use small crops, while masked views remove large portions of the global view.
- Macro Framework: An EMA teacher encodes global views while a learned student encodes local and masked views, stabilizing training as pretext tasks become harder.The asymmetric design reduces the risk that the student collapses or is misled by challenging views.
- SSL criteria: The method replaces contrastive and generative criteria with self-distillation driven by Sinkhorn-Knopp centering and KoLeo regularization.The change addresses scalability limits from pairwise similarity and representational anchoring to predefined cues.
- Scaling and evaluation: Training scales from 23k data and a 39M PTv3 model to 140k assets and a 108M PTv3 model, with progressive ablations evaluated through linear and decoder probing.Evaluation freezes the encoder for linear probing and trains only a lightweight decoder for decoder probing.
- Micro Design: Mask size increases from 10 cm to 40 cm and mask ratio from 30% to 70% during the first 5% of training, progressively shifting reliance toward input features.This schedule is designed to prevent increasing task difficulty from redirecting reliance to point coordinates.
5. Main Results
Sonata is evaluated as a reliable and efficient point-cloud representation through linear, decoder, and full fine-tuning probes across indoor and outdoor tasks. It shows strong low-data performance, zero-shot semantic structure, and state-of-the-art results, while large-class recognition remains limited.
- Comparison with image self-supervised models: Sonata’s linear probing reaches 72.5% on ScanNet, exceeding DINOv2 features at 63.1%, while combining both reaches 76.4%.The comparison evaluates representations transferred to ScanNet semantic segmentation.
- Data efficiency: 19.5% improvement occurs with 1% of ScanNet scenes, and linear probing surpasses previous SOTA by 12.5% in that setting.With 20 points per scene, the improvement over training from scratch is 10.4%.
- Parameter efficiency: A single linear layer uses fewer than 0.02% of total parameters, while decoder probing uses 13% and achieves higher accuracy.Reported examples include 79.1% on ScanNet and 81.5% on S3DIS 6-fold cross-validation for decoder probing.
- Parameter efficiency: Decoder probing remains limited on ScanNet200 and ScanNet++, indicating difficulty distinguishing large numbers of classes.The limitation concerns the learned representation’s class discrimination rather than the smaller-class benchmarks.
- Indoor semantic segmentation: Full fine-tuning reaches 79.4% on ScanNet and 82.3% on S3DIS 6-fold cross-validation, consistently advancing indoor semantic-segmentation SOTA.Full fine-tuning remains necessary for the highest benchmark performance.
- Indoor instance segmentation: Linear probing improves instance-segmentation mAP50 by 10× on ScanNet and 21× on ScanNet200, with decoder probing reaching 12× and 33×.Full fine-tuning further boosts these results and achieves SOTA.
- Outdoor semantic segmentation: Outdoor full fine-tuning sets new SOTA mIoU scores of 81.7, 72.9, and 72.6 across nuScenes, Waymo, and SemanticKITTI.Decoder-only probing recovers 95%, 97%, and 94% of full fine-tuning performance, respectively.
- Zero-shot representation across scenes: Zero-shot PCA and dense matching across two floors and 12 rooms show semantically coherent features across objects and indoor spaces.Representative queries include sofa arms, chairs, tables, pillows, and side tables.
6. Conclusion and Discussion
Sonata targets reliable self-supervised point representations that support zero-shot semantic correspondence and downstream grouping. The paper attributes prior failures to geometric shortcuts and addresses them with multi-scale spatial obfuscation, feature emphasis, self-distillation, and scaling, while identifying semantic richness and scenario unification as future directions.
- Conclusion: Sonata aims to zero-shot correspond semantically similar 3D points to the instance level despite spatial and visual perturbations.The representation is intended to support semantic and instance-level grouping in 3D tasks.
- Discussion: The geometric shortcut collapses representations toward low-level spatial features, limiting reliability in prior point-cloud SSL approaches.The paper identifies this as a problem unique to 3D.
- Conclusion: Figure 8’s two-floor, 12-room HM3D visualization shows semantic grouping, granular object patterns, and unsupervised dense matching across indoor scenes.The examples include sofa arms, chairs, tables, pillows, and side tables.
- Method: Sonata addresses the shortcut with coarser-scale SSL losses, disturbed spatial information for featureless masked points, progressive task difficulty, self-distillation, and scaling to 140k point clouds.These changes are presented as enabling effective scaling of the training framework.
- Limitations and future work: Future work includes adding object-level assets to deepen semantic meaning and unifying indoor and outdoor pre-training without the added challenge of a domain gap.The current training separates indoor and outdoor scenarios.
Appendix
The appendix provides detailed implementation guidance, including the procedure for generating views with crop, photometric, and spatial augmentations.
- A.1. View Generation: View generation uses global random crops covering 40% to 100% of the relevant raw point-cloud size, combined with photometric and spatial augmentations.Photometric augmentation is shared across global views, while spatial augmentation is applied independently.
A. Additional Implementation
The additional implementation appendix lists sections on point self-distillation and out-of-distribution perception, but the supplied passages provide no substantive procedures or findings.
- A.2. Point Self-distillation: Appendix A.2 is dedicated to point self-distillation.The supplied text identifies the section but does not describe its implementation.
- B. Additional Applications: Appendix B.1 covers out-of-distribution perception, while B.2 covers surface reconstruction.The supplied text provides section titles only.
A.1. View Generation
Sonata generates global, local, and masked views through crop-based sampling, augmentation, and masking, then trains with point self-distillation across multiple view pairs.
- Self-distillation: The training pipeline encodes student and teacher networks, computes online-cluster similarities, matches corresponding original-coordinate neighbors, and backpropagates their entropy loss.Masked points receive Gaussian coordinate noise before encoding, while teacher parameters are updated by momentum.
- Self-distillation: The displayed loss computes cross-entropy between sharpened teacher assignments and centered student assignments, averaged across points.The pseudocode applies Sinkhorn-Knopp centering and separate student and teacher temperatures.
- View construction: Random photometric and spatial augmentations are applied to every view, with shared photometric and independent spatial transforms for global views.Masked views additionally apply random patch masks to global views.
- Self-distillation: Point self-distillation uses four local, two masked, and two global views, pairing local views with the principal global view and masked views with every global view.The eight point self-distillation pairs are evenly weighted.
B.1. Out-of-distribution (OOD) Perception.
Sonata is evaluated on sparse, out-of-distribution AEO point clouds, where linear probing trails training from scratch but fine-tuning shows substantial improvement.
- Evaluation setting: The AEO evaluation uses 25 sparse SLAM-generated point clouds with 17 object categories, reserving three samples for validation and 22 for training.The setup tests perception under unseen data patterns and sparse inputs.
- Results and limitation: 32.0% mIoU from Sonata linear probing remains 2.9% below the 34.9% mIoU achieved by training from scratch.This comparison identifies a limitation in the current linear-probing setting.
- Results and limitation: Insufficient diversity in training data patterns is identified as a current limitation because pretraining includes only dense indoor point clouds.The authors describe the framework as focused on reliability without adding further domain-gap challenges.
- Results and limitation: Fine-tuning Sonata produces a 21.0% improvement over training from scratch on the challenging out-of-distribution perception task.The result is presented as evidence of robustness and adaptability in this setting.
B.2. Surface Reconstruction.
The paper examines whether frozen Sonata features encode dense geometric priors by using them for surface reconstruction with signed-distance regression.
- Experiment: Frozen Sonata features are used in a surface regression experiment to investigate whether the representations capture dense geometric priors.The experiment follows earlier evidence of semantic information in Sonata representations.
- Experiment: The reconstruction target is a TSDF volume defined in a 4m×4m×4m local coordinate system at 96 × 96 × 96 resolution.This corresponds to approximately 4cm per voxel.
- Experiment: The TSDF volume is patchified into 8 × 8 × 8 patches before projection toward reconstruction from frozen features.The passage describes patch-based processing within the local volume.
- Result: The surface reconstruction experiment is presented as demonstrating strong geometric priors and cross-domain generalization.The result extends the representation evaluation beyond semantic grouping and segmentation.