Source-linked AI summary
Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations
Yujia Zhang, Xiaoyang Wu, Yixing Lao, Chengyao Wang, Zhuotao Tian, Naiyan Wang, Hengshuang Zhao
TL;DR
Concerto addresses whether multisensory-inspired joint learning can produce richer spatial representations than independent 2D or 3D self-supervision. It combines 3D self-distillation with 2D-3D joint embedding, outperforming single-modality and concatenated features in 3D perception and reaching state-of-the-art results with fine-tuning.
Problem
Independently learned 2D and 3D representations capture complementary spatial information, raising whether multimodal synergy can produce a superior shared representation space.
Method
Concerto combines 3D intra-modal self-distillation with 2D-3D cross-modal joint embedding, and adds video-lifted training and a linear translator into CLIP’s language space.
Results
Concerto outperforms standalone 2D and 3D models and their feature concatenation in linear probing, while achieving state-of-the-art performance across multiple benchmarks, including 80.7% mIoU on ScanNet.
Takeaways & Limitations
Joint 2D-3D self-supervision yields coherent spatial representations with fine-grained geometric and semantic consistency for 3D scene understanding.
Takeaways & Limitations
The study uses only 2D images and 3D point clouds, limiting alignment with language learned from a richer blend of modalities.
Abstract
from arXiv · showhide
Humans learn abstract concepts through multisensory synergy, and once formed, such representations can often be recalled from a single modality. Inspired by this principle, we introduce Concerto, a minimalist simulation of human concept learning for spatial cognition, combining 3D intra-modal self-distillation with 2D-3D cross-modal joint embedding. Despite its simplicity, Concerto learns more coherent and informative spatial features, as demonstrated by zero-shot visualizations. It outperforms both standalone SOTA 2D and 3D self-supervised models by 14.2% and 4.8%, respectively, as well as their feature concatenation, in linear probing for 3D scene perception. With full fine-tuning, Concerto sets new SOTA results across multiple scene understanding benchmarks (e.g., 80.7% mIoU on ScanNet). We further present a variant of Concerto tailored for video-lifted point cloud spatial understanding, and a translator that linearly projects Concerto representations into CLIP's language space, enabling open-world perception. These results highlight that Concerto emerges spatial representations with superior fine-grained geometric and semantic consistency.
1 Introduction
Concerto addresses complementary information in independently learned 2D and 3D representations by combining them through multisensory-inspired joint self-supervision. It produces stronger spatial features and improves 3D scene perception over single-modality and concatenated baselines.
- Independent image and point-cloud representations capture complementary spatial information, motivating a richer shared feature space.
- Human multisensory learning inspires representations that integrate geometry, texture, and semantics yet can be recalled from a single modality.
- Concerto couples 3D intra-modal self-distillation with 2D-3D cross-modal joint embedding, using 40k point clouds and 300k images for pretraining.
- Concerto achieves 77.3% mIoU on ScanNet with linear probing, improving over Sonata by 4.8% and surpassing Sonata-DINOv2 feature concatenation by 1.4%.
- With full fine-tuning, Concerto reaches 80.7% mIoU on ScanNet and achieves state-of-the-art performance across multiple scene perception tasks.
- Table 1 compares 2D, 3D, concatenated, and joint 2D-3D features, with Concerto achieving the best performance across all reported metrics.
2 Beyond Single Modality: Toward a New World of Representations
The paper asks whether joint 2D-3D learning can form a richer spatial representation than independently learned or concatenated features. It further considers whether such multimodal representations can connect spatial concepts to language despite limited sensory coverage.
- The central question is whether synergy between 2D images and 3D point clouds can produce a superior representation space beyond single-modality learning.
- A pilot fusion experiment shows that concatenating DINOv2 image features with Sonata point-cloud features improves linear-probed 3D semantic segmentation, indicating complementary information.
- Concatenation remains limited because it does not integrate modalities during learning or fully capture jointly formed, single-modality-retrievable embeddings.
- The proposed framework learns spatial information through both intra-modal refinement and cross-modal prediction.
- Linear projection into CLIP’s language space is proposed both to evaluate concept alignment and to extend open-vocabulary spatial perception.
3 Concerto: Joint 2D-3D Self-Supervised Learning
Concerto combines 3D self-distillation with image-conditioned cross-modal prediction to learn spatial representations beyond naive feature fusion. Its design uses a stable 3D teacher-student foundation and cross-modal alignment to expand representation capacity.
- 3.1 Intra-Modal Self-Distillation: The intra-modal branch trains a Point Transformer V3 student against a momentum-updated teacher using clustering-based consistency across augmented point-cloud views.
- 3.1 Intra-Modal Self-Distillation: Micro-designs mitigate geometric shortcuts in sparse point clouds, encouraging the model to learn geometric and structural priors from input features.
- The cross-modal objective predicts point-cloud embeddings that match image representations, following a JEPA-like conditional latent-prediction design.
- Camera parameters condition the predictor to establish correspondences between image pixels and point-cloud points.
- Concerto combines intra-modal self-distillation and cross-modal joint embedding to learn representations more expressive than concatenated 2D and 3D features.
- The architecture also supports training on point clouds without paired images, enabling hybrid self-supervised learning on large-scale 3D datasets.
4 Experiments
Concerto is evaluated across scene perception, video-lifted data, language probing, scaling, and ablations, with strong results across segmentation benchmarks and evidence that design choices affect performance.
- Main Results: 80.7% mIoU on ScanNet, 39.2% on ScanNet200, and 50.7% on ScanNet++ establish Concerto’s SOTA semantic-segmentation results under full fine-tuning.The model also achieves SOTA performance across the listed datasets and evaluation settings.
- Main Results: Concerto demonstrates the strongest instance-segmentation performance across four benchmarks and all evaluation protocols.Decoder probing on ScanNet can outperform full fine-tuning, indicating strong pretrained representations.
- Video Processing and Scaling: Concerto’s video-trained variation supports video-lifted point clouds, while the large model with additional video data shows potential for further scaling.Video-lifted point clouds are reconstructed from videos and added to training data.
- Language Probing: 44.56% mIoU is achieved on ScanNet zero-shot segmentation by linearly translating Concerto representations into language space without ground-truth labels.This remains below the 77.3% mIoU obtained with supervised linear probing.
- Ablation Studies: Scaling pretraining from 23k to 40k point clouds and increasing the backbone to 108M parameters both improve performance.The default ablation uses 23k data and a 39M backbone, while the scaled setting uses 40k data and a larger model.
5 Related Work
Prior work learns self-supervised representations mainly within 2D images or 3D point clouds, while multimodal 3D methods often transfer or imitate image features rather than fully modeling cross-modal interaction.
- 2D image self-supervised methods learn invariant representations from transformations or augmentations and have advanced visual representation learning.
- Methods that incorporate image features into 3D learning include lifting, rendering, distillation, attention fusion, and text-aligned encoders.These approaches target multimodal transfer from image representations into 3D systems.
- Existing approaches often imitate image features in point clouds or require image features during inference, whereas Concerto targets fuller multimodal interaction.
6 Conclusion and Discussion
Concerto delivers strong spatial representation performance across benchmarks and extends joint 2D-3D learning toward video and language, while its current scope leaves native multimodal training and deep language grounding for future work.
- Concerto achieves SOTA performance across multiple benchmarks as a joint 2D-3D self-supervised spatial representation model.
- The work adds a video-spatial variation and explores alignment of Concerto representations with text spaces.
- The current recipe freezes the image encoder, leaving joint native multimodal pretraining with trainable image and point-cloud encoders as future work.
A Additional Implementation
The implementation establishes 3D–2D correspondences through camera projection and depth-based visibility checks, then supports cross-modal learning across several data sources and views.
- Cross-Modal Joint Embedding: Training uses four local, two masked, and two global views, with masked-view representations predicting corresponding image representations.Cross-modal cosine similarity is computed at upcast level 3, while intra-modal clustering loss is computed at level 2.
- A.2 Correspondence Between Pixels and Points: 3D points are projected into 2D pixels using camera intrinsics and extrinsics, producing pixel coordinates and projected depth.
- A.2 Correspondence Between Pixels and Points: Visibility is verified by comparing projected depth with the camera depth map, rejecting out-of-bounds or invalid-depth correspondences.The depth tolerance is set to 0.01.
- A.2 Correspondence Between Pixels and Points: The resulting point–pixel mapping enables direct correspondence between 3D points and ViT patches for cross-modal joint embedding prediction.
- Data Sources: Depth maps come from RGBD channels, rendered ground-truth meshes, or per-view reconstruction outputs for video-lifted point clouds.
- Augmentation: Image augmentations use color jittering, random grayscale, and Gaussian blur while excluding geometric transformations to preserve pixel–point alignment.This setting yields 75.27% mIoU on ScanNet, lower than using the original images.
A.4 Experimental Setting
The experiments use specified open-source 3D datasets, broad pretraining splits, and standardized optimization and evaluation procedures on substantial computing resources.
- Software and hardware environment: Base-model pretraining without video data takes 85 hours on 16 Nvidia H20 GPUs, with evaluation using eight H20 GPUs.
- Data license: The study uses ScanNet, ScanNet++, S3DIS, Structured3D, ARKitScenes, HM3D, and RealEstate10K under their respective dataset licenses.
- Training details: Pretraining uses train, validation, and test splits, while downstream evaluation trains on train data and tests on validation data across four scene-understanding benchmarks.
- Training details: AdamW optimization, cosine annealing, a maximum learning rate of 0.004, and 100 pretraining epochs define the main training procedure.
B Additional Results
Additional experiments extend Concerto to video-lifted point clouds and language localization, while comparing image-encoder choices for semantic segmentation.
- Language locate: Text-aligned linear probing enables zero-shot localization of specified objects in HM3D scenes.
- Different 2D encoders: The image-encoder comparison evaluates Concerto variants across four datasets for semantic segmentation.
- Video perception: Concerto adapts to both single-view and multi-view video-lifted point clouds, supporting offline reconstruction and single-view forward settings.
- Language locate: A linear translator aligns Concerto representations with CLIP’s text space through LSeg image-encoder outputs without ground-truth labels.
B.3 Results with Different 2D Encoder
Across four semantic-segmentation benchmarks, Concerto performs best in general when its 2D image encoder uses DINOv2 with self-distillation, while LoRA evaluates parameter-efficient adaptation.
- Results with Different 2D Encoder: The comparison evaluates DINOv2, SigLIPv2, and RADIO image encoders using Concerto variants pretrained on 40k non-video data.
- LoRA fine-tuning: LoRA fine-tuning substantially improves performance with few learnable parameters, supporting the reliability of pretrained Concerto representations.
- Results with Different 2D Encoder: DINOv2-based Concerto with self-distillation achieves the highest mIoU in general across the four-dataset comparison.The results suggest that intra-modal self-distillation provides the strongest synergy with cross-modal learning.
B.4 Results with LoRA Finetuning
Concerto’s LoRA fine-tuning improves performance and efficiency across data-limited ScanNet settings and multiple benchmarks. It achieves strong results with relatively few learnable parameters.
- B.4 Results with LoRA Finetuning: LoRA-based fine-tuning outperforms linear probing and full fine-tuning in mIoU across most ScanNet data-efficiency scenarios.It does so despite only a small increase in learnable parameters over original linear probing.
- B.4 Results with LoRA Finetuning: Linear probing with LoRA matches decoder probing in full evaluation and trails full fine-tuning by only 0.9% mIoU while improving training efficiency.The setting uses LoRA rank 8, alpha 16, and dropout 0.1.
- B.4 Results with LoRA Finetuning: LoRA fine-tuning performs comparably to decoder probing across four benchmarks with relatively small learnable parameters.These results support efficient adaptation of Concerto representations across varied scene-understanding tasks.