PaperlayerView exact revision on arXiv

Source-linked AI summary

Utonia: Toward One Encoder for All Point Clouds

Yujia Zhang, Xiaoyang Wu, Yunhan Yang, Xianzhe Fan, Han Li, Yuechen Zhang, Zehao Huang, Naiyan Wang, Hengshuang Zhao

arXiv:2603.03283v2cs.CV

TL;DR

Point-cloud pretraining remains fragmented across domains, limiting progress toward shared sparse-3D representations. Utonia addresses this by jointly pretraining one self-supervised point transformer encoder, yielding cross-domain perception benefits and transfer to spatial reasoning and robotic manipulation.

  • Problem

    Point-cloud pretraining remains siloed by domain, motivating a single encoder that learns shared representations across diverse 3D observations.

  • Method

    Utonia jointly pretrains one self-supervised point transformer encoder across diverse point-cloud domains using domain-agnostic designs for modality, granularity, and geometric mismatches.

  • Results

    Utonia yields emergent cross-domain benefits, reaches 81.1% mIoU on ScanNet, and transfers to spatial reasoning and robotic manipulation.

  • Takeaways & Limitations

    Utonia provides evidence that sparse point clouds can support unified spatial cognition across perception, multimodal reasoning, and embodied manipulation.

  • Takeaways & Limitations

    Utonia focuses on transferable 3D geometry from static point clouds, leaving spatiotemporal pretraining for dynamic 4D settings as future work.

Abstract

from arXiv · show

We dream of a future where point clouds from all domains can come together to shape a single model that benefits them all. Toward this goal, we present Utonia, a first step toward training a single self-supervised point transformer encoder across diverse domains, spanning remote sensing, outdoor LiDAR, indoor RGB-D sequences, object-centric CAD models, and point clouds lifted from RGB-only videos. Despite their distinct sensing geometries, densities, and priors, Utonia learns a consistent representation space that transfers across domains. This unification improves perception capability while revealing intriguing emergent behaviors that arise only when domains are trained jointly. Beyond perception, we observe that Utonia representations can also benefit embodied and multimodal reasoning: conditioning vision-language-action policies on Utonia features improves robotic manipulation, and integrating them into vision-language models yields gains on spatial reasoning. We hope Utonia can serve as a step toward foundation models for sparse 3D data, and support downstream applications in AR/VR, robotics, and autonomous driving.

1. Introduction

Point clouds offer a compact, geometry-explicit alternative to dense visual data, but their sparse, unstructured nature and severe domain shifts make universal learning difficult. Utonia takes a first step toward a shared encoder across diverse point-cloud domains, revealing cross-domain benefits and downstream gains in spatial reasoning and robotic manipulation.

  • Motivation: Point clouds provide a compact, geometry-explicit representation of the physical world that complements or conditions dense images and videos.Dense visual pipelines remain dominant but are redundant carriers of 3D information and do not explicitly enforce geometric consistency.
  • Challenge: Sparse, unstructured point clouds exhibit extreme domain shifts in scale, density, sampling pattern, and auxiliary modalities across sensors and preprocessing.Outdoor LiDAR and object-centric scans differ substantially in sparsity, spatial extent, scan patterns, color, and normal availability.
  • Challenge: Prior point-cloud SSL methods remain domain-fragmented, typically training within one domain and transferring poorly despite all point clouds sampling the same 3D reality.The introduction attributes this fragmentation to practical design choices rather than the structure of the physical world.
  • Utonia: Utonia takes a first step toward one encoder for diverse point clouds by addressing mismatches in input channels, sparsity and density, and perceptual granularity.The model is presented as an exploration that identifies what breaks joint training and distills a minimal set of fixes.
  • Emergent behaviors: Joint pretraining lets object-, indoor-, and outdoor-scale data benefit under one encoder while preserving scene-level gravity alignment and largely gravity-irrelevant object geometry.These behaviors emerge when the domains are trained jointly rather than separately.
  • Downstream impact: Utonia representations improve spatial reasoning in VLM backends and robotic manipulation when used to condition VLA policies.These results extend the encoder’s benefits beyond 3D perception.

2. Pilot Study: What Prevents Unification?

The pilot study finds that naive joint pretraining across point-cloud domains is often unstable and suboptimal because domain-specific spatial scales, gravity conventions, and auxiliary modalities create harmful cues. These mismatches degrade overall performance and weaken cross-domain transfer unless training aligns granularity and controls domain priors and modality dependence.

  • Pilot study: Naive joint training across merged datasets is often suboptimal and unstable, motivating a pilot study of three recurring failure symptoms.The study identifies sensitivity to granularity shifts, bias toward gravity convention, and inconsistent modality availability.
  • Sensitivity to granularity shifts: Domain-specific grid sizes alter neighborhood statistics and local topology, coupling learned features to scale and impairing direct cross-domain transfer.Per-domain optimal settings can work separately, while global grid sizing with coordinate rescaling substantially improves joint training and transfer.
  • Bias toward gravity convention: Gravity-aligned scene conventions make height a domain cue, which can hurt transfer to fine-grained object-centric scans.The proposed direction is to retain upright structure for scene-scale scans while encouraging rotation invariance for fine-grained objects.
  • Inconsistent modality availability: Naive pretraining exploits colors and normals when available, causing representations and performance to degrade when modalities are missing, noisy, or defined differently.This modality dependence is unstable because domains expose different auxiliary channels beyond coordinates.
  • Conclusion: Together, spatial coordinate discrepancies and inconsistent modality availability let domain-specific cues dominate joint training, degrading overall performance and weakening cross-domain transfer.The two spatial discrepancies are discretization granularity and bias toward a gravity-aligned convention.

3. Utonia

Utonia pretrains a single Point Transformer V3 encoder across heterogeneous point-cloud domains using teacher-student self-distillation. Its multi-domain design unifies input modalities, aligns perceptual granularity, and applies RoPE to granularity-aligned coordinates for cross-domain robustness.

  • 3.1: Utonia pretrains a single Point Transformer V3 encoder across indoor scans, outdoor LiDAR, remote sensing, object CAD, and video-lifted point clouds.The method follows a teacher-student self-distillation pretraining recipe.
  • 3.1: A unified modality interface concatenates coordinates, colors, and normals, substituting default zeros when modalities are missing.This enables one encoder to accept inputs from all domains while exposing noise and shortcut-learning risks.
  • 3.2: Perceptual Granularity Rescale maps point clouds with different spatial extents and sampling patterns into a comparable coordinate space before positional encoding.The design addresses scale mismatch, which can otherwise encode domain-specific geometry into local operators and positional hints.
  • 3.3: RoPE computed from granularity-aligned, augmented coordinates is applied in every attention layer to rotate queries and keys under cross-domain shifts.Anisotropic coordinate jittering and isotropic scaling discourage binding semantics to a particular unit convention or axis scaling.

4. Experiments

Utonia performs strongly across indoor, outdoor, and object-centric 3D benchmarks, with results showing benefits from task-specific readouts, unified geometric processing, and robustness to missing modalities. Its representations also transfer beyond perception to robotic manipulation and spatial reasoning in multimodal models.

  • Indoor semantic segmentation: Utonia reaches 81.1% mIoU on ScanNet and 78.1% on S3DIS with full fine-tuning, while decoder probing largely closes linear-probing gaps on finer-grained indoor benchmarks.The results indicate that non-linear spatial heads improve indoor segmentation beyond a single linear layer.
  • Outdoor semantic segmentation: Utonia achieves the best outdoor mIoU among compared methods under linear and decoder probing and slightly surpasses Concerto with full fine-tuning.Reported Sonata results use additional supervised tuning and extra data, so they are not directly comparable to Utonia’s target-dataset fine-tuning.
  • Object part segmentation and classification: Object classification shows strong linear-probing transfer, whereas part segmentation improves mainly with full fine-tuning and a task decoder.This supports global readouts for object semantics and query-based decoders for fine-grained part cues.
  • Robustness to missing modalities: Utonia remains more stable than Concerto when color or normal inputs are missing or unreliable, demonstrating robustness across ScanNet and NuScenes.Concerto degrades substantially when modalities are absent, especially color.
  • Ablations: Unified grid sizing with coordinate rescaling stabilizes multi-domain training, while RoPE consistently improves performance across ScanNet200, Waymo, and PartNetE.RoPE’s largest reported gain occurs on outdoor Waymo, where density, scale, and coordinate conventions differ substantially across domains.
  • Embodied and multimodal transfer: Utonia features improve robotic manipulation and provide consistent gains in 3D visual grounding and question answering while remaining competitive on dense captioning.The manipulation features remain coherent in cluttered RGB-D scenes, including support surfaces and partially observed objects.

5. Related Works

Prior point-cloud self-supervised learning largely targets individual domains, while only limited work pursues unified pretraining across point-cloud types. Related work also spans domain adaptation and 3D positional encodings, with Utonia’s focus positioned differently from earlier RoPE applications.

  • Point self-supervised learning: Most point-cloud self-supervised methods focus on a single domain, including object-level representations and indoor or outdoor scene applications.These approaches are organized around reconstruction and contrastive learning.
  • Point self-supervised learning: Recent methods address geometry shortcuts and cross-modal prediction, but learning one encoder that generalizes across domains remains underexplored.Sonata handles geometry shortcuts, while Concerto incorporates cross-modal prediction with point SSL.
  • Unified models in point clouds: Only a limited number of works explicitly target unified point-cloud pretraining across various point-cloud types.Examples include adapting between voxels and points, pretraining on outdoor data, and mixture-of-domain-experts modeling.
  • 3D Rotary Position Embedding: The paper distinguishes its focus from prior RoPE work in 3D point transformers.The supplied passage introduces this distinction but ends before specifying the paper’s full RoPE focus.

6. Conclusion and Future

Utonia is presented as a first step toward a unified point-cloud encoder that learns shared representations across diverse domains and transfers beyond 3D perception. Future work targets more expressive task interfaces, 4D spatial cognition, and scalable sparse backbones.

  • Conclusion: Utonia jointly pretrains one point encoder across diverse point clouds, addressing cross-domain mismatches with minimal domain-agnostic designs.This yields a shared representation space with emergent cross-domain benefits.
  • Conclusion: Utonia features transfer beyond 3D perception to spatial reasoning in vision-language-model backends and robotic manipulation when conditioning vision-language-action policies.
  • Query-based task interfaces: Contrasting linear-probing trends across part segmentation and object classification suggest that a single linear probe is overly restrictive for diverse downstream tasks.The gap between linear probing and full fine-tuning on part segmentation indicates that task-relevant cues may be present but not linearly accessible under a fixed readout.
  • Query-based task interfaces: Future adaptation could introduce global registers for object-level semantics and provide a clean global interface for classification.
  • 4D spatial cognition: Moving toward 4D spatial cognition requires spatiotemporal pretraining objectives for dynamic, sequence-centric settings.Frame augmentation already improves sequence-dataset performance such as Waymo through pose-aligned multi-frame teacher aggregation and a single-frame student view.
  • A scalable next-generation backbone: Scaling multi-domain, task-conditioned, and 4D pretraining motivates a more efficient and deployable sparse backbone.Sparse convolution can limit token budget, resolution, and sequence length, while operator stacks may create deployment friction through kernel and system dependencies.

Appendix · A. Additional Implementation · A.1. Data Preparation and Training Details

Utonia uses balanced multi-dataset pretraining with cross-modal joint prediction when paired images are available. To address optimization challenges from domain gaps and varying noise, training begins with a stable initialization before continuing pretraining.

  • A.1. Data Preparation and Training Details: Utonia is pretrained on a diverse mixture of datasets summarized in Tab. 12.The passage identifies the training mixture as spanning multiple dataset sources.
  • A.1. Data Preparation and Training Details: 90k Cap3D instances are randomly subsampled per training epoch to balance the mixture because Cap3D is much larger than other sources.This balancing step is motivated by Cap3D’s dataset size.
  • A.1. Data Preparation and Training Details: Cross-modal joint prediction uses paired images when available, following Concerto [55].This procedure is not applied to PartNet, ScanObjectNN, or HK Remote.
  • A.1. Data Preparation and Training Details: For outdoor datasets without point-cloud color, image colors are projected onto points using the provided calibration.The passage describes this as part of outdoor-data preparation.
  • A.1. Data Preparation and Training Details: Joint training is optimization-challenging because large domain gaps and varying noise levels increase gradient variance and can destabilize training.These effects motivate the two-stage pretraining schedule.
  • A.1. Data Preparation and Training Details: Stage 1 pretraining uses ScanNet [7], Structured3D [60], Waymo [31], and PartNet [23] to obtain a stable initialization.These datasets also correspond to the ablation setting.

A.2. RoPE Implementation Details

The implementation augments canonical point coordinates with axis-wise jittering and isotropic scaling before applying separable 3D rotary positional embeddings. This perturbation reduces dependence on fixed axis scaling and strengthens cross-domain transfer, while requiring model channels divisible by 6.

  • Coordinate augmentation: Canonical coordinates receive coordinate jittering and scaling, following DINOv3, to improve cross-domain generalization.The method samples axis-wise multiplicative jitter with degree γ > 1 and isotropic multiplicative rescaling with degree η > 1.
  • RoPE construction: The 3D rotary positional embedding extends standard 1D RoPE separably across axes by splitting each feature vector into three equal subvectors.Each subvector is rotated using the corresponding augmented coordinate component.
  • Attention integration: Applying augmented coordinates to rotate queries and keys in every attention layer prevents semantic binding to fixed axis scaling and strengthens cross-domain transfer.The coordinate perturbation is applied directly within the attention layers.
  • Implementation constraint: Model channels must be divisible by 6 because RoPE is applied in the attention layer.The current model settings are provided in Table 11.

A.3. Downstream Tasks Implementation

Utonia is implemented across robotics manipulation, open-world part segmentation, and spatial reasoning tasks using domain-specific simulation, promptable decoding, and geometry-aware multimodal fusion. The downstream setups extend existing pipelines with Utonia or related 3D features.

  • Robotics manipulation: Robotics manipulation uses simulated tabletop clutter with RGB-D observations, synthesizing 10,680 Objaverse LVIS object instances across 290 categories.Each episode generates a physically plausible cluttered layout on a 0.4m × 0.5m tabletop, with expert trajectories produced using BoDex grasp enumeration and CuRobo collision-free planning.
  • Robotics manipulation: Robotics evaluation reports the three-attempt success rate, defined as the probability that the gripper successfully grasps an object within three opening-and-closing attempts.Results and discussions are reported in Section 4.2.
  • Open-world part segmentation: Open-world part segmentation replaces P3SAM’s MLP-based mask decoder with a SAM-like decoder to improve support for flexible and interactive prompting.The pipeline uses Sonata as a 3D feature extractor and integrates the SAM-like decoder architecture to enhance promptability.
  • Spatial reasoning: Spatial reasoning extends Video-3D LLM by encoding image-patch centers as pseudo point clouds and adding Utonia geometry-aware features to the original 2D visual features.The resulting 3D features are fused with the 2D features by addition, injecting explicit geometric cues into the pipeline.
Loading 2603.03283v2…