PaperlayerView exact revision on arXiv

Source-linked AI summary

PAGER: Partial-to-global Alignment via Geometric and Relational Distillation

Akira-Miranda Adeyomi Adeniran-Lowe, Binod Singh, Lars Arnold Dethlefsen, Lazaros Nalpantidis, Theodora Kontogianni

arXiv:2610.01589v1cs.CV

TL;DR

Pretrained 3D representations are usually developed on globally reconstructed scenes, while embodied perception uses partial camera-coordinate observations. PAGER adapts partial features to a frozen global semantic space using paired geometry and no partial-view labels. It outperforms supervised PEFT in-domain and surpasses fully fine-tuned Sonata in zero-shot ScanNet→ScanNet++ transfer.

  • Problem

    Pretrained 3D point-cloud representations are primarily studied on globally reconstructed scenes, leaving their behavior on partial camera-coordinate observations comparatively underexplored.

    Source: §1 · p. 1

  • Method

    PAGER uses paired partial/global geometry to align partial-view features with a frozen global semantic space, keeping the encoder and probe frozen and requiring no partial-view labels.

    Source: §1 · p. 2

  • Results

    53.93 vs. 48.09 mIoU in zero-shot ScanNet→ScanNet++ transfer shows PAGER surpassing fully fine-tuned Sonata; it also outperforms label-supervised PEFT in-domain.

    Source: §1 · p. 3

  • Takeaways & Limitations

    PAGER’s results suggest that aligning partial features to a frozen global representation can preserve transferability better than adapting the backbone to a labeled source domain.

    Source: §1 · p. 3, §6 · p. 9

    PAGER requires paired partial/global observations during training, which are available for RGB-D datasets with camera poses but not universally, and it inherits the frozen probe’s label taxonomy.

    Source: §5.3 · p. 8

Abstract

from arXiv · show

Pretrained 3D encoders are typically developed on globally reconstructed scenes expressed in a consistent world coordinate frame, whereas embodied systems must reason from partial, viewpoint-dependent observations in camera coordinates. We show that this shift from globally learned 3D feature spaces to realistic partial observations exposes a severe representation mismatch, which we find consistently across representative state-of-the-art encoders, including Sonata and Concerto. A frozen Sonata encoder with a global linear probe achieves 72.47 mIoU on full ScanNet scenes, but 2.57 mIoU on single-frame camera-coordinate inputs. Training-free gravity alignment recovers performance to 41.64 mIoU, showing that coordinate-frame mismatch is a dominant source of degradation but cannot be fully resolved through canonicalization alone. We introduce PAGER, a label-free adaptation method that aligns partial-view features with a frozen global 3D semantic space using only paired partial/global geometry. It learns lightweight adaptation modules while keeping the pretrained encoder and global segmentation probe frozen. Matched-point feature alignment anchors partial features to their global counterparts, while relational supervision preserves their similarity structure with respect to the global representation. Global geometry provides supervision only during training. Inference operates directly on the partial observation. Without partial-view labels, PAGER outperforms label-supervised PEFT on both Sonata and Concerto, and in zero-shot ScanNet$\rightarrow$ScanNet++ transfer surpasses fully fine-tuned Sonata ($53.93$ vs.\ $48.09$ mIoU), suggesting that preserving the frozen global representation can improve cross-dataset transfer.

1 INTRODUCTION

Globally pretrained 3D encoders face a representation gap on partial camera-coordinate observations, a setting not yet well studied for point-cloud representations. PAGER adapts these observations to a frozen global semantic space using paired geometry rather than partial-view labels.

1 INTRODUCTION

  • Existing 3D encoders are primarily developed on globally reconstructed scenes, unlike embodied perception from partial, occluded, limited-FOV camera-coordinate point clouds.The effect of this viewpoint gap on pretrained 3D point-cloud representations remains underexplored.Source: §1 · p. 1
  • A frozen Sonata encoder with a global linear probe scores 72.47 mIoU on full ScanNet scenes but 2.57 mIoU on single-frame camera-coordinate views.The same partial views score 67.45 mIoU in world coordinates.Source: §1 · pp. 1–2
  • Fully supervised fine-tuning and existing PEFT methods depend on costly target-domain semantic labels, which may also limit cross-dataset generalization when labeled views are scarce.The paper notes that existing PEFT had not been applied to partial-view segmentation.Source: §1 · p. 2
  • PAGER aligns partial camera-coordinate observations with a frozen globally pretrained model’s semantic feature space using paired partial/global geometry.Matched-point feature anchoring and point-specific relational supervision transfer global structure; the encoder and probe stay frozen, and inference needs neither the complete scene nor camera pose.Source: §1 · p. 2
  • 65.46 mIoU on ScanNet and 53.93 vs. 48.09 mIoU on ScanNet→ScanNet++ show PAGER outperforming supervised PEFT in-domain and fully fine-tuned Sonata under domain shift.PAGER uses 1.3M trainable parameters and no partial-view labels.Source: §1 · p. 3

2 RELATED WORK

Prior 3D representation work largely targets globally reconstructed scenes, while view-constrained point-cloud segmentation remains comparatively unexplored. PAGER addresses this deployment gap with label-free adaptation from paired partial/global geometry.

3D representation learning

  • Self-supervised 3D pretraining and scene-level representations have advanced, but models are trained and evaluated primarily on globally reconstructed scenes.This leaves a deployment gap when applying them to single-view partial observations.Source: §2 · p. 3

Egocentric and cross-view understanding

  • Unlike video, where viewpoint invariance is established, the analogous gap in pretrained 3D point-cloud representations has received little attention.Dense per-point segmentation from one partial camera-coordinate cloud without multi-frame fusion or reconstruction remains largely unexplored.Source: §2 · p. 3

Efficient adaptation and PEFT for 3D

  • Standard PEFT approaches can transfer poorly to point clouds by ignoring local spatial structure, while GEM uses geometry-aware adapters but assumes reconstructed scenes and semantic supervision.GEM is described as achieving performance close to full fine-tuning with fewer trainable parameters.Source: §2 · p. 3

PAGER

  • PAGER targets camera-coordinate partial clouds with a lightweight translator learned from paired geometry, transferring local and scene-level structure through similarity relations to shared global keys.The setting combines limited context and a coordinate frame different from pretraining, without relying on semantic labels.Source: §2 · p. 3

3 REPRESENTATION GAP ANALYSIS

The analysis separates geometric incompleteness from coordinate-frame mismatch and finds the latter has the larger effect on frozen Sonata performance. Training-free canonicalization partly recovers the loss but remains below PAGER.

Evaluation conditions

  • The four input conditions progress from Global to Room Slice, View Crop, and Single Frame, with world- and camera-coordinate variants.Camera—Global transforms the complete accumulated scene, while View Crop and Single Frame use their corresponding camera poses.Source: §3 · pp. 3–4

Two separable sources of degradation

  • Within camera coordinates, reducing geometry further lowers performance from 5.14 to 2.57 mIoU, so both factors contribute.The coordinate frame is the larger and qualitatively distinct effect.Source: §3 · p. 4

Can training-free canonicalization solve the gap?

  • Gravity alignment recovers frozen-model performance to 41.64 mIoU, but remains below PAGER at 65.46 mIoU.PCA-based axis alignment reaches 8.38 mIoU, so coarse orientation recovery helps but does not fully restore the frozen global representation.Source: §3 · p. 4

Two separable sources of degradation

  • −70 mIoU under camera-coordinate shift versus −5 mIoU under world-coordinate geometric reduction summarizes the larger representation gap from frame mismatch.With a complete scene transformed to camera coordinates, mIoU falls from 72.47 to 5.14; single-frame camera-coordinate input scores 2.57.Source: §3 · p. 4, Table 1 · p. 5

Motivation

  • Matched partial/global feature supervision can transfer structure learned in complete-scene context, particularly for close-range views with visually similar planar regions.Those global targets were produced in the context of the complete reconstructed scene.Source: §3 · p. 5

4 METHODOLOGY

PAGER adapts camera-coordinate partial features to a frozen global feature space without partial-view semantic labels or updating the encoder and probe. Paired partial/global geometry supplies matched-point and relational supervision.

4 METHODOLOGY

  • PAGER uses paired partial/global geometry to adapt partial-view features without semantic labels while keeping the pretrained encoder and global probe frozen.Adaptation targets compatibility with the global feature space and frozen classifier.Source: §4 · p. 5
  • Compared with supervised adaptation, PAGER avoids dense partial-view labels and class-taxonomy assumptions, while pointwise regression leaves inter-feature relationships unconstrained.Full fine-tuning also updates the entire encoder per target setting and may risk corrupting pretrained global relational structure.Source: §4 · p. 5

4.1 PAGER

  • A lightweight residual adapter aggregates local voxel-neighborhood information, while scene-level information is supplied through the loss rather than global attention.The adapter uses a parameter-efficient bottleneck.Source: §4.1 · pp. 5–6

4.2 TRAINING OBJECTIVE

  • Matched-point cosine anchoring aligns each partial feature with its corresponding global feature, while KL supervision matches similarity distributions over randomly sampled global-scene keys.The teacher distribution is sharpened with a lower temperature; direct similarity regression is also a valid parameterization, with KL retained as the default.Source: §4.2 · p. 6

4.3 WHY THE TWO LOSSES ARE COMPLEMENTARY

  • The relational loss complements pointwise anchoring by constraining inter-feature structure, and random scene-wide keys avoid the correlated similarities expected from spatially adjacent keys.Random sampling is O(M) and parallelizable; the cited ablation reports a drop when k-NN keys replace random keys.Source: §4.3 · p. 6

5 EXPERIMENTS

PAGER is evaluated as label-free partial-view adaptation across pretrained encoders and datasets, including zero-shot cross-domain transfer. Ablations examine its alignment objectives and key sampling, while the limitations section notes training-data and in-domain performance constraints.

5 EXPERIMENTS

  • Evaluation covers ScanNet, ScanNet200, and ScanNet++, comparing label-requiring baselines with PAGER across Sonata and Concerto.The reported baselines include adapters, GEM, linear probing, and partial or full encoder fine-tuning.Source: §5 · p. 7

5.1 RESULTS

  • 65.46 mIoU with Sonata and 71.67 with Concerto versus GEM's 59.81 and 69.90, respectively, on ScanNet without partial-view labels.PAGER also closes most of Sonata's in-domain gap to full fine-tuning with only 1.2% of the trainable parameters.Source: §5.1 · p. 7
  • Sonata and Concerto both collapse on partial camera-coordinate inputs despite different pretraining, while PAGER uses the same adapter and objectives for both.Concerto's consistent gains over supervised GEM across three benchmarks suggest PAGER benefits from improvements in the underlying pretrained encoder.Source: §5.1 · pp. 7–8
  • 53.93 vs. 48.09 mIoU: PAGER outperforms fully fine-tuned Sonata in zero-shot ScanNet-to-ScanNet++ transfer.Models are trained on ScanNet partial views and evaluated on ScanNet++ partial views.Source: Table 3 · p. 8, §5.1 · p. 8

5.2 ANALYSIS EXPERIMENTS

  • Combining cosine anchoring and relational distillation reaches 65.46 mIoU, 2.15 above cosine-only and 9.88 above KL-only on ScanNet Partial.The two losses target distinct aspects of partial-to-global alignment.Source: §5.2 · p. 8
  • Replacing random keys with k-NN keys drops ScanNet Partial performance from 65.46 to 49.71 mIoU, while key-count variation stays under 1.5 mIoU.M = 4096 was the best value found in the sweep, not a sharply defined optimum; sampling cost is linear in M and independent of scene size.Source: §5.2 · p. 8
  • D-DITR checkpoints are applied directly to partial inputs, so the authors frame Table 5's performance gap primarily as distribution shift, not a direct algorithmic comparison.The evaluation includes original ScanNet partial scenes and RGB-D reconstructed counterparts.Source: §5.2 · p. 8, Table 5 · p. 9

5.3 LIMITATIONS.

  • PAGER requires paired partial/global observations during training and remains below in-domain Sonata full-plus-linear performance at 65.46 vs. 69.26 mIoU.It inherits the frozen probe's label taxonomy; new vocabularies require remapping or a new probe.Source: §5.3 · pp. 8–9

6 CONCLUSION

PAGER aligns partial camera-coordinate features with a frozen global semantic space using paired geometry and no semantic labels. The paper reports improved partial-view predictions and label-free transfer results.

6 CONCLUSION

  • PAGER aligns partial camera-coordinate 3D features with a frozen global semantic space using paired geometry and no semantic labels.The conclusion describes geometric self-distillation as combining a matched-point cosine anchor with a point-specific relational objective, without modifying the backbone.Source: §6 · p. 9
  • PAGER outperforms label-supervised PEFT in-domain and fully fine-tuned Sonata in zero-shot cross-dataset transfer using 1.3M trainable parameters across two pretrained encoders.The paper's qualitative ScanNet Partial results show predictions closer to ground truth than frozen Sonata and label-supervised GEM, especially under severe FoV limitations and camera-coordinate inputs.Source: §6 · p. 9, Figure 4 · p. 9
  • The authors state that they will release code upon publication and use publicly available datasets and resources.They report that dataset, preprocessing, and training settings are detailed in the appendices.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 ROBUSTNESS ACROSS CAMERA POSE

PAGER remains strong across common camera-pitch bins and produces coherent in-the-wild segmentations from reconstructed single-image and stereo inputs.

A.1 ROBUSTNESS ACROSS CAMERA POSE

  • PAGER scores 61–66 mIoU in the steep-down, downward, and level bins covering 98.9% of evaluation frames, beating supervised GEM by 5–9 mIoU in each.Both methods degrade on the steep-up bin, which contains only 30 frames; targeted augmentation for underrepresented orientations is suggested as future work.Source: §A.1 · p. 13, Table 6 · p. 13

A.2 APPLICATION: IN-THE-WILD SEGMENTATION.

  • On inferred point maps from single RGB images, PAGER identifies a door without ground-truth depth, while Sonata (linear) and GEM label the region as wall.The input is reconstructed point geometry without camera intrinsics.Source: §A.2 · p. 13
  • On stereo point clouds from a mobile robot in unseen environments, PAGER produces coherent segmentations, with plausible labels across 183 recorded frames.The evaluation has no ground truth; stereo depth is projected using known camera intrinsics.Source: §A.2 · p. 13

A.3 PER CLASS ANALYSIS.

The appendix reports classwise results across partial-view datasets, a trade-off between relational-loss parameterizations, the encoder-selection rationale, and a visualization of PAGER losses.

A.3 PER CLASS ANALYSIS.

  • PAGER improves over GEM on most ScanNet-Partial classes, especially shower curtain, curtain, bathtub, sink, and door, while GEM leads on sofa and picture.Table 7 reports per-class IoU for 20 ScanNet classes.Source: §A.3 · p. 13, Table 7 · p. 14
  • On ScanNet++-Partial cross-evaluation, PAGER improves over GEM on most classes, with gains up to +28.11 IoU points, while GEM leads on toilet, picture, and window.Table 8 reports 20 class IoUs for models trained on ScanNet-Partial and evaluated on ScanNet++-Partial.Source: §A.3 · pp. 13–14, Table 8 · p. 14
  • On ScanNet200-Partial, PAGER leads on several structural and mid-sized categories, while GEM is stronger on some fine-grained or rare classes.Table 9 shows 20 representative classes, selected as the 10 largest gains for each method.Source: §A.3 · p. 14, Table 9 · p. 15
  • On ScanNet++-Partial, PAGER gains on categories including coat hanger, blanket, refrigerator, bed, shower wall, and towel, while GEM leads on several fine-grained objects.Table 10 presents 20 of 100 semantic classes for visualization.Source: §A.3 · p. 14, Table 10 · p. 15

A.4 RELATIONAL LOSS PARAMETERIZATION

  • KL and sampled Gram relational losses both improve over the cosine anchor, but trade off class-balanced mAcc against overall mIoU and allAcc.Gram regression raises mIoU by +1.81 on Sonata and +2.47 on Concerto; KL raises mAcc by +2.75 and +2.51, respectively. The paper uses KL in its main configuration.Source: §A.4 · p. 15, Table 11 · p. 15

A.5 CHOICE OF ENCODER

  • Sonata is the strongest pretrained representation in the tested partial setup, so it is used for the main experiments.The comparison also evaluates Sonata with the PTv3 decoder for a controlled comparison.Source: §A.5 · p. 15, Table 12 · p. 16

B.1 VISUAL REPRESENTATION OF THE LOSS.

  • Figure 6 provides a graphical visualization of the PAGER losses.The figure caption identifies it as a loss visualization.Source: §B.1 · p. 15, Figure 6 · p. 16

B.2 IMPLEMENTATION DETAILS.

The implementation follows established codebases and a shared baseline training protocol; preprocessing constructs camera-frame partial clouds, labels, and tracked correspondences to global geometry.

B.2 IMPLEMENTATION DETAILS.

  • The implementation uses Pointcept and follows official Sonata and GEM implementations; baselines train for 100 epochs with the same batch size on one NVIDIA H100 PCIe GPU with 80GB memory.The authors report that all baselines had converged by 100 epochs.Source: §B.2 · p. 16, Table 13 · p. 16

C DATASET AND PREPROCESSING DETAILS

  • RGB-D pixels are back-projected and invalid depths discarded; retained points form dense partial clouds with RGB, mesh-derived semantic and instance labels, and camera-frame normals.Mesh correspondences use a nearest valid vertex within 0.01 m, and unmatched RGB-D points are discarded.Source: C DATASET AND PREPROCESSING DETAILS · pp. 16–17

C.1 CORRESPONDENCES BETWEEN PARTIAL POINT CLOUDS AND THE GLOBAL MESH

  • Nearest-neighbor matches in world coordinates assign partial points to global-scene coordinates, while persistent origin IDs propagate correspondences through indexing, voxelization, and augmentation.Voxel representatives retain links to their original points so voxel-level correspondences stay aligned after subsequent transforms.Source: §C.1 · p. 17

C.2 MOBILE ROBOT DATA ACQUISITION

The mobile-robot acquisition uses calibrated stereo depth to form partial point clouds, with distant points filtered and normals estimated.

C.2 MOBILE ROBOT DATA ACQUISITION

  • A Husky A200 UGV carrying a Roboception RC-Visard 160c stereo camera provides synchronized RGB images and onboard disparity and depth maps for real-world evaluation.The camera is mounted on a rigid front-facing tower; only its disparity and depth maps are used.Source: §C.2 · p. 17
  • Calibrated intrinsics are used to back-project stereo depth into partial point clouds, and points beyond 7 m are discarded.The stereo system outputs rectified images, so no additional geometric calibration is required.Source: §C.2 · p. 17
  • Surface normals for the acquired points are estimated with the method cited as Bae & Davison (2024).Source: §C.2 · p. 17

C.3 DATASET STATISTICS

The evaluation uses ScanNet, ScanNet200, and ScanNet++ with dataset-specific frame-sampling protocols. The paper also notes privacy and monitoring considerations for deployment in human-centred environments.

C.3 DATASET STATISTICS

  • ScanNet comprises 1,513 indoor scenes and about 2 million views, while ScanNet++ provides sub-millimeter high-resolution scans with comprehensive 100-class annotations.ScanNet annotations cover 20 classes or 200 classes in the ScanNet200 version.Source: §C.3 · p. 17
  • Partial-variant training samples every 100th frame on ScanNet and every 60th on ScanNet++, with evaluation sampling every 30th and 60th frame, respectively.The ScanNet++ validation evaluation uses about 7,000 frames; methods use the same subsets for fair comparison, and cross-dataset experiments train with the ScanNet protocol and evaluate on ScanNet++ validation.Source: §C.3 · pp. 17–18, Table 15 · p. 18

C.4 BROADER IMPACTS

  • Deployment in human-centred environments raises privacy and monitoring considerations that warrant care in real-world use.Source: §C.4 · p. 18
Loading 2610.01589v1…