PaperlayerView exact revision on arXiv

Source-linked AI summary

SAGE: Semantic-Augmented Geometric Experts for Point Cloud Registration

Tomoyasu Shimada, Kazuhiko Murasaki, Shoichiro Takeda, Shogo Sato, Riku Inoue, Toshihiko Nishimura, Ryuichi Tanida

arXiv:2610.36658v1cs.CV

TL;DR

Low-overlap registration is difficult because geometry-only methods can confuse structures that look alike but differ semantically. SAGE combines frozen point-cloud semantic representations with semantic-guided expert routing; it achieves the highest registration recall among directly comparable methods on 3DLoMatch while remaining competitive on 3DMatch and outdoor scenes.

  • Problem

    Low overlap limits spatial context, making geometrically similar but semantically distinct structures difficult to distinguish for correspondence-based registration.

    Source: §1 · p. 1

  • Method

    SAGE combines frozen pre-trained semantic representations with semantic-geometric features, routing SG-MoE experts by semantic features without extra sensors or encoder retraining.

    Source: §1 · p. 2

  • Results

    SAGE achieves the highest registration recall among directly comparable methods on low-overlap 3DLoMatch while remaining competitive on 3DMatch and outdoor scenes.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Frozen semantic representations provide effective cues for resolving geometric ambiguity in low-overlap registration.

    Source: §1 · p. 2, §5 · p. 8

    At 50% semantic-feature replacement, SAGE's RR declines to 73.5%, indicating sensitivity to plausible but incorrect semantic features.

    Source: F. Limitations · p. 23

Abstract

from arXiv · show

Point cloud registration is an essential task for scene reconstruction, requiring the alignment of partially overlapping point clouds. Under low-overlap conditions, this task remains challenging because existing geometry-only methods cannot distinguish geometrically similar but semantically distinct structures (e.g., TV corners vs. desk corners and chair legs vs. table legs), resulting in incorrect correspondences. To address this geometric ambiguity, we present Semantic-Augmented Geometric Experts (SAGE), which integrates frozen pre-trained point cloud semantic representations into the registration pipeline without requiring additional sensors or fine-tuning the semantic encoder. To effectively leverage these semantic distinctions, SAGE employs a Semantic-Guided Mixture of Experts (SG-MoE) architecture. Through semantic-aware routing, SG-MoE enables each expert to specialize in distinct semantic-geometric patterns, reducing mismatches that confound geometry-only methods. With a structure-aware estimator, SAGE achieves the highest registration recall on 3DLoMatch while remaining competitive on 3DMatch. When evaluated with the same number of refinement iterations, SAGE further achieves the highest inlier ratio and registration recall on both benchmarks. These results indicate that frozen semantic representations provide effective cues for resolving geometric ambiguity in low-overlap registration.

1. Introduction

Low overlap leaves too little spatial context to distinguish semantically different structures with similar geometry, undermining correspondence-based registration. SAGE combines frozen semantic representations with semantic-guided expert routing and reports strong results on 3DLoMatch and other benchmarks.

1. Introduction

  • Low overlap makes correspondence-based registration difficult because geometrically similar but semantically distinct structures lack enough surrounding points for disambiguation.Incorrect correspondences can result when methods rely only on geometric features.Source: §1 · p. 1
  • SAGE integrates frozen pre-trained semantic representations with geometric features without additional sensors or semantic-encoder fine-tuning.It extracts semantic features, combines them with point clouds through a geometric encoder, and refines the resulting features using SG-MoE.Source: §1 · p. 2
  • SG-MoE routes experts using frozen semantic features while experts refine semantic-geometric features, preserving semantic and geometric information during refinement.This design addresses the dominance of geometric knowledge under registration losses.Source: §1 · p. 2
  • SAGE achieves the highest registration recall on 3DLoMatch among single-pass methods with a structure-aware estimator, alongside strong feature-matching, inlier-ratio, and KITTI translation-error results.It performs on par with iterative methods.Source: §1 · p. 2

2. Related Work

Registration research has progressed from handcrafted and learned geometric descriptors to transformer-based global context, but geometric similarity can still obscure semantic differences. SAGE draws on pre-trained semantic representations and routes MoE experts by semantics to address that ambiguity.

2. Related Work

  • Handcrafted descriptors and RANSAC/ICP pipelines established coarse-to-fine registration, while learned local features improved discriminability and robustness but still struggle when low overlap limits spatial context.Low overlap means little shared observable area, distinct from incomplete observations within individual scans.Source: §2 · p. 2
  • Transformers add global geometric context for low-overlap registration, yet purely geometric features still struggle to distinguish structures with similar patterns and different meanings.GeoTransformer [26] introduced geometric transformers and superpoint matching.Source: §2 · p. 3
  • Prior semantic-registration methods use additional modalities or task-specific labels, whereas SAGE integrates frozen point-cloud semantic representations without those additions.Examples include RGB-D or calibrated images, external semantic models, and explicit semantic labels.Source: §2 · p. 3
  • PSReg [19] routes MoE experts using overlap information to specialize for geometric patterns, while SAGE incorporates semantic representations into routing to distinguish similar geometry with different meanings.SAGE's semantic-aware routing supports specialization across both geometric and semantic characteristics.Source: §2 · pp. 3–4

3. Method

SAGE estimates the rigid transformation aligning two point clouds through a five-stage pipeline: semantic and geometric feature extraction, superpoint-level aggregation, iterative feature refinement, and correspondence-based registration. Its SG-MoE uses semantic features to route experts that refine semantic-geometric features.

3.1. Problem Formulation

  • The registration task is to estimate a rigid transformation that aligns point cloud Q to point cloud P.The transformation comprises rotation and translation.Source: §3.1 · p. 4

3.2. Two-Stage Feature Extraction

  • A frozen Sonata encoder extracts point-level semantic representations, which KPConv combines with point coordinates to produce semantic-geometric features.The two-stage design lets KPConv's conditional convolutions adapt to the joint semantic-geometric space.Source: §3.2 · p. 4

3.3. Spatial Feature Aggregation

  • Spatial feature aggregation maps semantic and semantic-geometric features to downsampled superpoints by pooling nearby point-level semantic representations.For each downsampled point, the method averages features from its 16 nearest neighbors in the original cloud.Source: §3.3 · p. 4

3.4. Semantic-Guided Mixture of Experts

  • SG-MoE routes using superpoint-level semantic representations while its experts refine semantic-geometric features, avoiding the semantic collapse that geometry-optimized routing can cause.The semantic router assigns each superpoint to its highest-probability expert; alternating geometric self-attention and SG-MoE cross-attention refine features before matching.Source: §3.4 · pp. 4–5

3.5. Training Objective

  • Training combines coarse correspondence supervision, fine point-match supervision, and a load-balancing loss, while keeping the Sonata encoder frozen.The load-balancing objective regularizes expert utilization; correspondence losses supervise matches at different spatial resolutions.Source: §3.5 · pp. 5–6

4. Experiments

SAGE is evaluated on 3DMatch and 3DLoMatch using single-pass and iterative protocols, with ablations and qualitative comparisons examining its components and correspondences. Under a shared six-iteration refinement budget, it attains the highest inlier ratio and registration recall on both benchmarks.

4.1. Implementation Details

  • Experiments train SAGE for 45 epochs on 3DMatch and compare point-cloud-only methods using direct prediction or iterative refinement; evaluation reports IR, FMR, and RR.The iterative comparison gives each method six refinement iterations; evaluation excludes semantic-enhanced methods requiring additional RGB images.Source: §4.1 · p. 6, §4.2 · pp. 6–7

4.2. Evaluation on 3DMatch and 3DLoMatch

  • With RANSAC-50K on 3DLoMatch, SAGE’s FMR is best or tied-best across sample counts (89.6–90.0%), while RR is 73.4–74.6% and can trail the best methods.On 3DMatch, its RR is highest among single-pass methods at 2500 samples (92.0%) and 1000 samples (91.9%).Source: §4.2 · pp. 6–7, Table 1 · p. 7
  • Under LGR, SAGE achieves 76.3% RR on 3DLoMatch, exceeding GeoTransformer by 2.3 points and RoITr by 2.9 points, while remaining competitive on 3DMatch (91.0%).Despite lower IR than RoITr, its correspondences exhibit stronger spatial consistency that LGR can exploit.Source: §4.2 · p. 7, Table 2 · p. 7
  • With the same six refinement iterations, SAGE attains the highest inlier ratio and registration recall on both 3DMatch and 3DLoMatch, reaching 92.2% and 77.6% RR respectively.PEAL records the highest FMR under this protocol.Source: §4.2 · p. 7, Table 2 · p. 7

4.3. Ablation Studies

  • Removing SG-MoE decreases RR by 1.6 points and IR by 3.9 points, while removing two-stage feature extraction drops FMR by 1.9 points and RR by 1.0 point.The complete model improves RR by 2.3 points over the baseline; two-stage extraction chiefly improves FMR (+1.4 points), whereas SG-MoE improves IR and RR (+0.3 and +1.3 points).Source: §4.3 · pp. 7–8, Table 3 · p. 7

4.4. Qualitative Results

  • On challenging 3DLoMatch scenes, GeoTransformer produces predominantly outlier correspondences for geometrically similar structures, whereas SAGE distinguishes them and achieves accurate final registration.Figure 4 colors inliers green and outliers red, with GeoTransformer shown above SAGE.Source: §4.4 · p. 8, Figure 4 · p. 8
  • Standard MoE produces fragmented, geometry-driven expert assignments, whereas SG-MoE assigns semantically similar regions to consistent experts without explicit supervision of expert roles.The highlighted furniture illustrates the coherent assignments.Source: §4.4 · p. 8, Figure 5 · p. 8

5. Conclusion

SAGE integrates frozen pre-trained semantic representations with a mixture of experts for point cloud registration, without additional sensors or semantic-encoder retraining. It distinguishes geometrically similar but semantically different structures and achieves the highest registration recall among directly comparable methods on low-overlap 3DLoMatch while remaining competitive on 3DMatch and outdoor scenes.

5. Conclusion

  • SAGE integrates frozen pre-trained semantic representations with a mixture of experts for registration, requiring neither additional sensors nor semantic-encoder retraining.It achieves the highest registration recall among directly comparable methods on low-overlap 3DLoMatch while remaining competitive on 3DMatch and outdoor scenes.Source: §5 · p. 8

A. Network Architecture Details

SAGE combines frozen continuous semantic features with geometric features, then uses semantic-guided expert routing to refine them before matching and transformation estimation. Its supplementary architecture also specifies indoor and outdoor backbones, matching losses, and expert-load balancing.

A.1. Semantic Representation Extractor (Sonata)

  • Sonata supplies frozen 1024-dimensional continuous semantic vectors that encode fine-grained relationships without dataset-specific category labels.The representations are used as features for both indoor and outdoor environments.Source: §A.1 · p. 12

A.2–A.3. Geometric Feature Extractor

  • The KPConv-FPN backbone incorporates Sonata features at its first layer; KITTI uses 30 cm voxels and a five-stage backbone for scenes spanning over 100 meters.For 3DMatch and 3DLoMatch, the first layer takes 1088 input dimensions; KITTI adds a downsampling stage to capture long-range relationships.Source: §A.2 · p. 12, §A.3 · p. 12

A.4. Semantic-Guided MoE Transformer

  • SG-MoE routes superpoints using frozen semantic features while its experts refine semantic-geometric features.The transformer stacks three geometric self-attention and SG-MoE cross-attention blocks; it uses four experts and top-1 routing.Source: §A.4 · p. 13

A.5. Superpoint Matching Module

  • Superpoint matching normalizes Gaussian feature correlations in both directions, then selects the top-scoring correspondences.Empty local patches are removed before matching; inference uses 256 putative superpoint matches.Source: §A.5 · p. 13

A.6. Point Matching Module

  • Point matching uses optimal transport with dustbin entries for unmatched points, then extracts mutual top-k assignments.The procedure uses k = 3 to produce approximately 6000 correspondences for Local-to-Global Registration.Source: §A.6 · p. 13

A.7. Local-to-Global Registration

  • Local-to-Global Registration estimates transformations by weighted SVD per superpoint pair, then selects and iteratively refines the transformation with the most inliers.The method is parameter-free and RANSAC-free; refinement uses five iterations.Source: §A.7 · p. 14

B.1–B.2. Matching Losses

  • Training uses overlap-aware circle loss for coarse superpoint matching and negative log-likelihood for point-level assignments.The coarse loss distinguishes positive patches with at least 10% overlap from negatives with no overlap; point-level supervision includes dustbin assignments for unmatched points.Source: §B.1 · p. 14, §B.2 · p. 14

B.3. Expert Load Balancing Loss

  • The MoE load-balancing loss minimizes KL divergence between batch-average expert routing and a uniform distribution.This discourages routing strategies that deviate substantially from uniformity and encourages use of all experts.Source: §B.3 · p. 14

B.4. Training Configuration

Training uses Adam for 45 epochs, with a learning rate that decays each epoch and specified loss hyperparameters.

B.4. Training Configuration

  • The model is trained with Adam for 45 epochs using an initial learning rate of 10^-4.The learning rate is multiplied by 0.95 after each epoch.Source: §B.4 · p. 14
  • The coarse-loss margins are Δp = 0.1 and Δn = 1.4, with γ = 80; the fine matching radius is τ = 0.05 m..Source: §B.4 · p. 14
  • The MoE load-balancing loss weight is λ_moe = 0.01, and the random seed is 7351.Source: §B.4 · p. 14

C. Evaluation Metrics

The evaluation uses correspondence-quality and registration-success metrics for indoor benchmarks, and transformation-error metrics for KITTI. Supplementary KITTI results report competitive registration performance using Local-to-Global Registration as the default estimator.

C. Evaluation Metrics

C.1. 3DMatch and 3DLoMatch

  • For 3DMatch and 3DLoMatch, IR measures the inlier fraction, FMR the fraction of pairs with sufficient inliers, and RR the fraction successfully registered.IR uses a 0.1 m inlier threshold; FMR uses a 5% threshold, and RR uses an RMSE success threshold of 0.2 m.Source: §C.1 · p. 15

C.2. KITTI

  • For KITTI, RTE and RRE measure translation and rotation errors, while RR counts pairs below both error thresholds.The thresholds are 2 m translation and 5° rotation; mean RTE and RRE are computed only over successfully registered pairs.Source: §C.2 · p. 15

D. Additional Experiments

  • Supplementary experiments use confidence-weighted Local-to-Global Registration, a RANSAC-free estimator, unless otherwise stated.The paper identifies it as the default estimator for supplementary evaluation.Source: D. Additional Experiments · p. 15

D.1. Outdoor Scene Performance on KITTI

  • SAGE reports RTE of 6.1 cm, RRE of 0.22° and RR of 99.8% on KITTI.The supplementary evaluation uses Local-to-Global Registration by default; Table C presents KITTI registration performance.Source: §D.1 · p. 15

D.2. Scene-wise Analysis

Scene-wise results show SAGE’s overall registration recall advantage is concentrated in environments with greater semantic diversity. Its gains are smaller or absent in more homogeneous scenes.

D.2. Scene-wise Analysis

  • SAGE achieves 87.9% overall RR, versus 86.4% for RoITr and 86.3% for GeoTransformer.Source: §D.2 · p. 16
  • In the diverse Study scene, SAGE reaches 80.3% RR versus 75.2% for GeoTransformer and 74.3% for RoITr; in Kitchen, it reaches 94.2% versus GeoTransformer’s 91.7%.The analysis attributes the Study-scene gain to expert specialization across semantically distinct objects.Source: §D.2 · p. 16
  • Gains are smaller in homogeneous scenes: Hotel 3 methods score 85.4–86.4%, while SAGE’s 81.2% RR in MIT Lab trails GeoTransformer’s 83.9%.The cited scenes contain uniform corridors and walls or repetitive structural elements with limited semantic variation.Source: §D.2 · p. 16

D.3. Overlap-wise Analysis

SAGE’s recall advantage is strongest at low-to-moderate overlap and narrows when geometry provides ample context. It incurs additional compute, largely from the frozen semantic encoder.

D.3. Overlap-wise Analysis

  • 73.4% RR at 10–20% overlap and 88.7% RR at 20–30% show SAGE outperforming RoITr and GeoTransformer in the most challenging bins.At 10–20%, SAGE exceeds RoITr by 1.2 points and GeoTransformer by 2.8; at 20–30%, it exceeds them by 3.2 and 2.3 points, respectively.Source: §D.3 · pp. 16–17
  • In the 50–60% overlap bin, SAGE achieves 98.4% registration recall, slightly below GeoTransformer’s 98.7%; ample geometric context makes registration substantially easier and semantic guidance’s benefit diminishes accordingly.Source: §D.3 · p. 16

D.4. Computational Efficiency

  • SAGE has 121.71M parameters versus GeoTransformer’s 9.83M, but the frozen Sonata encoder accounts for 108.41M of SAGE’s total.Excluding Sonata, SAGE has 13.3M parameters; its routing networks and expert FFNs add approximately 3.5M beyond the GeoTransformer backbone.Source: §D.4 · p. 17
  • 179ms per pair makes SAGE’s inference time comparable to RoITr’s 140ms, while SAGE adds 2.3 percentage points of RR for 104ms of additional latency over GeoTransformer.RoITr adds 65ms without improving RR (73.4% vs. 74.0%).Source: §D.4 · p. 17
  • 16.69 GB peak memory fits within the cited 24 GB RTX 4090, while incorporating a pre-trained semantic encoder carries computational overhead.The authors describe the overhead as inherent to using pre-trained semantic encoders and suggest a more compact encoder could replace Sonata without changing SG-MoE.Source: §D.4 · p. 17

D.5. Additional Ablation Study

The ablations examine encoder adaptation, routing choices, expert count, and semantic-feature degradation; qualitative analyses show how SAGE performs and how experts specialize. Baseline reproduction uses a matched protocol but has limits where original implementations or settings are unavailable.

D.5. Additional Ablation Study

  • Adapter-tuned Sonata yields 74.8% RR, a 1.5-point drop from the frozen baseline’s 76.3%, and the authors attribute this to a geometric shortcut that weakens semantic distinctions.The adapted features also reduce FMR from 89.4% to 86.4% and IR from 44.3% to 42.6%.Source: §D.5 · pp. 17–18
  • SCAR achieves 75.0% RR, below vanilla SG-MoE’s 76.3%, while maintaining similar IR (44.4% vs. 44.3%).The authors suggest cross-attention routing introduces premature correspondence assumptions; SG-MoE routes each cloud independently from its own semantic features before later cross-attention.Source: §D.5 · pp. 18–19
  • Four experts gives the best reported result at 76.3% RR; performance falls to 75.5% with five experts and 75.7% with six.Two experts achieve 74.9% RR and three achieve 75.1%; the authors suggest too many experts may over-specialize with insufficient training samples per expert.Source: §D.5 · p. 19
  • SAGE maintains RR above 75% under 50% semantic-feature noise or dropout, but structured replacement with plausible incorrect features can mislead routing and progressively degrade performance.The authors attribute resilience to the router learning to bypass noise and prioritize stable semantic cues; structured replacements instead cue irrelevant semantic contexts.Source: §D.5 · p. 19

D.6. Qualitative Visualization

  • On low-overlap pairs below 20%, SAGE uses semantic cues and distinct objects to establish correct correspondences where GeoTransformer often fails.Figure B compares their correspondences and registration results, including cases where GeoTransformer fails and SAGE achieves accurate alignment.Source: §D.6 · p. 19, §D.7 · p. 20

D.7. Expert Routing Analysis

  • Across six scenes, experts show recurring roles: Expert 4 handles large salient objects, Expert 2 mid-scale objects, Expert 1 small distinctive regions, and Expert 3 low-saliency backgrounds.The assignments emerge without explicit supervision, and source-target views route the same semantic regions consistently, including across viewpoints and occlusion.Source: §D.7 · pp. 19, 21, 22

E. Baseline Reproduction Details

  • Matched training and architecture settings are used for PEAL, PSReg, and SAGE, but PEAL’s prior-generation procedure is undocumented and PSReg’s official implementation and expert count are unavailable.The authors retrain or re-implement baselines under shared settings and report matched-protocol comparisons because reproduced PEAL and PSReg results fall below their papers’ reported values.Source: E. Baseline Reproduction Details · pp. 22–23

F. Limitations

SAGE’s benefits depend on reliable semantic representations and scene diversity; its guidance is less effective when semantics are misleading or geometric context is already sufficient.

F. Limitations

  • 50% Gaussian noise leaves SAGE at 76.2% RR, while 50% plausible-but-incorrect semantic-feature replacement lowers it to 73.5% RR.The authors link this sensitivity to structured semantic errors that can mislead the router, including representations from underrepresented scene or object categories.Source: F. Limitations · p. 23
  • In the repetitive MIT Lab scene, SAGE reaches 81.2% versus GeoTransformer’s 83.9%, while Hotel 3 methods are similar at 85.4–86.4%.The scene-wise analysis ties SAGE’s improvement over baselines to each scene’s semantic diversity.Source: F. Limitations · p. 23
  • When overlap is 50–60% and geometric context is sufficient, SAGE’s 98.4% is marginally below GeoTransformer’s 98.7%.The authors say uniform planar overlap regions provide little semantic variation for the router to use.Source: F. Limitations · p. 23
  • Severe occlusion or truncated semantic objects can leave featureless or nondiscriminative overlap regions, a challenge the authors describe as fundamental to correspondence-based methods.Temporal cues from sequential scans or multi-view constraints remain an open direction for these environments; the analyses clarify when semantic guidance is more or less effective.Source: F. Limitations · p. 23

Related papers

Loading 2610.36658v1…