PaperlayerView exact revision on arXiv

Source-linked AI summary

C-GenReg: Training-Free 3D Point Cloud Registration by Multi-View-Consistent Geometry-to-Image Generation with Probabilistic Modalities Fusion

Yuval Haitman, Amit Efraim, Joseph M. Francos

arXiv:2604.16680v1cs.CV

TL;DR

Learning-based 3D registration struggles across sensing modalities, sampling differences, and environments. C-GenReg transfers geometry into a multi-view-consistent RGB domain, combines image and geometric correspondence posteriors without fine-tuning, and reports strong zero-shot performance across indoor RGB-D and outdoor LiDAR benchmarks, including real LiDAR data.

  • Problem

    Learning-based 3D registration methods have limited cross-domain generalization across sensing modalities, point density, and acquisition environments.

  • Method

    C-GenReg uses a frozen World Foundation Model for multi-view-consistent RGB generation, a task-specific VFM for dense matching, a geometric branch, and posterior-level Match-then-Fuse fusion.

  • Results

    C-GenReg achieves strong zero-shot performance across indoor RGB-D and outdoor LiDAR benchmarks and successfully operates on real LiDAR data.

  • Takeaways & Limitations

    The framework is a plug-and-play, training-free registration module that combines complementary generative-image and geometric correspondence cues.

  • Takeaways & Limitations

    Generated RGB views need geometric coherence with the input depth and consistency across viewpoints; otherwise, correspondences may become unreliable.

Abstract

from arXiv · show

We introduce C-GenReg, a training-free framework for 3D point cloud registration that leverages the complementary strengths of world-scale generative priors and registration-oriented Vision Foundation Models (VFMs). Current learning-based 3D point cloud registration methods struggle to generalize across sensing modalities, sampling differences, and environments. Hence, C-GenReg augments the geometric point cloud registration branch by transferring the matching problem into an auxiliary image domain, where VFMs excel, using a World Foundation Model to synthesize multi-view-consistent RGB representations from the input geometry. This generative transfer, preserves spatial coherence across source and target views without any fine-tuning. From these generated views, a VFM pretrained for finding dense correspondences extracts matches. The resulting pixel correspondences are lifted back to 3D via the original depth maps. To further enhance robustness, we introduce a "Match-then-Fuse" probabilistic cold-fusion scheme that combines two independent correspondence posteriors, that of the generated-RGB branch with that of the raw geometric branch. This principled fusion preserves each modality inductive bias and provides calibrated confidence without any additional learning. C-GenReg is zero-shot and plug-and-play: all modules are pretrained and operate without fine-tuning. Extensive experiments on indoor (3DMatch, ScanNet) and outdoor (Waymo) benchmarks demonstrate strong zero-shot performance and superior cross-domain generalization. For the first time, we demonstrate a generative registration framework that operates successfully on real outdoor LiDAR data, where no imagery data is available.

1. Introduction

C-GenReg addresses domain-dependent 3D matching by transferring geometry into a multi-view-consistent image domain while retaining a geometric branch. It combines pretrained generative and correspondence models without fine-tuning, then fuses their correspondence posteriors.

  • Point cloud registration estimates a rigid transformation aligning source and target scans, but performance remains limited primarily by imprecise feature matching.
  • The geometric branch independently extracts structural features from raw point clouds, complementing the generated-RGB branch.
  • Learned 3D extractors vary substantially with sensing modality, point density, and acquisition settings, limiting generalization from indoor RGB-D to outdoor LiDAR.
  • C-GenReg uses a World Foundation Model to generate geometrically aligned, multi-view-consistent RGB views and a task-specific VFM to estimate dense image correspondences.
  • “Match-then-Fuse” combines independent image and geometric correspondence posteriors to produce calibrated, robust matches without learning.
  • The plug-and-play pipeline operates zero-shot across indoor RGB-D and outdoor LiDAR settings, including successful operation on real LiDAR data.

2. Related Work

Prior registration work spans handcrafted descriptors, learned geometric features, multimodal RGB-D models, and generative approaches. C-GenReg is positioned as a zero-shot framework that combines generated imagery, geometric features, and probabilistic fusion.

  • Hand-crafted Registration Methods: Handcrafted descriptors such as FPFH and SHOT encode neighborhood geometry and are typically matched with RANSAC and refined using ICP.
  • Learning-based Registration Methods: Learned geometric methods replace handcrafted descriptors with data-driven features for dense correspondence and improved robustness under challenging overlap conditions.
  • RGB-D Registration Methods: Learnable RGB-D approaches jointly use color and depth through architectures or consistency objectives, but require real RGB inputs and task-specific training.
  • C-GenReg: C-GenReg instead generates multi-view-consistent RGB images from point-cloud depth sequences, extracts correspondence features, and fuses them with geometric descriptors.
  • Generative Based Registration Methods: Generative registration methods use diffusion priors and structural conditioning to bridge geometry and image modalities, but prior formulations rely on real RGB observations or different registration settings.

3. Method

C-GenReg estimates rigid point-cloud alignment by combining a geometry-to-image correspondence branch with a raw geometric branch. Its frozen, probabilistic fusion integrates independently derived correspondence posteriors before transformation estimation.

  • Registration formulation: Point cloud registration estimates a rigid transformation aligning source and target scans, with reliable correspondences as the central practical challenge.Standard pipelines typically extract features, match points, and estimate the pose robustly.
  • Geometry-to-image branch: C-GenReg renders depth maps into multi-view-consistent RGB images, extracts dense matching features, and lifts the resulting image features back to 3D.The original depth maps provide the correspondence between image pixels and 3D points.
  • Match-then-Fuse fusion: Each modality converts feature similarities into correspondence posteriors, which are fused into a unified probability map used to estimate the final rigid transformation.The image branch evaluates multiple view pairs, while the fusion stage preserves independently computed modality evidence.
  • Input construction: C-GenReg uses temporal depth sequences for indoor data and virtual-camera projections for LiDAR point clouds to provide the WFM depth-video input.Source and target sequences are concatenated temporally before generation, while prompts provide correlated-sequence and scene guidance.
  • Geometric branch: The parallel geometric branch processes the raw point clouds with a frozen registration-oriented 3D feature extractor to produce complementary per-point descriptors.GeoTransformer is selected as the final geometric backbone after ablation comparisons.
  • Match-then-Fuse fusion: The selected Noisy-AND fusion favors matches jointly supported by the image and geometric branches, increasing confidence through mutual agreement.Ablations compare Noisy-AND and Noisy-OR, with Noisy-AND chosen for the final design.

4. Experiments

C-GenReg is evaluated across indoor RGB-D and outdoor LiDAR benchmarks, including standard, cross-dataset, and ablation settings. It generally achieves strong registration performance and benefits from task-specific visual features and probabilistic fusion.

  • 3DMatch Benchmark: 3DMatch results show C-GenReg achieves the best overall performance across most rotation and translation metrics against handcrafted and learning-based baselines.It nearly halves mean RTE compared with GeoTransformer while retaining advantages across most metrics despite isolated baseline strengths.
  • 3DMatch Benchmark: C-GenReg achieves comparable results to RGB-D registration methods using real images while operating solely on 3D point-cloud inputs.It outperforms PointMBF and is compared with an oracle variant using real RGB as an empirical upper bound.
  • ScanNet Benchmarks: On unseen ScanNet data, C-GenReg ranks first or second on most metrics across the Hard and SuperGlue splits.GPCR slightly leads on median RRE and mean RTE in the Hard split, while RoITr has a marginally lower median RRE on SuperGlue.
  • Waymo Benchmark: On Waymo outdoor LiDAR, C-GenReg substantially outperforms methods trained on KITTI in rotation and translation accuracy.Each LiDAR scan is projected through a virtual camera into a depth image before applying the same registration pipeline.
  • Ablation Studies: Task-specific VFMs outperform a general-purpose VFM, achieving roughly 2× lower mean RTE and up to 3× lower mean RRE in the image-only ablation.MASt3R is selected because its denser feature outputs integrate more effectively with probabilistic fusion.
  • Ablation Studies: The generated-RGB branch improves FCGF, Predator, and GeoTransformer baselines, while Match-then-Fuse outperforms feature concatenation across all geometric backbones.With GeoTransformer features, probabilistic fusion provides up to 5× improvement in mean RRE and RTE; Noisy-AND is selected for higher-precision matches.

5. Conclusions

C-GenReg combines world-scale generative priors, registration-oriented VFMs, and raw geometric features in a training-free registration framework. Its probabilistic fusion produces geometry-aware correspondences and supports zero-shot use across indoor and outdoor settings.

  • Framework: C-GenReg transfers point-cloud geometry into multi-view-consistent RGB views, extracts dense VFM correspondences, and lifts them back to 3D using depth maps.The framework then fuses generated-RGB and geometric correspondence posteriors through Match-then-Fuse without additional learning.
  • Framework: The framework is zero-shot and plug-and-play, with all modules operating without fine-tuning.Its two correspondence branches preserve complementary modality inductive biases and provide calibrated confidence.

A. Probabilistic Fusion Derivation

The appendix derives probabilistic fusion and details the geometry-to-image pipeline. It explains how depth sequences are formatted, views are selected, LiDAR is projected, and correspondence posteriors are combined.

  • Probabilistic Fusion: Noisy-AND fuses image and geometric correspondence posteriors under a conditional-independence assumption.The derivation proceeds through odds-form Bayes updates and yields the closed form p_ij = 1−(1−pimg_ij)(1−pgeo_ij).
  • Probabilistic Fusion: Noisy-OR models modality activations as independent given their respective similarity signals and represents support from at least one modality.The derivation uses the complementary event to obtain the disjunctive posterior.
  • WFM Input Formatting: Temporal concatenation of source and target depth sequences preserves the WFM’s pretrained multi-view consistency priors better than horizontal concatenation.Horizontal concatenation introduces an artificial spatial discontinuity, whereas temporal formatting remains closer to natural camera motion; safeguard frames near the midpoint are discarded.
  • View Selection: Selecting a small number of generated views limits quadratic pairwise feature evaluation while retaining sufficient viewpoint diversity.RRE and RTE saturate as K increases, indicating that K much smaller than L is sufficient.
  • LiDAR Conversion: For LiDAR, scans are projected onto an f-θ virtual camera to create depth images suitable for the WFM.The wide-FOV projection avoids the extreme nonlinear distortions of pinhole projection, and multiple overlapping virtual cameras can extend the approach to 360° LiDAR.

B.4. Fusion Method - Point Matching Performance

The point-matching ablation evaluates Noisy-AND and Noisy-OR on 3DMatch using precision–recall curves. Noisy-AND is favored because it emphasizes correspondences supported confidently by both modalities.

  • Point Matching: Noisy-AND emphasizes matches simultaneously confident in both modalities, whereas Noisy-OR admits a larger set of candidate correspondences.This precision-oriented behavior motivates using Noisy-AND for registration.
  • Point Matching: Noisy-AND consistently achieves higher precision than Noisy-OR across the entire recall range on 3DMatch.A correspondence is correct when matched points lie within 5cm under the ground-truth transformation.

C. Additional Implementation Details

C-GenReg uses frozen pretrained components for RGB generation, visual correspondence extraction, and geometric feature extraction across indoor and outdoor settings.

  • Cosmos-Transfer1-7B variants generate RGB outputs from depth at dataset-specific indoor and outdoor resolutions.Indoor inputs use 960×704 resolution, while outdoor inputs use 1280×640.
  • MASt3R extracts image descriptors from RGB inputs resized to 512×384.The VFM pathway uses an Encoder ViT-L, Decoder ViT-B, and descriptor dimension dimg=24.
  • GeoTransformer extracts geometric descriptors after voxelization at 2.5cm indoors and 5cm outdoors.The geometric descriptor dimension is dgeo=256, using official 3DMatch and KITTI checkpoints for indoor and outdoor scenes, respectively.
  • Cosmos, MASt3R, and GeoTransformer use publicly released pretrained weights and remain completely frozen.The implementation therefore does not fine-tune any component.

D. Runtime Analysis

C-GenReg’s runtime is dominated by World Foundation Model video generation, while correspondence extraction and geometric registration are lightweight; model distillation could substantially reduce this cost.

  • Noisy-AND consistently achieves higher precision than Noisy-OR at similar recall rates.The comparison uses precision–recall curves for point matches judged correct within 5cm under the ground-truth transformation.
  • 507s: World Foundation Model generation accounts for almost the entire runtime per registration problem.The VFM requires less than one second, while geometric matching and pose estimation take only a fraction of a second.
  • A reported 72× inference speedup from Cosmos distillation would reduce the pipeline runtime to approximately ∼7s.Lowering the video generation rate may provide additional speedups.

E.1. ScanNet Original Benchmark

On the original ScanNet benchmark, C-GenReg improves registration across all reported metrics, with especially strong gains in translation accuracy over geometric baselines.

  • C-GenReg achieves clear improvements across all metrics on the original ScanNet benchmark.The benchmark uses relatively easy pairs sampled 20 frames apart, producing modest source–target motion.
  • The strongest gains occur in translation accuracy compared with FCGF and GeoTransformer.The results support a consistent boost from augmenting geometric features with the RGB-generated branch.

E.2. Low-Overlap Benchmarks

C-GenReg remains effective when point-cloud overlap is below 30%, outperforming GeoTransformer on Lo3DMatch and low-overlap Waymo while using geometrically coherent generated views.

  • Low-Overlap Results: 14.57° versus 21.10°: C-GenReg reduces Lo3DMatch rotation error relative to GeoTransformer in the low-overlap setting.Translation error also decreases from 53.46cm to 45.49cm.
  • Low-Overlap Results: 4.95° versus 19.72°: C-GenReg reduces low-overlap Waymo rotation error relative to GeoTransformer.Waymo translation error decreases from 9.04m to 1.66m.
  • Low-Overlap Results: C-GenReg consistently outperforms GeoTransformer across the Lo3DMatch and low-overlap Waymo benchmarks.Performance still degrades compared with high-overlap cases because the scans share less geometric overlap.
  • Mechanism: Consistent WFM-generated views and probabilistic Match-then-Fuse correspondences support reliable matching despite sparse overlap.The generated RGB views preserve shared scene regions and geometric coherence across viewpoints.
  • Qualitative Results: Qualitative examples visualize generated RGB views, color-coded correspondences, and their lifted matches on 3D point clouds.Figures 9 and 10 report resulting rotation and translation errors for 3DMatch and Waymo examples.
  • Qualitative Results: Figures 11–13 show multi-view-consistent RGB generation from depth across 3DMatch, ScanNet, and Waymo.The examples preserve scene layout, structure, or geometry across source and target viewpoints.
Loading 2604.16680v1…