PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Geometry: GT-Free and Plug-and-Play Test-Time Adaptation for Geometrically Consistent 3D Vision Foundation Models

Seokhyun Youn, Dahyeon Kye, Sung-Ho Bae, Jihyong Oh

arXiv:2608.10708v1cs.CV

TL;DR

Pretrained VFMs can be geometrically inconsistent because explicit multi-view consistency is too costly to enforce during pretraining, while prior implicit test-time signals provide limited gains. Self-Geometry uses correspondence-based explicit geometric constraints with GDO, FAN, and LoRA-based Lightweight TTA, consistently improving pose and geometry across six VFMs and four benchmarks, though matcher failures and non-real-time latency remain limitations.

  • Problem

    Explicit multi-view geometric consistency is omitted from VFM pretraining because it is computationally costly, while implicit test-time self-consistency offers limited improvement, especially on inaccurate scenes.

  • Method

    Self-Geometry performs GT-free, plug-and-play test-time adaptation using 2D pixel correspondences as pseudo-ground truth, GDO, FAN, and LoRA-based Lightweight TTA.

  • Results

    Self-Geometry consistently improves pose and geometry estimation across six VFMs and four benchmarks, including Mean pose AUC@3 +8.3% and geometry w/o p. +5.1% on π3.

  • Takeaways & Limitations

    The pipeline provides per-scene explicit geometric adaptation without ground-truth annotations or an additional training dataset.

  • Takeaways & Limitations

    Adaptation quality degrades when the external feature matcher struggles, and the few-minutes latency remains far from real-time requirements.

Abstract

from arXiv · show

Recent Vision Foundation Models (VFMs) predict depth, camera pose, and pointmap in a single forward pass without per-scene optimization, achieving strong generalization. However, enforcing explicit multi-view geometric consistency, e.g., through bundle adjustment, is computationally costly and is thus not imposed during VFM pretraining, so such inconsistency can arise. To address this, implicit self-consistency derived from model outputs (e.g., pointmaps, features), though enforced at test-time in prior work, delivers inherently limited performance gain, especially on scenes where the pretrained VFM is highly inaccurate. In contrast to this implicit signal, we propose Self-Geometry, a plug-and-play test-time adaptation pipeline that directly imposes explicit multi-view geometric constraints using 2D pixel correspondences as pseudo ground-truth. Our proposed Self-Geometry consists of Geometric Disentanglement Optimization, which combines Multi-View Consistency and Epipolar Consistency losses with Gradient Disentanglement to prevent gradient conflict; Frame Angular-Neighbor, a view sampler based on SO(3) geodesic distances for lightly imposing these constraints; and Lightweight TTA, which adapts VFMs via LoRA. Our method achieves consistent improvements in both pose and geometry estimation across six VFMs (VGGT, $π^3$, DA3-Giant/Large/Base/Small) and four benchmarks (7Scenes, ETH3D, ScanNet++, HiRoom).

I. Introduction

Recent VFMs provide efficient, strong zero-shot multi-view predictions, but omit explicit geometric consistency during pretraining. Self-Geometry addresses this gap with GT-free, plug-and-play test-time adaptation using explicit geometric supervision from 2D correspondences.

  • Recent VFMs predict depths, camera poses, and pointmaps in one forward pass with strong zero-shot performance across multiple geometry benchmarks.
  • Explicit multi-view consistency is omitted during VFM pretraining because enforcing it, such as through bundle adjustment, is prohibitively costly.
  • Implicit self-consistency methods barely improve pose and depth, yielding only marginal pointmap gains where pretrained predictions are highly inaccurate.
  • Self-Geometry uses test-time 2D pixel correspondences as pseudo-ground truth to impose explicit multi-view geometric constraints without GT annotations.
  • GDO combines MVC and EC losses with gradient disentanglement, while FAN and LoRA-based Lightweight TTA make adaptation efficient and plug-and-play.
  • Self-Geometry consistently improves pose and geometry estimation across six VFMs and four datasets, completing scene-wise adaptation within two minutes per scene.

C. Test-Time Adaptation

Self-Geometry addresses geometric inconsistency in pretrained VFMs by using 2D correspondences to impose explicit multi-view constraints during plug-and-play test-time adaptation. Its GDO combines complementary losses, filters mismatches, and adapts predictions without ground-truth annotations or teacher distillation.

  • Pretrained VFMs may produce geometrically inconsistent pointmaps because explicit multi-view consistency is not enforced during pretraining.
  • 2D pixel correspondences provide explicit multi-view geometric supervision for pretrained VFM predictions without auxiliary priors or teacher distillation.
  • Self-Geometry combines GDO, FAN, and Lightweight TTA into a GT-free, plug-and-play pipeline that adapts scenes on a single NVIDIA RTX PRO 6000 within two minutes.
  • GDO extracts and filters pseudo-correspondences, then applies point-to-point MVC and point-to-line EC losses to constrain predicted poses and depths.Correspondences are extracted once with LightGlue and reused during subsequent adaptation iterations.
  • MVC jointly supervises poses and depths but retains pose-depth ambiguity, whereas depth-independent EC supervises camera poses through epipolar constraints.
  • Gradient Disentanglement projects the MVC gradient away from the EC-gradient direction, preserving EC while preventing conflict between the two losses.

C. Frame Angular-Neighbor (FAN)

Frame Angular-Neighbor sampling uses rotation-only SO(3) geodesic distances to select views with broad and uniform angular coverage. GRV first maximizes active-bin coverage, then selects the candidate with the highest angular entropy, while ANS samples from each bin.

  • SO(3) geodesic distances provide a scene-scale-invariant basis for sampling views while avoiding the distinct units of rotation and translation in SE(3).
  • GRV partitions SO(3) geodesic distances into B bins and assigns each view to the interval containing its angular distance.
  • GRV first selects candidate target views maximizing the number of active SO(3) bins occupied by source views.
  • Among candidates, GRV selects the target with maximum bin entropy so source views are distributed uniformly across active angular regions.
  • ANS samples source views from every bin around the GRV-selected target, preventing angular concentration and maintaining uniform scene coverage at each adaptation iteration.

D. Lightweight Test-Time Adaptation (Lightweight TTA)

Lightweight TTA specializes a pretrained VFM to each target scene through parameter-efficient LoRA adaptation while preserving the pretrained geometric prior. It updates only adapter parameters and completes adaptation within two minutes on a single GPU.

  • Lightweight TTA adapts a pretrained VFM to the target scene without additional training data while preserving its pretrained geometric prior.
  • Combined with FAN, Lightweight TTA completes per-scene adaptation on a single GPU within two minutes.
  • The LoRA adapter is inserted only into attention-block QKV weights, with the pretrained VFM frozen and only LoRA parameters updated.

IV. Experiments

The experiments evaluate Self-Geometry across multiple VFM architectures, scales, datasets, and pose and geometry estimation settings, using established metrics and comparison baselines.

  • Evaluation Metrics: Pose estimation is measured with pairwise pose-error AUC@1, AUC@3, and AUC@30, averaged across benchmark scenes.The AUC thresholds are reported in degrees.
  • Evaluation Metrics: Geometry estimation reports F1-score in posed mode using ground-truth poses and unposed mode using predicted poses.The unposed setting evaluates predicted pose and depth, while the posed setting evaluates depth only.
  • Evaluation Setup: Evaluation covers VGGT, π3, and DA3 Giant, Large, Base, and Small variants across four benchmark datasets.The datasets are 7Scenes, ETH3D, ScanNet++, and HiRoom.
  • Baselines: Self-Geometry is compared with frozen pretrained VFMs, Free-Geometry, and TCO for each model-dataset pair.The comparison is designed to assess generalizability across VFM architectures and scales.
  • Implementation: Each scene is adapted with identical optimization settings using LoRA while pretrained parameters remain frozen.The total loss combines two primary losses with three self-supervised auxiliary regularizers, and optimization uses AdamW for 50 TTA iterations.

Training Details.

Across six pretrained VFMs, Self-Geometry improves pose and geometry estimation relative to competing adaptations and frozen baselines, while gains vary by model, dataset, and metric.

  • Overall Results: Self-Geometry outperforms Free-Geometry and TCO on most Mean columns and achieves the largest Mean improvements over frozen baselines in both tasks.The reported tasks are Pose Estimation and Geometry Estimation.
  • Overall Results: π3 with TCO degrades Mean pose AUC@3 by -91.5% and Mean geometry w/o p. by -60.0% across four datasets.The paper attributes this to divergence when TCO derives its pose prior from the pretrained VFM prediction.
  • Overall Results: π3 with Self-Geometry gains +8.3% in Mean pose AUC@3 and +5.1% in Mean geometry w/o p.These results are reported across the four evaluated datasets.
  • Pose Estimation: On ETH3D, Self-Geometry improves VGGT AUC@30 / AUC@3 by +9.2% / +37.3% and π3 by +5.0% / +25.1%.The reported result concerns wide-baseline outdoor scenes, and relative pose gains grow as the AUC threshold tightens.
  • Geometry Estimation: On HiRoom, DA3-Base and DA3-Small improve geometry w/o p. by +85.2% and +70.4%, respectively.The gains are reported on a synthetic benchmark with varied indoor illumination and abundant fine structures.
  • Limitations: VGGT+Ours shows subset-column degradations on ScanNet++ and HiRoom, including geometry w/o p. changes of -10.6% and -7.5%.The corresponding absolute drops are within 6 pp, while Mean pose improvements remain positive: AUC@3 Mean +3.3% and AUC@30 Mean +1.4%.

C. Qualitative Results

Self-Geometry improves qualitative depth and geometry reconstruction over pretrained VFMs and competing test-time methods, while ablations identify complementary losses, filtering, and view selection as important components.

  • Depth Estimation Results: Self-Geometry substantially reduces depth-error regions across VGGT, π3, and DA3-Giant compared with Original, TCO, and Free-Geometry.The qualitative reduction is consistent with F1-score gains reported in Tab. III.
  • Geometry Estimation Results: Self-Geometry produces cleaner fused pointclouds with fewer geometry-error regions than Original, TCO, and Free-Geometry.TCO shows especially severe scattered outliers on π3.
  • Ablations for GDO: 42.4% of TTA iterations exhibit gradient conflict between the MVC and EC losses, motivating Gradient Disentanglement in GDO.Combining both losses without Gradient Disentanglement reduces w/ p. from 0.50 to 0.45 relative to the EC-only variant.
  • Ablations for Pseudo-Correspondence Filtering: Sequential EC-then-MVC pseudo-correspondence filtering outperforms EC-only filtering on Precision and downstream metrics by trimming large-error mismatches.The two filters remove different mismatch types, so their combination is complementary.
  • Ablations for FAN: GRV target selection improves substantially over Fixed t = 0 on every reported column, whereas dynamically updating the target underperforms GRV.FAN remains robust across 15°, 30°, and 45° SO(3) bin widths and across source-view orderings.

A. Complexity Comparison

Self-Geometry is parameter-efficient and completes practical per-scene adaptation within a short budget across the evaluated VFMs and datasets.

  • Efficiency: LoRA adds only 0.7%–3.4% parameters, while Self-Geometry completes adaptation within two minutes for up to 40 ETH3D views on DA3-Giant.The method also remains within a practical scene-wise adaptation budget on 7Scenes, ScanNet++, and HiRoom.
  • Overall Comparison: Across six pretrained VFMs and four benchmarks, Self-Geometry consistently improves both pose and geometry estimation.The method is presented as a practical recipe for adding explicit multi-view consistency at test-time.

S.II. Preliminary: Epipolar Geometry

Epipolar geometry constrains corresponding pixels across two views through camera pose, with the fundamental matrix mapping source pixels to target-view epipolar lines. The Sampson distance measures violations of this constraint and is used in the Epipolar Consistency Loss.

  • Epipolar geometry defines the relation that corresponding pixels from two views must satisfy under their relative camera pose.The setup uses target and source images, camera intrinsics, relative rotation and translation, and homogeneous pixel coordinates.
  • The essential matrix Ei←j = [ti←j]×Ri←j expresses the epipolar relation in calibrated normalized coordinates.The fundamental matrix then expresses the same relation in pixel coordinates.
  • The fundamental matrix maps a source correspondence to an epipolar line in the target view, where the matching target pixel must lie.This yields the epipolar constraint used to assess geometric consistency between correspondences and relative pose.
  • The Sampson distance provides a residual for noisy correspondences and fundamental matrices by quantifying epipolar-constraint violation.It is used directly as the residual in the Epipolar Consistency Loss.

S.III. Total Loss Formulation

Self-Geometry optimizes a total loss combining explicit multi-view and epipolar consistency with auxiliary regularizers, robustification, dynamic weighting, and gradient disentanglement. The formulation preserves geometric supervision while limiting depth drift and outlier sensitivity.

  • The total loss combines MVC and EC geometric losses with photometric, edge-aware smoothness, and baseline-depth consistency regularizers.The auxiliary terms suppress excessive drift of pretrained depth predictions during test-time adaptation.
  • Photometric consistency compares the target image with a source image warped using predicted poses and depth, mixing SSIM and L1 with α = 0.85.Edge-aware smoothness encourages depth discontinuities at image edges and smoothness elsewhere.
  • Baseline depth consistency anchors adapted depth to the pretrained prediction on the confident-pixel set Vc, defined by the top q-quantile of baseline confidence.The default confident-region quantile is q = 0.5.
  • Huber robustification is applied to per-point residuals of MVC, EC, photometric, and baseline-depth losses, while edge-aware smoothness remains unmodified.The Huber threshold is initialized from baseline forward residuals for primary losses and re-estimated each iteration for baseline-depth consistency.
  • Gradient Disentanglement projects the MVC gradient onto the orthogonal complement of the EC gradient before combining losses.This preserves EC while removing only the EC-direction component from MVC, allowing pose-only and joint pose-depth refinement to remain complementary.

S.IV. Algorithm

The per-scene algorithm initializes correspondence-based supervision and robust thresholds, selects target views, and iteratively performs lightweight test-time adaptation. A shared hyperparameter configuration is used across all evaluated models and datasets.

  • Scene Initialization: Scene initialization extracts pseudo-correspondences with LightGlue, applies correspondence filtering, and estimates per-scene Huber thresholds.These quantities seed the subsequent adaptation loop.
  • TTA Loop: The procedure then selects target views before entering the iterative test-time adaptation loop.Each iteration computes primary and auxiliary losses and updates the learnable adaptation parameters.
  • Lightweight TTA: LoRA is inserted into every attention block’s QKV weights while the remaining pretrained VFM parameters stay frozen.The same hyperparameters are used across all six VFMs and four benchmark datasets without per-model or per-dataset tuning.

S.VI. Limitations

Self-Geometry’s main practical boundaries are dependence on external feature matching and non-real-time adaptation latency. Qualitative evaluations compare original models and competing adaptation methods across multiple benchmarks.

  • Dependence on External Feature Matcher: Adaptation quality degrades when LightGlue struggles with repetitive textures, textureless surfaces, or wide-baseline views with limited overlap.The external matcher supplies the 2D correspondence supervision, so weak matching weakens the geometric signal.
  • Adaptation Latency: Per-scene adaptation completes within a few minutes on one GPU but remains far from real-time requirements.The reported latency is therefore a practical boundary despite being limited to a few minutes.
  • Qualitative Comparisons: The qualitative figures compare Original, +TCO, +Free-Geometry, and +Self-Geometry across depth and geometry error maps.Rows represent six pretrained VFMs, with scenes selected for the largest F1 improvement for each model.
Loading 2608.10708v1…