Source-linked AI summary
Free Geometry: Refining 3D Reconstruction from Longer Versions of Itself
Yuhang Dai, Xingyi Yang
TL;DR
Feed-forward 3D reconstruction models remain fixed after training and cannot adapt to new scenes without costly 3D supervision. Free Geometry uses full-view features to supervise masked-view features through consistency and relational constraints, optimized with lightweight LoRA updates. Across four benchmarks, it consistently improves Depth Anything 3 and VGGT in pose and reconstruction quality with under two minutes of adaptation per dataset.
Problem
Fixed feed-forward models cannot adapt to test scenes, while collecting diverse high-quality 3D ground truth for retraining is prohibitively expensive.
Method
Free Geometry masks test-sequence frames and adapts lightweight LoRA parameters by matching partial-observation features to full-observation features while preserving cross-frame relations.
Results
3.73% average camera pose accuracy and 2.88% point map prediction improvements were achieved across four benchmarks for state-of-the-art foundation models.
Takeaways & Limitations
Free Geometry provides fast, label-free geometric recalibration for unseen scenes with minimal test-time training overhead and parameter footprint.
Abstract
from arXiv · showhide
Feed-forward 3D reconstruction models are efficient but rigid: once trained, they perform inference in a zero-shot manner and cannot adapt to the test scene. As a result, visually plausible reconstructions often contain errors, particularly under occlusions, specularities, and ambiguous cues. To address this, we introduce Free Geometry, a framework that enables feed-forward 3D reconstruction models to self-evolve at test time without any 3D ground truth. Our key insight is that, when the model receives more views, it produces more reliable and view-consistent reconstructions. Leveraging this property, given a testing sequence, we mask a subset of frames to construct a self-supervised task. Free Geometry enforces cross-view feature consistency between representations from full and partial observations, while maintaining the pairwise relations implied by the held-out frames. This self-supervision allows for fast recalibration via lightweight LoRA updates, taking less than 2 minutes per dataset on a single GPU. Our approach consistently improves state-of-the-art foundation models, including Depth Anything 3 and VGGT, across 4 benchmark datasets, yielding an average improvement of 3.73% in camera pose accuracy and 2.88% in point map prediction. Code is available at https://github.com/hiteacherIamhumble/Free-Geometry .
1 Introduction
Feed-forward reconstruction models are efficient but fixed after training, leaving test-scene geometry errors and no practical route for retraining without costly 3D supervision. Free Geometry uses more reliable full-view predictions to supervise masked-view features, enabling fast test-time adaptation with consistent benchmark gains.
- Fixed zero-shot models can produce plausible but geometrically erroneous reconstructions under occlusions, specularities, and ambiguous visual cues.
- Large-scale 3D ground-truth collection is prohibitively expensive, making direct retraining impractical.
- More views add geometric constraints and reduce ambiguity, so full-view predictions can supervise masked-view predictions.
- Free Geometry optimizes lightweight LoRA updates by matching full- and partial-observation encoder features while preserving relations implied by held-out frames.
- 3.73% average camera pose accuracy and 2.88% point map prediction improvements were achieved across benchmarks, with adaptation taking under 2 minutes per dataset on one GPU.
- The method is presented as a plug-and-play, label-free adaptation framework that consistently improves reconstruction quality and pose accuracy.
2 Related Work
Feed-forward reconstruction replaces iterative multi-view optimization with efficient learned prediction, while test-time adaptation methods provide ways to adjust models using unlabeled test data. Free Geometry combines feature consistency, relational structure, and parameter-efficient adaptation for multi-view reconstruction.
- COLMAP and MVSNet use iterative optimization or cost volumes and require known camera poses or substantial computation.
- DUSt3R, VGGT, and Depth Anything 3 directly predict 3D geometry in feed-forward architectures, with VGGT and Depth Anything 3 targeting efficient multi-view inference.
- Test-time adaptation methods such as TENT, TTT, TTT++, and MEMO update models using test-data signals including entropy, auxiliary tasks, or augmentation consistency.
- Relational knowledge distillation transfers structural relationships such as angles and distances in addition to per-sample feature information.
- Free Geometry combines multi-view-to-partial-view feature consistency with architecture-guided self-distillation and LoRA-based parameter-efficient adaptation.
3 Longer is Better as Free Supervision
Free Geometry treats full-observation features as stronger supervision for partial observations because additional views improve geometric constraints and reduce ambiguity. It recalibrates the partial branch through feature matching and cross-frame relational losses while updating only lightweight components.
- The adaptation objective adjusts a pretrained multi-view model to target geometry using a test sequence without ground-truth 3D annotations.
- Full observations provide more view-consistent and geometrically reliable representations than partial observations because cross-view attention aggregates more correspondences.
- Feature Consistency and Self-Supervised Distillation: The teacher processes all frames with a frozen backbone, while the student processes unmasked frames through the same backbone with trainable lightweight adapters.
- Feature Consistency and Self-Supervised Distillation: Student features on unmasked frames are matched to teacher features to distill full-observation representations into the partial-observation setting.
- Efficient Adaptation via LoRA: LoRA adapters are inserted into multi-view transformer blocks while the original backbone remains frozen, and learnable camera tokens are updated.
- Self-Supervised Geometric Recalibration: The intra-frame loss aligns corresponding feature magnitudes and directions, while the cross-frame loss preserves geometric relationships involving masked-frame anchors.
- Cross-frame Relational Loss: Cross-frame relational supervision penalizes changes in pairwise similarity distributions and virtual-triangle angles, transferring masked-view geometry to the partial branch.
4 Experiments
Free Geometry is evaluated on four diverse benchmarks using pose and reconstruction metrics against frozen Depth Anything 3 and VGGT baselines. It consistently improves performance, with especially clear gains in low-observation settings and difficult visual conditions.
- Experimental Setup: Four benchmarks cover indoor, outdoor, cluttered, repetitive-texture, reflective, and challenging-lighting scenes.The evaluation includes ETH3D, ScanNet++, 7Scenes, and HiROOM.
- Experimental Setup: Pose accuracy uses AUC metrics, while reconstructed point clouds are evaluated with F-score after alignment to ground truth.Views are randomly sampled with fixed seeds, and reconstruction uses predicted poses and depths.
- Baselines: The primary comparison is against frozen, pretrained Depth Anything 3 Giant and VGGT models without test-time adaptation.Free Geometry performs per-dataset test-time optimization relative to this zero-shot baseline.
- Quantitative Results: Free Geometry consistently improves both pose and geometry over frozen baselines across datasets and view counts.The comparison reports pose accuracy and reconstruction quality, with means averaged over three seeds.
- Quantitative Results: Low-observation scenes show the strongest gains, including VGGT ETH3D improvements from 0.157 to 0.178 AUC@3 and from 0.102 to 0.110 F1.DA3 on ETH3D improves from 0.286 to 0.305 AUC@3; gains are smaller on ScanNet++ where baselines already perform strongly.
3D Reconstructions Results.
Free Geometry improves aligned 3D reconstructions across view counts, with the largest benefits when observations are scarce. Qualitative error maps show fewer outliers and cleaner surfaces, especially in difficult regions.
- Qualitative Results: Free Geometry produces fewer error regions and cleaner surface structures than frozen baselines after alignment to ground truth.The qualitative comparison highlights scattered baseline outliers and locally distorted surfaces that are reduced after adaptation.
- 3D Reconstruction Results: Free Geometry improves reconstruction quality across 4, 8, 16, and 32 views, with consistent F1 gains and reduced Chamfer distance.Although optimized with an 8-view-to-4-view consistency signal, the gains transfer across tested view counts.
- 3D Reconstruction Results: The 4-view setting benefits most, while gains diminish at 32 views as additional observations provide stronger geometric constraints.Sparse observations increase reliance on learned geometric priors, whereas abundant views reduce ambiguity.
5 Ablation Studies and Analysis
Ablations show that feature consistency and cross-frame relational constraints provide complementary supervision. Feature-distance analysis further indicates that adaptation brings partial-observation features closer to full-observation features.
- Ablations on Loss Components: Removing either loss component degrades performance, while the full objective achieves the best pose accuracy and reconstruction quality.The two terms therefore act as complementary supervision signals in test-time optimization.
- Ablations on Loss Components: Dropping the relational term reduces F1 from 0.2475 to 0.2190 under sparse ETH3D views.This result identifies cross-view relational constraints as important for resolving geometric ambiguities.
- Feature Consistency Analysis: Adapted features have lower MSE and higher cosine similarity than frozen-baseline features at both evaluated encoder layers.Features from partial observations become measurably closer to full-observation features.
6 Conclusion
Free Geometry adapts feed-forward multi-view 3D reconstruction models on unseen scenes without 3D ground truth. Across four benchmarks, it improves pose and reconstruction quality with fast, lightweight test-time recalibration.
- Conclusion: Free Geometry uses full-observation predictions as supervision for masked observations and optimizes lightweight LoRA parameters at encoder feature level.This enables self-supervised geometric recalibration without 3D ground truth and with under two minutes of overhead per dataset on one GPU.
- Conclusion: Experiments across four benchmarks show consistent improvements for Depth Anything 3 and VGGT in both pose accuracy and reconstruction quality.Ablations support the importance of feature consistency and cross-frame relational constraints, while feature analysis verifies improved cross-view consistency.
1 Method Details
Free Geometry adapts a frozen feed-forward reconstruction model through two branches of the same scene: a full-view teacher and a partial-view student. Its geometric objective combines local feature matching with cross-frame relational constraints, optimized through lightweight test-time parameters.
- Test-Time Adaptation: The full branch provides detached teacher features, while the partial branch uses trainable LoRA modules and camera tokens to recover scene geometry.The remaining backbone parameters stay frozen during dataset-wise adaptation.
- Intra-frame Consistency Loss: The intra-frame loss aligns student and teacher tokens on shared unmasked frames using Huber distance and cosine similarity.It matches both patch tokens and the camera token.
- Cross-frame Relational Loss: The cross-frame loss preserves pairwise relation distributions and triangle geometry across corresponding token triplets.It uses KL-divergence terms and an L1 angular-geometry term to transfer constraints implied by masked views.
- Overall Objective: The overall objective sums intra-frame and cross-frame consistency losses to stabilize visible-frame adaptation and inject masked-frame structure.The formulation is L_geo = L_intra + L_cross.
- Adaptation Pipeline: The dataset-wise pipeline initializes teacher and student models from the pretrained network, freezes the teacher and backbone, and optimizes LoRA parameters over masked-view inputs.The default example uses 8 full views and 4 even-indexed unmasked views.
2 Experiment Setup
Experiments evaluate dataset-wise test-time adaptation on four processed benchmarks using the pretrained-model protocol, repeated frame sampling, pose metrics, and geometry metrics.
- Implementation: Free Geometry is implemented on a single RTX Pro 6000 GPU, optimizing LoRA parameters and trainable camera tokens while keeping the remaining model frozen.The LoRA configuration uses rank 32 and α = 32.
- Evaluation Protocol: The evaluation follows the Depth Anything 3 benchmark pipeline, selecting fixed frame budgets and retaining a maximum of 100 frames for longer scenes.The protocol evaluates camera pose and geometry reconstruction under a unified setting.
- Evaluation Protocol: Each frame-budget evaluation is repeated three times with random seeds 43, 44, and 45, and average performance is reported.Repeated sampling reduces dependence on a single frame subset, particularly in sparse-view settings.
- Metrics: Camera pose is measured with AUC@3 and AUC@30, covering strict angular precision and more tolerant pose correctness.Reporting both metrics captures fine-grained and coarse robustness.
- Metrics: Reconstruction is evaluated with point-set accuracy, completeness, Chamfer Distance, precision, recall, and threshold-based F1-score.The protocol uses predicted poses, pose-based alignment, and RANSAC-based robustness to outliers.
- Datasets: The study uses four processed benchmark datasets and excludes DTU because it already achieves strong Depth Anything 3 results and is object-oriented.The remaining datasets target novel indoor and outdoor test scenes.
3 Additional Analysis
Additional analyses examine anchor selection and adaptation capacity, showing that mixed anchor selection and moderate LoRA capacity provide the strongest supported configuration.
- Anchor Selection: The ablation compares top, random, and mixed masked-frame anchor selection strategies in the cross-frame relational loss.The analysis evaluates both pose and reconstruction outcomes.
- Anchor Selection: The mixed anchor strategy achieves the best reconstruction results by combining similar and contrasting masked patches.The authors report that top-only sampling misses relational structure, while random sampling is less informative.
- LoRA Rank: All tested LoRA variants outperform the frozen baseline on most metrics, with rank 32 improving pose accuracy and F1 over rank 8.Rank 32 achieves the best overall AUC@3, AUC@30, and F1 results.
- LoRA Rank: Increasing LoRA rank to 64 reduces pose and reconstruction quality rather than providing further gains.The analysis supports moderate adaptation capacity over a larger trainable parameter set.
4 More Reconstruction Results
Across view-count settings and foundation-model baselines, Free Geometry generally improves or matches pretrained performance, with qualitative gains in structural coherence and depth alignment.
- Quantitative Results: Free Geometry consistently improves or matches pretrained baselines across most view-count settings for both VGGT and Depth Anything 3.Gains are especially stable on ETH3D and HiRoom, while remaining competitive on ScanNet++ and 7-Scenes.
- Quantitative Results: The adaptation improves geometric quality without requiring 3D supervision across the evaluated settings.This is the reported overall trend of the quantitative comparisons.
- View-Count Transfer: Consistent improvements transfer from 8-view-to-4-view adaptation to evaluations with more input views.The adapted representation remains useful in denser-view inference settings.
- Qualitative Reconstruction: Qualitative point-based reconstructions show fewer broken fragments and better preservation of large structural surfaces, thin vertical regions, room layouts, and wall boundaries.Planar support surfaces are also reported as less scattered.
- Qualitative Depth: Multi-view depth predictions are better aligned around depth discontinuities and elongated structures, yielding more faithful geometry with reduced error relative to ground truth.Figures 2 qualitative error maps mark significant depth deviations in red and within-threshold agreement in gray.
- Qualitative Reconstruction: Figure 1 presents qualitative 3D reconstruction comparisons, while its error maps mark geometry deviations from ground truth in red and within-threshold regions in gray.The comparison is between baseline and Free Geometry reconstruction behavior.