PaperlayerView exact revision on arXiv

Source-linked AI summary

Does VGGT Know What It Does Not Know?

Adityan Thirumani Srinivasan, David Nordström

arXiv:2610.35806v1cs.CVcs.LG

TL;DR

Feed-forward reconstruction models such as VGGT provide no uncertainty for their predicted camera poses, leaving the paper to ask whether the frozen model’s features reveal pose failure. The authors probe those features and find that they predict failure more accurately than VGGT’s own confidence on HardMatch, with transfer to rotation failures on other datasets.

  • Problem

    Feed-forward reconstruction models predict cameras and geometry in one pass but provide no uncertainty for their camera poses.

    Source: §1 · p. 1

  • Method

    The authors train linear probes on frozen VGGT activations across network blocks and evaluate them with category-grouped cross-validation against signals from VGGT’s outputs.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    93.5% ROC-AUC for a frozen-feature probe versus 79.7% for VGGT’s own confidence on HardMatch, with the signal also transferring to rotation failures on other datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The probe provides a signal for selective prediction and transfers without retraining to rotation failures on PhotoTourism and ScanNet-1500.

    Source: §1 · p. 2

    The evaluation covers VGGT and π3 on image pairs; with 16 PhotoTourism views, failures are rare and the probe is no longer clearly better than VGGT’s confidence.

    Source: §5 · p. 8

Abstract

from arXiv · show

Feed-forward 3D reconstruction models such as VGGT regress cameras and geometry in a single forward pass, but they report no uncertainty for the camera poses they predict: an incorrect relative pose looks no different from a correct one. We ask whether the frozen model nonetheless knows when its pose is wrong. On HardMatch, a benchmark of 1,000 deliberately difficult image pairs on which VGGT fails on 35% of pairs, a linear probe on VGGT's frozen features predicts pose failure on held-out categories with ROC-AUC 93.5%, compared with 79.7% for VGGT's own point-map confidence and 77.7% for a classifier trained on all of VGGT's outputs. The probe catches failures VGGT is confident about, lets a downstream system abstain on the pairs it distrusts, and transfers without retraining to rotation failures on PhotoTourism and ScanNet-1500, beyond what co-visibility explains. This knowledge is absent from the per-image encoder: it is computed partway through the layers that fuse the two views, and the same holds for a second model, $\pi^3$. In essence, we find that VGGT knows more about its own failures than it reports.

1. Introduction

Feed-forward reconstruction models predict cameras and geometry in one pass but provide no camera-pose uncertainty, even though VGGT often fails on difficult pairs. The paper tests whether frozen features reveal pose failure and reports that a linear probe outperforms VGGT’s outputs, including on failures the model is confident about.

Introduction

  • Feed-forward models predict cameras, depth, and point maps in one pass, but VGGT provides no camera-pose uncertainty for downstream systems to distinguish reliable poses from badly wrong ones.VGGT does provide per-pixel confidence for depth and point maps.Source: §1 · p. 1
  • On HardMatch, VGGT’s predicted geometry is grossly wrong on 35% of pairs, motivating the question of whether frozen features reveal when its pose is wrong.The benchmark uses deliberately difficult, hand-annotated image pairs.Source: §1 · pp. 1–2
  • A linear probe on frozen VGGT features reaches 93.5% ROC-AUC on HardMatch, versus 79.7% for VGGT’s own confidence, and the signal supports selective prediction and transfers to other datasets.The probe is evaluated with category-grouped cross-validation, leaving every test category unseen during training; it also detects failures VGGT is confident about.Source: §1 · p. 2

2. Related Work

Prior feed-forward reconstruction work predicts geometry and exposes geometric confidence, while this paper focuses on whether internal features indicate the correctness of predicted relative poses. It distinguishes pose failure from scene co-visibility and compares its frozen-feature probes with prior probing and failure-prediction approaches.

Related Work

  • VGGT predicts cameras, depth, point maps, and tracks in one pass, while related models extend feed-forward reconstruction and expose per-pixel geometric confidence but no camera-pose uncertainty.DUSt3R [25] regresses aligned point maps, and MASt3R adds dense matching.Source: §2 · p. 2
  • Prior uncertainty work evaluates or estimates uncertainty for per-point geometry, whereas this paper targets relative-pose correctness using internal features rather than outputs or input perturbations.Hillemann et al. [9] evaluate VGGT’s per-pixel confidence for depth and point errors; ReVar3R [16] perturbs inputs to estimate per-point uncertainty.Source: §2 · p. 2
  • Co-visibility describes a scene property, whereas failure describes a prediction; the probe predicts failure beyond ground-truth co-visibility on PhotoTourism.Related work includes probing VGGT layers for co-visibility [28] and studies of what 3D foundation models encode [27] [19].Source: §2 · p. 2
  • Linear probes assess what frozen features make available, while prior language-model work studies internal representations of answer correctness and failure-prediction methods typically train dedicated heads.This work keeps the model frozen and compares the probe with confidence it already emits.Source: §2 · p. 2

3. Method

The method evaluates frozen VGGT features and model outputs for pose failure on image pairs, using HardMatch’s correspondence-based protocol and direct pose errors on datasets with ground-truth poses. Category-grouped cross-validation tests whether probes generalize to unseen categories.

3. Method

  • The study processes image pairs with frozen VGGT and also evaluates π3, decoding VGGT’s predicted extrinsics and intrinsics and fitting a focal length for π3.VGGT receives two frames in one forward pass; the paper also tests up to 16 views in a later section.Source: §3 · p. 2

3.1. Defining failure

  • On HardMatch, failure is a median symmetric epipolar distance above 50 px across annotated correspondences, with pairs lacking at least eight correspondences or labelled dynamic excluded.The resulting set contains 903 pairs in 97 categories; PhotoTourism and ScanNet-1500 instead use ground-truth pose errors, including geodesic rotation error and translation-direction angle.Source: §3.1 · p. 2

3.2. Probing for failure

  • The probes use pooled frozen-network features from encoder, aggregator, and camera-head stages, with category-grouped 10-fold cross-validation and category-level bootstrap confidence intervals.For transfer, a probe trained on all HardMatch pairs is applied unchanged to the other datasets.Source: §3.2 · p. 3
  • The baselines include VGGT’s confidence, image sharpness, predicted relative rotation and focal-length ratio, plus classifiers trained on confidence statistics or all model outputs.The all-output classifier tests whether the probe merely re-reads information VGGT already outputs.Source: §3.2 · p. 3

4. Experiments

Experiments test whether frozen-model features reveal pose failures on HardMatch and transfer to other datasets and failure types. The results locate this signal in fused features and show that its usefulness depends on the failure mode.

4.1. Setup

  • HardMatch supplies the main training and analysis set, while PhotoTourism and ScanNet-1500 test transfer without retraining; probes use frozen VGGT and category-grouped cross-validation.After filtering, HardMatch contains 903 pairs in 97 categories; the probe reads the last global-attention block unless stated otherwise.Source: §4.1 · p. 3, §4.2 · p. 3

4.2. HardMatch: does VGGT encode its own failures?

  • The failure signal is weak in the per-image encoder, rises sharply in middle fusion layers, and is also linearly readable in π3, though its training-free cosine cue does not rise.VGGT’s probe rises from 69% after block 11 to 93% after block 15; π3’s reaches 92% by block 33.Source: §4.2 · pp. 4–5, Figure 3 · p. 5
  • 93.5% ROC-AUC for the last-block probe exceeds VGGT’s point-map confidence (79.7%) and an all-outputs classifier (77.7%) on HardMatch; VGGT fails on 34.9% of pairs.The probe reaches 93–94% across four late layers, while adding confidence statistics leaves its score unchanged at 93.5%.Source: §4.2 · pp. 3, 6, Table 2 · p. 5
  • Within VGGT-confidence quartiles, confidence ranks failures near chance (AUC 47%, 61%, 64%, 57%), while the probe reaches 90%, 91%, 92%, and 82%.On the most confident half, where 12.2% still fail, the probe reaches 90.0% [85.4, 94.2] versus 69.9% [61.2, 77.8].Source: §4.2 · p. 5, Figure 4 · p. 6
  • Keeping the 50% of pairs the probe trusts most yields a 3.8% failure rate and mAA@10 of 34.9%, versus 12.4% and 29.7% when ranked by confidence.Across all coverages, the risk–coverage area is 0.096 for the probe and 0.154 for confidence.Source: §4.2 · p. 6, Figure 6 · p. 6

4.3. PhotoTourism: transfer to rotation failures

  • On PhotoTourism, the unchanged HardMatch probe predicts rotation failures with AUC 94.6% at 10° and 97.5% at 15°, beyond what co-visibility alone explains.Among barely overlapping pairs, the probe reaches 86.9% and 89.5%, above confidence (79%) and co-visibility (74–76%).Source: §4.3 · p. 7, Table 5 · p. 7
  • With more views, rotation failures above 5° fall from 3.6% with two views to 0.3% with 16, leaving probe and confidence rankings difficult to distinguish.With 28–35 failures per setting, neither is clearly better; the authors note that many-view evaluation needs inputs on which failures remain denser.Source: Table 6 · p. 7, §4.3 · pp. 7–8

4.4. ScanNet-1500: failure modes differ

  • On ScanNet-1500, errors are mostly translation-direction failures at short baselines; the HardMatch probe transfers to rotation failures (79.5% AUC) but not translation failures (44–48%).A ScanNet-trained probe detects translation-dominated failures with AUC 90.5–91.0%; translation errors above 15° occur on 47% of pairs below 10 cm versus fewer than 1% above 40 cm.Source: §4.4 · p. 8, Table 7 · p. 8

5. Limitations

The study is limited to VGGT and π3, two-view image pairs, and failure labels whose definition varies by dataset. Its probe transfers to some failure modes, but not short-baseline translation errors, and filtering edges did not improve rotation averaging.

5. Limitations

  • A single probe does not cover short-baseline translation errors.VGGT's fused features encode these failures, but a probe trained on HardMatch's gross failures does not learn their signal.Source: §5 · p. 8
  • The analysis covers only VGGT and π3; larger or newer models may behave differently.The training-free cosine reading also differs between the two studied models.Source: §5 · p. 8
  • HardMatch failure labels use epipolar error from about 11 annotated correspondences per pair, mixing rotation, translation, and intrinsics errors.Ground-truth datasets confirm the result for rotation but contain few failures, including 22 PhotoTourism pairs above 15°.Source: §5 · p. 8
  • The study evaluates image pairs; with 16 PhotoTourism views, failures fall to 0.3% and the two-view probe is no longer clearly better than confidence.Filtering edges by the probe also did not improve rotation averaging.Source: §5 · p. 8
  • The probe requires failure labels, unlike VGGT's free confidence, so the study also compares classifiers trained on VGGT outputs and tests transfer without retraining.Source: §5 · p. 8

6. Conclusion

VGGT’s frozen features reveal pose failures more clearly than its reported outputs, with the signal emerging midway through VGGT and π3. A linear readout offers a practical reliability signal and transfers to rotation failures on other datasets.

6. Conclusion

  • A linear readout of frozen features predicts incorrect relative poses better than VGGT’s own confidence or other outputs, including when VGGT is confident.The signal emerges abruptly in the middle of the network in both VGGT and π3.Source: §6 · p. 8
  • The feature signal supports selective prediction and transfers to rotation failures on other datasets.Source: §6 · p. 8
  • Adding one linear layer to already-computed features provides a practical reliability signal for systems using feed-forward reconstruction.Source: §6 · p. 8
  • The findings indicate that self-knowledge about output correctness is also present in 3D foundation models.Source: §6 · p. 8

Related papers

Loading 2610.35806v1…