PaperlayerView exact revision on arXiv

Source-linked AI summary

A Calibration Audit of Confidence in Feed-Forward 3D Reconstruction Models

Nanxing Nick Deng, Qing Cheng, Niclas Zeller, Daniel Cremers

arXiv:2608.29705v3cs.CVcs.AI

TL;DR

Feed-forward reconstruction systems use native per-pixel confidence as an uncertainty signal, but its uncertainty magnitude has not been measured against prediction errors. This paper audits seven backbones across 13 datasets and tests post-hoc calibration; confidence ranks errors well, yet its scale is overconfident and remains tied to training conditions.

  • Problem

    Whether native confidence trained as a loss weight represents the magnitude of reconstruction error is unmeasured, despite downstream use for filtering and weighting predictions.

    Source: §1 · p. 2

  • Method

    The authors audit seven backbones on 13 datasets across ranking, level, slope, and coverage, interpreting confidence as uncertainty through the training-loss optimum and comparing post-hoc corrections.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Across seven backbones, confidence ranks error well but uncertainty level is off by a median factor of 2.4; a power-law correction reaches dataset-level targets, and held-out-dataset fitting brings the median factor to 1.35.

    Source: §1 · p. 2, §6 · p. 7, §7 · p. 8

  • Takeaways & Limitations

    Native confidence remains useful for ordering errors within images, but its absolute uncertainty scale is reliable only under the training conditions.

    Source: §7 · p. 8

    No correction scored without labels reaches the approximately 87% ceiling for held-out-scene coverage, while the successful scene-fitted correction requires that scene’s labels.

    Source: §1 · p. 2, §6 · p. 8

Abstract

from arXiv · show

Feed-forward 3D reconstruction models output a per-pixel confidence that is used by downstream systems as an uncertainty signal. The confidence is trained to serve as a weight in the training loss of models. Whether the confidence can be used as an uncertainty magnitude has not been measured. We audit seven backbones on 13 datasets and score the confidence on four properties, i.e., ranking of error, ratio of error to uncertainty on average, slope of this ratio across the confidence range, and coverage of the implied error distribution. Although the confidence ranks error quite well, the uncertainty decoded from the confidence is too small compared to the actual error. The uncertainty has the right size only under the exact training conditions. The median case is off by at least 2.4x across all seven models, while the uncertainty is further off the more confident the model is. Our work shows that the overconfidence appears on unseen scenes even when the model reaches its loss's optimum. As a post-hoc repair we fit a power law on the confidence with two constants per backbone--dataset pair. The repair brings all four audited properties to target at the dataset level, while leaving ranking untouched. Fitted with the target dataset held out, the constants bring the median case from 2.4x off to 1.35x. The repair does not hold below the dataset level, where two-thirds of held-out scenes are still more than five points off in coverage. We attribute what the repair cannot reach to the model, which carries neither the scale of the error nor the shape of its distribution across predictions. We release the audit protocol, its results, and the fitted constants per backbone-dataset pair.

1. Introduction

The paper asks whether native per-pixel confidence can serve as an uncertainty magnitude, since downstream systems use it to filter and weight 3D predictions. Across seven backbones and 13 datasets, the audit finds overconfidence and tests post-hoc corrections.

1. Introduction

  • Downstream filtering and weighting rely on confidence as an uncertainty signal, but neither the fixed-threshold nor relative-weight assumption had been evaluated against model errors.Fixed thresholds presume the same error for a confidence value across predictions; MP-SfM uses converted confidence as a depth-prior standard deviation.Source: §1 · p. 2
  • The audit evaluates ranking, level, slope, and coverage of uncertainty decoded from native confidence, then compares post-hoc corrections without retraining.It keeps backbone weights frozen and uses the conversion implied by the training-loss optimum.Source: §1 · p. 2
  • The uncertainty level is off by a median factor of 2.4, despite confidence ranking errors well; overconfidence persists on held-out scenes even at the training-loss optimum.The authors attribute it to changes in the target, prediction normalization, or data between training and deployment.Source: §1 · p. 2
  • Only a power law with two constants per backbone–dataset pair corrects level and slope without changing ranking; the best correction reaches nominal coverage across a dataset.No compared correction brings individual scenes to nominal coverage, and the successful correction is fitted using each scene’s ground truth.Source: §1 · p. 2
  • With the target dataset held out, the fitted constants reduce the median level error to a factor of 1.35.The constants are fitted per backbone using the other twelve datasets.Source: §1 · p. 3

2. Related Work

Prior dense-uncertainty evaluations often emphasize ranking, while ranking metrics cannot establish whether uncertainty has the right scale. Related work also covers recalibration of predictive distributions and uncertainty in geometric systems.

Measures of dense uncertainty

  • Ranking metrics are invariant to multiplicative scale error, so even their best scores can coexist with uncertainty that understates error by a constant factor.Prior work on VGGT found overconfidence using a monotonic, scale-free proxy on DTU alone; this audit measures its size across seven backbones and 13 datasets.Source: §2 · p. 3

Replacing the head

  • Trust3R reports ranking improvements from a retrained uncertainty head, but those results leave the absolute certainty of that head unresolved.Its reported AURC and AUSE gains are measured on a refined pointmap and do not establish whether the head is on average too certain or uncertain.Source: §2 · p. 3

Post-hoc recalibration

  • Prior recalibration methods include scalar or partition-based corrections, but transfer beyond tuning data has not been measured for MP-SfM’s depth-prior rescaling.The paper leaves weights untouched and corrects confidence from released backbones; it reports no prior method for feed-forward pointmap reconstruction.Source: §2 · p. 3

3. Confidence Training

The audited models train per-pixel confidence as part of a confidence-weighted regression loss. The loss optimum makes confidence inversely related to the mean residual among pixels sharing that confidence, under the training conditions.

3. Confidence Training

  • The DUSt3R family predicts a 3D point and confidence at each pixel, and trains them with a confidence-weighted regression loss.The loss’s first term weights the residual by confidence, while its second term pushes confidence upward.Source: §3 · p. 3
  • The residual is the distance between predicted and target points, and the confidence head outputs x such that C = 1 + e^x, so C > 1.Source: §3 · p. 3
  • Normalization changes what errors the model learns: five backbones rescale predictions by their own pointmap factor, so they do not learn absolute scale.MASt3R and CUT3R use the ground-truth factor on metric-scale datasets, where wrong overall size is penalized.Source: §3 · p. 4
  • At the loss optimum, confidence is inversely proportional to the mean residual of pixels sharing that confidence, conditional on the training target, normalization, and data.Thus, the confidence-to-uncertainty interpretation is tied to those three training conditions.Source: §3 · p. 4

4. Audit Protocol

The protocol decodes native confidence as uncertainty using the training-loss optimum, then tests ranking, level, slope, and coverage over confidence bins and datasets. It evaluates pixel-level behavior while accounting for scene-level variation and checks coverage on held-out scenes.

Confidence to uncertainty

  • The implied uncertainty is σ̂ = α/C, which represents the mean training residual for pixels with the same confidence; target and normalization match training where known, while data conditions vary.The audit keeps model weights frozen and adds no head.Source: §4 · p. 4

Grid

  • The grid covers 168 backbone–dataset settings across 13 datasets, scored on valid pixels; each dataset uses eight-frame clips, with 2.786×10^10 pixels overall.Three datasets are close relatives, leaving 10 independent sources; one backbone cannot run CO3Dv2.Source: §4 · p. 4

Rank correlation

  • Ranking uses Spearman correlation between uncertainty and error, measured over pooled pixels and within images; the latter is the median across images and excludes floored confidences and out-of-range bins.Within-image scoring rules out confidence that tracks only image-to-image error scale.Source: §4 · p. 4

Calibration level

  • The calibration level is the delivered-error-to-predicted-uncertainty ratio in each confidence bin; scene-resampled intervals fail high above 1.10 or low below 0.90.A bin is reported for a backbone only when at least half its settings contain pixels in that bin.Source: §4 · p. 4

Slope and coverage

  • The slope fit separates a common level offset from how error changes across bins, while coverage tests whether uncertainty-scaled intervals contain errors at nominal rates.Calibrated slope parameters are c = 0 and b = 1; coverage is tested at 11 levels on held-out scenes using both a fitted Gamma distribution and an empirical within-bin distribution.Source: §4 · pp. 4–5

5. Calibration Audit

Across seven backbones, confidence generally ranks error, but its implied uncertainty is too small, worsens at higher confidence, and yields undercoverage. The audit examines candidate explanations and alternative uncertainty designs, finding that neither training convergence nor ground-truth quality accounts for the main overconfidence.

5.1. Results

  • Within-image median ρ is 0.43 and 166 of 168 settings pass; pooled median ρ is 0.46 and 161 pass, while seven pooled failures pass within images.The seven settings lie on CO3Dv2 or BlendedMVS; in the worst setting, ρ is negative in 44% of images.Source: §5.1 · p. 5
  • The median setting is overconfident by 2.4× across the grid; all 1,224 failing bins among five ground-truth-trained backbones are overconfident.Intervals are too narrow at all 11 nominal levels, and at nominal 10% no backbone delivers more than 3%.Source: §5.1 · p. 5
  • Ranking holds on every backbone, but level, slope, and coverage fail on every backbone read against ground truth; the level worsens toward the most confident bins.Table 1 reports medians over datasets.Source: Table 1 · p. 5
  • The slope b is below one in 123 of 168 settings, indicating that overconfidence grows with confidence; the slope remains below one when measured within single frames.For b < 1, predicted uncertainty varies more across bins than delivered error does.Source: §5.1 · p. 5

5.2. Candidate explanations

  • Content splits by indoor/outdoor, static/dynamic, or real/synthetic shift level by at most 1.6×, whereas training normalization shifts CUT3R from 3.0× to 15.6× and MASt3R from 12.6× to 31.8×.Neither model is calibrated on either normalization branch.Source: §5.2 · p. 5
  • A DUSt3R confidence head and the released model both reach calibration on training frames yet remain overconfident on held-out frames; the level reaches 1.10× after roughly 300 updates.The runs used datasets from DUSt3R’s training diet, and an entirely new domain was not tested.Source: §5.2 · p. 6
  • Synthetic datasets with exact rendered ground truth remain overconfident, while conservative measurement noise accounts for at most 12.7% of failing bins.The figure shows uncertainty following error structure but sitting below error on 84–97% of scored pixels; post-hoc correction removes most of the gap.Source: §5.2 · p. 6, Figure 2 · p. 6

5.3. Other formulations

  • Marigold’s sampled-prediction uncertainty is overconfident on all 13 datasets, while UniDepth v2’s directly regressed error is underconfident by a factor of 3 to 34 on metric-ground-truth datasets.Marigold’s nominal 90% interval covers 17–41% of pixels; only 10 datasets permit UniDepth v2 scoring in metres.Source: §5.3 · p. 6

6. Post-hoc Recalibration

The section compares post-hoc corrections and finds that a two-constant power law repairs level and slope without changing ranking. Its dataset-level gains do not transfer fully to individual scenes, where coverage remains label-dependent.

6. Post-hoc Recalibration

  • Only the power law repairs level and slope while leaving ranking untouched; per-bin fits can disrupt pixel order.The table reports medians over 168 settings, fitted and scored on disjoint scene halves.Source: §6 · pp. 6–7, Table 2 · p. 7
  • The power-law factor varies with confidence; its two constants set level and slope, and an exponent clamp of 0.05 preserves ranking.The correction uses the audited slope as its exponent; the clamp leaves ranking unchanged on all 168 settings.Source: §6 · p. 7
  • 1.35× is the grid-median level ratio using per-backbone constants with the target dataset held out; 74% of settings move closer to one.The constants are also released for each backbone–dataset pair.Source: §6 · p. 7, Table 3 · p. 8
  • Both corrections reach nominal 90% coverage over a dataset, but only a median ≈33% of held-out scenes per setting fall in band.No correction scored on held-out scenes of the same dataset raises that fraction, including isotonic recalibration.Source: §6 · p. 7
  • Scene-specific labels raise the in-band fraction to ≈87%, while scene-fitted quantiles put only 41% of clips and 26% of frames in band.Without labels for the scene at hand, no scored correction reaches the ≈87% ceiling.Source: §6 · p. 8
  • Confidence-bin refinements do not beat uniform calibration-frame weighting; a frozen-feature correction brings slope 94% of the way to one on held-out scenes without improving ranking.The refinements improve 15–36% of settings while degrading a comparable share.Source: §6 · p. 8

7. Conclusion

Across seven released models and 13 datasets, native confidence orders error within images but is overconfident, especially at higher confidence. A two-constant power-law correction preserves ranking, but cannot capture scene-to-scene error variation without scene labels.

7. Conclusion

  • Across seven backbones and 13 datasets, confidence orders error within images, but models are overconfident wherever ground truth permits comparison, most at higher confidence.Training longer does not fix the overconfidence.Source: §7 · p. 8
  • A confidence head reaches the training-loss optimum and is calibrated on training frames yet overconfident on held-out frames.The optimum is defined against the training target, normalization, and data, each of which deployment can change.Source: §7 · p. 8
  • A two-constant power law repairs most overconfidence without changing ranking, but one pair cannot follow scene-to-scene variation in error spread.The authors could not predict a scene’s error spread without its labels and leave that question open.Source: §7 · p. 8
Loading 2608.29705v3…