PaperlayerView exact revision on arXiv

Source-linked AI summary

Pumpire: Unified Benchmark for Metric Distance Estimation

Siyu Chen, Zehan Wang, Jiayang Xu, Yihan Wu, Jialei Wang, Junming Chen, Ziang Zhang, Yutong Ying, Zhou Zhao

arXiv:2610.12423v1cs.CV

TL;DR

Existing benchmarks do not directly assess metric distances between arbitrary 3D point pairs or unify comparisons across model settings. Pumpire builds a physically annotated real-world dataset and evaluates image- and video-level models; its analyses reveal accuracy–robustness trade-offs across model families and sensing conditions.

  • Problem

    Existing depth, camera-intrinsics, and point-cloud benchmarks do not directly evaluate metric distance estimation between arbitrary point pairs.

    Source: §1 · pp. 1–2

  • Method

    Pumpire combines a physically measured dataset with a protocol that back-projects predicted geometry and compares point-pair distances across image- and video-level estimation and completion models.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    Experiments across model configurations reveal distinct accuracy–robustness trade-offs across model families and sensing conditions, as well as effects of multi-view context and depth priors on distance accuracy.

    Source: §5.2 · p. 6, §7 · p. 10

  • Takeaways & Limitations

    Pumpire provides a unified way to compare metric point-pair distance estimation across image- and video-level model settings.

    Source: §1 · p. 2, §7 · p. 10

Abstract

from arXiv · show

We present Pumpire, a unified benchmark for evaluating metric point-pair distance estimation capability of both image- and video-level 3D foundation models, with or without depth priors. In contrast to previous approaches that normally evaluate depth and camera intrinsics separately or evaluate point-clouds with geometric similarity metrics, which cannot directly reflect models' point-to-point distance estimation capability, Pumpire directly assesses point-to-point distances from the reconstructed geometry. To this end, we collect a large-scale and diverse dataset (pumpire-6k) comprising 100 real-world scenes, each annotated with physically measured point-pair distances and containing 64 frames, for a total of 6,400 frames. Building on this dataset, we establish a holistic evaluation protocol that covers both image- and video-level 3D foundation models and enables direct assessment of point-pair distance errors and cross-setting comparison. We conduct extensive experiments across 29 baseline configurations of representative 3D foundation models and provide a comprehensive analysis of the results. By offering this benchmark, we target the more fundamental ability to perceive and estimate physical scale in the reconstructed 3D space, which prior evaluation protocols have largely overlooked. The project page can be found at https://pumpire.github.io/

1 INTRODUCTION

Pumpire addresses the lack of direct evaluation of metric distances between arbitrary 3D point pairs and the fragmentation of existing benchmarks across model settings. It introduces a unified benchmark and dataset, then evaluates representative 3D foundation models.

1 INTRODUCTION

  • Metric point-pair distance estimation matters in applications including autonomous driving, robotic manipulation, and vision-language spatial reasoning.The passage connects driving translation errors with collision rates and robotic grasp success with 3D spatial measurements.Source: §1 · p. 1
  • Existing depth and camera-intrinsics benchmarks, even when considered jointly, do not necessarily measure point-pair distance accuracy; point-cloud and object-level reasoning benchmarks also lack direct point-level evaluation.Point-cloud benchmarks emphasize geometric fidelity or point-wise reconstruction errors, while spatial reasoning benchmarks evaluate distances at coarse object-level granularity.Source: §1 · pp. 1–2
  • Existing benchmarks are fragmented across image- and video-level estimation and completion, limiting unified comparisons across these settings.Source: §1 · p. 2
  • Pumpire introduces a unified benchmark for directly evaluating point-pair distance estimation across image- and video-level estimation and completion settings.Source: §1 · p. 2
  • The benchmark uses 100 diverse real-world scenes and 6,400 frames with physically measured point-pair distances, and evaluates 29 baseline configurations of representative 3D foundation models.Source: §1 · p. 2

2 RELATED WORK

The related work reviews image- and video-level approaches to geometry estimation and geometry completion with depth priors. Their varied input configurations and inference strategies motivate unified evaluation of metric point-pair distance estimation.

2.1 GEOMETRY ESTIMATION

  • Image-level geometry estimation methods use diverse training data or camera geometry, while video-level approaches exploit geometric constraints across pairs or multiple views and may maintain a recurrent scene representation.Source: §2.1 · p. 3

2.2 GEOMETRY COMPLETION WITH DEPTH PRIORS

  • Geometry completion methods incorporate depth measurements as metric constraints, addressing incomplete observations and sensor noise through varied conditioning and test-time adaptation strategies.Approaches include image-level refinement and sparsity generalization, plus video-level depth-conditioned inference and test-time adaptation.Source: §2.2 · p. 3

3 PUMPIRE-6K

Pumpire-6k combines real-world scenes, physically measured point-pair distances, and frame-level annotations to support evaluation across varied surfaces and viewing conditions. Its collection design addresses sensor-derived label errors and samples diverse camera motion and viewpoints.

3 PUMPIRE-6K

  • 6,400 frames come from 100 real-world scenes—40 indoor and 60 outdoor—with 64 uniformly sampled frames from sequences of 200–300 frames per scene.Source: §3 · pp. 3–4
  • Each frame includes RGB images, aligned depth maps, camera intrinsics, sticker pixel coordinates, and the physically measured distance between sticker centers.Sticker coordinates are annotated through manual anchor-frame annotation, CoTracker3 propagation, and frame-by-frame verification.Source: §3 · p. 4
  • Physical measurements avoid propagating sensor-depth errors into ground truth and support coverage of challenging optical, textural, and geometric conditions.The paper identifies reflective or transparent surfaces, thin structures, object boundaries, and ordinary surfaces affected by sensor noise.Source: §3 · pp. 3–4, Table 1 · p. 4
  • Randomized camera trajectories vary distance, orientation, sticker position, and viewing angle; 10 mm-radius circular stickers aid localization across capture distances.Source: §3 · p. 4

4 PUMPIRE PROTOCOLS

Pumpire back-projects annotated point pairs into camera coordinates to estimate their 3D distance, then compares predictions with physically measured ground truth. It reports scene-averaged distance errors and within-sequence consistency, with results also separated by scene difficulty.

4 PUMPIRE PROTOCOLS

  • Predicted point-pair distances are computed by back-projecting the two annotated pixels into camera coordinates and measuring their Euclidean separation.Depth maps are used directly when available; otherwise, depth comes from the predicted point map’s z-coordinates.Source: §4 · p. 4
  • Ground-truth point-pair distances come from physical measurements, and back-projection uses the camera intrinsics associated with the prediction.Source: §4 · p. 4
  • The protocol reports Absolute Error, Relative Error, Logarithmic Error, and Threshold Accuracy, averaging scene-level values for each benchmark metric.Source: §4 · pp. 4–5
  • Figure 3’s challenging examples are drawn from the subset identified using an RSD threshold greater than 0.1.Source: Figure 3 · p. 5
  • Relative Standard Deviation measures normalized variation in predicted distances within a sequence to assess cross-view consistency.Source: §4 · p. 5

5 BENCHMARKING METRIC 3D FOUNDATION MODELS

Pumpire evaluates absolute-scale 3D models across image- and video-level estimation and completion, with separate normal and challenging scene subsets. Results show that category leaders and rankings vary by setting and scene difficulty.

5.1 SCENE DIFFICULTY SPLIT VIA CROSS-VIEW CONSISTENCY

  • The benchmark evaluates 29 absolute-scale configurations across four settings and separates scenes by sensor-depth consistency for robustness analysis.The split uses an RSDsensor threshold of 0.1, yielding 24 challenging and 76 normal scenes; predicted depth is back-projected with intrinsics associated with its camera space.Source: §5.1 · p. 5, §5.2 · p. 6

5.2 QUANTITATIVE RESULTS FOR 3D FOUNDATION MODELS ON OUR BENCHMARK

  • Image-level estimation leaders differ by scene difficulty: Metric3D v2 leads on normal scenes, while Depth Pro and MoGe-3 rank first and second on challenging scenes.Metric3D v2 drops substantially in average rank on the challenging subset.Source: §5.2 · p. 6, Table 3 · p. 6
  • Depth Anything 3 ranks first on both subsets for video-level estimation; pi3x rises from fourth to second on challenging scenes, while MUSt3R drops from third to fifth.Both pi3x and MapAnything also support depth-prior conditioning, with their completion results reported separately.Source: §5.2 · p. 6, Table 4 · p. 6
  • Lingbot-Depth ranks first on both subsets for image-level completion, while CDM variants rise and Omni-dc and LDCM fall on challenging scenes.CDM-L515 and CDM-D435 move from eighth and seventh on normal scenes to second and third on challenging scenes.Source: §5.2 · p. 6, Table 5 · p. 7
  • The three CAPA variants take the top three average ranks on both subsets, with CAPA-VGGT first overall.On challenging scenes, CAPA-MoGe2 has the lowest AE, RE, and RSD; CAPA-VGGT has the lowest LE and highest threshold accuracies.Source: §5.2 · p. 7, Table 6 · p. 7

6 COMPREHENSIVE INSIGHTS

The analysis tests why direct point-pair distance evaluation matters and compares model accuracy and robustness across sensing conditions and settings. It finds distinct trade-offs among sensor depth, estimation, completion, and depth-prior use.

Direct evaluation of point-pair distance estimation

  • Depth and intrinsics benchmark rankings do not reliably predict point-pair distance performance, and replacing predicted intrinsics with ground truth degrades most cases.MoGe-2 has the best combined depth–intrinsics rank but is outperformed by Unidepthv2 on Pumpire.Source: §6 · p. 7, Table 7 · p. 8

Sensor depth versus learned 3D foundation models

  • Raw RealSense depth is strong on normal scenes, but 16 of 18 configurations achieve lower RE on challenging scenes, where learned models surpass the sensor.Only five configurations approach or surpass sensor δ1.05 on normal scenes, and all five are completion models.Source: §6 · pp. 7–8, Figure 4 · p. 8

Distinct robustness patterns of estimation and completion

  • Completion models reach higher peak accuracy but degrade by 20–210× on challenging scenes, while estimation models degrade by 1–20× and keep errors more bounded.The completion models’ challenging-scene outliers inflate RE despite their high δ1.05; multi-view estimation models lie near the y = x line in the robustness comparison.Source: §6 · p. 8, Figure 4 · p. 8

Sparsely-sampled depth prior for image-level estimation models

  • More sampled depth points do not consistently improve image-level completion, because performance depends on handling both sparse constraints and sensor noise.Some models perform best at intermediate densities rather than with full depth maps.Source: §6 · p. 9, Figure 5 · p. 9

Offering models with different numbers of depth priors

  • Increasing the fraction of depth-prior views lowers RE for both MapAnything and pi3x, with a substantially larger improvement for MapAnything.The comparison indicates that models differ in their ability to integrate metric cues across views.Source: §6 · p. 9, Figure 5 · p. 9

Comparison across model settings

  • Video-level completion combining depth priors and multi-view context achieves the lowest RE and RSD overall; multi-view estimation improves consistency, while depth-conditioned completion improves accuracy and consistency.Metric3D v2 and Depth Anything 3 have RE of 0.106 and 0.109, respectively, while their RSD is 0.067 and 0.029; CAPA-VGGT performs best overall.Source: §6 · p. 9, Figure 5 · p. 9

7 CONCLUSION

Pumpire introduces a unified benchmark and real-world dataset for direct metric point-pair distance evaluation. Its analyses show that model performance and trade-offs vary with scene difficulty, model family, multi-view context, and depth priors.

Dataset

  • Pumpire-6k contains 100 real-world scenes and 6,400 frames annotated with physically measured point-pair distances.The scenes span indoor and outdoor environments, surface properties, and viewing conditions.Source: §7 · p. 10

Benchmark

  • The benchmark enables unified comparison of image- and video-level geometry estimation and completion, across 29 baseline configurations.The evaluation directly measures metric point-pair distance estimation.Source: §7 · p. 10

Findings

  • The analyses reveal accuracy–robustness trade-offs across model families and sensing conditions, and examine how multi-view context and depth priors affect distance accuracy and cross-view consistency.Source: §7 · p. 10

A PIXEL-COORDINATE ANNOTATION PIPELINE

The appendix describes the annotation pipeline and evaluation setup. It covers semi-automated sticker tracking, baseline counts, deterministic video sampling, and aligned comparisons across model settings.

A PIXEL-COORDINATE ANNOTATION PIPELINE

  • The annotation pipeline propagates manually clicked sticker centers with CoTracker3, then uses human inspection, new anchors after excessive drift, and pixel-level refinement.Sequential inspection uses temporal coherence between adjacent frames to identify localization errors and make local corrections.Source: A PIXEL-COORDINATE ANNOTATION PIPELINE · p. 15

B BASELINE CONFIGURATIONS

  • The benchmark lists 26 method entries across four settings, representing 29 configurations when sensor-specific CDM variants and CAPA backbones are counted separately.The evaluation includes only metric 3D foundation models that recover absolute-scale scene geometry.Source: B BASELINE CONFIGURATIONS · p. 15, Table 8 · p. 16

C VIDEO-LEVEL MODEL SAMPLING STRATEGY.

D ADDITIONAL DETAILS FOR COMPARISON ACROSS MODEL SETTINGS

  • Image- and video-level comparisons use shared multi-view samples and weights, giving all four models 24,320 frame occurrences over 76 normal scenes.Scene-level RE and RSD are computed from aligned predictions and averaged equally across scenes; no additional image-level inference is required.Source: D ADDITIONAL DETAILS FOR COMPARISON… · pp. 15–16

E DETAILED EXPLORATION RESULTS FOR DIRECT DISTANCE EVALUATION

Depth and intrinsics benchmark rankings do not reliably predict point-pair distance performance. Replacing predicted intrinsics with ground-truth intrinsics also degrades most evaluated model–metric pairs.

E DETAILED EXPLORATION RESULTS FOR DIRECT DISTANCE EVALUATION

  • Five image-level models were evaluated on depth metrics across NYUD, KITTI, ETH3D, and DIODE, and intrinsics AUC across LaMAR, MegaDepth, Stanford2D3D, and TartanAir.Table 10 presents depth benchmark results for image-level metric geometry estimation models.Source: E DETAILED EXPLORATION RESULTS FOR… · p. 16, Table 10 · p. 17
  • MoGe-2 ranks best overall across depth and intrinsics benchmarks but is consistently outperformed by Unidepthv2 on distance estimation.Unidepth ranks best for depth but fourth for intrinsics, while MoGe-2 shows the opposite task-specific pattern.Source: E DETAILED EXPLORATION RESULTS FOR… · p. 16
  • 47 of 56 model–metric pairs degraded when predicted intrinsics were replaced with ground-truth intrinsics.The analysis used the normal subset; Table 9 reports changes relative to normal-subset results in Table 3.Source: E DETAILED EXPLORATION RESULTS FOR… · p. 16, Table 9 · p. 17

F EFFECT OF MULTI-VIEW SAMPLING STRIDE

Wider frame-sampling strides generally benefit video-level geometry estimation, though reduced visual overlap can offset the gain. The best stride differs for some models.

F EFFECT OF MULTI-VIEW SAMPLING STRIDE

G ADDITIONAL QUALITATIVE RESULTS

The qualitative results illustrate depth–intrinsics coupling and the effects of different depth-prior inputs. They show model-specific robustness and improvement patterns across the prior settings.

G ADDITIONAL QUALITATIVE RESULTS

  • Table 11 reports intrinsics AUC@1/5/10° and vertical and horizontal FoV; methods trained on evaluated datasets are excluded from rankings.The qualitative examples complement aggregate results with depth–intrinsics coupling and depth-prior analyses.Source: G ADDITIONAL QUALITATIVE RESULTS · p. 17, Table 11 · p. 18
  • Figure 8 compares ground-truth annotations with back-projections using ground-truth versus predicted intrinsics and depth maps.The figure encodes annotations in red, ground-truth intrinsics in blue, and predicted intrinsics in yellow.Source: Figure 8 · p. 18
  • Lingbot-Depth and PriorDA remain robust across prior-depth patterns, while CDM-L515 drops substantially under a pattern domain gap.The gap is between CDM-L515’s training data and the evaluated prior patterns.Source: Figure 9 · p. 19
  • MapAnything shows continuous performance improvement as the fraction of views supplied with depth priors increases.Figure 10 compares varying prior-input ratios; its “Sensor Captured” distance averages sensor-measured distances from ten sampled views.Source: Figure 10 · p. 19
Loading 2610.12423v1…