PaperlayerView exact revision on arXiv

Source-linked AI summary

HiP3R: Hierarchical Pose Graphs for Long-Horizon 3D Reconstruction

Zunnan Xu, Shangchen Zhou, Yihang Luo, Chen Change Loy

arXiv:2610.31635v1cs.CV

TL;DR

Long-video reconstruction must reconcile overlapping local predictions, but chunk-level alignment cannot resolve conflicting frame poses and global frame-level optimization is costly. HiP3R first recovers chunk layout and scale, then refines a shared camera per frame; across benchmarks it improves trajectory accuracy with both chunk-based and streaming backends.

  • Problem

    Long videos produce overlapping local reconstructions with drift and conflicting estimates of shared frames, while chunk-level transforms cannot resolve those pose disagreements.

    Source: §1 · p. 1

  • Method

    HiP3R optimizes a chunk-level Sim(3) graph for global layout, then holds that layout fixed while refining a shared SE(3) camera per frame.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4

  • Results

    Across seven datasets, VGGT-Omega ATE falls 74%, from 18.555 to 4.906 m, and LingBot-MAP macro ATE falls 52%, from 8.288 to 4.017 m.

    Source: §4.1 · p. 6

  • Takeaways & Limitations

    Evaluations report improved overall trajectory accuracy and surface reconstruction with both VGGT-Omega and LingBot-MAP.

    Source: §5 · p. 9

Abstract

from arXiv · show

Feed-forward models accurately estimate camera motion and 3D geometry from short video windows. However, computational and memory costs prevent direct inference on sequences containing thousands of frames. Long videos must therefore be processed as overlapping chunks. This division produces independent local reconstructions with inconsistent coordinate frames and duplicate estimates of shared cameras. Achieving global consistency requires propagating constraints from adjacent overlaps and distant revisits across the sequence. This requirement naturally motivates pose graph optimization. However, the choice of graph level creates a trade-off. Chunk-level graphs efficiently resolve scale drift and close large loops. Yet, their rigid transformations cannot correct internal errors or conflicting frame predictions. Conversely, frame-level optimization offers local flexibility, but scaling it globally is computationally expensive and can break metric scale. HiP3R resolves this trade-off using a hierarchical pose graph. HiP3R first optimizes a robust chunk-level $\mathrm{Sim}(3)$ graph to establish the global layout and scale. It then holds this layout fixed and estimates one shared $\mathrm{SE}(3)$ camera per frame, turning overlapping window predictions into a single trajectory. Experiments across multiple benchmarks demonstrate the effectiveness of HiP3R in improving camera trajectories and 3D reconstruction. Our method applies to both chunk-based and streaming backends without additional training or fine-tuning. The codes are available at https://kkakkkka.github.io/HiP3R.

1 INTRODUCTION

Feed-forward models handle short clips well, but long-video reconstruction faces computational limits, drift, and inconsistent frame poses across overlapping chunks. HiP3R addresses these issues with a hierarchical pose graph that recovers global chunk layout before refining one shared camera per frame.

1 INTRODUCTION

  • Long-video chunking bounds GPU memory, but sequential alignment is computationally costly and compounds scale and orientation drift; applying distant loop constraints before alignment can destabilize optimization.Dense overlapping points make all-pixel alignment expensive, while loop constraints may cause divergence or poor local minima.Source: §1 · p. 1
  • Chunk-level similarity transforms preserve internal camera configurations up to scale, so they cannot resolve conflicting pose estimates for the same frame across overlapping windows.Choosing one prediction discards multi-view evidence and can cause velocity jumps; full bundle adjustment is expensive, while relative motion alone drifts from the global layout.Source: §1 · pp. 1–2
  • HiP3R first optimizes a chunk-level Sim(3) graph over overlaps and distant loop closures, then holds the recovered layout fixed while optimizing a shared-camera SE(3) graph per frame.Placement priors keep cameras on the recovered layout, while within-chunk motion factors reconcile overlapping-window predictions.Source: §1 · p. 2
  • The method is evaluated on seven public benchmarks, improves long-horizon trajectory accuracy over loop-enabled VGGT-Long under matched backends and windows, and also works with streaming backends.The authors also report that bounded point sampling reduces graph construction and optimization time with little accuracy change.Source: §1 · p. 2
  • HiP3R’s contributions include progressive loop closure and bounded point sampling for stable, scalable long-sequence optimization without retraining.Its hierarchical graph first recovers global chunk layout and scale, then estimates one shared SE(3) camera pose per frame.Source: §1 · p. 2

2 RELATED WORK

Related work spans feed-forward geometry models, streaming video systems, and multi-window fusion. HiP3R is positioned as post-processing that combines global loops with shared per-frame pose consistency.

Feed-forward 3D vision models

  • Feed-forward models predict dense pointmaps without per-scene numerical optimization, but quadratic attention restricts them to short windows and leaves long sequences as overlapping local reconstructions.The cited model families include DUSt3R, MASt3R, VGGT, and VGGT-Omega.Source: §2 · pp. 2–3

Multi-window fusion and Structure-from-Motion

  • HIP3R’s overview pairs frozen backends’ overlapping chunk predictions with a chunk-level Sim(3) graph, followed by one shared SE(3) camera per frame under the recovered layout.The stages respectively recover layout and scale, then estimate the per-frame camera trajectory.Source: Figure 2 · p. 3

Streaming and recurrent video geometry

  • Streaming and recurrent methods extend temporal context through persistent memory, causal processing, or structured geometric context, while HIP3R adds global loops and per-frame pose consistency without retraining.The passage notes that causal streaming still accumulates drift on long trajectories.Source: §2 · p. 3

Multi-window fusion and Structure-from-Motion

  • Chunk-based fusion methods align adjacent chunks with pairwise Sim(3), which preserves internal cameras up to scale and cannot resolve disagreements between overlapping-frame predictions.HIP3R instead uses a shared frame-level pose graph after chunk alignment, without optimizing 3D tracks.Source: §2 · p. 3

3 METHOD

HiP3R processes overlapping feed-forward reconstructions in two stages: first aligning chunks into a global layout, then refining one shared camera pose per frame while keeping that layout fixed.

3.1 PROBLEM FORMULATION

  • HiP3R defines a single SE(3) camera trajectory and consistent pointmaps from independently processed overlapping chunks.The backend predicts pointmaps, confidence maps, and camera-to-chunk poses; the two-stage optimization first aligns chunks and then refines shared frame poses with the layout fixed.Source: §3.1 · pp. 3–4

3.2 COARSE LAYOUT OPTIMIZATION WITH PROGRESSIVE LOOP SCHEDULING

  • The coarse stage optimizes a chunk-level Sim(3) graph, using overlap and loop constraints to recover global layout and scale.Confidence-weighted IRLS with bounded point sampling estimates overlap similarities; loop weights are bounded by registration residuals, and loop constraints are introduced progressively while adjacent edges remain active.Source: §3.2 · p. 4

3.3 SHARED-CAMERA TRAJECTORY REFINEMENT

  • The fine stage assigns one shared SE(3) variable to each physical frame and combines placement priors with within-chunk relative-motion constraints.Overlapping chunk predictions jointly constrain the shared pose; motion constraints preserve within-chunk relative poses, while placement priors keep the trajectory close to the recovered layout. For long sequences, HiP3R optimizes a keyframe subgraph and interpolates corrections for other frames.Source: §3.3 · pp. 4–5

4 EXPERIMENTS

Across seven public benchmarks, HIP3R improves trajectory accuracy, while controlled comparisons and ablations examine long-horizon reconstruction, dense geometry, and the roles of its optimization components.

4.1 BENCHMARK EVALUATION ACROSS PUBLIC DATASETS

  • On VGGT-Omega across seven datasets, HIP3R lowers macro-averaged ATE by 74% (18.555 to 4.906 m) and RPEtrans by 94% (0.9878 to 0.0622 m), while RPErot drops from 1.820° to 0.590°.Source: §4.1 · p. 6

4.2 CONTROLLED LONG-HORIZON RECONSTRUCTION

  • HIP3R achieves lower ATE than both VGGT-Long variants on all three long-horizon datasets, reducing KITTI and VBR errors to 9.730 m and 22.44 m.VGGT-Long with loops records 10.19 m on KITTI and 41.27 m on VBR; HIP3R uses a jointly optimized sparse layout graph with progressive loop scheduling.Source: §4.2 · p. 6, Table 2 · p. 7

4.3 END-TO-END POSE ACCURACY AND 3D GEOMETRY

  • With VGGT-Omega, HIP3R reduces Chamfer from 0.0759 to 0.0272 m and raises F1 from 65.92 to 91.26.Stage-wise results attribute most of the Chamfer reduction to coarse alignment, with shared-camera refinement adding gains on both backends.Source: §4.3 · p. 7, Table 3 · p. 8

4.4 QUALITATIVE DENSE 3D RECONSTRUCTION

  • On Oxford Bodleian Library and Keble College, coarse layout optimization aligns distant revisits and collapses offset surfaces onto single facades; refinement reconciles overlapping camera predictions.Across four long sequences with identical VGGT-Omega points, HIP3R recovers closed street networks with lower trajectory error.Source: §4.4 · p. 7, Figure 3 · p. 7, Figure 4 · p. 9

4.5 ABLATION STUDIES

  • Bounding sampled points cuts graph build and solve time by 83.62%, from 12232.18 to 2004.20 s, while ATE changes from 21.7751 to 21.7569 m.Reported total time includes measurement construction and layout solves.Source: §4.5 · p. 8, Table 4 · p. 8
  • Progressive loop scheduling reduces graph build and solve time from 2077.49 to 1477.59 s; the complete configuration reaches 17.4708 m ATE in 753.43 s.The complete configuration's final solve takes 41.28 s and improves both runtime and trajectory accuracy in this comparison.Source: §4.5 · p. 8, Table 4 · p. 8
  • Shared placement priors and motion factors reduce relative errors over Single and Unary on both backends, while Motion without placement priors raises ATE to 11.678 and 6.501 m.Placement priors preserve the recovered layout; motion factors improve local smoothness.Source: §4.5 · p. 8, Table 5 · p. 8
  • Layout Only reduces ATE by 87% on KITTI, 84% on Oxford, and 46% on VBR; Full further reduces RPErot by 24%, 30%, and 5%, and RPEtrans by 29%, 40%, and 22%, respectively.Full adds shared-camera refinement on the fixed coarse layout.Source: §4.5 · pp. 8–9, Table 6 · p. 9

5 CONCLUSION

HiP3R combines chunk-level layout recovery with shared-camera refinement to produce globally consistent long-video reconstructions. The supplementary material details implementation and experiments, including sparse Jacobian construction for the pose graphs.

5 CONCLUSION

  • The hierarchical graph first recovers global layout and scale, then refines one shared camera per frame with chunk transforms fixed.Distant loop closures address accumulated drift while shared-camera refinement reconciles conflicting estimates across windows.Source: §5 · p. 9

A APPENDIX

  • The supplementary material covers derivations, solver settings, dataset splits, runtime breakdowns, and extended tracking and geometry analyses.It also includes window diagnostics and solver ablations.Source: A APPENDIX · p. 12

B GRAPH COLORING AND CLOSED-FORM JACOBIAN BLOCKS

  • Sparse Jacobians exploit graph topology: coloring groups independent variables to reduce finite-difference residual sweeps, while closed-form SE(3) derivatives and automatic Sim(3) differentiation provide graph blocks.For a 735-frame scene, coloring reduced post-processing from 10,228 to 172 seconds; Jacobian construction scales linearly with edges.Source: B GRAPH COLORING AND CLOSED-FORM… · pp. 12–13

C LONG-SEQUENCE KEYFRAME SUBGRAPH

For long sequences, HiP3R optimizes a keyframe subgraph and interpolates corrections to recover dense per-frame poses. Solver settings adapt loop scheduling and restart behavior to the constraints and convergence state.

C LONG-SEQUENCE KEYFRAME SUBGRAPH

  • For sequences exceeding 2,000 frames, the fine stage optimizes an active keyframe subgraph and interpolates rotation and translation corrections between keyframes.Selection includes regular-stride targets, chunk boundaries, and inserted overlap frames when existing samples are insufficient.Source: C LONG-SEQUENCE KEYFRAME SUBGRAPH · pp. 13–14
  • On KITTI, dense solving improves RPErot and RPEtrans by 6% and 8%, respectively, with unchanged ATE; stride 5 yields comparable errors on Courthouse and downtown1.The default retains the keyframe subgraph on long sequences.Source: Table 9 · p. 14, C LONG-SEQUENCE KEYFRAME SUBGRAPH · p. 14

D HYPERPARAMETERS AND SOLVER SETTINGS

  • The solver expands loop stages when loop spans vary, and restarts from the latest estimate when scale bounds remain active.The fine stage holds recovered chunk transformations fixed; backend inputs may include poses and pointmaps or unprojected depth.Source: D HYPERPARAMETERS AND SOLVER SETTINGS · pp. 15–16

E DATASETS AND SPLITS

HiP3R is evaluated on seven public benchmarks spanning 61 sequences, from room-scale scenes to long driving and campus traversals. Pipeline timings identify pairwise point alignment as a major VGGT-Long cost, while HIP3R uses bounded sampling and a sparse solver.

E DATASETS AND SPLITS

  • The seven benchmarks cover 61 diverse sequences, including indoor rooms, high-precision multi-scale scenes, outdoor structures, and extended traversals.VBR sequences span 8,815–18,846 frames, while the KITTI odometry sequence contains 4,541 frames.Source: E DATASETS AND SPLITS · p. 16
  • Trajectory metrics are averaged within each dataset and equally across seven datasets; dense geometry metrics are averaged equally across 38 sequences.Additional keyframe and layout-solver experiments use three Droid-W downtown sequences.Source: E DATASETS AND SPLITS · p. 16

F REPORTED PIPELINE-STAGE TIMINGS

  • Pairwise 3D point alignment is a dominant computational bottleneck for VGGT-Long; HIP3R uses bounded sampling and a sparse graph solver to accelerate alignment.The timing comparison covers Oxford, KITTI, and VBR, with HIP3R component sums including inference and both graph solves.Source: F REPORTED PIPELINE-STAGE TIMINGS · p. 17, Table 12 · p. 17

G PER-DATASET COMPARISON

Per-dataset tracking results show relative rotation and translation accuracy gains across all seven datasets with VGGT-Omega, while ATE gains are largest on extended outdoor traversals. LingBot-MAP also improves macro-averaged ATE.

G PER-DATASET COMPARISON

H PER-DATASET DENSE 3D GEOMETRY

The dense-geometry evaluation reports sequence-macro reconstruction metrics over 38 sequences with dense 3D ground truth. Both evaluated backends show lower Chamfer distance and higher F1 for HiP3R than their baselines under dataset-macro averaging.

H PER-DATASET DENSE 3D GEOMETRY

  • VGGT-Omega baseline versus HIP3R scores are 0.0796 versus 0.0269 for Chamfer distance and 65.97 versus 92.06 for F1 score.The comparisons use unweighted dataset-macro averages across the three datasets.Source: H PER-DATASET DENSE 3D GEOMETRY · p. 18, Table 16 · p. 19
  • LingBot-MAP baseline versus HIP3R scores are 0.1439 versus 0.1014 for Chamfer distance and 61.34 versus 74.58 for F1 score.The comparisons use unweighted dataset-macro averages across the three datasets.Source: H PER-DATASET DENSE 3D GEOMETRY · p. 18
  • The dense-geometry evaluation covers 38 sequences: 11 from ETH3D, 9 from Neural RGB-D, and 18 from 7-Scenes.Table 3 reports sequence-macro averages to weight each physical evaluation scene equally.Source: H PER-DATASET DENSE 3D GEOMETRY · p. 18
  • Table 16 reports Chamfer distance and F1 score for VGGT-Omega versus HIP3R on datasets with dense 3D ground truth.F1 is one of the reported dense reconstruction metrics.Source: Table 16 · p. 19

I CHUNK AND OVERLAP EXPERIMENTS

The chunk-and-overlap experiments compare selected window sizes with fixed defaults and evaluate long-sequence overlap consensus. Selected sizes yield slightly lower macro-averaged ATE for VGGT-Omega and essentially unchanged ATE for LingBot-MAP.

I CHUNK AND OVERLAP EXPERIMENTS

  • The defaults are 60/30 frames for VGGT-Omega and 120/32 for LingBot-MAP, denoting chunk and overlap sizes.Each configuration generates chunk predictions and an optimized coarse layout held fixed during camera processing.Source: I CHUNK AND OVERLAP EXPERIMENTS · p. 19
  • For sequences longer than 2000 frames, overlap consensus averages rotations and translations at stride-10 frames, retains the final frame, and interpolates corrections to other frames.This replaces the main benchmark’s stride-5 keyframe optimization for long sequences.Source: I CHUNK AND OVERLAP EXPERIMENTS · p. 19
  • Selected versus default macro-averaged ATE is 4.403 versus 4.908 m for VGGT-Omega and 3.822 versus 3.825 m for LingBot-MAP.The averages span seven datasets.Source: I CHUNK AND OVERLAP EXPERIMENTS · p. 19
  • On KITTI, LingBot-MAP achieves 3.508 m with overlap consensus, compared with 6.063 m for Full in Table 13.The comparison is specific to KITTI.Source: I CHUNK AND OVERLAP EXPERIMENTS · p. 19

J ABLATION ON THE COARSE-ONLY LAYOUT

The controlled layout-solver comparison reports lower clean mean ATE for the complete solver than the baseline, while a coarse-only layout comparison finds lower rotational and translational RPE after shared-camera refinement.

J ABLATION ON THE COARSE-ONLY LAYOUT

  • On three long sequences, shared-camera refinement lowers RPErot and RPEtrans on all three while leaving ATE close to the coarse-stage result.Table 19 compares Layout Only with Full on the same feed-forward predictions.Source: J ABLATION ON THE COARSE-ONLY LAYOUT · p. 19

K CONTROLLED LAYOUT-SOLVER COMPARISON

  • The layout-solver study uses ten long sequences and holds model predictions, candidate revisit pairs, and initialization identical across configurations.Camera refinement and point corrections are disabled to isolate layout optimization; the full coarse solve allows at most 5000 function evaluations.Source: K CONTROLLED LAYOUT-SOLVER COMPARISON · p. 20
  • The five solver configurations test baseline pairwise alignment, bounded point sampling, residual-fit weighting, progressive loop scheduling, and the complete optimization solver.The final step combines adaptive continuation, Huber loss, and bounded restarts.Source: K CONTROLLED LAYOUT-SOLVER COMPARISON · p. 20
  • The complete layout solver reduces clean mean ATE by 19.77% relative to the baseline.The improvement concentrates in VBR, where mean error falls from 28.6017 to 22.4461 m; peak host memory remains comparable across configurations.Source: K CONTROLLED LAYOUT-SOLVER COMPARISON · p. 20
  • Synthetic loop noise perturbs 25% of detected edges with 10° rotation error, 25% translation offset, and 0.10 log-scale distortion.The evaluation reports efficiency gains from bounded sampling and accuracy and runtime gains from the complete solver.Source: K CONTROLLED LAYOUT-SOLVER COMPARISON · p. 20

L PLACEMENT AND MOTION ABLATION SETTINGS

The placement-and-motion ablation defines controls for resolving multiple frame predictions under identical optimized coarse layouts. Across dense-geometry benchmarks, shared-camera refinement adds Chamfer and F1 gains on both backends.

L PLACEMENT AND MOTION ABLATION SETTINGS

  • Layout Only selects the first coarse-placed pose in chunk order, while Single selects the occurrence farthest from either chunk boundary.Unary robustly averages all coarse-placed occurrences using deterministic Huber IRLS; Motion optimizes within-chunk relative-motion factors without placement priors; Shared jointly optimizes placement priors and relative-motion constraints.Source: L PLACEMENT AND MOTION ABLATION SETTINGS · p. 20
  • Table 19 compares trajectory accuracy for Layout Only versus Full on the same feed-forward predictions.The configurations are described as sharing identical evaluation protocols and macro-averaging across seven datasets.Source: L PLACEMENT AND MOTION ABLATION SETTINGS · p. 20, Table 19 · p. 21
  • Table 20 ablates layout-solver components on extended sequences, with total time including measurement construction and all layout solves.Its final-solve time is the duration of the final least-squares call.Source: Table 20 · p. 21

M DENSE GEOMETRY ACROSS OPTIMIZATION STAGES

  • Shared-camera refinement produces additional gains in Chamfer and F1 on both backends while preserving predicted depth and chunk scale.Full adds shared-camera corrections to pointmaps on the same coarse layout; Native performs sequential overlap stitching, while Layout Only uses coarse placements without frame-level corrections.Source: M DENSE GEOMETRY ACROSS OPTIMIZATION… · p. 21, Table 21 · p. 21

Related papers

Loading 2610.31635v1…