PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast Pose Tracking of Rigid Objects with Compact Pose Graph Optimization

Xiaojie Zhang, Tom Fischer, Viktor Larsson, Eddy Ilg

arXiv:2610.11815v1cs.CV

TL;DR

Long-horizon 6D pose tracking for novel objects must balance real-time operation with accuracy despite occlusions and fast motion, while existing approaches typically require onboarding or ongoing reconstruction. The paper uses a compact pose graph of uncertainty-weighted relative-pose constraints, reporting comparable accuracy to reconstruction-based trackers at substantially lower optimization cost across four benchmarks.

  • Problem

    Novel-object pose tracking must run online while remaining accurate and temporally consistent over long horizons, but existing approaches typically require per-object onboarding or ongoing reconstruction.

    Source: §1 · p. 1

  • Method

    The tracker compresses correspondences into relative-pose measurements and optimizes only pose constraints, weighted by uncertainty derived from geometric alignment.

    Source: §1 · p. 1

  • Results

    Across four benchmarks, the method achieves near-best accuracy among non-CAD methods at a fraction of reconstruction- and pose-graph-based baselines’ optimization cost, remaining stable beyond a thousand frames.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The results suggest stable long-sequence object tracking can use compact relative-pose measurements without retaining dense correspondences throughout optimization.

    Source: §5 · p. 8

    The method assumes rigid objects, and its local covariance approximation cannot represent well-separated alternative pose solutions such as those for textureless symmetric objects.

    Source: §6 · p. 8

Abstract

from arXiv · show

Tracking a novel object's 6D pose over long horizons currently requires either expensive onboarding or a reconstruction maintained throughout the sequence. This makes current trackers impractical for robotic manipulation and augmented reality, which need trackers that are ready to use and run in real time. We show that a lightweight tracking module can be applied on top of a wide range of correspondence estimation methods to keep drifts bounded while maintaining fast runtime. Our key idea is to avoid point-based optimization in the pose graph and operate only on relative pose constraints, which we weight by a derived uncertainty from the geometric alignment. This makes optimization independent of the number of correspondences while avoiding the direct inclusion of noisy point measurements, leading to fast and robust long-term tracking. Across four real-world benchmarks, our approach achieves tracking accuracy comparable to reconstruction-based trackers with a fraction of the optimization cost. Overall, these results suggest that a compact and reliable pose graph optimization can provide long-horizon consistency at substantially lower computational cost.

1. Introduction

The introduction motivates long-horizon, real-time pose tracking and presents a compact, uncertainty-weighted pose graph as a way to meet both requirements without maintaining a 3D representation. It situates this approach among relative-pose estimators and prior pose-graph trackers.

2.1. Pose Estimation

  • Relative-pose estimators can avoid per-object onboarding and generalize to unseen objects, but pairwise errors compound over time and cause drift.Other long-horizon approaches use online reconstruction or pre-onboarded CAD models or posed references; the former is too slow for real-time use, while the latter must be repeated for each object.Source: §1 · p. 1, §2.1 · p. 2

1. Introduction

  • The method compresses correspondences into relative-pose measurements, optimizes only poses, and weights graph constraints by uncertainty derived from geometric alignment.This keeps update cost independent of the number of inliers and avoids exposing the global objective directly to noisy point measurements.Source: §1 · p. 1
  • Across four real-world RGB-D benchmarks, accuracy is comparable to state-of-the-art tracking-by-reconstruction methods at a fraction of the optimization cost.The tracker is reported to remain stable over sequences longer than a thousand frames.Source: §1 · p. 2

2.2. Pose Graph Optimization

  • Pose-graph optimization combines pairwise constraints into absolute poses, and iSAM2 supports low-latency incremental updates as new frames arrive.Object-centric RGB-D tracking has more commonly used alignment-based objectives than pose-consistency terms in SE(3).Source: §2.2 · p. 2

2.3. Object 6D Pose Tracking

  • Prior object pose-graph trackers can be accurate, but alignment-based objectives couple dense correspondences to global optimization and make cost scale with points or image resolution.The proposed compact pose graph uses correspondences only to generate relative-pose measurements, preserving modularity with respect to the estimator.Source: §2.3 · p. 2

3. Methodology

The method tracks objects by incrementally optimizing an SE(3) pose graph of relative constraints, with per-edge uncertainty derived from geometric alignment. It selects past keyframes for loop closures, estimates relative poses, and updates the graph online.

3. Methodology

  • Each frame adds an odometry constraint and up to K loop-closure constraints, yielding a compact tracker that can use different correspondence methods.Each edge is weighted by uncertainty derived from geometric alignment.Source: §3 · pp. 2–3

3.1. Problem Formulation

  • Tracking is online: at step i, the tracker estimates each frame’s pose relative to frame 0 using only frames 0 through i.Source: §3.1 · p. 3

3.2. Object Pose Graph Optimization

  • The pose graph represents frame poses as nodes and relative pose measurements as edges; anchoring the first pose at identity fixes gauge freedom.Source: §3.2 · p. 3

3.3. Frame Selection

  • Loop closures constrain drift beyond chained odometry; similar-rotation candidates are sampled across timesteps to combine easier short edges with longer-range trajectory anchors.Source: §3.3 · p. 3

3.4. Relative Pose Estimation

  • Relative poses are estimated from correspondences, with RANSAC selecting inliers for weighted Umeyama alignment; alignment uncertainty downweights poorly conditioned measurements.Thin objects and heavy occlusion can make the alignment poorly conditioned.Source: §3.4 · pp. 3–4

3.5. Online Pose Graph Optimization

  • iSAM2 incrementally updates the graph through selective relinearization, while the memory pool retains keyframes with sufficiently distinct views.Source: §3.5 · p. 4

4. Evaluation

Across four benchmarks, the method is among the strongest non-CAD trackers while running substantially faster than reconstruction-based and pose-graph baselines. The evaluation also tests correspondence quality, long-sequence drift, and the effects of uncertainty weighting and keyframe selection.

4. Evaluation

  • The evaluation compares tracking quality and runtime across four real-world benchmarks and examines correspondence sensitivity, long-sequence stability, and component contributions.Pose quality uses ADD-S and ADD AUC, with inference speed reported in frames per second.Source: §4 · pp. 4–5, §4.1 · p. 5

4.3. Comparison to State-of-the-Art

  • The method leads FoundationPose on three of four benchmarks on ADD; on HOI4D, the authors hypothesize that loop closures and uncertainty-weighted information propagation explain the difference.FoundationPose uses CAD models, while this comparison is not directly applicable to all novel-object settings.Source: §4.3 · pp. 5–6
  • Among non-CAD methods, the tracker leads on YCBInEOAT, YCB-Video, and HOI4D, is comparable to BundleSDF on HO3D, and runs substantially faster than reconstruction-based and pose-graph baselines.The comparison reports ADD-S and ADD AUC, BOP Average Recall where available, and average inference speed across four benchmarks.Source: Table 1 · p. 6, §4.3 · p. 6
  • Across four benchmarks on the same H100 GPU, the method averages 10.0 fps, running 5.5 to 7 times faster than the TbR (BundleSDF) and PGO (BundleTrack (R.)) baselines.On YCB Video, frame selection, PGO, and memory pool update together add only 2.7 ms, versus 75 ms for BundleTrack’s dense-alignment PGO step under the same matcher.Source: §4.3 · p. 6

4.4. Sensitivity to Correspondence Quality

  • With LoFTR, the compact method leads BundleTrack on all datasets and both metrics, most notably by 6.4 ADD points on HO3D; with RoMa v2, the objectives perform comparably.The comparison uses the same feature matcher for both optimization objectives.Source: §4.4 · p. 7, Table 5 · p. 7

4.5. Drift Analysis

  • On the full 1718-timestep YCB Video sequence, the method has the lowest mean rotation and translation error throughout, with no sustained failure episode.YCB Video is the longest sequence among the four benchmarks.Source: §4.5 · p. 7, Figure 4 · p. 7

4.6. Ablation Studies

  • Both trace and full-covariance weighting improve accuracy over unweighted edges, but anisotropic covariance is not consistently better than magnitude-only weighting on two of four benchmarks.With LoFTR, covariance weighting gains more over unweighted edges; uncertainty correlates positively with true error, though heavy occlusion on symmetric objects and end-on views of thin objects can produce overconfident estimates.Source: §4.6 · pp. 7–8, Table 6 · p. 7, Figure 5 · p. 8
  • Removing the memory pool makes selected frames redundant, while replacing farthest-frame sampling with adjacent frames causes severe pose degeneration.The results indicate that loop-closure value comes from spanning the sequence rather than increasing the number of edges.Source: §4.6 · p. 8

5. Conclusion

The paper concludes that compact pose-graph tracking can achieve near-best accuracy among non-CAD methods at lower optimization cost, while remaining stable over long sequences.

5. Conclusion

  • The framework uses iSAM2 to optimize a compact pose graph of relative pose measurements weighted by geometry-derived per-edge uncertainty.Dense correspondences produce relative pose measurements rather than being fused into the optimization objective.Source: §5 · p. 8
  • Across four benchmarks, the method reaches near-best accuracy among non-CAD methods at a fraction of reconstruction- and pose-graph-based baselines’ optimization cost.The reported tracking stability extends to sequences longer than a thousand frames.Source: §5 · p. 8
  • The results suggest that dense correspondences need not be retained during optimization to provide comparable accuracy at substantially lower cost.Source: §5 · p. 8

6. Limitation

The method assumes rigid objects, and its local covariance approximation cannot represent well-separated alternative pose solutions.

6. Limitation

  • The method assumes rigid objects and leaves articulated and deformable-object tracking for future work.Source: §6 · p. 8
  • Its covariance is local to the RANSAC-Umeyama solution, capturing flat, poorly constrained directions but not well-separated alternatives such as those for textureless symmetric objects.Source: §6 · p. 8
  • A learning-based 6D pose uncertainty estimator is proposed as future work to represent multimodal pose distributions.Source: §6 · p. 9

7. Overview

The supplementary material covers preprocessing, shared hyperparameters, parameter sensitivity, and the derivation of per-edge pose uncertainty.

8.1. Input Preprocessing

  • Input preprocessing uses XMem masks on HO3D and ground-truth masks elsewhere, with object-centric crops resized to 320 × 320 for RoMa v2 and 400 × 400 for LoFTR.Source: §8.1 · p. 11

8.2. Hyperparameters

  • The shared settings use a Cauchy loss with k=1.0, τk=25° for RoMa v2 and 15° for LoFTR, 1000 correspondences, τm=5°, and RANSAC threshold dr=2e−3.These parameters are applied across benchmarks without tuning.Source: §8.2 · p. 11

8.3. Ablation of Parameters

  • LoFTR is more sensitive to larger frame-selection thresholds τk than RoMa v2, while thresholds that are too small suppress performance for both matchers.The sweep compares the two feature matchers; the text suggests their difference may reflect generalization to large viewpoint changes.Source: §8.3 · p. 11, Figure 7 · p. 11

9. Uncertainty Estimation

  • Per-edge covariance estimates uncertainty in the geometric alignment from the inlier correspondences, using a Gauss–Newton approximation and residual-variance scale calibration.The covariance is a 6 × 6 matrix.Source: §9 · pp. 11–12

10. Metrics

Tracking quality is evaluated with ADD and ADD-S area-under-curve metrics, with trajectories aligned to ground truth at the first frame.

10. Metrics

  • ADD and ADD-S AUC are computed over distance thresholds from 0 to 0.1 m after aligning each video’s predicted trajectory to ground truth at the first frame.Source: §10 · p. 12
  • The metrics compare the estimated and ground-truth poses against the object model point set.Source: §10 · p. 12
  • ADD-S uses closest-point distances rather than one-to-one correspondences, making it more suitable for symmetric objects with symmetry ambiguity.Source: §10 · p. 12

11. Comparison to Different Tracking Approaches

The comparison evaluates direct frame-to-frame tracking against BundleTrack and the proposed pose-graph module with different correspondence and pose-estimation methods. Pairing the module with each backend consistently improves accuracy, with larger gains for weaker backends.

11. Comparison to Different Tracking Approaches

  • Pairing the pose-graph module with each backend improves accuracy consistently, with larger gains when the backend is weaker.The comparison includes direct frame-to-frame tracking, BundleTrack, and the proposed module paired with different pose estimators.Source: §11 · p. 12, Table 8 · p. 12
  • RoMa v2 and LoFTR are more robust and stable than UNOPose among the direct frame-to-frame results.Each query pose in this setting is estimated relative to the previous frame alone.Source: §11 · p. 12

12. Runtime Analysis

The runtime analysis compares the compact pose-graph updates with BundleTrack and reports rotation accuracy by category on HOI4D. Updates take about 2 milliseconds per frame on average, while the method leads on three of five non-CAD categories and on average.

12. Runtime Analysis

  • Around 2 milliseconds per frame on average, compact PGO updates are significantly faster than BundleTrack using the same feature matcher and correspondence count.The runtime comparison is on YCB Video; iSAM2 relinearization time grows gradually but stays in the low-millisecond range over its 1,718-frame sequence.Source: §12 · p. 12, Figure 8 · p. 13, Figure 9 · p. 13

13. Results on HOI4D

  • Our method leads on three of five non-CAD categories and on average for HOI4D rotation accuracy.The strongest category results are reported for knife and toy car.Source: Table 10 · p. 13, §13 · p. 13

14. Qualitative Results of Uncertainty-Aware PGO

The qualitative comparison shows that uncertainty-aware PGO improves robustness in geometrically ambiguous cases. The challenge-case results also identify failure settings alongside several difficult conditions in which tracking succeeds.

15. Qualitative Results on Challenge Cases

14. Qualitative Results of Uncertainty-Aware PGO

  • Uncertainty modeling improves robustness in geometrically ambiguous cases, including textureless and thin objects.The comparison is between uncertainty-aware PGO and unweighted PGO using identity as uncertainty.Source: §14 · p. 13, Figure 10 · p. 14
Loading 2610.11815v1…