PaperlayerView exact revision on arXiv

Source-linked AI summary

RoPT: Robust 6DoF Object Pose Tracking in Video

Yongqing Liang, Yuzhong Chen, Sihang Li, Junsuk Kim, Chase Kidder, Hui-ping Wang, Yunzhi Xu, Raihan Haque, Yubiao Zhang, Chen Feng

arXiv:2610.28962v1cs.CVcs.RO

TL;DR

Robotic interaction needs full 6-DoF pose information, while conventional monocular trackers can diverge under rapid motion. RoPT combines visual perception, temporal kinematics, and failure-triggered recovery in a closed loop; it is reported to maintain trajectory consistency where state-of-the-art trackers diverge during rapid motion.

  • Problem

    Video localization and segmentation methods operate in the image plane, while robotic interaction requires full spatial orientation and geometric grounding; conventional open-loop trackers can also diverge under rapid motion.

    Source: §1 · pp. 1–2

  • Method

    RoPT couples visual perception with a continuous kinematic factor graph for pose forecasts and multi-hypothesis verification, plus deterministic, failure-triggered self-recovery.

    Source: §1 · p. 2

  • Results

    RoPT maintains trajectory consistency during rapid, erratic motion where existing state-of-the-art trackers catastrophically diverge.

    Source: §1 · p. 2

  • Takeaways & Limitations

    RoPT is a closed-loop 6-DoF tracking framework engineered for rapid, high-acceleration dynamics in embodied systems.

    Source: §5 · p. 9

    RoPT assumes an exact geometric CAD prior and RGB-D sensing, limiting applicability to highly deformable objects, transparent materials, or specular surfaces.

    Source: §5 · p. 9

Abstract

from arXiv · show

Accurate 6-DoF object pose tracking is essential for robotic interaction, yet conventional monocular pipelines frequently suffer from catastrophic divergence under rapid object motion. We present RoPT, a robust tracking framework engineered to overcome this limitation by unifying visual perception, temporal kinematics, and global re-localization into a tightly coupled closed loop. Unlike existing open-loop trackers, RoPT delivers three critical capabilities that fundamentally alter system resilience. First, the visual perception component tracks object masks to robustly handle partial and severe occlusions. Second, a kinematic factor graph regularizes frame-to-frame updates, forecasts forward pose predictions, and drives a parallel multi-hypothesis verification scheme to bypass local minima. Finally, an active re-localization module transforms the pipeline into a closed-loop system, making it failure-aware and capable of deterministic self-recovery. We demonstrate the effectiveness of our framework across challenging benchmarks alongside real-world robotic demonstrations, achieving significantly higher tracking survivability under rapid motion.

1 Introduction

Video localization and masks do not provide the full spatial orientation and geometric grounding needed for robotic control, while conventional open-loop trackers can diverge under rapid motion. RoPT addresses these challenges with a closed-loop framework combining temporal kinematics, multi-hypothesis verification, and failure-triggered recovery.

1 Introduction

  • Video understanding methods that localize and segment objects in 2D lack the full spatial orientation and geometric grounding required for reliable robotic interaction.The resulting discrepancy is between existing video understanding frameworks and the requirements of robotic control.Source: §1 · p. 1
  • Conventional open-loop trackers can lose objects during sudden high-acceleration motion and lack the capacity to detect their own divergence.The resulting tracking loss can cause policy execution and the manipulation process to fail.Source: §1 · p. 2

Resilience to High-Acceleration Dynamics

  • RoPT maintains trajectory consistency during rapid, erratic object motion where existing state-of-the-art trackers catastrophically diverge.The framework is designed to bridge fragile visual perception and the demands of dynamic robotic manipulation.Source: §1 · p. 2

Kinematic Factor Graph-Enabled Multi-Hypothesis

  • A continuous temporal graph backend drives parallel pose-hypothesis verification to bypass local geometric minima and severely degraded 2D masks.This mechanism is intended to help the tracker cope with sudden motion and visual degradation.Source: §1 · p. 2

Failure-Aware Active Self-Recovery

  • A deterministic re-localization module detects catastrophic tracking divergence and self-recovers by matching observations against predictive synthetic CAD renderings.Source: §1 · p. 2

2 Related Work

Prior 6-DoF tracking work supports robotic applications, but model-based methods commonly rely on local refinement and small inter-frame pose changes. RoPT’s overview combines perception, temporal pose forecasts, scoring, and recovery in a closed loop.

6-DoF Perception in Robotic Manipulation

  • 6-DoF tracking supports applications including aerial manipulation, grasping, visual servoing, dynamic manipulation, human-robot interaction, catching flying objects, and online replanning.These applications remain dependent on the accuracy of the underlying pose estimator.Source: §2 · p. 2

RoPT pipeline overview

  • RoPT’s pipeline couples mask-guided perception and kinematic pose forecasts with candidate scoring, then invokes CAD-rendering-based re-localization after tracking loss.The overview describes 3D-2D correspondences with the incoming RGB frame for recovery.Source: Figure 2 · p. 3

6-DoF Perception in Robotic Manipulation

  • Depth may be unavailable in web video or invalid at close range because of active sensors’ minimum working distance.Source: §2 · p. 3

6DoF Pose Estimation and Tracking

  • Model-based pose methods use explicit 3D geometry for alignment, but typically focus on local refinement and assume small pose changes between frames.The paper positions its tracking framework as moving beyond that assumption.Source: §2 · p. 3

Foundation Models for Visual Perception

  • Recent foundation models provide zero-shot generalization for 2D segmentation and feature matching, which RoPT uses as a perception backbone for robotics.Source: §2 · p. 3

3 Method

RoPT combines visual refinement, mask tracking, kinematic prediction, lighting adjustment, and active re-localization in a closed loop for 6-DoF tracking in RGB-D video. Its components address local refinement failures, occlusion and rapid motion, appearance mismatch, and tracking divergence.

3 Method

  • RoPT links render-and-compare refinement, mask tracking, a kinematic factor graph, global lighting optimization, and self-recovery in a closed loop.The pipeline targets continuous 6-DoF tracking in RGB-D video.Source: §3 · pp. 3–4

3.1 Render-and-Compare Pose Refinement

  • The baseline refines pose hypotheses against each RGB-D frame, scores their alignment, and selects the highest-scoring pose; this open-loop configuration is V0.The refinement network starts from the previous frame’s estimate, and large inter-frame displacements can place the pose outside its local convergence basin.Source: §3.1 · p. 4

3.2 Occlusion-Robust Mask Tracking

  • V1 uses a tracked 2D mask and camera intrinsics to back-project object depth pixels into a 3D centroid for an initial translation, improving tolerance to partial occlusions and abrupt translations.Imperfect masks can corrupt the translation prior, and mask-based anchoring does not address abrupt rotations.Source: §3.2 · p. 4

3.3 SE(3) Factor Graph with Multi-Hypothesis Selection

  • V2 adds a sliding-window factor graph with a constant-velocity prior and robust observation weighting to predict poses and limit the influence of visual outliers.The graph tracks generalized velocity with an interval-adaptive EMA and uses a Huber estimator for observation covariance.Source: §3.3 · pp. 4–5
  • Four refinement hypotheses combine previous or predicted rotation with previous or mask-derived translation, then selection uses maximum rotational divergence or the score network.The hypotheses include the previous optimized pose, the graph’s forward prediction, and combinations of predicted or previous rotation with mask-derived translation.Source: §3.3 · p. 5

3.4 Global Lighting Optimization

  • V3 optimizes 27 second-order spherical-harmonics lighting coefficients at first-frame initialization with the object pose fixed, then locks the parameters for the rest of tracking.The module addresses the synthetic-to-real appearance gap attributed to unmodeled illumination and material properties.Source: §3.4 · pp. 5–6

3.5 Failure-Aware Self-Recovery

  • V4 triggers recovery when the multi-hypothesis alignment score exceeds a threshold, matching a predictive synthetic CAD rendering to the real observation and using PnP with RANSAC to re-anchor the factor graph.Dense 2D matches are lifted to 3D-to-2D correspondences using the pixel-aligned rendered color buffer and object-frame coordinates.Source: §3.5 · p. 6

4 Experiments

Experiments evaluate RoPT on YCBInEOAT with motion-scaled sequences, then assess component ablations and robotic manipulation. Reported findings include state-of-the-art tracking accuracy under slow and fast motion, gains from three components, and recovery when near-field depth readings fail.

4 Experiments

  • Benchmark evaluation downsamples sequences by factors of 2× to 16× to simulate normal-to-high velocities and reports ADD and ADD-S AUC.The framework uses dataset CAD models for zero-shot tracking without finetuning.Source: §4 · p. 6

4.1 Benchmark Comparison

  • RoPT achieves state-of-the-art tracking accuracy under both slow and fast motion on enhanced YCBInEOAT.Table 1 is identified as the quantitative robustness and accuracy comparison.Source: Table 1 · p. 7, §4.1 · p. 7

4.2 Ablation Study

  • Mask-guided tracking, the kinematic factor graph, and active recovery drive the largest survivability and accuracy gains; lighting optimization adds only marginal quantitative improvement.The authors attribute the lighting plateau to the baseline neural refinement module’s illumination invariance, while retaining lighting optimization to reduce the domain gap for synthetic-to-real feature matching.Source: §4.2 · pp. 7–8, Table 3 · p. 8

4.3 Real-World Robotic Experiments

  • In robotic manipulation, RoPT maintains lock during erratic target and sensor motion, then uses RGB matching when near-field depth readings fail.The gripper approaches close enough for the depth sensor to exceed its minimum working distance.Source: Figure 6 · p. 8, §4.3 · pp. 8–9

5 Conclusion

RoPT integrates visual perception, a continuous kinematic factor graph, and active self-recovery into a closed-loop tracker for rapid, high-acceleration dynamics. Its scope is constrained by model and sensing assumptions, prolonged occlusion, and computational tuning.

5 Conclusion

  • RoPT combines visual perception, a continuous kinematic factor graph, and failure-aware active self-recovery in a closed-loop 6-DoF tracker for rapid dynamics.The paper presents the framework for embodied systems and real-world robotic manipulation.Source: §5 · p. 9
  • The method assumes an exact CAD prior and RGB-D sensing, limiting applicability to highly deformable objects, transparent materials, and specular surfaces.Source: §5 · p. 9
  • Severe, prolonged occlusion beyond the factor graph’s predictive horizon can place the target beyond the multi-hypothesis spread-gate’s search radius, preventing autonomous recovery.Source: §5 · p. 9
  • Although RoPT runs at 18.6 FPS, computational scalability is bottlenecked by tuning factor-graph optimization parameters to balance temporal resilience and overhead.The authors identify learning-based graph tuning and feature distillation as future work.Source: §5 · p. 9

Appendix A Implementation Details

The implementation combines pretrained perception and pose modules with a GTSAM kinematic factor graph, empirical recovery triggers, and first-frame lighting estimation.

Appendix A Implementation Details

  • The V4 implementation uses pretrained FoundationPose refinement and scoring, EdgeTAM mask tracking, and Mast3r feature matching; further end-to-end finetuning is outside this paper’s scope.Source: Appendix A · p. 13
  • The kinematic factor graph uses GTSAM ISAM2 and NonlinearFactorGraph, maintains pose in SE(3), and optimizes after adding each estimated pose.Initial observation sigmas are {0.02, 0.02, 0.02, 0.10, 0.10, 0.10}; motion noise is set to 0.05.Source: Appendix A · p. 13
  • Recovery is triggered when FoundationPose’s score exceeds τs = 140 or frame-to-frame pose change exceeds 50 mm or 15 degrees.The score threshold was selected empirically; the paper reports it works across different objects and testing cases.Source: Appendix A · p. 13
  • For V3 and V4, spherical-harmonics lighting coefficients are estimated on the first frame and reused for later rendering.The authors report lighting-coefficient computation overhead below 1 second.Source: Appendix A · p. 13

Appendix B Experiments on YCB-Video Benchmark

On YCB-Video, RoPT is evaluated in scenes with rapid sensor motion, changing lighting, and occlusion. The reported comparison favors RoPT over FoundationPose and FoundationPose++ for stability and tracking performance.

Appendix B Experiments on YCB-Video Benchmark

  • YCB-Video sequences include rapid sensor motion, varying lighting, and complex occlusions, conditions in which FoundationPose often drifts or fails.Source: Appendix B · p. 13
  • RoPT outperforms FoundationPose and FoundationPose++ on YCB-Video, with superior stability and adherence to object boundaries despite lacking depth cues.Table 4 reports quantitative results averaged over three object-tracking results per video.Source: Table 4 · p. 13, Appendix B · p. 13

Appendix C Real-World Robotic Demonstrations

Real-world experiments used a FANUC CRX robot and an Astra 2 RGB-D sensor to evaluate RoPT’s pose tracking and failure-aware self-recovery.

Appendix C Real-World Robotic Demonstrations

  • Experiments used a FANUC CRX robot and an Astra 2 RGB-D sensor in a third-view setup.Source: Appendix C · p. 14
  • The real-world evaluation covered 6-DoF pose tracking and failure-aware self-recovery.Source: Appendix C · p. 14
  • Standard pick-and-place tasks are discussed in the main manuscript, while this appendix reports additional real-world evaluations.Source: Appendix C · p. 14

C.1 New 6-DoF Pose Tracking Benchmark

The new real-world benchmark tests 6-DoF tracking under robot-controlled motion, and RoPT successfully tracks object pose at both slow and fast speeds. Existing methods show qualitative difficulty with rapid camera motion and estimating translation and orientation.

C.1 New 6-DoF Pose Tracking Benchmark

  • The benchmark uses random robotic-arm trajectories capped at 10, 15, and 20 cm/s, with video recorded at 25 fps.Source: §C.1 · p. 14
  • FoundationPose often loses the object when the camera moves rapidly, while existing methods frequently struggle to estimate correct translation and orientation.Source: Figure 8 · p. 15, Figure 9 · p. 15
  • The robot’s calibrated Tool Center Point location serves as 6-DoF pose ground truth for evaluating tracking methods.Source: §C.1 · p. 15
  • The benchmark targets highly dynamic robotic kinematics, unlike existing benchmarks that predominantly feature artificially slow movements.Source: §C.1 · p. 15
  • RoPT successfully tracked 6-DoF object pose at both slow and fast speeds in the real-world benchmark.Source: §C.1 · p. 15

C.2 Failure-Aware Self-Recovery Evaluation

The evaluation examines tracking during intentional dynamic occlusions. RoPT signals tracking failure during full occlusion and re-establishes the accurate 6-DoF pose when the object reappears.

C.2 Failure-Aware Self-Recovery Evaluation

  • The evaluation uses video sequences with intentional, dynamic occlusions that block the target object.Source: §C.2 · p. 15
  • During full occlusion, open-loop methods output erratic, unreliable poses, whereas RoPT correctly signals tracking failure.Source: Figure 10 · p. 16
  • Upon the target’s reappearance, RoPT successfully recovers the accurate 6-DoF pose.Source: Figure 10 · p. 16
  • RoPT explicitly detects tracking failures and re-establishes tracking when the object reappears.Source: §C.2 · p. 16
Loading 2610.28962v1…