PaperlayerView exact revision on arXiv

Source-linked AI summary

RoPE-Flow: Monocular Vision-based Robot Pose Estimation via 2D Graph-Conditioned 3D Flow Matching

Liangjing Shao, Wanhao Liu, Zhiwei Fang, Rulin Zhou, Quanlu Zhang, Changjing Liu, Beilei Cui, Yiming Huang, Hongliang Ren

arXiv:2610.21086v1cs.CVcs.RO

TL;DR

Estimating articulated robot poses from monocular RGB is challenging because kinematic-chain links self-occlude and have appearance ambiguity under changing viewpoints and lighting. RoPE-Flow uses multi-scale keypoint-centered features and graph-guided conditional flow matching, with bone-length and ray-depth losses and joint-wise weighting. RoPE-Flow outperforms state-of-the-art methods across robot types and cameras, and the authors report strong zero-shot cross-scene performance on the LabRoPE benchmark.

  • Problem

    Estimating articulated robot poses from monocular RGB is challenging because kinematic-chain links self-occlude and have appearance ambiguity under changing viewpoints and lighting.

    Source: §1 · p. 2

  • Method

    RoPE-Flow uses multi-scale keypoint-centered features and graph-guided conditional flow matching, with bone-length and ray-depth losses and joint-wise weighting.

    Source: §1 · p. 2

  • Results

    RoPE-Flow outperforms state-of-the-art methods across robot types and cameras, and the authors report strong zero-shot cross-scene performance on the LabRoPE benchmark.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The authors present RoPE-Flow as a monocular camera-to-robot pose estimation approach and introduce LabRoPE for evaluating zero-shot cross-scene generalization.

    Source: §1 · p. 2, §5 · p. 10

    The authors identify prediction of self-rotating and distant joints, as well as inference efficiency, as areas for future optimization.

    Source: §5 · p. 10

Abstract

from arXiv · show

Camera-to-robot pose estimation from a monocular image is significant for vision-based robot navigation and control. However, recovering 3D structural features from 2D observations remains a challenging task. Existing camera-to-robot pose estimation methods either rely on time-consuming iterative rendering or involve sophisticated 2D-3D registration which suffers from projection ambiguity, accumulated errors for distal joints and imbalanced learning for different joints. In this work, we propose RoPE-Flow, an end-to-end flow matching-based framework for camera-to-robot pose estimation. From the predicted 2D heatmaps, we propose a Keypoint-centered Feature Extractor to obtain positional features along with contextual representations from keypoints-centered multi-scale patches. Moreover, we introduce Graph-guided Condition Generator, which embeds a learnable adjacency map to generate condition for flow matching based on correlation among keypoints. Besides, we introduce Bone Length Loss and Ray-Depth Loss for topological consistency and scale stability, with joint-wise weighting to mitigate varied prediction difficulty across joints. The proposed method outperforms state-of-the-art methods for pose estimation across different types of robots with mutiple cameras. Besides, we establish a novel cross-scene benchmark to demonstrate the outstanding performance of RoPE-Flow on zero-shot cross-scene robot pose estimation. Meanwhile, animated visualizations are shown on the website: ropeflow.netlify.app

1 INTRODUCTION

RoPE-Flow targets monocular camera-to-robot pose estimation, where recovering a robot’s 3D structure from images is difficult. It combines keypoint-centered features, graph-guided flow matching, and training losses for structural consistency and scale stability; the authors report strong benchmark results, including zero-shot cross-scene evaluation.

Motivation

  • Monocular robot pose estimation must recover a kinematic chain whose links can self-occlude and appear ambiguous under changing viewpoints and lighting.The paper identifies applications including vision-based navigation, human–robot collaboration, closed-loop manipulation, and multi-robot collaboration.Source: §1 · p. 2

Prior approaches

  • Marker-based calibration can drift after impacts or repeated operation and as environmental conditions, such as vibration, change.Conventional methods use image marker coordinates, forward kinematics, and PnP to estimate the camera-to-robot transformation.Source: §1 · p. 2
  • Existing approaches use iterative rendering or recover 3D structure from 2D keypoints with PnP and forward kinematics, while 2D–3D registration can remain ambiguous because projections may match multiple depth configurations.The paper also notes limited zero-shot cross-scene evaluation on real robots in prior work.Source: §1 · p. 2

RoPE-Flow

  • RoPE-Flow is a conditioned flow-matching approach that directly predicts 3D keypoints, addressing the need to account for rigid structure, kinematic priors, inter-joint relationships, and varied joint difficulty.Figure 2 visualizes prior-method comparisons, prediction-error distributions, projection ambiguity, and potential solutions in this work.Source: §1 · p. 2, Figure 2 · p. 3
  • The Keypoints-centered Feature Extractor combines positional embeddings with multi-scale contextual representations at keypoint locations.Source: §1 · p. 2
  • The Graph-guided Condition Generator conditions 3D flow matching on keypoint-centered features and relationships among keypoints.Source: §1 · p. 2
  • Bone Length Loss and Ray-Depth Loss target structural consistency and scale stability, respectively, while joint-wise weighting balances differences in prediction difficulty across joints.Source: §1 · p. 2

Results

  • RoPE-Flow outperforms state-of-the-art methods across robot types and backgrounds in DREAM, and its zero-shot cross-scene performance is evaluated on the new LabRoPE benchmark.The reported DREAM comparisons cover varied robots and cameras.Source: §1 · p. 2

2 RELATED WORK

Camera-to-robot pose estimation has been approached with both marker-based calibration and markerless methods. The latter aim to avoid repeated offline calibration, but existing approaches face inference and robustness constraints.

2.1 MARKER-BASED CAMERA-TO-ROBOT POSE ESTIMATION

  • Marker-based calibration detects fiducials across multiple known robot poses and uses PnP to estimate the camera-to-robot transformation; a shift in relative pose requires full offline recalibration.Source: §2.1 · p. 3

2.2 MARKERLESS CAMERA-TO-ROBOT POSE ESTIMATION

  • Markerless methods include iterative rendering, keypoint-based PnP, temporal cues, masked-image pretraining, and depth priors, but may require a predefined robot ROI and remain sensitive to noise, occlusion, or projection ambiguity.RoPE-Flow is described as predicting camera-frame 3D pose from the full image and modeling uncertainty for occluded joints.Source: §2.2 · p. 3

3 PROPOSED METHOD

RoPE-Flow combines a pretrained image encoder and keypoint heatmaps with keypoint-centered and graph-guided conditioning, then uses conditional flow matching to predict camera-frame 3D keypoints. The pipeline also includes a joint-angle prediction pathway.

3 PROPOSED METHOD

  • The pipeline combines a pretrained image encoder, two groups of LoRA adapters, heatmap decoders, keypoint and graph-guided conditioning, a joint head, and 3D conditional flow matching.The two adapter groups are LoRAkp and LoRAq.Source: §3 · pp. 3–4

3.1 2D KEYPOINTS PREDICTION

  • A frozen pretrained DINOv3 ViT-L/16 produces heatmaps whose argmax peaks give keypoint coordinates and whose peak values provide confidence scores.The encoder feature map has 1024 channels and spatial dimensions 16×16.Source: §3.1 · p. 4

3.2 KEYPOINTS-CENTERED FEATURE EXTRACTOR

  • The Keypoints-centered Feature Extractor samples patches of sizes 3, 7, and 11 around each heatmap peak, maps them through MLPs, and combines them with position and confidence.The resulting representation concatenates keypoint coordinates, confidence, and patch features.Source: §3.2 · p. 4

3.3 GRAPH-GUIDED CONDITION GENERATOR

  • The Graph-guided Condition Generator processes keypoint features through three graph-convolution layers using a learnable row-softmax adjacency, then adds an embedding of normalized camera intrinsics.The adjacency is initialized as the identity matrix, and the intrinsics embedding is added residually to form the flow condition.Source: §3.3 · p. 4

3.4 3D CONDITIONED FLOW MATCHING

  • Conditional flow matching generates camera-frame 3D keypoints by predicting velocity from the interpolant, time token, and graph condition, starting from Gaussian noise and applying Euler steps.The initial noise is standard Gaussian and the number of Euler steps is N.Source: §3.4 · p. 5

3.5 JOINT ANGLE PREDICTION

  • A separate joint-angle pathway uses LoRAq in the last six ViT blocks, a dedicated keypoint decoder and condition generator, and a residual MLP head iterated four times from zero.The joint-angle pathway is distinct from 3D keypoint estimation.Source: §3.5 · p. 5

3.6.1 TRAINING ON SYNTHETIC DATA

  • Synthetic supervision uses three stages—2D keypoint prediction, 3D keypoint prediction, then joint-angle prediction; the final stage trains the joint-angle pathway with normalized-joint-space MSE while freezing the 3D-keypoint pathway.Stage II uses weighted joint-wise velocity loss; geometric loss includes weighted joint-wise 3D L1, bone-length loss, and FK alignment.Source: §3.6.1 · pp. 5–6

3.6.2 SIM-TO-REAL FINE-TUNING

  • On real data, they fine-tune 3D keypoints with reprojection filtering, pose-invariant bone lengths, and ray-depth alignment, then adapt joint-angle prediction using reprojected keypoints.Samples below a 100-pixel residual are inliers; otherwise the lowest 50% of finite residuals are retained.Source: §3.6.2 · p. 6

4 EXPERIMENTS AND RESULTS

RoPE-Flow is evaluated on DREAM across robot types and cameras, on joint-angle estimation, and on the zero-shot LabRoPE benchmark. Ablations report benefits from the full multi-scale model, graph-guided flow matching, and the loss components.

4.1 IMPLEMENTATION DETAILS

  • The DREAM experiments train separately on synthetic Franka, Kuka, and Baxter splits, with evaluation across specified DR, Photos, and real-camera splits.Franka uses DR-pretrained weights for Azure sim-to-real fine-tuning before evaluation on Kinect360, RealSense, and ORB data.Source: §4.1 · p. 6

4.2 CAMERA-FRAME POSE ESTIMATION

  • RoPE-Flow surpasses state-of-the-art methods on every DREAM split and robot-camera combination, and reports the most accurate joint positions without joint priors.It also achieves competitive joint-position results against methods using known joint states.Source: §4.2 · p. 7

4.3 JOINT ANGLE ESTIMATION

  • On DREAM, RoPE-Flow achieves the best joint-angle accuracy on the Franka and Kuka splits, including against methods using ground-truth or preprocessed bounding boxes.URDF-based reconstructions are reported as more accurate than prior state-of-the-art models.Source: §4.3 · p. 7, Table 3 · p. 8, Figure 4 · p. 8

4.4 ZERO-SHOT CROSS-SCENE ESTIMATION

  • LabRoPE evaluates zero-shot cross-scene estimation on 50 RealSense samples from diverse views, and Table 4 is reported to verify RoPE-Flow’s strong generalization.The comparison uses mIoU and Dice between input-image robot masks and masks from reconstructed 3D models.Source: §4.4 · p. 8, Table 4 · p. 8

4.5 ABLATION STUDIES

  • Ablations report that the full multi-scale model with position embeddings performs best overall, graph-guided 3D flow matching outperforms PnP-FK, and each loss component contributes substantially.Larger single patches favor synthetic data, while smaller patches favor sim-to-real fine-tuning; sim-to-real ablations also examine Bone Length Loss and Ray-depth Loss.Source: §4.5 · p. 9, Table 5 · p. 9

5 CONCLUSION

RoPE-Flow combines multi-scale keypoint features with graph-guided flow matching for monocular camera-to-robot pose estimation. The authors report accurate pose and joint-angle predictions, strong zero-shot results, and remaining challenges with self-rotating and distant joints and inference efficiency.

5 CONCLUSION

  • RoPE-Flow combines multi-scale keypoint-centered features and graph-guided flow matching for monocular RGB camera-to-robot pose estimation.The conclusion describes it as the first flow matching-based approach for this task.Source: §5 · p. 10
  • The authors report accurate robot poses and joint angles, and superior practical performance against state-of-the-art approaches on a benchmark for zero-shot generalization.The benchmark evaluates zero-shot generalization.Source: §5 · p. 10
  • Prediction of self-rotating and distant joints, along with inference efficiency, remains an area for future optimization.Source: §5 · p. 10

A DETAILS OF PROPOSED DATASET LabRoPE

LabRoPE is a 50-image cross-scene dataset of a Franka Panda robot, captured from five views against varied laboratory backgrounds. It includes human-robot interaction and occlusion, and uses detected and segmented robot masks for zero-shot evaluation.

A DETAILS OF PROPOSED DATASET LabRoPE

  • LabRoPE contains 50 RealSense images of a Franka Panda from five views and varied laboratory backgrounds, including human-robot interaction and occlusion.A DIY laser transmitter replaces the standard grasper; GroundingDINO and SAM2 are used to detect and segment ground-truth robot masks.Source: A DETAILS OF PROPOSED DATASET LabRoPE · p. 12

B SUPPLEMENTARY COMPARISON RESULTS

The supplementary results include ADD curves and qualitative 3D-keypoint comparisons. The latter compare RoPE-Flow with RoboPEPP and HPE using reconstructed skeletons.

B.1 CURVES OF ADD

  • Figure 7 presents curves of ADD, providing a visual view of the ADD evaluation metric.Source: Figure 7 · p. 12

B.2 QUALITATIVE COMPARISON OF 3D KEYPOINTS PREDICTION

  • Reconstructed skeletons compare RoPE-Flow’s 3D keypoint predictions with those of RoboPEPP and HPE, illustrating the reported performance qualitatively.Source: §B.2 · p. 13, Figure 8 · p. 13

B.3 JOINT-WISE ERROR FOR JOINT ANGLES PREDICTION

The joint-wise analysis reports differing prediction accuracy across robot joints, while supplementary examples cover DREAM and LabRoPE, including human–robot interaction and occlusion.

B.3 JOINT-WISE ERROR FOR JOINT ANGLES PREDICTION

  • RoPE-Flow has relatively inaccurate predictions on several Franka joints but achieves the best average error and all-joints prediction for Kuka.The comparison reports mean MAE (°) for each robot joint.Source: §B.3 · p. 13, Table 7 · p. 15

B.4 SUPPLEMENTARY EXAMPLES FROM DREAM

  • Supplementary DREAM examples were randomly selected from nine listed robot and data-split combinations.The examples include Franka, Panda, Kuka, and Baxter splits, as well as Azure, Kinect360, RealSense, and ORB data.Source: §B.4 · p. 13, Figure 9 · p. 14

B.5 SUPPLEMENTARY EXAMPLES FROM LABROPE

  • LabRoPE supplementary examples include two samples per view, with some showing human–robot interaction and occlusion.The qualitative results are shown in Figure 10.Source: §B.5 · p. 15

C.1 QUALITATIVE COMPARISON FOR ABLATION STUDIES

The qualitative ablations compare removing Ray-depth Loss with using single-scale rather than multi-scale keypoint-centered patch embedding.

C.1 QUALITATIVE COMPARISON FOR ABLATION STUDIES

  • Without Ray-depth Loss, results remain relatively accurate in the original camera view but degrade significantly from another perspective.The authors state that this verifies the projection ambiguity motivating their work.Source: §C.1 · p. 15
  • The Ray-depth Loss ablation illustrates projection ambiguity: accuracy in the original view does not carry over to a different perspective.Source: §C.1 · p. 15
  • The full model with multi-scale keypoint-centered patch embedding provides more accurate pose estimation than single-scale patch embedding.The comparison is shown in Figure 12.Source: §C.1 · p. 15

C.2 ZERO-SHOT ABLATION STUDIES ON LABROPE DATASET

The zero-shot LabRoPE ablation study uses settings selected on the RealSense split of DREAM and reports that ablated components affect generalization.

C.2 ZERO-SHOT ABLATION STUDIES ON LABROPE DATASET

  • Ablation settings were selected by their performance on the RealSense split of DREAM for zero-shot evaluation on LabRoPE.Source: §C.2 · p. 15
  • The LabRoPE zero-shot ablations demonstrate that the ablated components critically influence RoPE-Flow’s generalization.The results are reported in Table 8 and Figure 13.Source: §C.2 · p. 15
  • The study evaluates generalization using zero-shot pose estimation on the LabRoPE dataset.Source: §C.2 · p. 15

C.3 ABLATION STUDIES ON FLOW MATCHING STEPS

The flow-step ablation favors 10 steps on the evaluated DREAM splits; runtime is lower in efficiency but reported to meet real-time requirements.

C.3 ABLATION STUDIES ON FLOW MATCHING STEPS

  • 10-step flow matching provides the best results among the tested settings on Panda and Kuka DREAM splits.The tested step counts were 1, 2, 5, 10, 12, 15, and 20; results are in Table 9.Source: §C.3 · p. 15, Table 9 · p. 19

D RUNTIME AND COMPUTATIONAL COST

Loading 2610.21086v1…