PaperlayerView exact revision on arXiv

Source-linked AI summary

Camera-to-Quadruped Robot Pose Estimation from Sequential Images

Seobin Hwang, Ji-Seong Beom, Yeong-Jun Cho

arXiv:2610.36599v1cs.ROcs.CV

TL;DR

External-camera monitoring of quadrupeds needs full-body pose, but monocular video and articulated, often occluded limbs make estimation ambiguous, and no public benchmark exists. QuadPose estimates an 18-DoF pose from video and QuadSyn supports evaluation; on QuadSyn, QuadPose reports strong pose accuracy at about 33 FPS.

  • Problem

    Monocular external-camera video makes quadruped full-body pose estimation challenging because articulated, symmetric limbs and self-occlusion obscure keypoint correspondence, and no public benchmark exists.

    Source: §1 · pp. 2–3, §3 · p. 5

  • Method

    QuadPose estimates a quadruped’s 18-DoF full-body pose from monocular video using keypoint-focused selection, skeleton-aware attention, temporal pose prediction, and refinement.

    Source: §1 · pp. 2–3

  • Results

    95.54% PCK-AUC and 66.98% ADD-AUC; QuadPose achieves the best performance on QuadSyn and runs at about 33 FPS.

    Source: §1 · p. 3, §6 · p. 15

  • Takeaways & Limitations

    QuadSyn provides a benchmark with full-body annotations for reproducible evaluation of camera-to-quadruped pose estimation.

    Source: §1 · p. 3

    The method assumes a URDF-based forward-kinematics model and per-frame robot 2D bounding boxes are given.

    Source: §3 · p. 5

Abstract

from arXiv · show

Quadruped robots are increasingly deployed in hazardous environments such as disaster response and facility inspection, where external-camera monitoring is essential for safe operation. This setting is challenging because quadrupeds are articulated systems with time-varying leg joints, leg symmetry, and frequent self-occlusions, which make keypoint correspondence ambiguous. Moreover, no public benchmark exists, making quantitative comparison difficult. To address these challenges, we propose QuadPose, which estimates the full-body 18-DoF pose (6-DoF base pose + 12-DoF joint angles) from monocular video sequences, and introduce QuadSyn, the first camera-to-quadruped robot pose benchmark. On QuadSyn, QuadPose achieves 92.18% PCK@2.5px, 95.54% PCK-AUC, 85.94% ADD@20cm, 66.98% ADD-AUC, and 3.65 degree mean joint-angle error at approximately 33 FPS.

1 Introduction

External-camera monitoring of quadrupeds requires full-body 3D pose estimation, but monocular video, articulation, symmetry, and self-occlusion make this difficult. QuadPose estimates an 18-DoF pose from video, and QuadSyn provides a benchmark for evaluation.

1 Introduction

  • External-camera monitoring needs full-body 3D pose, since image-plane location alone cannot show whether a robot is stable, moving as intended, or approaching danger.The setting often provides only monocular video, without robot-side sensors or camera extrinsic calibration.Source: §1 · p. 2
  • Prior camera-to-robot methods use mesh alignment or keypoints, but often target fixed-base manipulators or rigid platforms that simplify pose estimation.Keypoint approaches may recover base pose through PnP or learn a direct image-to-pose mapping.Source: §1 · p. 2
  • Quadruped estimation requires 18 DoF—6-DoF base pose and 12 joint angles—and faces symmetric correspondences, gait-related self-occlusion, and no public benchmark.The framework uses 12 keypoints per frame.Source: §1 · p. 2
  • QuadPose uses keypoint-focused token selection for symmetric-limb ambiguity and skeleton-aware masked attention for self-occlusion, then predicts joint angles and base pose before refining reconstructed 3D keypoints.The pipeline reconstructs 3D keypoints with forward kinematics before its refinement head corrects residual errors.Source: §1 · pp. 2–3
  • QuadPose achieves 95.54% PCK-AUC and 66.98% ADD-AUC on QuadSyn while running at about 33 FPS.The paper reports consistent improvements over existing baselines.Source: §1 · p. 3
  • The paper contributes an 18-DoF full-body framework, keypoint-focused recovery components, and QuadSyn, which it describes as the first public camera-to-quadruped pose benchmark.QuadSyn is introduced to enable reproducible evaluation.Source: §1 · p. 3

2 Related Works

Camera-to-robot pose estimation recovers a robot’s position and orientation from monocular RGB, with articulated robots also requiring joint configurations. Prior work and datasets largely address manipulators or drones, leaving external-camera full-body quadruped estimation without a public dataset or evaluation protocol.

2 Related Works

  • Table 1 categorizes camera-to-robot methods as mesh-based or keypoint-based and summarizes their robot types, inputs, and outputs.Source: Table 1 · p. 3
  • Camera-to-robot estimation recovers position and orientation in the camera frame; articulated robots additionally require joint configurations, unlike rigid platforms.Quadrupeds combine a 6-DoF base pose with 12 leg-joint angles.Source: §2 · pp. 3–4
  • Mesh-based methods align images with rendered CAD models, while keypoint-based methods use detected joints, kinematic structure, and 2D–3D correspondences.The cited discussion notes that mesh methods require iterative test-time optimization; keypoint approaches include PnP-based recovery and structure-aware methods.Source: §2 · p. 4
  • Prior work typically estimates joint angles for fixed-base manipulators or treats drones as rigid bodies; the paper identifies no prior work jointly estimating base motion and articulation.Existing datasets and protocols mainly target manipulators and drones, not external-camera full-body quadruped pose.Source: §2 · p. 4

3 Proposed Methods

QuadPose estimates a quadruped’s 18-DoF state from image sequences using a 2D keypoint detector followed by a 3D pose estimator. Its pipeline uses joint-aware attention, temporal base-pose estimation, forward kinematics, and refinement.

3 Proposed Methods

  • QuadPose models each frame with a 6-DoF base pose and 12 joint angles, assuming a URDF-based forward-kinematics model and per-frame robot bounding boxes.The framework combines a 2D keypoint detector with a 3D pose estimator.Source: §3 · pp. 4–5

3.1 2D Keypoint Detector

  • Learnable selector tokens attend to joint-specific visual regions, while skeleton-aware masked attention restricts information exchange to connected joints.The detector encodes cropped robot images and predicts 12 joint keypoints.Source: §3.1 · pp. 5–7
  • The detector predicts keypoint coordinates from independent 1D distributions along the x and y axes rather than direct coordinate regression.The coordinates are computed as expectations of the predicted distributions.Source: §3.1 · p. 7

3.2 3D Pose Estimator

  • The base-pose head uses camera rays, predicted joint angles, and temporal features from consecutive frames to estimate the 6-DoF base pose.Learnable weighted pooling prioritizes frames that provide stronger depth and translation cues.Source: §3.2 · pp. 7–8
  • Forward kinematics reconstructs 3D keypoints from the estimated base pose and joint angles, and a refinement module predicts residual keypoint offsets.The refinement input combines coarse 3D keypoints, 2D keypoints, and predicted joint angles.Source: §3.2 · p. 8

3.3 Loss Functions

  • The estimator loss combines kinematic, base-pose, and refinement terms, while the 2D detector is trained independently.The kinematic term covers joint angles and FK-derived keypoints; refinement losses include refined-keypoint accuracy, reprojection error, and residual-size regularization.Source: §3.3 · pp. 8–9

4 QuadSyn: A New Benchmark for Quadruped Robots

QuadSyn addresses the lack of a public camera-to-quadruped pose benchmark with a synthetic dataset of robot images and pose annotations. Its data pipeline is designed to make rendered images resemble real camera observations.

4 QuadSyn: A New Benchmark for Quadruped Robots

  • QuadSyn is introduced as the first benchmark dataset for 18-DoF camera-to-quadruped pose estimation, addressing the lack of public images with pose-estimation labels.The missing labels include 2D/3D joint annotations and camera setup information.Source: §4 · p. 9
  • The rendering pipeline is designed to make synthetic images closely match real camera observations, reducing the domain gap; the authors report transfer to real data without domain adaptation.The real-data transfer is confirmed in Section 5.4.Source: §4 · p. 10
  • QuadSyn contains 51,697 synthetic 1280 × 720 RGB images across 52 sequences and background scenes, generated in NVIDIA Isaac Sim.The dataset uses Boston Dynamics Spot and panoramic textures built from collected real-world images; robot trajectories, gaits, and camera-relative poses vary across scenes.Source: §4 · p. 10
  • Each frame is annotated with 2D/3D leg keypoints, base pose, joint angles, camera parameters, and a 2D bounding box, with 2D keypoints projected from 3D for correspondence.Figure 5 shows the annotations and example scenes with 30-frame leg-keypoint trajectories and robot paths.Source: Figure 5 · p. 9, §4 · p. 10

5 Experimental Results

On QuadSyn, QuadPose outperforms baselines across 2D keypoint, 3D base-pose, and joint-angle evaluation; ablations isolate detector and estimator gains, while Harper tests real-world 2D transfer.

5.1 Settings

  • QuadSyn supplies diverse synthetic scenes with 2D/3D keypoints, base poses, and joint angles; Harper evaluation is limited to manually annotated 2D keypoints.Harper lacks external-view camera intrinsics and robot annotations; the pipeline runs at approximately 33 FPS.Source: §5.1 · pp. 10–11

5.2 Results on QuadSyn Dataset

  • QuadPose scores 92.18 PCK@2.5 versus DREAM's 24.45, leads ADD accuracy at all thresholds, and has 3.65° mean joint-angle error.It achieves the lowest error on 11 of 12 joints.Source: Table 2 · p. 11, §5.2 · pp. 11–12, Table 3 · p. 12
  • QuadPose localizes keypoints more precisely than SGTAPose under self-occlusion and symmetric leg configurations, and has lower MPJPE than RoboPEPP across viewpoints and gait patterns.The comparisons are visualized in Figures 6 and 7.Source: Figure 6 · p. 12, §5.2 · p. 12, Figure 7 · p. 13

5.3 Ablation Study

  • SimCC adds +7.99 PCK@2.5 over encoder-only; +ST reaches 91.77 PCK@2.5, and +SA reaches 95.54 AUC.With +SA, median error is 0.41 px; PCK@5/10 drops slightly near saturation.Source: Table 4 · p. 13, §5.3 · p. 13
  • Freezing JointNet raises ADD@10 from 45.40 to 53.80; temporal modeling reaches 67.47, and refinement reaches 69.49 ADD@10 and 66.98 AUC.The study attributes the temporal gain to multi-frame context and the refinement gain to residual correction.Source: Table 5 · p. 13, §5.3 · p. 14

5.4 Results on Real-world Data: Harper

  • On Harper, QuadPose exceeds YOLOv8 by +3.36 OKS mAP (64.13) and reduces temporal jitter from 171.5 to 24.32.The model was trained only on synthetic data without explicit domain adaptation; Harper evaluation is 2D-only.Source: Table 6 · p. 14, §5.4 · pp. 14–15

6 Conclusions

QuadPose recovers quadruped 18-DoF pose from external monocular RGB video, and QuadSyn provides the first benchmark with full-body annotations. QuadPose achieves the best performance on QuadSyn; qualitative Harper results suggest promising sim-to-real transfer.

6 Conclusions

  • QuadPose achieves the best performance on QuadSyn by improving 2D keypoint localization under leg symmetry and self-occlusion and leveraging temporal modeling.The keypoint-focused selector tokens and skeleton-aware masked attention improve robustness under challenging test scenarios.Source: §6 · p. 15
  • QuadPose recovers the 18-DoF full-body state from external monocular RGB video sequences, while QuadSyn is the first benchmark for this task with full-body annotations.Source: §6 · p. 15
  • Qualitative results on Harper suggest promising sim-to-real transfer in external-camera monitoring scenarios.Source: §6 · p. 15
  • Future work includes extending QuadSyn to more diverse motions and robot platforms and collecting real-world 3D ground truth.Source: §6 · p. 15

Related papers

Loading 2610.36599v1…