PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision as Proprioception: Estimating Humanoid Robot Pose from Video

Jiaxi Jiang, Matthias Heyrman, René Zurbrügg, Marco Hutter, Christian Holz

arXiv:2610.25288v1cs.ROcs.AIcs.CV

TL;DR

When onboard state is inaccessible and video lacks joint-state labels, estimating humanoid pose from video remains a challenge. HumanoidPose combines human-pretrained features and robot kinematics with synthetic supervision and detector self-training. It improves pose accuracy over evaluated baselines on real humanoids, while multi-robot scenes with sustained mutual occlusion remain unexplored.

  • Problem

    Humanoid pose recovery from video must work without onboard state and address missing joint-state labels, differing robot kinematics, occlusion, depth ambiguity, and camera motion.

    Source: §I · p. 1

  • Method

    HumanoidPose combines frozen human-pretrained features with robot-kinematic queries in a shared decoder and uses synthetic data plus self-training on unlabeled videos to adapt its detector.

    Source: §I · p. 2

  • Results

    HumanoidPose improves joint-angle and joint-position accuracy over evaluated baselines on real humanoids and supports transfer and adaptation to unseen embodiments.

    Source: §I · p. 2, §VI · p. 7

  • Takeaways & Limitations

    The findings support vision as an external source of robot-state information when onboard measurements are inaccessible.

    Source: §VI · p. 7

    The evaluation focuses on individual robots; multi-robot scenes with sustained mutual occlusion and downstream effects on prediction, coordination, and learned behaviors require further evaluation.

    Source: §V · p. 7

Abstract

from arXiv · show

Estimating a humanoid robot's internal state from video has a wide range of real-world applications but remains unexplored. During interactions with a robot, its onboard state information may be unavailable due to privacy, ownership, or communication constraints. Meanwhile, online and AI-generated robot videos provide rich sources of data but typically lack corresponding joint configurations, limiting their direct use in robot learning. To address these issues, we present HumanoidPose, a new framework that treats vision as proprioception to recover full-body joint configurations in world space from video alone. Specifically, we combine pretrained human-pose features with robot kinematics in a unified model that predicts each embodiment's joint configurations. We retarget human motions to sixteen different humanoid embodiments, enabling scalable learning entirely from synthetic data. To improve real-world generalization, we further leverage unlabeled online videos through self-training to develop a 2D robot-pose detector. In our evaluation, HumanoidPose improves joint-angle and joint-position accuracy over the evaluated baselines and reduces joint-angle error by 18-32% relative to the best-performing baseline on each of the three real humanoid platforms. We further show improved global localization and generalization to unseen embodiments. HumanoidPose enables visual understanding of other robots during interaction and extraction of robot motion from online and generated videos.

I. INTRODUCTION

The paper frames video-based humanoid pose recovery as visual proprioception for settings where onboard state is inaccessible and video lacks joint-state labels. HumanoidPose combines human-pretrained features with robot kinematics and synthetic supervision to estimate poses across embodiments.

I. INTRODUCTION

  • Video-based recovery addresses settings where onboard measurements are inaccessible and online or AI-generated videos lack joint-state logs, although such videos capture motion in varied environments.Recovering motion in robot joint coordinates makes it available for analysis, prediction, and learning.Source: §I · p. 1
  • Humanoid pose recovery must accommodate differing body proportions, joint layouts, and mechanical limits while resolving occlusion, monocular depth ambiguity, and camera motion.The paper identifies learned priors that generalize across embodiments, motions, and visual conditions as needed for these challenges.Source: §I · p. 1
  • The framework combines frozen human-pretrained image features with robot-kinematic queries in a shared decoder that predicts joint configurations and root pose across embodiments.It combines the predicted root pose with estimated camera motion to recover world-space trajectories.Source: §I · p. 2
  • The synthetic pipeline generates 128,000 images across sixteen humanoids with paired 2D and 3D labels, then adapts its 2D detector through self-training on online videos.The paired labels train the 3D pose model and 2D detector.Source: §I · p. 2
  • HumanoidPose improves joint-angle and joint-position accuracy on three real humanoid platforms, with improved root translation on G1 and transfer and adaptation to unseen embodiments.The reported comparison is against evaluated baselines.Source: §I · p. 2

II. RELATED WORK

Prior work addresses robot pose estimation, human pose and motion recovery, and synthetic-data transfer, but humanoid full-body robot-state recovery requires direct estimation of actuated coordinates across embodiments.

II. RELATED WORK

  • Robot pose work uses calibration, articulation refinement, geometric lifting, feed-forward regression, and learned articulation representations; cited humanoid methods estimate 2D image keypoints across robot platforms, not full 3D articulation or actuated joint configurations.Source: §II · p. 2
  • Human-body recovery methods use parametric models, kinematics, temporal priors, and camera-motion handling, but human representations remain indirect for robot-state recovery because humanoids have different morphology, joint limits, and actuated coordinates.HumanoidPose directly estimates robot joint coordinates and world-space motion from video.Source: §II · p. 2
  • Synthetic rendering, motion archives, domain randomization, and retargeting support scalable supervision, and HumanoidPose pairs rendered robot images with exact joint configurations to train entirely on synthetic images and transfer to real videos.The passage connects motion retargeting and robot-specific rendering to direct supervision of actuated coordinates.Source: §II · p. 2

III. METHOD

HumanoidPose takes video and a robot kinematic description to predict configurations and root pose, combining per-frame pose decoding with detector adaptation from unlabeled online videos. Synthetic retargeting and rendering provide the pose decoder’s supervision.

A. Problem Formulation and Overview

  • Each frame yields a configuration for the robot’s actuated joints and a root-to-camera transform, which are combined with camera motion to recover world-space trajectories.The input includes RGB video and a URDF or MJCF kinematic description; root estimation uses camera intrinsics and nominal metric robot height.Source: A. Problem Formulation and Overview · p. 2
  • At inference, the adapted detector crops the robot, and a frozen human-pretrained encoder and shared decoder use kinematic joint queries to predict configurations and root pose per frame.Temporal information supports detector self-training and camera estimation, while pose decoding itself is per-frame.Source: A. Problem Formulation and Overview · p. 2, Figure 2 · p. 3

B. Training Data and Detector Adaptation

  • Retargeted AMASS motions and randomized rendering produce paired images and robot states across sixteen humanoids, with camera intrinsics, root poses, joint configurations, and 2D annotations.A common compiled robot model keeps coordinate conventions, joint limits, forward kinematics, labels, descriptors, and training losses consistent.Source: B. Training Data and Detector Adaptation · p. 3
  • The detector is trained on synthetic images with 15 shared body landmarks, then adapted using pseudo-labeled frames from unlabeled online videos over two rounds; the 3D decoder trains solely on synthetic poses.Pseudo-label generation retains single-robot pairs with overlapping boxes and excludes evaluation recordings.Source: B. Training Data and Detector Adaptation · p. 3
  • At inference, the adapted detector defines a square RGB crop using a robot-height fraction of ρ = 0.55, and predicted 2D landmarks supply observations for the geometric fitting baseline.The 3D decoder predicts pose from crop features and robot descriptors.Source: B. Training Data and Detector Adaptation · p. 3

C. Kinematics-Conditioned 3D Pose Estimation

HumanoidPose conditions a shared pose decoder on robot-specific joint descriptors, enabling predictions across different humanoid kinematic structures. It combines visual features with joint and root-pose predictions, then trains these outputs with joint-, body-, and root-level supervision.

C. Kinematics-Conditioned 3D Pose Estimation

  • The shared encoder maps each actuated joint’s description to a query, allowing one decoder to handle variable joint layouts across embodiments.Each query encodes joint geometry, motion range, and position in the kinematic tree.Source: C. Kinematics-Conditioned 3D Pose… · pp. 3–4
  • The synthetic training corpus contains retargeted motion from sixteen humanoids spanning 0.60–1.78 m in height and 19–43 degrees of freedom.Images were rendered with randomized scenes, illumination, and viewpoints, paired with the configuration used to render each sample.Source: Figure 3 · p. 4
  • Joint descriptors combine local geometry and base-frame identity cues, with normalized offsets and robot scale supporting reuse on new embodiments.A shared MLP maps each 27-dimensional descriptor to a query; new embodiments supply their descriptors and output ranges while reusing learned weights.Source: C. Kinematics-Conditioned 3D Pose… · p. 4
  • Learned root and auxiliary tokens are shared across robots; four pre-normalized transformer blocks combine token self-attention, cross-attention to X, and feed-forward updates with residual connections, while a shared linear head maps normalized joint tokens to scalar zj.The root token predicts root pose; the auxiliary token has no separate prediction target.Source: C. Kinematics-Conditioned 3D Pose… · p. 4
  • The root-pose readout uses nominal robot height and crop intrinsics to estimate camera-relative translation, with learned corrections for posture and image evidence.The readout also predicts a 6D orientation; the offsets adjust the crop-center ray and log-depth.Source: C. Kinematics-Conditioned 3D Pose… · p. 4
  • Training uses joint-angle, forward-kinematics body-position, terminal-body orientation, and root-transform losses.Differentiable forward kinematics propagates body-position errors to joint predictions; terminal-orientation supervision addresses rotations that leave terminal body origins unchanged.Source: C. Kinematics-Conditioned 3D Pose… · p. 4

D. World-Space Motion Recovery

World-space recovery combines the predicted root-to-camera transform with the camera-to-world transform. Forward kinematics then places the articulated robot in world coordinates.

D. World-Space Motion Recovery

  • Combining the predicted root-to-camera transform with the camera-to-world transform expresses recovered poses in a common world frame.For a fixed camera, the camera-to-world transform is constant; for a moving camera, the method uses the PromptHMR camera pipeline with DROID-SLAM and metric depth for translation scale.Source: D. World-Space Motion Recovery · p. 5
  • Together with predicted joint configurations, the combined transform places the articulated robot in world coordinates through forward kinematics.The world frame is defined independently for each continuous shot.Source: D. World-Space Motion Recovery · p. 5
  • For moving-camera estimation, the robot is masked to exclude its motion from camera estimation.Metric depth supplies translation scale.Source: D. World-Space Motion Recovery · p. 5

IV. EXPERIMENTS

The experiments assess HumanoidPose as a visual source of robot-state information. They examine pose and global-motion recovery, generalization across embodiments and domains, design choices, and adaptation with additional visual data.

IV. EXPERIMENTS

  • The evaluation measures recovery of joint configurations and global motion from visual observations.The experiments frame HumanoidPose as an external source of robot-state information.Source: §IV · p. 5
  • The experiments examine accuracy and generalization across embodiments and domains.Source: §IV · p. 5
  • The evaluation also studies the contribution of the main design choices and adaptation with additional visual data.Source: §IV · p. 5

A. Experimental setup

The setup evaluates synthetic data and held-out real videos, with pose metrics computed separately from global root translation. Comparisons include adapted human-pose pipelines, keypoint fitting, and robot-pose models trained for each robot.

A. Experimental setup

  • Real evaluation uses held-out, encoder-synchronized videos of G1, PM01, and N1, with synchronized optical motion capture on G1 for global root motion.The videos include locomotion, arm movements, crouching, and supported poses; global root translation is evaluated separately.Source: A. Experimental setup · p. 5
  • MPJAE averages absolute wrapped joint-angle errors, while MPJPE averages Euclidean forward-kinematics body-origin errors in the robot base frame.Both root poses are fixed to zero translation and identity rotation for these metrics.Source: A. Experimental setup · p. 5
  • The baselines are PromptHMR followed by GMR, its synthetic-finetuned variant, keypoint fitting, HoRoPose, and RoboPEPP.The same real-data square crops use a robot-height fraction of 0.55; GMR retargeting is solved independently for each frame.Source: A. Experimental setup · p. 5
  • Keypoint fitting optimizes joint angles and base pose against 15 detected 2D keypoints using confidence-weighted reprojection error, forward kinematics, and joint limits.It starts from the zero configuration and a box-based translation estimate, shares detections and crops with other methods, and requires no training.Source: A. Experimental setup · p. 5
  • HoRoPose and RoboPEPP are adapted from manipulator evaluations and trained separately for each humanoid.HoRoPose keeps its original losses with a joint-angle branch registered per humanoid; RoboPEPP pretrains its embedding on the synthetic renders before finetuning.Source: A. Experimental setup · p. 5

B. Comparisons with Baselines

HumanoidPose outperforms evaluated baselines on synthetic and real humanoid pose estimation, and improves global root localization on G1 recordings. Its predictions cover varied motions and robot structures.

B. Comparisons with Baselines

  • 18–32% lower MPJAE than the best-performing baseline on each of three real platforms; HumanoidPose also leads MPJAE and MPJPE on each platform.Figure 4 shows recovered locomotion, arm movements, and supported poses across robot geometries.Source: B. Comparisons with Baselines · p. 5, Table I · p. 6, Figure 4 · p. 6
  • 40% lower root-mean-square error (RMSE) than height-scaled PromptHMR on G1 global root-position evaluation.Trajectories receive one rigid alignment without rescaling; evaluation uses phone-camera recordings and motion-capture ground truth.Source: B. Comparisons with Baselines · p. 5, Table II · p. 6
  • 35% lower average MPJAE than RoboPEPP on the held-out synthetic test set, with the lowest average MPJAE and MPJPE.The shared model leads both metrics on each reported embodiment.Source: B. Comparisons with Baselines · p. 5, Table I · p. 6
  • The root estimator uses known robot size and learns image-dependent position corrections to account for posture changes beyond fixed height-based scaling.Source: B. Comparisons with Baselines · p. 6

C. Ablation studies

Ablations indicate that direct robot-state prediction, kinematic conditioning, and human-pretrained features each contribute to performance. Few-shot synthetic data also reduces held-out G1 test error.

C. Ablation studies

  • Removing joint identity information increases errors in both domains, with a larger effect on synthetic data.The removed descriptor includes a base-frame rest anchor, joint axis, and side indicator.Source: C. Ablation studies · p. 6, Table III · p. 7
  • G1 test error decreases as the number of synthetic training examples increases.The horizontal axis is log(1 + N).Source: Figure 5 · p. 6
  • Replacing direct robot-state prediction with PromptHMR followed by GMR retargeting causes the largest increase in both errors among compared variants.The comparison covers synthetic data from sixteen robots and real data from three platforms.Source: C. Ablation studies · p. 6, Table III · p. 7
  • Replacing kinematic query conditioning with learned joint embeddings worsens both metrics in both domains, and replacing the human-pretrained encoder with DINOv2 increases both errors.The authors report that the geometric descriptors and human-pretrained features provide useful information for robot articulation.Source: C. Ablation studies · p. 6, Table III · p. 7

D. Transfer to new embodiments

The pose decoder transfers to held-out robot embodiments using their kinematic descriptions, and adding synthetic examples reduces held-out G1 test error. Detector self-training improves real-video coverage while retaining keypoint accuracy.

D. Transfer to new embodiments

  • 11.27° pooled real MPJAE on G1 and 11.44° on PM01, with median root errors of 8–16 cm across their sequences.The target robot's kinematic description supplies joint queries without learning new embeddings; the trained detector includes target-robot synthetic images and public videos.Source: D. Transfer to new embodiments · p. 7
  • G1 synthetic test error decreases when the held-out decoder is finetuned with more synthetic G1 data, without real pose labels.The image encoder remains frozen.Source: D. Transfer to new embodiments · p. 7

E. Self-training from online videos

  • Self-training improves real-video detector coverage while retaining keypoint accuracy and yielding the lowest pose error across platforms.Synthetic finetuning first improves keypoint localization; the comparison holds the pose decoder fixed.Source: E. Self-training from online videos · p. 7

V. LIMITATIONS

HumanoidPose improves real-humanoid pose accuracy over evaluated baselines and supports transfer to new embodiments, but multi-robot occlusion and downstream behavioral effects remain unresolved.

V. LIMITATIONS

  • Evaluation focuses on individual robots; multi-robot scenes with sustained mutual occlusion remain unexplored and require identity maintenance and pose consistency under intermittent visual evidence.The authors identify temporal modeling and tracking as motivating needs for these scenes.Source: §V · p. 7
  • Further evaluation is needed to establish how pose errors affect motion prediction, coordination, and learned behaviors.Source: §V · p. 7

VI. CONCLUSION

  • Experiments show improved pose accuracy over evaluated baselines and support transfer and adaptation to new embodiments.The authors present vision as an external source of robot-state information when onboard measurements are inaccessible.Source: §VI · p. 7

Related papers

Loading 2610.25288v1…