PaperlayerView exact revision on arXiv

Source-linked AI summary

MoCapAnything V2: End-to-End Motion Capture for Arbitrary Skeletons

Kehong Gong, Zhengyu Wen, Dao Thien Phong, Mingxi Xu, Weixia He, Qi Wang, Ning Zhang, Zhengyu Li, Guanli Hou, Dongze Lian, Xiaoyu He, Mingyuan Zhang, Hanwang Zhang

arXiv:2604.28130v3cs.CV

TL;DR

Arbitrary-skeleton motion capture remains challenging because joint positions alone do not determine rotations across rest poses and local coordinate conventions. MoCapAnything V2 jointly learns video-to-pose and pose-to-rotation using a reference pose–rotation pair, reducing average rotation angle error from ∼17◦ to ∼10◦ on Truebones Zoo and Objaverse.

  • Problem

    Arbitrary-skeleton motion capture is difficult because joint positions leave rotations, including bone-axis twist, ambiguous across rest poses and local coordinate conventions.

  • Method

    The framework jointly learns Video-to-Pose and Pose-to-Rotation, conditioning rotation prediction on the target asset’s rest pose and reference pose–rotation pair.

  • Results

    Average rotation angle error decreases from ∼17◦ with V1’s factorized learned-V→P plus analytical-IK pipeline to ∼10◦ on Truebones Zoo and Objaverse.

  • Takeaways & Limitations

    Direct video-to-pose prediction without a mesh intermediate achieves approximately 20× faster inference than mesh-based pipelines.

  • Takeaways & Limitations

    The decoder can drift toward typical per-skeleton configurations during unnatural retargeting, and occlusion cases remain unexplored.

Abstract

from arXiv · show

Recent methods for arbitrary-skeleton motion capture from monocular video follow a factorized pipeline, where a Video-to-Pose network predicts joint positions and an analytical inverse-kinematics (IK) stage recovers joint rotations. While effective, this design is inherently limited, since joint positions do not fully determine rotations and leave degrees of freedom such as bone-axis twist ambiguous, and the non-differentiable IK stage prevents the system from adapting to noisy predictions or optimizing for the final animation objective. In this work, we present the first fully end-to-end framework in which both Video-to-Pose and Pose-to-Rotation are learnable and jointly optimized. We observe that the ambiguity in pose-to-rotation mapping arises from missing coordinate system information: the same joint positions can correspond to different rotations under different rest poses and local axis conventions. To resolve this, we introduce a reference pose-rotation pair from the target asset, which, together with the rest pose, not only anchors the mapping but also defines the underlying rotation coordinate system. This formulation turns rotation prediction into a well-constrained conditional problem and enables effective learning. In addition, our model predicts joint positions directly from video without relying on mesh intermediates, improving both robustness and efficiency. Both stages share a skeleton-aware Global-Local Graph-guided Multi-Head Attention (GL-GMHA) module for joint-level local reasoning and global coordination. Experiments on Truebones Zoo and Objaverse show that our method reduces rotation error from ~17 degrees to ~10 degrees, and to 6.54 degrees on unseen skeletons, while achieving ~20x faster inference than mesh-based pipelines. Project page: https://animotionlab.github.io/MoCapAnythingV2/

1 Introduction

MoCapAnything V2 introduces a fully end-to-end arbitrary-skeleton motion-capture framework that jointly learns pose and rotation from monocular video. A reference pose–rotation pair anchors the skeleton-specific rotation coordinates, while direct joint prediction and skeleton-aware attention improve efficiency and generalization.

  • 1 Introduction: Arbitrary-skeleton motion capture remains challenging because rotations depend on rest poses and local axis conventions, unlike more skeleton-shared joint positions.The same joint positions can correspond to different rotations across skeletons, making direct video-to-rotation prediction prone to overfitting and poor generalization.
  • 1 Introduction: The proposed framework jointly learns Video-to-Pose and Pose-to-Rotation, replacing the conventional learned pose predictor plus analytical inverse-kinematics pipeline.This makes the full arbitrary-skeleton motion-capture system end-to-end trainable.
  • 1 Introduction: A reference pose–rotation pair, together with the rest pose, anchors the rotation coordinate system and converts pose-to-rotation prediction into a well-constrained conditional problem.The reference is available for any rigged skeleton and resolves ambiguity caused by different rest poses and local coordinate frames.
  • 1 Introduction: End-to-end rotation supervision reshapes the intermediate pose representation to support rotation recovery, allowing gradients to flow into the visual encoder.The pose representation is therefore optimized beyond positional accuracy alone.
  • 1 Introduction: Approximately 20× faster inference results from directly predicting joint positions from video instead of using noisy mesh intermediates, while GL-GMHA coordinates local kinematic and global motion reasoning.Experiments cover seen, rare, and unseen skeletons in Truebones Zoo, plus Objaverse, with especially strong rotation-accuracy improvements on unseen skeletons.

2 Related Work

Prior work estimates poses or reconstructs motion and deformable surfaces, but does not fully resolve pose-to-rotation ambiguity across varying skeleton definitions. MoCapAnything V2 addresses this with a reference pose–rotation pair and end-to-end learning of the V→P→R pipeline.

  • Existing approaches must bridge observable visual cues such as joint positions and skeleton-specific motion parameters such as rotations.
  • The method trains the entire V→P→R pipeline end-to-end rather than relying on a separate analytical rotation-recovery stage.This enables bidirectional coupling between pose and rotation for improved robustness and animation-ready motion synthesis.
  • Pose estimation: Existing pose-estimation methods span heatmap-based, DETR-style, and category-agnostic architectures, but operate in 2D or assume fixed keypoints.
  • Motion capture: Monocular motion-capture research includes parametric human models, feed-forward and transformer regressors, model-free reconstruction, and video-based deformable-surface methods.
  • MoCapAnything V2 resolves P→R ambiguity with a reference pose–rotation pair that defines the underlying coordinate system.This casts P→R as a learnable conditional problem.

3 Method

MoCapAnything V2 formulates arbitrary-skeleton motion capture as a jointly learnable Video-to-Pose-to-Rotation pipeline, using rest-pose and reference pose–rotation conditioning to resolve rotation-coordinate ambiguity. It predicts joint positions directly from video and uses shared skeleton-aware attention to coordinate local structure and global motion.

  • Problem formulation: The method targets arbitrary tree-structured skeletons with a single root, representing per-frame joint rotations in 6D and using root-relative 3D joint positions as an intermediate representation.Joint count and topology are otherwise arbitrary, with skeletons padded to 150 joints.
  • End-to-end architecture: Both Video-to-Pose and Pose-to-Rotation are learnable neural modules trained jointly end-to-end, replacing the prior non-differentiable analytical IK stage.This enables rotation-loss gradients to flow through predicted poses into the visual encoder.
  • Skeleton-aware attention: A shared Global-Local Graph-guided Multi-Head Attention module alternates kinematic-chain-restricted local layers with fully connected global layers to model limb dependencies and whole-body dynamics across skeleton topologies.The mechanism is shared by both stages and adds no additional parameters.
  • Learned modules and training objective: The Video-to-Pose module predicts joint positions directly from video, while the rotation decoder uses rest-pose conditioning, temporal attention, GL-GMHA spatial reasoning, and reference-anchor cross-attention.The total objective combines position, rotation, angular-velocity, and root-rotation terms.
  • Reference-conditioned rotation: Rest pose and a reference pose–rotation pair jointly specify local coordinate-frame origins and axes, converting the ambiguous pose-to-rotation mapping into a well-constrained conditional problem.The reference pair is sampled from the same asset and anchors the skeleton-specific local rotation convention.

4 Experiments … 4.5 Pose-to-Rotation Module Analysis

Experiments across diverse seen, rare, unseen, and non-animal skeletons show that reference-conditioned end-to-end learning improves rotation quality over learned baselines and V1’s factorized mesh-and-IK pipeline. Ablations further show that gradient coupling and explicit reference conditioning are especially important for unseen skeletons.

  • 4.1 Dataset and Evaluation Protocol: The evaluation spans Truebones Zoo’s 1,038 animal sequences and Objaverse’s 1,000 structurally distinct targets, including unseen species and non-animal skeletons.Truebones test sequences are stratified into Seen, Rare, and Unseen splits.
  • 4.1 Dataset and Evaluation Protocol: The study reports MPJPE, MPJVE, Ang. Err, and AngV Err, using normalized samples, unified evaluation scale, and per-joint masking for varying joint counts.The metrics cover spatial accuracy, velocity accuracy, rotation error, and angular velocity error.
  • 4.2 Comparison with Baselines: Reference-conditioned learning reduces learned-baseline rotation error from near 20◦ to approximately 10◦ across splits, avoiding artifacts such as joint spinning.Compared learned baselines receive the same reference inputs and use end-to-end V→P→R, but do not effectively exploit reference and topology cues.
  • 4.3 End-to-End vs. Factorized Design: Comparison with V1: Compared with V1, the method removes the mesh intermediate and replaces analytical IK with an end-to-end learnable V→P→R design.V1 combines a 4D mesh bridge with factorized learned V→P and analytical-IK P→R.
  • 4.3 End-to-End vs. Factorized Design: Comparison with V1: The mesh-free end-to-end model substantially outperforms V1 variants on rotation while achieving pose accuracy competitive with V1 using ground-truth mesh.V1’s predicted-mesh configuration propagates noise through the mesh bridge, whereas ground-truth mesh is unavailable at inference.
  • 4.4 Training Strategy Analysis: Validating the End-to-End Claim: 7.82◦ to 6.54◦: joint training lowers Zoo-Unseen angle error when gradients flow from P→R back to V→P.GT-only training reaches 13.28◦ on Zoo-Unseen, while Pred-only training is unstable because early predictions are noisy; mixed-pose training gives the best balance.
  • 4.5 Pose-to-Rotation Module Analysis: 24.05◦–24.26◦: removing the reference pair causes a large Zoo-Unseen error increase, while rest-pose encoding supplies only the coordinate origin and leaves axes ambiguous.On Zoo-Seen and Zoo-Rare, all four reference/rest configurations remain comparable in the 10–15◦ range because axis conventions are represented in training.

4.6 Role of Pose as Intermediate Representation … 4.9 Qualitative Results

Ablations show that explicit joint positions and alternating local-global attention are essential for generalization across skeletons, while removing mesh prediction and analytical IK yields major efficiency gains. Qualitative results further show stable rotation recovery, cross-domain robustness, and unified retargeting across diverse skeletons.

  • 4.6 Role of Pose as Intermediate Representation: 6.54° on Zoo-Unseen: the explicit-pose model generalizes better than Direct (V→R) and Latent + Aux variants without an explicit pose intermediate.Direct (V→R) and Latent + Aux achieve 23.73° and 23.57° on Zoo-Unseen, respectively.
  • 4.6 Role of Pose as Intermediate Representation: Joint positions are skeleton-shared, whereas rotations are skeleton-dependent, making explicit pose a structural bottleneck between transferable motion and skeleton-specific parameterization.The explicit pose remains competitive on Seen/Rare while improving Zoo-Unseen performance.
  • 4.7 GL-GMHA Attention Analysis: 16.97° on Zoo-Rare and 11.60° on Zoo-Unseen: All-local attention performs worst by losing cross-branch coordination.Removing graph biases also trails uniform GMHA, while alternating GL-GMHA outperforms uniform GMHA across all four splits.
  • 4.7 GL-GMHA Attention Analysis: GL-GMHA improves attention by combining kinematic-chain locality with skeleton-global coordination, outperforming uniform GMHA across all four splits.All-local captures intra-branch dependencies but loses cross-branch coordination; uniform GMHA attends globally without explicitly modeling kinematic-chain locality.
  • 4.8 Efficiency Analysis: ∼20× speedup: for 120-frame sequences, V1 takes over 20 minutes, whereas the proposed method predicts pose and rotation in a single pass in ∼10 seconds and finishes under 1 minute.The gain comes from eliminating mesh prediction and the analytical IK solver while retaining the same feature-extraction cost.
  • 4.9 Qualitative Results: V2 produces natural, temporally continuous rotations, unlike V1’s frequent joint spinning and limb flipping on joints with large twist components.The comparison attributes V1’s failures to per-frame IK lacking temporal context and reference priors.
  • 4.9 Qualitative Results: Across Objaverse, Truebones Zoo, and in-the-wild videos, the method produces accurate, temporally consistent 3D motion despite varied appearance, shape, and capture conditions.The results are rendered from multiple viewpoints and indicate generalization to arbitrary subjects rather than one training distribution.
  • 4.9 Qualitative Results: A single input video can simultaneously drive mocap and retarget the same motion onto diverse humanoid and animal skeletons without skeleton-specific training.The retargeted motion preserves the rhythm and semantics of the source across the illustrated dance clips.

5 Limitations

The Pose-to-Rotation decoder learns per-skeleton motion priors that can limit unnatural motion retargeting. In such cases, predicted rotations may drift toward typical configurations for the target skeleton despite accurate Video-to-Pose recovery.

  • 5 Limitations: Unnatural retargeting can cause predicted rotations to drift toward typical configurations for the target skeleton, even when Video-to-Pose recovers the intended pose.The passage illustrates this with transferring a bird’s flapping motion to a dog, producing a dog that “flies” with its forelegs spread out.

6 Conclusion

The paper concludes with a fully end-to-end arbitrary-skeleton motion-capture framework that jointly learns Video-to-Pose and Pose-to-Rotation using a reference pose–rotation pair. It achieves lower rotation error and faster inference than prior pipelines, including on unseen skeletons.

  • Conclusion: The framework jointly trains learnable Video-to-Pose and Pose-to-Rotation modules for arbitrary-skeleton monocular motion capture.This is presented as the first fully end-to-end framework of its kind.
  • Conclusion: A single reference pose–rotation pair anchors each joint’s local coordinate axes and makes pose-to-rotation prediction well-constrained.The pair resolves ambiguity that rest-pose structure alone cannot disambiguate, particularly for unseen skeletons with differing axis conventions.
  • Conclusion: ∼20× faster inference results from removing the mesh intermediate used by prior work as a video-to-joint bridge.The paper attributes this to avoiding compounded predicted-mesh errors in the pipeline.

A Implementation Details

The model uses a frozen DINOv2 ViT visual backbone and jointly trains Video-to-Pose with Pose-to-Rotation end-to-end. Training processes 48-frame sequences with temporal attention windows of 5 and uses an 8-block rotation decoder with reference cross-attention in its first 6 layers.

  • A Implementation Details: The implementation freezes a DINOv2 ViT encoder and jointly optimizes the Video-to-Pose and Pose-to-Rotation modules end-to-end.The rotation decoder contains L=8 blocks, with reference cross-attention in its first L_cross=6 layers.
  • A Implementation Details: The model processes sequences of T=48 frames, using a temporal attention window of 5 for each joint.
  • A Implementation Details: Training uses Adam on 8× V100 GPUs for 60 epochs with batch size 8 and takes roughly one day.

B Warm-up Schedule Sensitivity

The mixed-pose warm-up length is robust across schedules from 20 to 50 epochs, with 30 epochs selected as the default.

  • B Warm-up Schedule Sensitivity: Performance remains consistent for mixed-pose warm-up lengths E_w∈[20, 50].This indicates robustness to the mixing schedule.
  • B Warm-up Schedule Sensitivity: The sensitivity study evaluates warm-up lengths E_w∈{10, 20, 30, 40, 50}.Table 7 reports angle and position metrics for these settings.
  • B Warm-up Schedule Sensitivity: The method uses E_w=30 as the default mixed-pose warm-up length.The default is chosen within the robust range.

C Model Depth

Jointly scaling both modules shows that model depth materially affects rotation accuracy: 8 layers perform best, while 12 layers degrade performance across all splits. The 8-layer configuration achieves 6.54° on Zoo-Unseen, indicating a favorable balance between capacity and optimization.

  • C Model Depth: Increasing depth from 6 to 8 layers improves rotation accuracy when scaling both modules jointly.The Video-to-Pose and Pose-to-Rotation modules are scaled together.
  • C Model Depth: Further increasing depth to 12 layers degrades performance across all splits.
  • C Model Depth: 6.54° on Zoo-Unseen is achieved by the 8-layer model, which delivers the best results.The result suggests a balance between model capacity and optimization.

D Cross-Attention Depth

The rotation decoder requires reference cross-attention, with six of eight layers providing the best reported Zoo-Unseen performance. Removing cross-attention causes failure, while using all layers offers no further gain and slightly degrades performance.

  • Cross-Attention Depth: 6.54° on Zoo-Unseen is achieved with reference cross-attention in 6 of the 8 rotation-decoder layers.The experiment varies reference cross-attention depth while keeping overall model depth fixed at 8 layers.
  • Cross-Attention Depth: 23.49° on Zoo-Unseen results without reference cross-attention, showing that reference conditioning is essential.This corresponds to L_cross=0.
  • Cross-Attention Depth: L_cross=8 provides no further gain and slightly degrades performance compared with the best setting.The best reported setting is L_cross=6.
Loading 2604.28130v3…