PaperlayerView exact revision on arXiv

Source-linked AI summary

MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos

Kehong Gong, Zhengyu Wen, Weixia He, Mingxi Xu, Qi Wang, Ning Zhang, Zhengyu Li, Dongze Lian, Wei Zhao, Xiaoyu He, Mingyuan Zhang

arXiv:2512.10881v2cs.CV

TL;DR

Species- and template-specific motion-capture pipelines leave a gap for animating arbitrary rigs from monocular video. MoCapAnything predicts joint trajectories from asset and video cues, then recovers asset-specific rotations with IK; it reports lower average test error than GenZoo and cross-species retargeting.

  • Problem

    Most motion-capture pipelines remain tied to a species or template, while creators need animation for heterogeneous and non-biological assets.

    Source: §1 · pp. 1–2

  • Method

    MoCapAnything encodes an asset and monocular video to predict 3D joint trajectories, then uses inverse kinematics to recover asset-specific rotations.

    Source: §1 · p. 2

  • Results

    0.4580 to 0.2549 average error on the overall test set, with lower CD-Skeleton errors than GenZoo across all categories.

    Source: §7 · p. 13

  • Takeaways & Limitations

    The framework reports in-the-wild generalization and cross-species motion capture and retargeting across heterogeneous rigs.

    Source: §1 · p. 2

    Performance depends on the pretrained image-to-3D reconstructor and a rig with known joint structure; the method primarily operates in camera space without explicit physics or contact reasoning.

    Source: §5 · p. 9

Abstract

from arXiv · show

Motion capture now underpins content creation far beyond digital humans, yet most existing pipelines remain species- or template-specific. We formalize this gap as Category-Agnostic Motion Capture (CAMoCap): given a monocular video and an arbitrary rigged 3D asset as a prompt, the goal is to reconstruct a rotation-based animation such as BVH that directly drives the specific asset. We present MoCapAnything, a reference-guided, factorized framework that first predicts 3D joint trajectories and then recovers asset-specific rotations via constraint-aware inverse kinematics. The system contains three learnable modules and a lightweight IK stage: (1) a Reference Prompt Encoder that extracts per-joint queries from the asset's skeleton, mesh, and rendered images; (2) a Video Feature Extractor that computes dense visual descriptors and reconstructs a coarse 4D deforming mesh to bridge the gap between video and joint space; and (3) a Unified Motion Decoder that fuses these cues to produce temporally coherent trajectories. We also curate Truebones Zoo with 1038 motion clips, each providing a standardized skeleton-mesh-render triad. Experiments on both in-domain benchmarks and in-the-wild videos show that MoCapAnything delivers high-quality skeletal animations and exhibits meaningful cross-species retargeting across heterogeneous rigs, enabling scalable, prompt-driven 3D motion capture for arbitrary assets. Project page: https://animotionlab.github.io/MoCapAnything/

1. Introduction

Existing motion-capture pipelines are often tied to a species or template, leaving creators without a general route from monocular video to animation for arbitrary rigs. The paper frames CAMoCap, introduces MoCapAnything and Truebones Zoo, and reports cross-species motion capture and retargeting.

1. Introduction

  • CAMoCap asks for rotation-based animation of an arbitrary rigged asset from monocular video, addressing the limits of species- and template-specific pipelines.The paper identifies uses including heterogeneous asset libraries, non-biological rigs, and avatars with changing topology.Source: §1 · pp. 1–2
  • Truebones Zoo contains 1,038 motion clips, each bundling a standardized rigged skeleton, mesh, and asset-aligned rendered video.The benchmark uses 60 held-out test clips and 978 training clips.Source: §1 · p. 2
  • CAMoCap must handle asset-local rotations across diverse rest poses, inject target-asset information into video-based keypoint estimates, and bridge dense RGB features with joint structure.The paper notes that naive multimodal bridging may reduce accuracy.Source: §1 · p. 2
  • MoCapAnything factorizes motion into 3D joint trajectories and asset-specific rotations, and the paper reports in-domain accuracy, in-the-wild generalization, and cross-species retargeting.Its modules use asset prompts, video and mesh cues, a motion decoder, and IK fitting.Source: §1 · p. 2

2. Related Works

Prior work spans human-specific pose and motion capture, category-agnostic 2D keypoint estimation, and species-specific animal reconstruction. The related-work section emphasizes that these lines do not jointly provide temporally coherent, animation-ready motion for arbitrary skeletons.

2.1. Pose Estimation

  • Human pose-estimation methods remain coupled to predefined human skeletons and keypoint sets, while CAPE-family approaches extend keypoint localization to unseen categories but remain static and 2D.The review also distinguishes heatmap, regression, query-based, and vision-language approaches, and describes support-based, matching, graph, and text-prompt CAPE variants.Source: §2.1 · pp. 2–3

2.2. Motion Capture

  • Human motion-capture pipelines recover parametric human models or global motion, but remain tied to a human template and can entangle camera movement with motion.The review contrasts camera-space monocular methods with calibrated multi-camera, inertial, and SLAM/visual-odometry approaches.Source: §2.2 · p. 3
  • Animal reconstruction includes model-free deformable-surface methods and model-based approaches, but the latter rely on species-specific templates such as SMAL and do not cover diverse nonanimal rigs.The review notes that video-based work extends model-free reconstruction beyond image collections.Source: §2.2 · p. 3

3. Method

The method targets motion capture and retargeting for arbitrary skeletal topologies from monocular video and a rigged asset. It predicts joint trajectories from reference and video cues, then uses inverse kinematics to recover asset-specific rotations.

3.1. Task Formulation

  • CAMoCap reconstructs motion for arbitrary 3D assets from monocular video, supporting both same-skeleton motion capture and retargeting when the asset differs from the video subject.The rig specification includes a mesh, a joint set with directed parent-child edges and offsets, canonical labels, and optional reference images.Source: §3.1 · p. 3

3.2. Overview

  • MoCapAnything avoids direct rotation regression by estimating 3D joint trajectories first and recovering rotations through IK, addressing asset-dependent angles, under-constrained monocular evidence, and temporal discontinuity.Its pipeline combines an asset prompt encoder, video features, a motion decoder, and IK fitting.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3.1. Reference Prompt Encoder

  • The Reference Prompt Encoder forms per-joint queries from skeleton, mesh geometry, and appearance cues, using topology-aware attention and cross-attention to asset information.Masked attention accommodates variable joint counts.Source: §3.2 · p. 4, §3.3.1 · pp. 4–5

3.3.2. Video Feature Extractor

  • The Video Feature Extractor pairs dense visual descriptors with a coarse deforming 4D mesh, whose geometry cues bridge RGB features and joint space.The visual stream uses a frozen DINOv2 encoder, while the geometry stream comes from a pretrained image-to-3D reconstructor.Source: §3.3.2 · p. 5

3.3.3. Unified Motion Decoder

  • The Unified Motion Decoder combines reference queries, visual tokens, and mesh tokens through topology-aware and temporal attention to predict joint-position trajectories.Its stages include graph-based self-attention, temporal video and point-cloud cross-attention, and per-joint temporal self-attention.Source: §3.3.3 · p. 5

3.4. Training Objective

  • Training uses masked joint-position regression for variable-size skeletons, without rotation-space or explicit temporal losses.Rotations are obtained afterward through the IK stage.Source: §3.4 · p. 5

3.5. IK Fitting Process

  • IK fitting initializes rotations by aligning rest-pose bone directions with predicted joint positions, then refines them to reduce forward-kinematics discrepancy while regularizing toward the initialization.Warm-starting from the previous frame supports temporal stability and suppresses unnecessary twist.Source: §3.5 · pp. 5–6

4. Experiments

On Truebones Zoo, MoCapAnything outperforms adapted baselines across seen, rare, and unseen species; ablations and qualitative results show robust motion prediction and retargeting across diverse rigs and settings.

4.1. Dataset and Evaluation Protocol; 4.2. Evaluation Metrics

  • Evaluation uses 60 stratified test sequences from 1,038 motion clips and reports MPJPE and MPJVE in centimeters after scale normalization.The test groups are seen, rare, and unseen species; ground-truth meshes provide joint positions for quantitative evaluation.Source: §4.1 · p. 6, §4.2 · p. 6

4.3. Compare with Baseline

  • MoCapAnything consistently outperforms adapted baselines on the seen, rare, and unseen Truebones Zoo splits.The baselines were retrained with unified skeleton representations and input modalities.Source: §4.3 · p. 6, Table 1 · p. 6

4.4. Ablation Study

  • Removing any modality or module reduces performance, especially on rare and unseen species; mesh and graph-attention branches support robust transfer to new species.The ablation tests remove image-set encoding, mesh features, or graph multi-head attention.Source: §4.4 · p. 6, Table 2 · p. 6

4.5. Qualitative Results on Truebones Zoo

  • Truebones Zoo examples show same-species mocap and retargeting to three other species, with temporally consistent, anatomically plausible motion despite appearance and shape variation.Figure 4 rows show the input, same-species results, and cross-species retargeting.Source: §4.5 · p. 7, Figure 4 · p. 7

4.6. Qualitative Results on Objaverse

  • The framework also supports human-like rigs, including human mocap and motion transfer between humans and animals.The examples include human-to-animal and animal-to-human retargeting.Source: §4.6 · p. 7

4.7. In-the-Wild Generalization

  • Across wild videos of birds, quadrupeds, and other animals, the method reconstructs plausible 3D skeletal motion for mocap and retargeting.The wild examples include chickens, eagles, seagulls, tigers, elephants, crabs, fish, and snakes.Source: §4.7 · pp. 7–8, Figure 5 · p. 8

4.8. Visualization of IK-Driven Asset Animation

  • Multi-view examples pair predicted skeletons with IK-driven asset animations for an ostrich, a goat, a crab, and a dog.The examples span bipedal, quadrupedal, and in-the-wild cases.Source: Figure 3 · p. 7, §4.8 · p. 8

4.9. Arbitrary Cross-Species Retargeting

  • Unconstrained transfer to unrelated rigs is not explicitly trained; without directly comparable baselines, evidence is qualitative, including creative, plausible motions.Examples include bird motion driving quadrupeds or pterosaurs and fish motion transferred to crocodiles or snakes.Source: §4.9 · p. 8, Figure 7 · p. 8

5. Conclusion

The conclusion presents CAMoCap and MoCapAnything as a prompt-based approach to animation across diverse rigs, while noting dependencies and scope limits.

5. Conclusion

  • CAMoCap uses a monocular video and arbitrary rigged asset as prompts to generate rotation-based animation tailored to the character.MoCapAnything predicts 3D joint trajectories, then recovers asset-specific rotations through constraint-aware inverse kinematics.Source: §5 · p. 8
  • The 1,038-clip Truebones Zoo benchmark provides standardized skeleton, mesh, and rendered-video triples for evaluation.The conclusion reports 60 test sequences.Source: §5 · p. 8
  • Performance depends on the pretrained image-to-3D reconstructor and a rig with known joint structure; motion is primarily camera-space without explicit physics or contact reasoning.Future work includes contact- and physics-aware IK and world-grounded trajectory recovery.Source: §5 · pp. 8–9

6. More Visualization Results

The visual results cover controlled multi-species motion, in-the-wild videos, cross-species retargeting, and IK fitting. They report smoother, coherent motion across varied skeletons and demonstrate transfers between animal and human rigs.

6. More Visualization Results

  • Visualizations span controlled multi-species datasets and in-the-wild videos, including flying birds and crocodiles.Objaverse examples include bipedal characters alongside diverse animal skeletons from Truebones Zoo.Source: §6 · p. 12
  • Retargeting examples include Zoo2Zoo, Human2Zoo, Zoo2Human, and animal-video motions transferred to human skeletons.The examples cover variation in morphology, topology, and motion dynamics.Source: §6 · p. 12
  • IK visualizations report approximately 17° average geodesic rotation error and improved temporal stability and orientation consistency.The stated improvements use geometric initialization, temporal warm-starting, and twist-regularized refinement.Source: §6 · p. 12
  • MoCapAnything produces smoother trajectories and stable, anatomically plausible motion across quadruped and non-quadruped skeletons, while GenZoo struggles beyond quadrupeds.The comparisons include birds, reptiles, bipeds, and non-biological assets.Source: Figure 8 · p. 12, §6 · p. 12

7. More Experiment Results

On Truebones Zoo, MoCapAnything achieves lower CD-Skeleton errors than GenZoo across categories, with qualitative comparisons showing smoother motion and broader skeletal coverage.

7. More Experiment Results

  • 0.4580 to 0.2549: MoCapAnything reduces average CD-Skeleton error on the overall test set compared with GenZoo.Table 4 reports Chamfer Distance results on Truebones Zoo.Source: Table 4 · p. 12, §7 · pp. 12–13
  • The comparison uses CD-Skeleton because GenZoo does not produce joint-aligned skeletons compatible with MPJPE or MPJVE evaluation.The reported metric measures structural accuracy of predicted skeletal motion.Source: §7 · p. 12
  • Figure 8 compares Truebones Zoo predictions and shows smoother trajectories and stable motion across quadruped and non-quadruped skeletons.The examples include bipeds, birds, reptiles, and non-biological assets; GenZoo relies on quadruped templates and struggles with broader categories.Source: §7 · p. 13

A. Dataset and Training Details

The section describes a heterogeneous benchmark and training setup, then details an IK fitting procedure that combines geometric initialization, temporal continuity, and twist suppression to recover stable rotations.

A. Dataset and Training Details

  • The 60-sequence benchmark includes 27 mammals, 9 birds, 12 dinosaurs+dragons, 7 reptiles, 2 aquatic animals, and 3 arthropods.Rest-pose meshes are normalized to unit volume; sequences are translated and scaled into [−1, 1]3, and in-the-wild inputs assume a fixed camera.Source: A. Dataset and Training Details · p. 13
  • Training uses same-species reference assets and 24-frame videos, supervises joint positions, and applies lightweight IK to recover rotations.The decoder has 12 layers, the prompt encoder 4 layers, and training runs for 60 epochs; sliding windows support arbitrarily long inference videos.Source: A. Dataset and Training Details · p. 13
  • Retargeting pairs are not used for training, yet the reference-conditioned representation supports cross-species retargeting at inference.Ground-truth mesh sequences are used during training; predicted video meshes at inference show negligible degradation relative to GT-mesh conditioning.Source: A. Dataset and Training Details · p. 13

B. Inverse Kinematics Fitting

  • Because forward kinematics is not injective, position constraints alone leave local orientation—especially bone-axis twist—underdetermined.The fitting combines geometric initialization, temporal warm-starting, and differentiable refinement with twist suppression.Source: B. Inverse Kinematics Fitting · p. 13
  • Closed-form initialization aligns rest-pose and observed bone directions, using axis–angle rotation for single-child joints and orthogonal Procrustes for multi-child joints.The latter provides orientations at branching structures such as the pelvis and shoulders.Source: B. Inverse Kinematics Fitting · p. 13
  • Refinement warm-starts each frame from the previous solution and penalizes bone-axis twist, suppressing candy-wrapper artifacts while preserving natural motion around long chains.The FK loss matches reconstructed joints to predicted positions, and the combined procedure yields stable, anatomically consistent rotations.Source: B. Inverse Kinematics Fitting · pp. 13–14

9. Evaluation Metrics

CD-Skeleton compares articulated skeletons using distances between joints and the other skeleton’s bone segments. Its symmetric form balances the two directions and reflects structural differences in pose and proportions.

Notation

  • Each skeleton is represented by 3D joint positions and a kinematic hierarchy stored as a parent array; the metric assumes corresponding joint indices even if parent arrays differ.A parent index of −1 marks a root joint.Source: §9 · p. 14, Notation · p. 14

Distance From Joint to the Other Skeleton

  • For each joint of Skeleton A, the metric measures distance to the closest point on Skeleton B’s bone segments.The segments are defined by Skeleton B’s kinematic tree.Source: Distance From Joint to the Other… · p. 14
  • The closest point is obtained by orthogonal projection onto a bone segment, with the projection parameter clipped so it stays on the segment.The distance is computed in the reverse direction as well, from Skeleton B’s joints to Skeleton A.Source: Distance From Joint to the Other… · p. 14

Skeleton-to-Skeleton Distance

  • The symmetric skeleton distance combines the two directed distances, rather than treating either skeleton as the sole reference.The section defines both an asymmetric distance and a symmetric distance.Source: Skeleton-to-Skeleton Distance · p. 14

Interpretation

  • The metric captures differences in global pose, limb orientation, and proportions; the symmetric version is balanced when neither skeleton is the reference.Source: Interpretation · p. 14
Loading 2512.10881v2…