PaperlayerView exact revision on arXiv

Source-linked AI summary

Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary Rigs

Noam Rotstein, Yoad Tewel, Chen Tessler, Haggai Maron, Gal Chechik, Or Litany, Yoni Kasten, Yuval Atzmon

arXiv:2610.28632v1cs.CVcs.AIcs.GRcs.LGcs.RO

TL;DR

Video models produce motion as pixels, but do not provide arbitrary-rig 3D motion and camera registration needed to replay it in a scene. JCA jointly generates video, rig motion, and a fitted camera using separately pretrained models and curated video–motion data. It supports diverse-rig animation and related conditional and keyframe-controlled tasks, with stronger text-driven rig-animation results reported.

  • Problem

    Video models depict motion as pixels, while existing joint video–motion approaches are limited to human skeletons or indirect pose-map triangulation and do not provide arbitrary-rig motion registered to the generated scene.

    Source: §1 · p. 2

  • Method

    JCA couples a separately pretrained graph-based rig-motion model to a pretrained video generator, learns joint image locations for camera fitting, and trains on curated asset, human, and robot video–motion pairs.

    Source: §1 · p. 2

  • Results

    JCA outperforms video-first and mesh-based baselines in text-driven rig animation and supports scene-aware, conditional, keyframe-controlled, and image-based animation.

    Source: §1 · p. 2, §4 · p. 7

  • Takeaways & Limitations

    The model generates motion for arbitrary rigs alongside video and a camera, and can also generate either modality from the other or use image keyframes to control 3D motion.

    Source: §1 · p. 2

    Camera alignment assumes a static inference camera, and the model does not observe 3D scene geometry, so contact is not enforced and penetration is not prevented.

    Source: §6 · p. 10

Abstract

from arXiv · show

Video generation models can now depict remarkably realistic motion. Yet it is unclear how to use this motion beyond pixels: to animate a specific 3D asset and replay the animation in its 3D scene. For instance, a video model can show a monkey jumping over a crossbar, but it cannot return that jump as replayable 3D motion placed in the 3D scene. Here we present a model that directly co-generates the video and 3D motion for any supplied rig, along with a camera that registers this motion to the scene. Prior video-motion models either handle only human skeletons, often adding billions of parameters, or triangulate 3D motion from generated 2D pose maps. We instead keep 3D motion in a separate model that supports arbitrary skeletons, and connect it to a pretrained video model through a small set of cross-attention layers. This separation lets the motion model first learn how skeletons move from motion-only data, which is easier to obtain and augment than paired video. Additionally, our model learns to locate each joint in the video frames, which lets us recover the camera and place the 3D motion in the scene. To train across many skeletons, we use a generative video editor to place animated 3D assets in scenes suited to their actions, curating the edits so their known motion remains valid. Together with human and robot video, this yields nearly 200K training pairs. Our evaluations demonstrate that our model animates diverse rigs from text more faithfully than video-first pipelines, and its motion responds to the surrounding scene. It also turns video into rig motion, rig motion into video, and enables keyframe-based control of the 3D motion. Videos and additional results are available on our project page: https://research.nvidia.com/labs/par/projects/jca/.

1 INTRODUCTION

Video models produce convincing motion but not replayable 3D rig motion registered to the scene; existing joint approaches are limited to human skeletons or indirect 2D-pose triangulation. JCA co-generates video, arbitrary-rig motion, and a camera, supported by a curated training corpus and multiple generation and control modes.

1 INTRODUCTION

  • Video models provide pixels without underlying rig motion, while replay in a 3D scene also requires a camera transformation that registers motion to the scene.Prior joint models are human-only or recover 3D motion by triangulating 2D pose maps.Source: §1 · p. 2
  • JCA co-generates video, motion for arbitrary 3D rigs, and a camera that places the motion in the generated scene.Its motion model treats the skeleton as a graph and is pretrained on motion data before connection to a pretrained video model through a few cross-attention layers.Source: §1 · p. 2
  • Learned joint localization supplies 2D–3D correspondences for fitting the camera that registers generated motion to the scene.At inference, bridge guidance contrasts predictions with cross-modal connections enabled and disabled, then amplifies their interaction.Source: §1 · p. 2
  • The training corpus combines curated edited asset animations with human and robot video to reach 200K video–motion pairs.Because editing can invalidate 3D labels, results are verified and failures regenerated.Source: §1 · p. 2
  • Evaluations cover text-driven animation of diverse rigs, scene-responsive motion, video-to-motion and motion-to-video generation, and keyframe-based control.A few reference image keyframes can specify a complete 3D rig trajectory and matching video.Source: §1 · p. 2
  • The work contributes graph-based arbitrary-rig modeling, bridge guidance, learned scene registration, and a data pipeline for labeled asset animations in action-suited scenes.The same model can generate either video or motion from the other modality.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses joint video–motion generation, diverse-rig animation, conditional generation, motion recovery, and training data through distinct representations and pipelines. JCA couples separately pretrained video and rig-motion models and adds localization for camera fitting.

2 RELATED WORK

  • Joint video–motion methods include human-focused models and systems for hand–object or robot actions, while AnimaX handles arbitrary skeletons through multi-view pose-map triangulation.JCA instead couples separately pretrained video and text-to-rig-motion networks, generating 3D joint parameters directly without mesh input or multi-view triangulation.Source: §2 · p. 3
  • Diverse-rig animation work spans shared motion priors, mesh-based motion, vertex trajectories, video-prior animation, and methods that avoid per-asset optimization.JCA jointly generates motion and video through coupled denoising, without tracking or fitting motion to video.Source: §2 · p. 3
  • Prior conditional-generation and guidance methods use structural cues, motion constraints, or contrasts between selected model inputs or layers.JCA supports both conditional directions for arbitrary rigs and uses bridge guidance to contrast predictions with cross-attention enabled and disabled.Source: §2 · p. 3
  • Video-to-motion recovery methods include human-specific approaches and arbitrary-skeleton methods, while JCA performs recovery within its joint generative framework.Source: §2 · p. 3
  • Training-data approaches pair video with estimated human motion or use diverse-skeleton asset animations, whose renders may lack natural scene context.JCA uses Bernini to place rendered animations in suitable scenes while retaining their known 3D motion as supervision.Source: §2 · p. 3
  • The architecture overview shows joint video and motion denoising, rig-graph features, bidirectional cross-attention bridges, and a localization head used for camera fitting.Source: Figure 2 · p. 4

3 METHOD

JCA combines a rig-conditioned motion model with a pretrained video generator and localization for camera alignment. Its method also builds a curated video–motion corpus across diverse rigs and trains the coupled components while keeping the base video weights frozen.

3 METHOD

  • JCA couples rig-conditioned motion and video generation, then uses joint localization to fit a camera and builds training pairs by editing animated assets into suited scenes.The method is organized around arbitrary-rig motion generation, video–motion coupling, joint localization and camera alignment, training data, and training.Source: §3 · p. 4

3.1 RIG-CONDITIONED MOTION GENERATION

  • The motion denoiser generates per-joint trajectories for varying joint counts and skeletal connectivity, conditioned on graph-encoded rig descriptors and pretrained on motion data.Its flow-matching path interpolates a clean sample toward Gaussian noise and uses the noise-minus-clean velocity as target; the representation stores each joint’s rotation, position, and frame-to-frame changes.Source: §3.1 · pp. 4–5

3.2 COUPLING VIDEO AND MOTION

  • Bidirectional cross-attention bridges connect separate pretrained video and motion branches, with zero-initialized outputs preserving their isolated behavior at the start of joint training.The design adds 178M parameters, compared with approximately 7B for motion-specific blocks in the cited hybrid design; bridge guidance contrasts connected and disconnected predictions, at four forward passes per denoising step.Source: §3.2 · p. 5

3.3 JOINT LOCALIZATION AND CAMERA ALIGNMENT

  • A joint-localization head predicts image positions from the video and motion features, and paired 2D–3D joints are used to fit one static perspective camera per clip.The camera fit estimates rotation, translation, and focal length while fixing the principal point at the image center.Source: §3.3 · p. 6

3.4 TRAINING DATA

  • The training pipeline edits asset animations into action-suited scenes and checks that edits preserve the source motion and its agreement with the video.The dataset contains 118K OiV videos from 43K assets, complemented by 60K human clips and 27K robot clips for approximately 200K video–motion pairs; motion-only pretraining also uses 125K sequences.Source: §3.4 · pp. 6–7

3.5 TRAINING

  • Training starts the motion denoiser and rig encoder from scratch, adapts the video backbone with LoRA, and jointly updates adapters, bridges, and localization while freezing base video weights.Captions, reference images with joint coordinates, and bridges are randomly dropped, and some steps keep one modality clean as a condition.Source: §3.5 · p. 7

4 EXPERIMENTS

JCA performs text-driven animation of diverse rigs, generates scene-aware motion, and supports human joint generation, cross-modal conversion, and keyframe control. Evaluations report stronger prompt adherence than the rig-animation baselines and improved synchronization after fine-tuning for human joint generation.

4 EXPERIMENTS

  • The evaluations show JCA outperforms video-first and mesh-based baselines in text-driven rig animation and, after fine-tuning, surpasses a human-specific model for joint motion–video generation.The broader evaluation reports motion that follows text, preserves asset geometry, stays synchronized with generated video, and responds to the surrounding scene.Source: §4 · p. 7

4.1 RIG ANIMATION

  • 58.4 versus 43.4 for the strongest baseline in prompt adherence; JCA also has the lowest mesh distortion and highest dynamic degree.AnimateAnyMesh is smoothest, but its motion has low dynamic degree.Source: §4.1 · p. 8

4.2 SCENE-AWARE RIG ANIMATION

  • Given a rigged asset, scene, initial location and pose, and text, JCA generates video and corresponding 3D motion that respects scene geometry.Examples include an ape jumping over a crossbar and a penguin sliding across snow.Source: §4.2 · p. 8

4.3 JOINT GENERATION FOR HUMANS

  • After fine-tuning, JCA SMPL-FT surpasses EchoMotion on all three synchronization metrics, reducing PA-MPJPE by 26% and SA-MPJVE by 37%.Both JCA variants produce better human anatomy (.93 vs. .73) and higher aesthetic quality; EchoMotion has higher dynamic degree but also a higher estimated non-static camera rate (74% versus about 45%).Source: §4.3 · p. 9

4.4 VIDEO-TO-MOTION AND MOTION-TO-VIDEO.

  • With a supplied video, JCA estimates motion for the rig; with a prescribed rig trajectory and text, it generates video that follows the motion across arbitrary rigs.Video generation works with or without a first-frame reference image.Source: §4.4 · p. 9

4.5 KEYFRAME-CONTROLLED MOTION.

  • RGB keyframes guide JCA to generate both video and an intervening 3D rig trajectory without additional training.No target 3D motion sequence or final-frame 3D pose is required.Source: §4.5 · p. 9

5 ABLATIONS

Ablations show that cross-branch connectivity and bridge guidance improve video–motion synchronization, while joint training improves text-driven animation metrics over motion-only pretraining.

5.1 STRENGTHENING VIDEO–MOTION COUPLING

  • Bridge guidance further improves synchronization over connected generation without guidance, while disconnecting the branches degrades synchronization.The ablation compares bridge guidance at ωbridge,i = 2 with connected generation at ωbridge,i = 1, and also tests fully disconnected branches.Source: §5.1 · p. 9

5.2 MOTION ONLY

  • Joint training raises prompt adherence from 51.6 to 58.4 and dynamic degree from .72 to .83 versus the motion-only pretrained checkpoint.Mesh distortion stays low for both models (0.14% and 0.20%).Source: §5.2 · pp. 9–10

6 CONCLUSION, LIMITATIONS, AND FUTURE WORK

JCA co-generates video, arbitrary-rig motion, and a camera alignment, using separate motion and video models linked by cross-attention. Its pipeline encodes rig motion and descriptors, couples the modalities, and localizes joints to fit a camera; limitations include static-camera alignment and no direct access to scene geometry.

6 CONCLUSION, LIMITATIONS, AND FUTURE WORK

  • JCA co-generates video and arbitrary-rig motion, with joint localization for camera fitting; its text-driven animations show stronger prompt adherence and lower mesh distortion than evaluated baselines.The framework also supports video-to-motion, motion-to-video, and RGB-keyframe guidance.Source: §6 · p. 10
  • Alignment assumes a static inference camera, and the motion branch never sees 3D scene geometry, so contact is unenforced and penetration is not prevented.Training rigs skew toward bipeds; interacted objects are not recovered in 3D, and each clip animates one rig.Source: §6 · p. 10

A.1 MOTION REPRESENTATION

  • Motion uses global root rotation and position, parent-local non-root rotations, and root-relative non-root positions, with fixed dataset-specific standardization.The representation stores frame-to-frame changes as well as joint rotations and positions.Source: §A.1 · p. 16

A.2 RIG ENCODER

  • Rig descriptors combine joint-name embeddings, rest-pose and reference-image geometry, and root/chain/leaf indicators, then encode skeletal connectivity and global joint interactions.Shared parameters let the encoder process different joint counts and skeletal topologies.Source: §A.2 · p. 16

A.3 MOTION DENOISER

  • The motion denoiser processes joint-frame tokens with global attention, rig-graph attention, and text conditioning, then decodes predicted motion and fits fixed bone lengths with inverse kinematics.Each denoiser block adds a projection of the corresponding rig descriptor to that joint’s tokens.Source: §A.3 · pp. 16–17

A.4 COUPLING VIDEO AND MOTION

  • Bidirectional cross-attention links paired video and motion blocks, exchanging features across their distinct denoising steps; inference bridge guidance independently scales cross-modal influence in each direction.Motion frames and video latent frames share a temporal coordinate system for attention.Source: §A.4 · pp. 17–18

A.5 JOINT LOCALIZATION AND CAMERA ALIGNMENT

  • A joint-localization head predicts 2D joint locations from motion and video features, which are paired with generated 3D joints to fit a static perspective camera.Training uses projected ground-truth joints for supervision; inference averages predictions across the second half of denoising steps before robust, reliability-weighted camera fitting.Source: §A.5 · p. 18

A.6 DATA

The data pipeline combines human and robotic video with animated 3D assets edited into action-appropriate scenes. It curates assets and edited clips to preserve usable motion labels and camera information.

A.6 DATA

  • Animated assets are rendered with varied viewpoints and lighting, then edited into environments chosen to fit each asset and action while preserving subject appearance, position, scale, pose timing, and motion.The editing instruction varies phrasing and local scene details to increase environmental diversity.Source: §A.6 · pp. 18–19
  • OiV contains 118K edited videos from 43K assets, spanning rigs with 2 to 256 joints, with asset variants kept within the same data split.The dataset is partitioned into 110K training, 4,000 validation, and 4,000 test examples.Source: §A.6 · p. 19
  • Edited clips are checked for camera drift, silhouette or image-motion changes, and physical plausibility; failed clips are regenerated, while only the first passing attempt is retained per asset and render variant.Ground-truth motion is rebuilt with fixed bone lengths, and finalized 3D joints are projected using the recorded camera to obtain 2D annotations.Source: §A.6 · pp. 18–19
  • Training combines 57K OpenVid and 3,500 HAA500 human clips; 25K DROID, 50 ALOHA, 912 RoboCerebra, and 570 TAVIS robotic clips; and 100K augmented motion sequences across 10,000 randomized rigs.Human clips receive reconstructed body and articulated-hand motion, while robotic joint trajectories are computed using each dataset’s kinematic model.Source: §A.6 · p. 19

A.7 TRAINING

Training proceeds through motion pretraining, video adaptation, and joint training of the motion, rig, bridge, localization, and video-adapter components. Randomized condition dropout supports generation with missing inputs and cross-modal conditioning.

A.7 TRAINING

  • Motion pretraining runs for 200,000 updates, and the video backbone is separately adapted with rank-32 LoRA for 10,000 updates; both stages use flow-matching objectives.Both stages use a learning rate of 10^-4; their global batch sizes are 256 for motion and 32 for video.Source: §A.7 · pp. 19–20
  • Joint training updates the motion denoiser, rig encoder, bridges, localization head, and video LoRA adapters while freezing the base video weights, VAE, and text encoder.Training combines each branch’s flow-matching loss with a smooth-L1 localization loss.Source: §A.7 · p. 20
  • Random condition dropout supports text-free, reference-free, and single-modality-conditioned generation; the reported checkpoint is from joint-training step 70,000.The video loss weight increases from 1 to 10 after 50,000 updates, while motion-flow and localization-loss weights remain 1.Source: §A.7 · p. 20
  • The motion denoiser, rig encoder, cross-modal bridges, localization head, and video LoRA adapters total 331.28M trainable parameters, versus an estimated 7.03B for motion-specific weights across 20 video blocks.The 7.03B estimate excludes motion input/output and conditioning modules.Source: §A.7 · p. 20

B RIG-ANIMATION EVALUATION DETAILS

The rig-animation evaluation compares methods on 20 assets and five text prompts each. Its setup aligns starting-pose information and guiding videos across methods, while noting possible training-data overlap for ActionMesh.

B RIG-ANIMATION EVALUATION DETAILS

B.1 BASELINE SETUP

  • The three baselines receive the initial 3D pose explicitly: mesh coordinates for AnimateAnyMesh and ActionMesh, and the same first-frame optimization initialization for Puppeteer; our method instead takes a rig rest pose and 2D projections.Our method does not receive posed 3D joints, mesh geometry, or skinning weights.Source: §B.1 · p. 20
  • Puppeteer and ActionMesh receive the same Wan2.2-generated guiding video for each asset-prompt pair, controlling the visual motion signal for comparison.The videos are generated from the same posed-asset render and action prompt.Source: §B.1 · p. 21
  • All methods produce animations of approximately equal duration, with the baselines receiving 25-frame, 16-FPS guiding videos to match JCA’s temporal configuration.AnimateAnyMesh produces 16 frames at 10 FPS, also approximately 1.6 seconds.Source: §B.1 · p. 21

B.2 EVALUATION PROTOCOL

The protocol renders each animation from three fixed views and evaluates prompt adherence through a two-turn judge. It also reports automatic motion and mesh metrics and the evaluation’s average cost.

B.2 EVALUATION PROTOCOL

  • Animations are rendered from front, right, and left views with fixed orthographic cameras, lighting, and materials; native frame counts and rates are preserved.The views are concatenated in that order for the prompt-adherence judge, with each view labeled.Source: §B.2 · p. 21
  • The GPT-6 Astra judge first describes motion from 48 sampled frames and an asset description, then scores it against the action prompt.It receives 16 evenly spaced frames per view and is not shown the rest pose or method identity.Source: §B.2 · p. 21
  • $0.38 per animation is the average estimated cost of the two-turn prompt-adherence evaluation.The estimate averages over 500 animations and includes about 30.4K input tokens and 189 output tokens per evaluation.Source: §B.2 · p. 21
  • Automatic evaluation uses VBench for motion smoothness and dynamic degree, while mesh distortion is the median edge-length change across edges and frames.Edges shorter than 10^-6 in the first frame are excluded from mesh distortion.Source: §B.2 · p. 21

B.3 EXAMPLE OF A COMPLETE EVALUATION

The example evaluates one cartoon-runner animation by first describing its motion across three views and then scoring it against the prompt to jog forward. The judge assigns a high score while noting that the motion is exaggerated and forward travel is unclear.

B.3 EXAMPLE OF A COMPLETE EVALUATION

  • The first request asks for an observations summary covering the initial pose, major movements, transitions, final pose, and chronological action labels.It directs the judge to use all three views together and not treat view transitions as subject movement.Source: §B.3 · p. 21
  • The example’s judge first describes motion chronologically from 48 frames across front, right, and left views, then compares that description with the action prompt.The figure shows two frames per view; the complete prompts and responses are given in section B.3.Source: Figure 6 · p. 22
  • 93/100 is the judge’s score for the cartoon runner’s match to “jogs forward.”The explanation cites continuous forward-oriented running and alternating strides, but notes exaggerated bounds and unclear measurable travel.Source: §B.3 · p. 22

C HUMAN JOINT-GENERATION EVALUATION DETAILS

The human joint-generation evaluation selects diverse observable actions from HumanML3D and defines synchronization and video-quality measures. It also specifies the SMPL fine-tuning variant and an additional first-frame input condition.

C HUMAN JOINT-GENERATION EVALUATION DETAILS

  • 250 HumanML3D test prompts are selected, with 25 prompts from each of ten action families and criteria including observable motion, physical validity, and diversity.For JCA, an appearance-only scene prompt and estimated SMPL proportions and projected 2D joints provide rest-skeleton and reference-pose conditions.Source: C HUMAN JOINT-GENERATION EVALUATION… · pp. 22–23
  • SMPL-FT fine-tunes the general checkpoint for 10K steps on the human subset, using the SMPL skeleton used by EchoMotion.The protocol also marks an additional condition that receives ground-truth first-frame joint positions and rotations.Source: C HUMAN JOINT-GENERATION EVALUATION… · p. 23
  • PA-MPJPE measures pose error after per-frame rigid alignment, while SA-MPJPE and SA-MPJVE measure root-centered position and velocity agreement under one sequence-level rotation.SA-MPJVE reports adjacent-frame joint displacement error in mm/s at each method’s native frame rate.Source: C HUMAN JOINT-GENERATION EVALUATION… · p. 23
  • Video evaluation reports VBench imaging quality, aesthetic quality, smoothness, human anatomy, and dynamic degree, alongside the non-static camera rate.The camera-rate measure provides context because dynamic degree can reflect articulated motion or camera movement.Source: C HUMAN JOINT-GENERATION EVALUATION… · p. 23

D VIDEO-TO-MOTION GENERATION

The video-to-motion evaluation tests recovery on unseen TrueBones Zoo clips, comparing JCA with TopoCap and MoCapAnythingV2. Fine-tuning makes JCA competitive with MoCapAnythingV2, with a lower PA-MPJPE but slightly higher sequence-aligned position and velocity errors, without a first-frame 3D pose.

D VIDEO-TO-MOTION GENERATION

  • For video-to-motion inference, JCA fixes video latents and denoises motion from noise using only the observed video and static rest-pose rig.The prompt and reference 2D joint coordinates are disabled, and no ground-truth initial pose is required.Source: D VIDEO-TO-MOTION GENERATION · p. 23
  • The evaluation uses the unseen TrueBones Zoo test split and compares JCA with TopoCap and MoCapAnythingV2; the latter receives ground-truth first-frame joint positions and rotations.For a controlled comparison, JCA is fine-tuned for 10K steps on the same training split as MoCapAnythingV2.Source: D VIDEO-TO-MOTION GENERATION · p. 23
  • Predicted motion is compared with ground-truth 3D trajectories across all 67 test clips.The table caption specifies benchmark-normalized millimeters for position errors and mm/s for velocity errors.Source: D VIDEO-TO-MOTION GENERATION · p. 23
  • Ours-FT is competitive with MoCapAnythingV2, with lower PA-MPJPE but slightly higher SA-MPJPE and SA-MPJVE, without receiving the first-frame 3D pose.Zero-shot JCA trails MoCapAnythingV2 but outperforms TopoCap on both sequence-aligned metrics.Source: D VIDEO-TO-MOTION GENERATION · p. 24
Loading 2610.28632v1…