Source-linked AI summary
3D-Aware Implicit Motion Control for View-Adaptive Human Video Generation
Zhixue Fang, Xu He, Songlin Tang, Haoxian Zhang, Qingfeng Li, Xiaoqiang Liu, Pengfei Wan, Kun Gai
TL;DR
Existing motion-control methods either bind motion to a driving viewpoint with 2D poses or impose inaccurate explicit 3D reconstructions. 3DiMo instead jointly learns view-agnostic motion tokens with a pretrained video generator using view-rich supervision and annealed geometric guidance, and experiments report faithful motion reproduction with flexible text-driven camera control and improved motion fidelity and visual quality.
Problem
2D poses rigidly bind motion to the driving viewpoint, while explicit 3D reconstructions suffer depth ambiguity and inaccurate dynamics that can constrain video generators.
Method
3DiMo jointly trains a motion encoder with a pretrained DiT generator to produce cross-attended view-agnostic motion tokens, using view-rich data and early SMPL/MANO supervision annealed to zero.
Results
Experiments demonstrate faithful driving-motion reproduction, 3D consistency across viewpoints, and flexible text-driven camera control.
Takeaways & Limitations
Implicit motion representations can leverage the generator’s intrinsic spatial priors while avoiding reliance on external 3D estimates at convergence.
Takeaways & Limitations
The current 480p resolution limits high-frequency details, causing blurred faces or missing hand texture when subjects occupy small regions of full-body frames.
Abstract
from arXiv · showhide
Existing methods for human motion control in video generation typically rely on either 2D poses or explicit 3D parametric models (e.g., SMPL) as control signals. However, 2D poses rigidly bind motion to the driving viewpoint, precluding novel-view synthesis. Explicit 3D models, though structurally informative, suffer from inherent inaccuracies (e.g., depth ambiguity and inaccurate dynamics) which, when used as a strong constraint, override the powerful intrinsic 3D awareness of large-scale video generators. In this work, we revisit motion control from a 3D-aware perspective, advocating for an implicit, view-agnostic motion representation that naturally aligns with the generator's spatial priors rather than depending on externally reconstructed constraints. We introduce 3DiMo, which jointly trains a motion encoder with a pretrained video generator to distill driving frames into compact, view-agnostic motion tokens, injected semantically via cross-attention. To foster 3D awareness, we train with view-rich supervision (i.e., single-view, multi-view, and moving-camera videos), forcing motion consistency across diverse viewpoints. Additionally, we use auxiliary geometric supervision that leverages SMPL only for early initialization and is annealed to zero, enabling the model to transition from external 3D guidance to learning genuine 3D spatial motion understanding from the data and the generator's priors. Experiments confirm that 3DiMo faithfully reproduces driving motions with flexible, text-driven camera control, significantly surpassing existing methods in both motion fidelity and visual quality.
1. Introduction
3DiMo reframes human motion control as implicit 3D reasoning: it learns view-agnostic motion representations aligned with a pretrained generator instead of imposing externally reconstructed constraints. View-rich supervision and annealed geometric guidance support motion reproduction across viewpoints with flexible text-driven camera control.
- Motivation: 2D pose conditioning binds motion to the driving viewpoint, while explicit parametric reconstructions remain vulnerable to depth ambiguity and inaccurate dynamics.These limitations motivate learning motion representations that preserve the generator’s intrinsic spatial understanding.
- 3D supervision: View-rich training spans single-view, multi-view, and moving-camera videos, supervising reconstruction and cross-view motion reproduction to encourage 3D-aware representations.This supervision goes beyond same-view reconstruction, which can learn view-dependent 2D patterns.
- Method: 3DiMo jointly trains a motion encoder with a pretrained DiT-based video generator to distill 2D driving frames into compact, view-agnostic motion tokens.The tokens are semantically injected through cross-attention rather than rigid projection-based alignment.
- Geometric guidance: Auxiliary SMPL and MANO supervision provides early geometric initialization, then anneals to zero so learning shifts toward the generator’s priors and view-rich data.The external models guide initialization without remaining strong constraints throughout training.
- Results: Experiments show faithful driving-motion reproduction with 3D consistency across varying viewpoints and flexible text-driven camera control.The reported results support expressive, 3D-aware motion representations aligned with the pretrained generator.
2. Related Work
The related work situates 3DiMo within diffusion-based video generation, emphasizing that pretrained DiT-based models provide scalable video synthesis with strong dynamics and 3D spatial reasoning.
- Diffusion-Based Video Generation: Latent diffusion models improve video-generation efficiency through compressed latent spaces, while DiT architectures enhance scalability and spatiotemporal consistency.These models form the broader generation paradigm underlying the paper’s backbone.
- Diffusion-Based Video Generation: Recent pretrained video diffusion models exhibit awareness and reasoning capabilities over both dynamics and 3D space.3DiMo builds on these capabilities to learn implicit 3D-aware motion representations.
3. Our Approach
3DiMo transfers motion from 2D driving videos to reference subjects while preserving text-guided camera control by jointly learning implicit motion tokens and a pretrained DiT generator. Its view-rich dataset, dual-scale encoding, augmentation, and staged supervision are designed to recover motion that remains consistent across viewpoints and camera trajectories.
- Task: The task transfers inherently 3D driving motion to a reference subject while disentangling it from camera trajectories entangled in 2D projections.The objective preserves flexible, text-guided camera control during generation.
- Framework: A motion encoder jointly optimized with a pretrained DiT generator distills view-agnostic tokens from 2D driving frames and injects them through cross-attention.The reference image and text prompt condition generation alongside the motion tokens.
- Video Generation Backbone: The DiT backbone uses latent diffusion with a causal 3D VAE and models latent video sequences using interleaved self-attention and feed-forward blocks.Reference-image latent tokens are concatenated with noised video tokens for cross-modal interaction.
- Parametric 3D Human Model: SMPL and MANO provide body and hand shape and pose parameters for auxiliary geometric alignment during early training.These parametric models supply initialization priors rather than the final motion representation.
- Implicit Motion Encoder: Random perspective and appearance augmentations encourage viewpoint-agnostic motion encoding while reducing reliance on view-specific projection and identity cues.The encoder uses separate body and hand streams to capture coarse motion and fine-grained gestures.
- Training Strategy: Single-view training supplies diverse motion dynamics, while cross-view objectives prevent the encoder from relying only on view-dependent 2D patterns.The progressive strategy begins with single-view self-reconstruction before adding richer cross-view supervision.
- View-Rich Supervision: The view-rich dataset combines same-view reconstruction, synchronized multi-view reproduction, and moving-camera reproduction to supervise expressive and 3D-consistent motion.Its sources include internet videos, UE5 renderings, and real-world multi-view captures, balancing diversity, realism, and geometric consistency.
4. Experiments
Experiments evaluate 3DiMo against 2D and SMPL-based baselines using quantitative metrics, visual comparisons, user studies, and ablations. Results indicate stronger visual quality, motion fidelity, physical plausibility, and view-adaptive control, while component removals degrade performance or 3D understanding.
- 4.2. Quantitative Evaluation: 3DiMo surpasses all baselines on LPIPS, FID, and FVD, indicating stronger visual quality and motion control.SSIM and PSNR are slightly lower than MTVCrafter, which the authors attribute to viewpoint-sensitive pixel metrics and the use of static-camera prompts.
- 4.3. Qualitative Evaluation: User-study MOS results consistently favor 3DiMo across motion accuracy, naturalness, 3D physical plausibility, and overall visual quality.Thirty participants evaluated ten cross-identity animation videos per method on a 5-point Likert scale.
- 4.3. Qualitative Evaluation: 3DiMo preserves physically plausible motion and spatial consistency across dynamic camera trajectories and varying viewpoints.The model combines precise video-driven motion control with text-guided camera manipulation.
- 4.4. Ablation Study and Analysis: Removing view-rich stages, auxiliary geometric supervision, or other components degrades camera control, training stability, visual quality, or fine-grained motion.Discarding the final two view-rich stages can slightly improve visual metrics but removes genuine 3D-aware motion understanding.
- 4.4. Ablation Study and Analysis: SMPL-based motion fails to preserve hand-hip contact from the side view, whereas the learned representation resolves this depth ambiguity.The comparison uses a frontal one-hand-on-hip driving motion.
5. Conclusion
The conclusion presents 3DiMo as an implicit 3D-aware motion-control framework that aligns a motion encoder with a pretrained video generator instead of relying on explicit parametric reconstruction. View-rich supervision and annealed geometric initialization yield view-agnostic motion representations that reproduce driving motion under flexible text-driven camera control and outperform 2D- and 3D-based baselines.
- 5. Conclusion: 3DiMo jointly trains a motion encoder and pretrained video generator to learn view-agnostic motion representations from 2D observations.The encoder discards view-dependent layouts while aligning with the generator’s intrinsic spatial priors.
- 5. Conclusion: View-rich supervision and gradually annealed geometric initialization help the framework internalize 3D spatial motion understanding without relying on external estimates.SMPL provides lightweight early guidance that is later removed.
- 5. Conclusion: Experiments show faithful driving-motion reproduction with flexible text-driven camera control and consistent superiority over 2D- and 3D-based baselines in motion fidelity and visual quality.
3D-Aware Implicit Motion Control for View-Adaptive Human Video Generation Supplementary Material
The supplementary material documents additional support for the main paper, including ethical considerations, broader applications, data collection, camera trajectories, and related technical details.
- Supplementary Material: The supplement is organized around ethics, broader applications, data collection, camera trajectories, and additional technical specifications.Listed applications include single-image novel-view synthesis, video stabilization, and automatic motion alignment.
A. Ethical Considerations
The paper acknowledges that realistic human-video generation raises privacy, intellectual-property, and deceptive-media concerns. It limits processed data, models, and results to academic use and excludes commercial or harmful applications.
- A. Ethical Considerations: Realistic human-video synthesis raises risks involving privacy violations, intellectual-property concerns, and deceptive deepfake media.
- A. Ethical Considerations: The authors state that processed data, models, and results are intended strictly for academic purposes and not authorized for commercial or harmful content.They frame ethical guidelines and legal frameworks as needed to address misuse without consent.
B. Broader Applications
3DiMo extends view-adaptive human video generation to downstream tasks by separating human motion from camera behavior through view-agnostic motion representations. It supports novel-view synthesis from a single image and video stabilization through text-specified camera constraints.
- Human Novel View Synthesis From a Single Image: 3DiMo generates single-image novel views by repeating the reference frame to impose zero motion while text specifies the camera trajectory.This setup avoids requiring explicit 3D scene reconstruction.
- Human Novel View Synthesis From a Single Image: 3DiMo addresses base I2V ambiguities by keeping the subject stationary and producing consistent geometric viewpoint changes.Base I2V models may hallucinate subject motion or animate the background instead of synthesizing a true view change.
- Video Stabilization: For video stabilization, the first shaky frame is used as reference, the full video supplies motion, and a static-camera prompt suppresses camera jitter.The generator reconstructs underlying human motion from a fixed viewpoint.
C. In-House Data Acquisition Setup
The in-house capture setup combines multiple camera viewpoints with diverse, independently assigned camera trajectories. Pairing the same human performance across these views strengthens supervision for view-agnostic motion learning.
- Multi-camera capture: The capture system uses a three-camera array positioned at diverse angles relative to the subject.Each camera receives a randomly sampled camera-motion type for every captured performance.
- Camera-motion diversity: Camera trajectories span static variants, linear translations, zoom actions, and complex paths such as vertigo and arc motions.The listed trajectories include handheld, aerial, variable-angle, and dynamic zoom variants.
- Downstream applications: The broader framework is demonstrated on single-image novel-view synthesis, video stabilization, and automatic motion-appearance alignment.The caption describes these as three downstream applications of 3DiMo.
- View-rich supervision: Identical human motions are paired with diverse, noncorrelated camera trajectories across three views to maximize view-agnostic motion supervision.This pairing separates human motion cues from camera-motion variation.
D. Limitations and Future Work
3DiMo remains limited by its 480p operating resolution and by incomplete modeling of complex human-object interactions. Future work targets higher-resolution generation and better handling of external objects and props.
- Resolution and Fine-Grained Details: At 480p, limited pixel capacity can blur facial features and reduce hand-texture detail, especially when the subject is small in full-body shots.The authors suggest 720p or 1080p backbones and cascaded super-resolution modules.
- Complex Human-Object Interactions: The motion encoders focus on human body and hand dynamics, so external objects and props are not explicitly modeled.Interactions with held objects may therefore be hallucinated even when human motion is faithfully reproduced.
E. Implementation Details
Training uses fixed-length 480 × 854-pixel clips, Adam optimization, and three staged training phases. Auxiliary geometric supervision is linearly removed during early training.
- Training configuration: Training uses 121-frame clips resized to a target area of 480 × 854 pixels while preserving the original aspect ratio.The total batch size is 64 and optimization uses Adam.
- Optimization schedule: The learning rate is 1e-5, and the three training stages run for 10K, 15K, and 5K steps.Training completes in approximately three days.
- Geometric supervision: Auxiliary geometric supervision is linearly annealed from 0.1 to 0 over the first 12K training steps.This schedule transitions away from the auxiliary geometric signal during training.