Source-linked AI summary
Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generation
Chenjie Cao, Jingkai Zhou, Shikai Li, Jingyun Liang, Chaohui Yu, Fan Wang, Xiangyang Xue, Yanwei Fu
TL;DR
Uni3C addresses separate camera and human-motion control and limited jointly annotated data in video generation. It introduces PCDController and globally aligned 3D guidance combining scene point clouds with SMPL-X characters. Experiments report that Uni3C significantly outperforms competitors in camera controllability and human-motion quality.
Problem
Existing approaches typically treat camera and human-motion control separately, while high-quality data jointly annotated for both aspects are limited.
Method
Uni3C combines a lightweight plug-and-play PCDController using monocular-depth point clouds with inference-time global 3D alignment of environmental point clouds and SMPL-X characters.
Results
Uni3C significantly outperforms competitors quantitatively and qualitatively in camera controllability and human-motion quality across the reported evaluations.
Takeaways & Limitations
The framework supports unified camera and human-motion control while allowing domain-specific modules to be trained without jointly annotated data.
Takeaways & Limitations
Uni3C depends on predefined camera trajectories and SMPL-X characters, and conflicting human motions can produce distortion, clipping, or disappearance near environmental obstacles.
Abstract
from arXiv · showhide
Camera and human motion controls have been extensively studied for video generation, but existing approaches typically address them separately, suffering from limited data with high-quality annotations for both aspects. To overcome this, we present Uni3C, a unified 3D-enhanced framework for precise control of both camera and human motion in video generation. Uni3C includes two key contributions. First, we propose a plug-and-play control module trained with a frozen video generative backbone, PCDController, which utilizes unprojected point clouds from monocular depth to achieve accurate camera control. By leveraging the strong 3D priors of point clouds and the powerful capacities of video foundational models, PCDController shows impressive generalization, performing well regardless of whether the inference backbone is frozen or fine-tuned. This flexibility enables different modules of Uni3C to be trained in specific domains, i.e., either camera control or human motion control, reducing the dependency on jointly annotated data. Second, we propose a jointly aligned 3D world guidance for the inference phase that seamlessly integrates both scenic point clouds and SMPL-X characters to unify the control signals for camera and human motion, respectively. Extensive experiments confirm that PCDController enjoys strong robustness in driving camera motion for fine-tuned backbones of video generation. Uni3C substantially outperforms competitors in both camera controllability and human motion quality. Additionally, we collect tailored validation sets featuring challenging camera movements and human actions to validate the effectiveness of our method.
1 Introduction
Uni3C unifies precise camera and human-motion control through lightweight 3D-enhanced modules and globally aligned conditions, addressing limited jointly annotated data and fragmented control.
- Camera and human-motion control are critical, interdependent aspects of controllable video generation.
- Existing methods often hack foundational video diffusion models, limiting generalization to out-of-distribution scenarios.
- Uni3C introduces PCDController, a plug-and-play camera module trained with a frozen backbone and unprojected monocular-depth point clouds.
- PCDController remains compatible with fine-tuned backbones, enabling camera and human-motion modules to be trained independently without jointly annotated data.
- Experiments on tailored camera and human-video test sets show Uni3C significantly outperforms competitors quantitatively and qualitatively.
- Uni3C aligns environmental point clouds and SMPL-X characters in a global 3D world for coherent joint control.
2 Related Work
Prior work studies camera and human control using explicit geometric or pose conditions, but generalized, joint, and physically coherent control remains unresolved.
- Camera-control methods commonly inject camera parameters or Plücker rays into video diffusion models.
- Point-cloud methods address camera control through training-based and training-free approaches.
- Existing approaches struggle with imperfect point clouds, out-of-distribution data, and motion scenarios involving humans or animals.
- Unified-control methods combine conditions such as poses, trajectories, point tracking, and 3D-aware representations.
3 Preliminary: Video Diffusion Models
Video diffusion models generate latent video representations through conditional diffusion or flow matching, with Wan2.1 providing the foundational DiT backbone used by Uni3C.
- VDMs model the conditional distribution of clean video latents given text and optional image conditions.
- Flow Matching interpolates between clean and noisy latents and predicts the corresponding velocity for the diffusion process.
- Wan2.1 is an open video diffusion model with DiT architecture trained using flow matching.
- Wan-I2V adds CLIP image features, while Uni3C primarily targets its 14B-parameter version and also works with Wan-T2V.
4 Method
Uni3C combines PCDController for geometric camera control with human-animation modules and global 3D world guidance that aligns scene and character conditions before rendering.
- Overview: Uni3C conditions video generation on a reference image, target camera trajectories, text, and PCDController camera guidance.
- PCDController: PCDController is a simplified lightweight DiT with reduced hidden size and zero-initialized projections to preserve backbone generalization.
- 3D Geometric Priors: Monocular depth is converted into metric point clouds, which are rendered under target camera intrinsics and extrinsics as geometric conditions.
- PCDController: PCDController remains robust with imperfect point clouds and retains precise camera control over fine-tuned Wan backbones without joint training.
- Human Animation: Human animation uses SMPL-X and Hamer conditions, with GVHMR recovering SMPL-X characters for control.
- Global 3D World Guidance: Global 3D world guidance addresses incompatible human and environmental coordinates by aligning SMPL-X to the environment and rerendering conditions under camera trajectories.
- Global 3D World Guidance: GeoCalib calibrates gravity and rigid transformations are applied across SMPL-X sequences, while low-confidence keypoints are discarded.
5 Experiments
Uni3C is evaluated on camera control, unified camera-human control, motion transfer, and ablations using challenging benchmarks and user studies. Results show strong camera precision, competitive video quality, effective unified control, and a controllability-generalization trade-off governed by guidance design and PCDController capacity.
- Experimental Setup: The camera-control evaluation combines an out-of-distribution benchmark, a 500-sample benchmark, and challenging multi-trajectory human-video tests.The out-of-distribution benchmark contains 32 images with four trajectories each, while the unified benchmark contains 50 videos with three trajectories each.
- Camera Control: Uni3C achieves the optimal balance between video quality and camera precision, improving ATE, RPE, and RRE over Wan2.1 and CogVideoX controls.Point clouds enhance controllability for both backbones, while combining Plücker rays and point clouds further helps on substantial viewpoint changes.
- Unified Control: Uni3C combines good VBench scores with impressive camera metrics, unlike RealisDance-DiT’s stronger visual quality but poor camera-trajectory accuracy.CamAnimate also suffers from conflicting human motions and backgrounds because body poses are misaligned with predefined camera trajectories.
- Point Clouds of Humans: Masking human point clouds slightly improves visual quality but harms camera precision, especially when humans occupy substantial image area.Retaining human point clouds is therefore essential for effective camera control, while Wan2.1 can still generate natural motion with fixed foreground point clouds.
- Exploratory Discussions: Gravity calibration corrects SMPL-X alignment with skewed human point clouds and eliminates error accumulation during long-distance human movements.User studies evaluate camera control on 128 samples and unified control on 150 samples with 20 unrelated volunteers.
- Motion Transfer: Uni3C controls camera trajectories and human motions when reference motions come from different videos or domains, including animation and real-world scenes.The framework is also applied with text-to-motion clips and retrieved motion conditions.
- PCDController Ablation: Using 20 external DiT layers balances controllability and generalization, whereas 30 layers slightly improve camera metrics but reduce visual quality and generalization through overfitting.The overfitting produces distorted point clouds.
6 Limitation and Future Work
Uni3C is constrained by predefined camera trajectories and SMPL-X human characters, limiting physical plausibility when prescribed motion conflicts with environmental obstacles or inaccurate coordinates.
- Limitations and Future Work: Predefined camera trajectories and SMPL-X characters constrain Uni3C, and obstructed human paths can produce distortion, clipping, or disappearance.The authors suggest physics-aware human motion generation as a mitigation; inaccurate SMPL or world coordinates can also cause undesired pose or motion direction.
7 Conclusion
Uni3C unifies 3D-enhanced camera and human-motion control through lightweight modules and aligned 3D guidance. Experiments report strong performance while reducing reliance on heavily annotated joint data.
- PCDController uses lightweight trainable modules and 3D point-cloud priors to manage camera trajectories without compromising foundational VDM capacities.
- Uni3C aligns environmental point clouds and human characters in a global 3D world space for joint camera and human-animation control.
- Experiments across diverse datasets show superior quantitative and qualitative performance compared with existing approaches for controllable video generation.
- The study focuses on technical aspects while highlighting privacy, consent, misleading content, and bias as responsible-use concerns for generative models.
B.1 Architecture Details of PCDController
PCDController is implemented as a simplified DiT-based external control branch, with architectural details illustrated in Figure 18.
- PCDController uses a 20-layer external DiT branch with 1024 hidden size to balance controllability and generalization.
- Its design follows CogVideoX’s DiT components, including AdaLayerNorm, 3D attention, and a Feed Forward Network, without a textual branch.
- Figure 18 provides the detailed model architecture of PCDController.
B.2 Architecture Details of Realisdance-DiT
Realisdance-DiT is a Wan-I2V-derived architecture that incorporates human-condition and reference-image features for unified camera and human-pose control. Uni3C also reports limited inference-time overhead and transfer from 480p training to 720p generation without specific fine-tuning.
- Realisdance-DiT is fine-tuned from Wan-I2V, adding SMPL-X and Hamer human-condition signals plus spatially concatenated reference-image features.
- Uni3C only increases inference time a little compared with the baseline method.
- PCDController trained at 480p also performs well for 720p video generation without specific fine-tuning.
- Inference efficiency is evaluated against RealisDance-DiT across two resolutions using official Wan2.1 settings.
C Details of Datasets
The training data use balanced sampling across domains, with greater attention to high-quality imagery and complex real-world camera trajectories.
- Sampling across epochs balances sample scales across different data domains.
- The training strategy emphasizes high-quality images and complex camera trajectories from real-world datasets such as DL3DV and UCo3D.
D Inference Details of Camera Control
Camera control begins by extracting monocular depth and defining the foreground depth medium as the rotation radius, then uses rotation and bounded translation primitives to construct trajectories.
- Inference setup: Monocular depth from the reference view defines the foreground depth medium, which serves as the rotation radius for initial camera placement.Foreground masks are extracted with CarveKit 2; if no foreground is detected, the entire image is treated as foreground.
- Evaluation scenario: Uni3C evaluates two-person scenarios using two camera trajectories while preserving the input video’s human motions.
- Training data: Training data for PCDController is drawn from nine datasets, with subsets dynamically sampled across training epochs.The listed datasets span Co3Dv2, DL3DV, RE10K, ACID, Tartainair, Map-Free-Reloc, WildRGBD, COP3D, and UCo3D.
- Camera primitives: The camera controller represents rotation through azimuth and elevation around the foreground-centered rotation radius.
- Camera primitives: Translation is applied along the x, y, and z axes, with values constrained to [0, 1.0] and scaled by the rotation radius.This scaling limits translations relative to the scene’s estimated foreground distance.
E Multiple-Person Discussion
Uni3C can operate in a two-person setting by separately aligning both SMPL-X representations, but additional people hinder performance because the backbone was trained for single-person scenarios.
- Scope: RealisDance-DiT is explicitly tailored for single-person scenarios, establishing the main scope of Uni3C’s human-motion backbone.
- Two-person control: Two-person control is supported by separately aligning each person’s SMPL-X representation using that person’s 3D keypoints.
- Limitation: Adding control signals for more people hinders performance because multi-person control lies outside RealisDance-DiT’s training scope.
- Future extension: The plug-and-play design leaves open integration with backbones natively designed for multi-person control.