Source-linked AI summary
4DAnyone: Create Anyone in 4D from a Casual Monocular Video
Yudong Jin, Tao Xie, Qihang Zhang, Zehong Shen, Zhen Xu, Yujun Shen, Hujun Bao, Xiaowei Zhou, Yinghao Xu
TL;DR
4DAnyone addresses the difficulty of maintaining multiview consistency when camera-controlled diffusion must generate the many views required for 4DGS from monocular video. It combines fixed-length reference-context packing, routed target-view groups, and skeleton conditioning, and reports stronger video quality and downstream 4DGS reconstruction than prior methods. The paper also identifies scope boundaries for loose garments and responsible use.
Problem
Camera-controlled video diffusion models do not maintain sufficient cross-view consistency when scaled to the tens of target views required for 4DGS reconstruction.
Method
4DAnyone uses Reference Context Packing, Target Context Routing, 3D skeleton conditioning, and mixed-source training data to generate reconstruction-grade multiview videos for 4DGS.
Results
4DAnyone significantly outperforms prior methods in both novel-view video quality and downstream 4DGS reconstruction.
Takeaways & Limitations
The framework supports high-fidelity 4D human reconstruction from monocular videos using reconstruction-grade multiview-consistent video generation.
Takeaways & Limitations
Loose garments can remain inconsistent across views because skeleton guidance is not informative for fabric moving far from the body.
Abstract
from arXiv · showhide
We present 4DAnyone, a framework for reconstructing 4D humans from an uncalibrated monocular video by generating reconstruction-grade multiview-consistent videos and lifting them into 4D Gaussian Splatting (4DGS). Existing camera-controlled video diffusion models synthesize plausible novel-view videos but fail to maintain consistency when scaled to the tens of target views required for 4DGS reconstruction. We identify this failure as a bounded-attention-context problem: when target views exceed the capacity of a single DiT forward pass, they must be split into groups, exposing two coupled bottlenecks. On the reference-context side, conditioning on all previously generated views grows as $O(N)$, weakening cross-view appearance guidance. On the target-context side, disjoint groups cannot directly exchange information, causing global structural drift. 4DAnyone addresses both bottlenecks with two complementary designs: Reference Context Packing (RCP) compresses growing reference views into a fixed-length mixed-resolution context with $O(1)$ reference-context complexity, while Target Context Routing (TCR) rotates target-view groupings during denoising to share context across groups at high-noise steps and stabilize details at low-noise steps. We further build the MVGameHuman dataset using our in-house game engine and combine it with light-stage and in-the-wild video datasets for training. Experiments on DNA-Rendering and DyMVHumans show that 4DAnyone outperforms prior methods in both novel-view video quality and downstream 4DGS reconstruction, with robust in-the-wild generalization. See our project page for video results and source code: https://4danyone.github.io.
1 Introduction
4DAnyone targets 4D human reconstruction from uncalibrated monocular video by generating consistent novel-view videos for 4DGS. It addresses reconstruction-scale consistency failures caused by bounded attention context with scalable reference conditioning and cross-group target routing.
- 4D human reconstruction remains challenging because current 4DGS construction requires dense multiview video from calibrated, static camera arrays.
- Camera-controlled diffusion models produce plausible novel views but struggle to maintain consistency across the tens of target views needed for 4DGS.
- When target views exceed one DiT pass, splitting them into groups creates O(N) reference-context growth and disjoint target contexts that cause structural drift.
- Reference Context Packing compresses generated reference views into a fixed-length mixed-resolution context, reducing reference-context complexity from O(N) to O(1) while preserving appearance guidance.
- Target Context Routing rotates target-view groupings during high-noise denoising for cross-group structure sharing, then fixes adjacent groups during low-noise denoising to stabilize details.
- 4DAnyone combines 3D-aware skeleton conditioning with the MVGameHuman dataset and light-stage and in-the-wild data to support reconstruction-grade consistency and generalization.
2 Related Work
Prior work spans camera-controlled video generation, multi-view diffusion, and 4D human reconstruction, while 4DAnyone targets reconstruction-grade novel-view consistency from monocular video. Its approach combines sparse 3D skeleton conditioning with scalable context management, and experiments report stronger consistency, reconstruction, and generalization than compared methods.
- Multi-View Diffusion: Multi-view diffusion extends viewpoint-conditioned generation from static images to dynamic videos for generation-then-reconstruction pipelines.The cited methods include Zero-1-to-3, MVDream, CAT3D, SV3D, Zero123++, SV4D, and CAT4D.
- 4D Human Avatar Reconstruction: Traditional dynamic-avatar reconstruction uses dense multi-view captures, whereas monocular methods reduce capture requirements but remain constrained by camera knowledge, occlusion, or appearance fidelity.DNA-Rendering exemplifies the expensive multi-camera setting with a 48-camera rig.
- 4DAnyone: 4DAnyone generates target-view videos from monocular input using 3D skeleton cues, RCP conditioning, and TCR-based exchange among target groups before 4DGS reconstruction.Its target viewpoints are prescribed static views distributed around the subject.
- Camera-Controlled Video Generation: Camera-controlled video methods use implicit or explicit camera conditioning, but dense geometry and camera parameters can be unreliable for in-the-wild videos.4DAnyone instead emphasizes accurate sparse geometric signals for viewpoint control.
- Evaluation: 4DAnyone outperforms all baselines across 4DGS reconstruction, generated-video consistency, and generated-video reconstruction on DNA-Rendering and DyMVHumans.The evaluation uses held-out DNA-Rendering sequences and out-of-distribution DyMVHumans data.
- Evaluation: Ablations associate RCP with appearance guidance, TCR with cross-group consistency, sliding routing with improved metrics, and depth buffering with better geometric consistency.In-the-wild and challenging-case evaluations also report robust target-view generation and 4DGS reconstruction.
5 Conclusion
4DAnyone reconstructs high-fidelity 4D humans from monocular videos using sparse 3D skeleton conditioning, scalable appearance guidance, and reduced cross-group drift. The authors report improved video quality and downstream 4DGS reconstruction, while noting risks from realistic human-video synthesis.
- 4DAnyone combines sparse 3D skeleton conditioning, Reference Context Packing, and Target Context Routing for monocular 4D human reconstruction.
- Realistic human-video synthesis poses risks of deepfake misuse, identity privacy violation, and copyright infringement.
Supplementary Material
The supplementary model details describe multiview attention, multi-scale RCP patchification, and a 3D-aware skeleton encoder that injects geometric residuals into DiT latents.
- Multiview self-attention rearranges tokens so views at the same timestep directly attend to one another.It shares architecture and weights with the base model’s temporal self-attention, differing in token arrangement.
- RCP 2× and 4× patchify layers use progressively larger spatial kernels and strides to reduce reference-token counts.Their kernels and strides are (1, 4, 4) and (1, 8, 8), respectively.
- The 3D-aware skeleton encoder processes depth-buffered RGB skeleton videos through Conv3d layers and projects them to DiT-resolution residuals.The encoder uses 10 Conv3d layers followed by a 1×1×1 projection, with 32× spatial and 4× temporal downsampling.
B Dataset Details
MVGameHuman is an in-house game-engine dataset providing synchronized, high-resolution multiview human videos across many actors and virtual camera viewpoints.
- MVGameHuman contains 38k synchronized multiview human videos rendered at 2560×1440 resolution.
- The dataset covers 318 actors captured by 24 virtual cameras per sequence.
C Training Details
Training fine-tunes Wan2.2 across staged datasets and camera configurations, using perceptual losses, body-part-aware sampling, and progressively broader video conditions.
- All training stages fine-tune Wan2.2-TI2V-5B at 704×1280 resolution with learning rate 1 × 10^-5 and LPIPS weight λ=0.25.The three stages take approximately 0.5, 1, and 1.5 days on 128 H20-3E GPUs.
- Body-part-aware LPIPS sampling prioritizes full-body, face, and hand regions while retaining uniform global coverage.Sampling probabilities are 0.2 for full body, 0.2 for face, 0.1 for each hand, and 0.4 uniform.
- Training samples camera and frame configurations while keeping target cameras multiplied by frame count approximately constant.Examples include 6 × 41 ≈ 4 × 61 ≈ 1 × 121 to maintain similar token counts per pass.
- Stage-specific settings: Stage 1 uses foreground-only DNA-Rendering and independently sampled source ranges to decouple pose from appearance.
- Stage-specific settings: Stage 2 adds unmasked MVGameHuman and SynCamVideo data, while Stage 3 adds Pexels and TedTalk monocular data and drops noisy finger keypoints.
D HMR Details
The HMR pipeline estimates a ground-aligned SMPL-X mesh sequence, then derives sparse 3D keypoints for skeleton conditioning and rendering.
- GVHMR estimates a ground-aligned SMPL-X mesh sequence from the monocular video.
- A sparse vertex-to-keypoint regressor extracts 70 Goliath-vocabulary 3D keypoints from the mesh.
- Skeleton rendering retains body, foot, and palm-level hand keypoints while excluding face and individual finger joints.
- Each Goliath keypoint is predicted as a convex combination of nearby SMPL-X vertices, reducing extraction to a sparse weighted sum.
E Inference Details
Inference takes multi-view skeleton conditions through target-view video generation and 4DGS reconstruction, with configurations and timings that scale camera coverage and runtime.
- The inference pipeline generates all target-view videos from multi-view skeleton conditions before reconstructing the final 4DGS model.
- Multi-GPU inference: Single-layer, two-layer, and three-layer configurations provide 16, 32, and 48 cameras for increasingly demanding view coverage.Sixteen cameras suit limited pitch angles, 32 cover most scenarios, and 48 accommodate complex clothing or extreme motions.
- 4DGS reconstruction: FreeTimeGS initializes 4D Gaussian primitives through space carving and optimizes 16-camera, 121-frame sequences for 50k iterations.Optimization uses Adam with a learning rate of 1.6 × 10^-4.
- Inference efficiency: Preprocessing takes approximately 2 minutes, multi-view generation approximately 7 minutes, and 4DGS training approximately 30 minutes.These timings use a single RTX 4090 for preprocessing and reconstruction, and a single H20 GPU for generation.
F Evaluation Details
Evaluation compares methods on matched DNA-Rendering and DyMVHumans test settings, while 4DAnyone generates reference views and routed target groups for reconstruction.
- Evaluation setup: All methods use the same DNA-Rendering and DyMVHumans test sequences, with 16 approximately uniformly distributed cameras and 98 frames per scene.The evaluation includes 10 DNA-Rendering scenes and 3 DyMVHumans scenes.
- Baselines: MV-Performer is evaluated zero-shot using source-view depth warped to target views with ground-truth camera parameters.
- Baselines: TrajectoryCrafter is evaluated zero-shot using source-video depth aligned to dataset camera scale before warping.
- Baselines: ReCamMaster is fine-tuned on the same training datasets and settings, with matching RCP and TCR for controlled conditioning comparison.
- 4DAnyone: 4DAnyone first generates 4 uniformly spaced reference views, then produces 16 target views in four-view groups using TCR and fixed RCP context.The model generates 121 frames and truncates to the first 98 for test-sequence matching.
F.1 Ablation Details
The ablation evaluates routing schedules on eight DNA-Rendering scenes and finds that sliding early denoising steps helps until a 0.2 switching ratio, after which gains saturate.
- Setup: The ablation uses eight DNA-Rendering scenes and omits reference views only for the w/o RCP condition.
- Routing variants: Sliding, Random, and Strided dynamically regroup azimuth-ordered target views during denoising, while w/o TCR keeps contiguous groups.
- Switching-time sweep: The ablation compares generated-video consistency across different numbers of sliding denoising steps.
- Switching-time sweep: t_s/T = 0.2 is selected because increasing sliding steps improves consistency until that point, while further decreases provide no measurable gain.Further decreases can slightly degrade individual metrics, with fluctuations comparable to 4DGS optimization variation.
G Additional Results
4DAnyone can extend from a single image to a 4D avatar by first synthesizing the subject’s target motion and then generating multiview-consistent videos for 4DGS reconstruction.
- Wan-Animate generates a monocular video of a subject performing target motion from a single image and driving motion video.
- 4DAnyone converts the synthesized monocular video into multiview-consistent videos for reconstructing a 4DGS avatar.
H Limitations and Failure Cases
The main failure cases arise when skeleton guidance is unreliable for non-body motion or when human mesh recovery mis-estimates unusual poses, although the pipeline remains robust to many occluded or blurred inputs.
- HMR typically predicts complete and plausible skeletons under occlusion or motion blur, so pose errors usually yield shifted but coherent humans.
- Loose garments: Skeleton guidance fails for large flowing garments that move far from the body, producing inconsistent views and degraded reconstruction.
- Inaccurate pose estimation: When HMR mis-estimates an unusual pose, all generated views faithfully inherit the incorrect skeleton.For an en-pointe dancer, HMR predicts flat feet instead of the correct pose.