Source-linked AI summary
ONE-SHOT: Compositional Human-Environment Video Synthesis via Spatial-Decoupled Motion Injection and Hybrid Context Integration
Fengyuan Yang, Luying Huang, Jiazhi Guan, Quanwei Yang, Dongwei Pan, Jianglin Fu, Haocheng Feng, Wei He, Kaisiyuan Wang, Hang Zhou, Angela Yao
TL;DR
Independent, precise control of human subjects and environments remains difficult, while 3D preprocessing, reduced flexibility, and short synthesis horizons limit existing approaches. ONE-SHOT disentangles motion, environment, camera, and appearance using canonical-space injection, Dynamic-Grounded-RoPE, Hybrid Context Integration, and lightweight LoRA fine-tuning. Experiments show a more favorable balance between structural control and creative diversity than prior methods.
Problem
Existing methods struggle to independently control human motion and environmental backgrounds while avoiding burdensome 3D preprocessing, reduced generative flexibility, and short synthesis horizons.
Method
ONE-SHOT factorizes human motion, environment geometry, camera trajectories, and appearance using canonical-space injection, Dynamic-Grounded-RoPE, Hybrid Context Integration, and lightweight LoRA tuning.
Results
Experiments and ablations show that ONE-SHOT achieves a more favorable balance between structural control and creative diversity than prior methods while requiring only lightweight LoRA fine-tuning.
Takeaways & Limitations
The framework sustains subject identity and scene coherence during minute-level long-horizon human-environment video generation.
Takeaways & Limitations
Performance depends on reconstructed scene point clouds and camera trajectories, and temporal drift may still accumulate in very long videos.
Abstract
from arXiv · showhide
Recent advances in Video Foundation Models (VFMs) have revolutionized human-centric video synthesis, yet fine-grained and independent editing of subjects and scenes remains a critical challenge. Recent attempts to incorporate richer environment control through rigid 3D geometric compositions often encounter a stark trade-off between precise control and generative flexibility. Furthermore, the heavy 3D pre-processing still limits practical scalability. In this paper, we propose ONE-SHOT, a parameter-efficient framework for compositional human-environment video generation. Our key insight is to factorize the generative process into disentangled signals. Specifically, we introduce a canonical-space injection mechanism that decouples human dynamics from environmental cues via cross-attention. We also propose Dynamic-Grounded-RoPE, a novel positional embedding strategy that establishes spatial correspondences between disparate spatial domains without any heuristic 3D alignments. To support long-horizon synthesis, we introduce a Hybrid Context Integration mechanism to maintain subject and scene consistency across minute-level generations. Experiments demonstrate that our method significantly outperforms state-of-the-art methods, offering superior structural control and creative diversity for video synthesis. Our project has been available on: https://martayang.github.io/ONE-SHOT/.
1 Introduction
ONE-SHOT addresses the limited precision, scalability, and flexibility of prior human-environment video synthesis methods by factorizing generation into distinct signals within a parameter-efficient framework. It combines canonical-space motion injection, Dynamic-Grounded-RoPE, and Hybrid Context Integration for independently controllable and temporally consistent synthesis.
- Motivation: Prior subject-environment editing methods lack precise, independent control over human movement and environmental backgrounds, limiting structurally consistent customization.These approaches rely on million-scale supervised fine-tuning but remain limited in professional workflows.
- Limitations of Existing Methods: Explicit 3D methods improve fine-grained motion control and generative freedom but require burdensome spatial alignment and unified 3D preprocessing.The paper identifies scalability and real-world applicability constraints alongside compromised generative abilities from rigid conditioning and narrow task fine-tuning.
- ONE-SHOT Framework: ONE-SHOT factorizes generation into human motion, environmental geometry, and camera trajectories within a unified compositional synthesis framework.The framework is designed as a parameter-efficient system built on pre-trained VFMs with sparse parameter optimization.
- Spatial-Decoupled Motion Injection: Canonical-space cross-attention injects human motion separately from environmental cues, using sparse condition tuning to reduce restrictive spatial coupling and VFM collapse risk.The architecture adopts a ControlNet-style design to leverage pre-trained VFM knowledge.
- Dynamic-Grounded-RoPE: Dynamic-Grounded-RoPE establishes spatial correspondences between 2D environment projections and canonical human poses without complex heuristic 3D coordinate alignment.The pipeline synthesizes an environment-only sequence under a controllable camera trajectory, then overlays a coarse human layout.
- Hybrid Context Integration: Hybrid Context Integration combines static reference tokens for subject identity with dynamic memory tokens for overall appearance, supporting minute-level consistency and reducing identity drift and environmental artifacts.The mechanism enables long-shot consistent human-environment video generation over minutes.
2 Related Work
Related work progresses from general video editing toward structured human reenactment and explicit human-environment synthesis. However, existing approaches remain limited by weak structural awareness or reliance on isolated/static settings and explicit 3D scene alignment.
- General Video Editing: Large-scale video generation has enabled general-purpose editing, but these methods often model humans as holistic pixel regions without sub-pixel physical accuracy or structural awareness.This limitation motivates specialized approaches using explicit human structural priors and 3D environments.
- Human-Centric Reenactment: Human-centric reenactment uses structured motion priors to drive realistic character animation, with methods incorporating SMPL-based 3D shape priors, aligned skeleton signals, and implicit facial features.Champ improves robustness to large viewpoint changes, while Wan-Animate targets controllability and expressiveness through multi-stage training.
- Human-Centric Reenactment: Realisdance-DiT modifies the DiT architecture and uses flexible fine-tuning to improve motion stability in rare poses and complex character-object interactions.Despite improved 2D fidelity, these reenactment methods primarily animate characters in isolated or static settings.
- Human-Environmental Video Synthesis: Human-environmental video synthesis grounds movement in explicit 3D scenes, with Uni3C aligning human motion to scene geometry and RealisMotion using decoupled world-space control with physical priors.These methods address complex interactions and mitigate artifacts such as interpenetration.
3 Preliminaries
The method builds on Wan2.1, a pretrained video foundation model trained with flow matching in latent space, and uses RoPE to encode token positions for attention. Its architecture adds a conditioning branch that injects environmental, identity, context, and disentangled motion signals into the VFM.
- Video Foundation Models: Wan2.1 encodes an input video into a latent tensor z_0 ∈ R^(T×H×W×C), where T, H×W, and C denote frames, spatial token grid, and channels.The VFM learns a time-dependent velocity field that transports a noise latent toward the encoded video latent.
- Video Foundation Models: Flow matching defines the ground-truth velocity as v_t = dz_t/dt = z_1 − z_0 along the rectified-flow path.This velocity describes transport between the noise and data latents.
- Video Foundation Models: The conditional velocity predictor v_θ is trained by minimizing mean squared error between its predicted and ground-truth velocities.The objective is evaluated over z_0, z_1, and t under conditioning signals c.
- Model Architecture: The model augments the pretrained VFM with a conditioning branch that encodes environmental conditions, identity appearance, and context memory while injecting disentangled human dynamics.Environmental conditions use 2D-projected point clouds and depth maps; motion enters through Decoupled Motion Cross-Attention.
- Rotary Position Embedding: RoPE applies position-dependent rotations to paired query and key channels, making attention sensitive to relative token offsets.For latent-grid positions p = (t, x, y), rotated Q and K are computed using R(p).
4 Methodology
ONE-SHOT formulates human–environment video synthesis as generating a reference identity performing specified motion coherently within a 3D scene and controllable camera trajectory. Its methodology decouples environmental, motion, appearance, memory, text, and mask conditions, then integrates them through ControlNet-style residual conditioning, cross-attention, Dynamic-Grounded-RoPE, and multi-task masking.
- Task Definition: The task synthesizes video x1:T from a scene point cloud, camera extrinsics, identity appearance, optional scene memory, human motion, and text instructions.The output may be represented as video x1:T or latent z0 ∈ R^T×H×W×C, with identity and memory embeddings maintaining visual coherence.
- Decoupled Conditioning Framework: The conditioning set comprises text, environmental RGB–D projections, canonical-space motion and coarse layout, identity, context memory, and a spatio-temporal synthesis mask.A lightweight ControlNet-style parallel branch injects control signals as hierarchical residual additions into selected layers of the frozen video foundation model.
- Environmental Condition Sequence and Hybrid Context Integration: Environmental conditions render per-frame RGB–D projections of the point cloud under target camera extrinsics, while identity and dynamically retrieved memory stabilize appearance across long sequences.Identity uses a static reference segment, whereas memory is optional and enabled for long-horizon generation from previously generated content under similar viewpoints.
- Decoupled Motion Cross-Attention: Motion control separates canonical-space local dynamics from video-grid global placement, injecting canonical motion tokens through dedicated cross-attention.Dynamic-Grounded-RoPE rescales query positions within each placement region while using canonical coordinates for motion keys, aligning the two spatial domains without 3D alignment.
- Multi-Task Training: Multi-task masking enables heterogeneous environmental and motion supervision by specifying which regions are synthesized versus preserved in the masked latent target.The target combines ground-truth latents in masked regions with preserved latents elsewhere through element-wise multiplication and broadcasting.
5 Experiments
Experiments show that ONE-SHOT achieves strong fidelity, temporal realism, motion smoothness, and scene consistency in self-reconstruction and cross-composition, while supporting flexible controls and long-horizon generation. Ablations further demonstrate that Dynamic-Grounded-RoPE is important for stable spatial correspondence and motion following.
- Datasets: Training uses heterogeneous datasets providing complementary supervision for dynamic-camera human and camera motion, diverse body motions, and scene information.The mixture includes EMDB2, a human-motion subset of MotionX, and ARKitScenes.
- Quantitative results: ONE-SHOT achieves the best FID and FVD on Traj100 self-reconstruction, with the highest Motion Smoothness and Background Consistency.RealisMotion achieves higher SubjectSim, but ONE-SHOT offers a more balanced trade-off across fidelity, motion stability, and background coherence.
- Quantitative results: In cross-composition identity-and-motion swaps across scenes, ONE-SHOT achieves the best FID/FVD, Motion Smoothness, and Background Consistency while remaining competitive in SubjectSim and FaceSim.These results indicate that the decoupled controls generalize beyond reconstruction.
- Qualitative results: ONE-SHOT produces coherent backgrounds and faithful motion dynamics, independently swaps identity, motion, environment, and camera trajectory, and supports text-guided editing and long-horizon generation.Long-horizon demonstrations preserve scene and human appearance consistency over time.
- Ablation studies: Removing Dynamic-Grounded-RoPE grounding or using naive label RoPE degrades FID/FVD, motion smoothness, spatial placement, and motion following.2D spatial coordinate–aware scaling is therefore important for stable spatial correspondence between canonical and other spatial domains.
6 Limitation
ONE-SHOT’s quality depends on accurate reconstructed scene point clouds, camera trajectories, and human estimates. It can also fail with severely misplaced user-specified bounding boxes and some temporal ...
- Reconstruction and estimation errors: Generation quality and scene consistency degrade when reconstructed point clouds or camera trajectories are inaccurate, sparse, or noisy.Such errors can arise from imperfect depth estimation or human removal.
- Reconstruction and estimation errors: Errors from the human estimation module, Human3R in this work, may propagate to the final results.
- Bounding-box grounding: The method can fail in extreme bbox-grounding cases, such as severely misplaced user-specified boxes.
7 Conclusion
ONE-SHOT is a parameter-efficient framework for compositional human-environment video synthesis that addresses burdensome 3D alignment, over-conditioning, and short-horizon generation. Its Dynamic-Grounded-RoPE and Hybrid Context Integration provide structural grounding and minute-level subject and scene consistency while preserving creative diversity through lightweight LoRA fine-tuning.
- Conclusion: ONE-SHOT targets burdensome 3D alignment, over-conditioning that limits generative flexibility, and short-horizon synthesis in human-centric VFMs.The framework is designed to address three practical bottlenecks identified in prior methods.
- Conclusion: Dynamic-Grounded-RoPE establishes spatial correspondences directly in attention, avoiding fragile pre-alignment while retaining precise structural grounding.This mechanism eliminates the requirement for 3D alignment across disparate spatial domains.
- Conclusion: Hybrid Context Integration combines static reference tokens and dynamic memory tokens to sustain subject identity and scene coherence for minute-level long-horizon generation.The mechanism supports consistency across extended video synthesis.
- Conclusion: Extensive experiments and ablations show that ONE-SHOT balances structural control and creative diversity more favorably than prior methods with lightweight LoRA fine-tuning.The conclusion attributes this balance to a parameter-efficient adaptation strategy.
A Implementation Details
The model uses a frozen Wan2.1-14B video foundation backbone and trains only lightweight components for conditioning and controllable motion injection.
- Backbone and trainable modules: The framework freezes the pretrained Wan2.1-14B VFM backbone parameters.The backbone is part of a pretrained video editing framework.
- Backbone and trainable modules: Training updates standard LoRA adapters in the conditioning branch and motion cross-attention modules for controllable motion injection.The passage identifies these as the only lightweight trainable components.
B Training Data and Pre-processing
ONE-SHOT trains on a balanced mixture of open-source and self-collected human-centric videos, emphasizing diverse outdoor motion and camera behaviors. Its evaluation uses Traj100 for self-reconstruction and a separate self-collected cross-composition set for novel identity–motion–scene recombination.
- Data sources: The dataset balances camera motion and subject movement to expose ONE-SHOT to broader compositional settings while remaining reproducible.The design includes videos with and without camera motion, and with roughly stationary subjects or clear global movement.
- Test sets: Evaluation uses Traj100 for self-reconstruction and a separate self-collected test set for cross-composition generalization.The cross-composition set addresses novel identity-motion-scene recombination, which standard self-reconstruction benchmarks cannot evaluate.
- Test sets: The cross-composition test set contains 5 identities, 10 motions, and 10 scenes, forming 83 plausible combinations.It is constructed separately from the self-reconstruction benchmark to evaluate recombination across identities, motions, and scenes.
C Details on 2D Bounding Box as Coarse Placement
ONE-SHOT uses 2D bounding boxes as soft coarse-placement cues rather than rigid constraints. Users can initialize placement manually or automatically, after which motion and camera geometry determine the subsequent boxes, with static boxes also supported in suitable shots.
- Bbox during Training: Training-time scaling and jittering make the bbox a flexible spatial guide, improving robustness to moderate misalignment and preserving natural foot-ground contact.The model is explicitly encouraged not to treat the bbox as a rigid constraint.
- General cases: In general cases, users specify only a coarse square first-frame bbox B0, while subsequent placement is automatically determined from the source SMPL-X motion and intrinsic parameters.The first-frame box serves as a soft signal of initial human placement.
- Automatic bbox construction: The method estimates the initial 3D human root from the first-frame bbox, propagates it using relative SMPL-X root motion, and projects each frame under target-camera extrinsics with depth-dependent bbox scaling.This automatically produces coarse boxes for arbitrary human motion and target-camera trajectories, requiring only coarse first-frame placement from the user.
- Scene-only cases: For scene-only tracking or hand-held shots, the bbox may remain static across frames because camera motion already provides a following perspective.This is described as more user-friendly than methods requiring complex and fragile 3D pre-processing.
D More Discussion · E Border Impact
The discussion explains that ONE-SHOT is evaluated with official baseline pipelines, preserves camera and scene structure through explicit RGB-D conditioning, and avoids fragile global alignment. The border-impact discussion highlights misuse risks and concerns associated with the extensive datasets underlying the model.
- D More Discussion: Baselines use official checkpoints and inference pipelines, with best-effort adaptation only when interfaces do not match the compositional task.The task involves disentangled control of motion, scene, camera, and identity.
- D More Discussion: WAN-VACE is fine-tuned on the same training data as ONE-SHOT and evaluated for self-reconstruction and cross-composition using SMPL-X controls.Table 3 specifies that lower FID/FVD is better, while higher values are better for the other metrics.
- D More Discussion: ONE-SHOT preserves camera motion and scene layout using per-frame RGB-D projections rendered from the target point cloud under the target camera trajectory.These projections provide explicit geometric conditioning.
- D More Discussion: ONE-SHOT relies on standard off-the-shelf preprocessing for scene and human reconstruction rather than fragile 3D global alignment.The preprocessing choice is discussed as part of pipeline clarity and dependence on 3D preprocessing.
- E Border Impact: The study presents ONE-SHOT as a parameter-efficient framework for compositional human-environmental video synthesis that supports creative expression and long-horizon production.The passage frames these capabilities as advances in controllable video generation.
- E Border Impact: The generative capabilities may facilitate misleading information or fabricated visuals, requiring high vigilance regarding potential misuse.The passage identifies misuse as a consequence of creative and long-horizon video-generation capabilities.
- E Border Impact: Because the model is built upon extensive datasets, the study states that associated concerns must be addressed.The supplied passage introduces this issue without specifying the concerns further.