Source-linked AI summary
Tri-Prompting: Video Diffusion with Unified Control over Scene, Subject, and Motion
Zhenghong Zhou, Xiaohang Zhan, Zhiqin Chen, Soo Ye Kim, Nanxuan Zhao, Haitian Zheng, Qing Liu, He Zhang, Zhe Lin, Yuqian Zhou, Jiebo Luo
TL;DR
Video diffusion still lacks unified, fine-grained control over scene composition, multi-view subject identity, and motion. Tri-Prompting combines multi-view references with dual-conditioned motion control in a two-stage framework, and outperforms specialized baselines while supporting 3D-aware insertion and manipulation. Its scope remains limited by symmetric-reference ambiguity, offline generation, and high inference cost.
Problem
Existing video methods often handle scene, subject, and motion controls separately, with limited multi-view identity preservation under arbitrary pose changes.
Method
Tri-Prompting uses multi-view references and dual conditioning—XYZ trajectories for backgrounds and low-resolution RGB proxies for foregrounds—within a two-stage video diffusion framework.
Results
Tri-Prompting significantly outperforms DaS in motion accuracy and Phantom in multi-view subject-to-video generation, including identity and 3D consistency.
Takeaways & Limitations
The framework supports jointly controlled video workflows including 3D-aware subject insertion and manipulation of existing subjects in scenes.
Takeaways & Limitations
Highly symmetric references can cause transient identity flips, while the 14B system remains offline and costly to run.
Abstract
from arXiv · showhide
Recent video diffusion models have made remarkable strides in visual quality, yet precise, fine-grained control remains a key bottleneck that limits practical customizability for content creation. For AI video creators, three forms of control are crucial: (i) scene composition, (ii) multi-view consistent subject customization, and (iii) camera-pose or object-motion adjustment. Existing methods typically handle these dimensions in isolation, with limited support for multi-view subject synthesis and identity preservation under arbitrary pose changes. This lack of a unified architecture makes it difficult to support versatile, jointly controllable video. We introduce Tri-Prompting, a unified framework and two-stage training paradigm that integrates scene composition, multi-view subject consistency, and motion control. Our approach leverages a dual-condition motion module driven by 3D tracking points for background scenes and downsampled RGB cues for foreground subjects. To ensure a balance between controllability and visual realism, we further propose an inference ControlNet scale schedule. Tri-Prompting supports novel workflows, including 3D-aware subject insertion into any scenes and manipulation of existing subjects in an image. Experimental results demonstrate that Tri-Prompting significantly outperforms specialized baselines such as Phantom and DaS in multi-view subject identity, 3D consistency, and motion accuracy.
1 Introduction
Tri-Prompting addresses the fragmented control landscape by jointly controlling scene, multi-view subject identity, and motion in one video diffusion framework. Its dual-conditioning design and two-stage training support disentangled foreground/background motion, 3D consistency, and competitive results against specialized baselines.
- Motivation: Existing methods typically address scene, subject, or motion separately, limiting unified control and multi-view identity preservation under changing poses.Motion-focused methods struggle with subject identity, while subject-to-video methods lack motion control.
- Method: Tri-Prompting integrates scene composition, multi-view subject consistency, and disentangled motion control within a single model.The framework uses first-frame and multi-view references together with motion conditioning.
- Method: Its motion module combines XYZ trajectories for background scenes with downsampled RGB grids for foreground subjects.The RGB grids encode primary subject movement while suppressing fine-grained motion details.
- Method: A two-stage strategy first learns scene and multi-view subject fusion, then trains ControlNet for dual-conditioning motion control.This separates foundational scene/identity control from subsequent motion-control learning.
- Advantages: Dual conditioning decouples background and foreground motion, while multi-view references preserve appearance and 3D consistency across large viewpoint changes.The design supports rigid and non-rigid objects and natural object-scene interactions.
- Results: Tri-Prompting surpasses DaS in motion accuracy and Phantom in multi-view identity preservation, while enabling 3D-aware insertion and object manipulation.Evaluations also report better PSNR and LPIPS than DaS and stronger video quality, identity, and 3D consistency than Phantom.
2 Related work
Related work develops specialized controls for spatial alignment, motion, or identity, with newer systems beginning to unify multiple signals. Tri-Prompting extends this direction through explicit, separated foreground/background conditioning and multi-view identity control.
- Video diffusion: Early text-to-video diffusion models add temporal layers to image diffusion backbones but can struggle with plausible physics, coherent motion, and complex dynamics.Transformer-based video diffusion models address these limitations through spatio-temporal tokens and full self-attention.
- Controllable generation: Controllable video methods use spatial signals such as pose, depth, segmentation, boxes, and strokes, alongside temporal signals such as camera trajectories, optical flow, and tracking cues.Identity control commonly uses reference images, with signals injected through encoders, attention modules, or ControlNet-style adapters.
- Unified control: Unified multimodal video models combine multiple control signals in one network, improving flexibility and efficiency over task-specific specialist models.Tri-Prompting extends unified temporal and identity control from single-view to multi-view subject customization.
- Control paradigms: Implicit control simplifies inference but requires precise annotations and large-scale data, whereas explicit control is more data-efficient but can hallucinate or overfit sparse or noisy signals.These trade-offs motivate careful design of explicit motion conditioning.
- Tri-Prompting: Tri-Prompting separates foreground and background motion using XYZ tracks and low-resolution RGB, targeting robust control under extreme poses and multi-view identity.This combination distinguishes the method from prior explicit-control approaches focused mainly on isolated motion or single-view identity.
3 Method
Tri-Prompting combines scene conditioning, multi-view subject identity, and disentangled motion control in a unified video diffusion pipeline. Its two-stage training and scheduled ControlNet guidance support controllable generation while addressing motion realism and subject consistency.
- Overview: Tri-Prompting jointly controls the scene, multi-view subject identity, and foreground and background motion using a unified video diffusion framework.The model uses first-frame and multi-view reference images together with dual-conditioning 3D motion cues.
- Two-stage training: The first training stage fuses the first-frame scene image with multi-view subject references, while the second adds explicit motion control through a ControlNet module.The base model is trained first for scene and identity conditioning; ControlNet is then added while the base weights remain fixed.
- Dual-conditioning motion control: The motion module assigns XYZ tracking trajectories to background scenes and downsampled RGB grids to foreground subjects.The two conditions are composited spatially exclusively; low-resolution RGB supplies subject-motion cues while multi-view references restore appearance details.
- Inference scale schedule: ControlNet guidance is gradually reduced during inference to balance motion controllability against video realism.For 50 denoising steps, the scale is linearly annealed during the first N_decay steps and then held at s_min.
- Inference workflows: The inference pipeline supports camera control, motion transfer, object manipulation, and workflows such as interactive 3D-aware subject motion.Users select the scene and multi-view character, then drive subject translation, rotation, and camera pose through an interactive interface.
4 Experiment
Experiments evaluate Tri-Prompting on motion controllability, multi-view subject consistency, and unified applications against specialized baselines. Ablations show that dual conditioning, multi-view references, and ControlNet scheduling improve fidelity, 3D consistency, and natural motion, while limitations remain in ambiguity, latency, and offline use.
- Evaluation Setup: The evaluation uses video reconstruction for motion controllability and multi-view subject-to-video generation for identity consistency and quality.DaS and Phantom provide specialized comparison points for these subtasks.
- Comparison with DaS and Phantom: Tri-Prompting maintains robust alignment under extreme motion where tracking-point-only methods hallucinate as points disappear.Multi-view references also recover details occluded in the first frame.
- Comparison with DaS and Phantom: 26.5% improvement in 3D consistency is achieved over Phantom, which exhibits structural distortions during drastic pose changes.Tri-Prompting preserves natural identity and 3D consistency across motion-controlled views.
- Applications and Generalization: Tri-Prompting generalizes to diverse scenes, subjects, motions, and anime, movie, and real-world styles beyond its mixed game and common-subject training videos.The reported applications include insertion and manipulation with natural foreground-background interactions.
- Ablation Study: Dual conditioning with RGB proxies achieves superior PSNR, SSIM, and LPIPS compared with XYZ tracking alone.RGB proxies provide more reliable guidance for extreme poses and 3D motion when tracking points are sparse or missing.
- Ablation Study: Three subject views reduce Ealign from 0.035 to 0.025 and consistently improve identity similarity compared with fewer views.More views reduce structural ambiguity and support identity preservation during pose changes.
- Ablation Study: ControlNet scale decay from 1 to 0.005 in 10 steps produces smooth realistic gaits, whereas a fixed 1.0 scale yields rigid translational motion.Low-resolution RGB also supports flexible interactions, while higher resolution increases motion rigidity.
- Limitations: The system remains limited by symmetric-reference ambiguity, offline-only generation, and the inference cost of its 14B backbone.The paper reports approximately 5 minutes on 8×A100 GPUs for 50 denoising steps at 480×832 resolution with 49 frames.
5 Conclusion
Tri-Prompting unifies scene, subject, and motion control in one video diffusion model. Its dual conditioning and multi-view references support 3D-consistent subjects, motion accuracy, insertion, manipulation, and diverse video styles.
- Conclusion: Tri-Prompting integrates scene, subject, and motion control within a single video diffusion model.The framework combines XYZ background coordinates with low-resolution RGB foreground proxies and multi-view references.
- Conclusion: The method outperforms specialized baselines in motion accuracy and multi-view subject-to-video generation.The paper highlights applications including 3D subject insertion and manipulation in scenes.
- Conclusion: Tri-Prompting supports arbitrary scene, subject, and motion control across general video styles.The authors identify reducing inference time as future work toward interactive applications.
Appendix
The appendix provides additional results, a keyboard motion-control interface, implementation details, and a limitations discussion.
- Appendix: The appendix contains more method results, a keyboard motion control GUI, implementation details, and a limitations discussion.These materials are organized into four appendix sections.
1 More Visualization Results
Additional visualizations demonstrate Tri-Prompting across games, movies, real-world scenes, and varied subjects. The results emphasize subject preservation, scene integration, motion tracking, and realistic interactions.
- More Visualization Results: Additional results cover games, movies, and real-world cases to illustrate performance across domains.The paper directs readers to the project page for video results.
- More Visualization Results: Scene and subject replacement preserve detailed multi-view appearance, harmonious interactions, shadows, and smooth movement.Examples include Times Square and old-street backgrounds, bear and anime-character replacements, object rotation, and camera movement.
- More Visualization Results: Subject insertion remains effective for humans, animals, and vehicles in real-world scenes.Inserted subjects remain consistent with input appearance and motion while blending naturally into their surroundings.
- More Visualization Results: Subject manipulation generalizes across anime, movie, and game scenes while preserving details and tracking input motion.The examples include realistic interaction effects for subjects such as a horse and man during turnarounds.
2 Keyboard Motion Control GUI
The keyboard motion GUI lets users select a background and 3D subject, manipulate both through keyboard controls, and record the resulting motion-control video for generation.
- Users select a background image and 3D subject, then control background/camera motion and subject motion through a keyboard interface.The GUI composites the background with the rendered 3D subject during manipulation.
- The recorded manipulation session is temporally resampled to 49 frames for use as motion-control video.
- The interface presents subject images, a keyboard-driven motion-control video, and the generated video as its core workflow.
3 Implementation Details
Tri-Prompting uses staged training and inference settings to combine scene, multi-view subject, and dual-conditioning motion control.
- Stage I fine-tunes the base Phantom 14B model with LoRA for scene and multi-view subject control.
- Stage II further trains a ControlNet module for dual-conditioning motion control.
- Inference includes a ControlNet scale schedule with decay-step settings.
4 Limitation
Tri-Prompting can produce transient identity-orientation inconsistencies when highly symmetric multi-view references provide insufficient asymmetric cues.
- Highly symmetric multi-view references can make the subject’s front and back identity orientation ambiguous during motion.
- The ambiguity may cause a short-lived inconsistency, such as a transient head-orientation flip during turning.The issue is described as non-persistent across the whole sequence.
Appendix 21
The appendix presents applications and a failure case covering subject insertion, subject manipulation, and motion-following outputs across real-world and fictional scenes.
- The appendix includes a failure case of the proposed method.
- Distinctive reference views or stronger motion constraints can often alleviate the ambiguity-related failure mode.
- Insertion examples show multi-view subject references, an original background image, motion-control video, and generated frames across scene, subject, and motion conditions.
- Subject insertion examples cover non-rigid humans and animals as well as rigid vehicles, while following motion instructions.
- Subject manipulation examples span anime, movie, and game scenes, demonstrating diverse video outputs across fictional domains.