PaperlayerView exact revision on arXiv

Source-linked AI summary

Controllable Video Generation: A Survey

Yue Ma, Kunyu Feng, Zhongyuan Hu, Xinyu Wang, Yucheng Wang, Mingzhe Zheng, Bingyuan Wang, Qinghe Wang, Xuanhua He, Hongfa Wang, Chenyang Zhu, Hongyu Liu, Yingqing He, Zeyu Wang, Zhifeng Li, Xiu Li, Sirui Han, Yike Guo, Wei Liu, Dan Xu, Linfeng Zhang, Qifeng Chen

arXiv:2507.16869v3cs.GRcs.CV

TL;DR

Text prompts alone provide limited control over complex, multimodal, and fine-grained video requirements. This survey synthesizes generative foundations, control mechanisms, taxonomies, applications, and research challenges, finding that diverse controls increasingly support simultaneous customization while robust multi-constraint systems remain difficult to build.

  • Problem

    Text prompts alone inadequately express complex, multimodal, and fine-grained video requirements, motivating controllable video generation.

  • Method

    The survey reviews generative foundations, representative models, control mechanisms, conditioning-signal taxonomies, applications, limitations, and future directions.

  • Results

    Universal control combines text, poses, depth, and motion to customize content, style, motion, and layout simultaneously.

  • Takeaways & Limitations

    The survey’s taxonomy and synthesis clarify methods while revealing challenges in combining diverse control signals.

  • Takeaways & Limitations

    Robust and scalable systems remain challenging because methods often optimize only subsets of structure, temporal consistency, style, or other constraints.

Abstract

from arXiv · show

With the rapid development of AI-generated content (AIGC), video generation has emerged as one of its most dynamic and impactful subfields. In particular, the advancement of video generation foundation models has led to growing demand for controllable video generation methods that can more accurately reflect user intent. Most existing foundation models are designed for text-to-video generation, where text prompts alone are often insufficient to express complex, multi-modal, and fine-grained user requirements. This limitation makes it challenging for users to generate videos with precise control using current models. To address this issue, recent research has explored the integration of additional non-textual conditions, such as camera motion, depth maps, and human pose, to extend pretrained video generation models and enable more controllable video synthesis. These approaches aim to enhance the flexibility and practical applicability of AIGC-driven video generation systems. In this survey, we provide a systematic review of controllable video generation, covering both theoretical foundations and recent advances in the field. We begin by introducing the key concepts and commonly used open-source video generation models. We then focus on control mechanisms in video diffusion models, analyzing how different types of conditions can be incorporated into the denoising process to guide generation. Finally, we categorize existing methods based on the types of control signals they leverage, including single-condition generation, multi-condition generation, and universal controllable generation. For a complete list of the literature on controllable video generation reviewed, please visit our curated repository at https://github.com/mayuelala/Awesome-Controllable-Video-Generation.

1 Introduction

Controllable video generation extends text-to-video foundation models with non-textual conditions to better express fine-grained user requirements. This survey organizes the field’s foundations, methods, applications, limitations, and future directions.

  • Motivation: Video generation foundation models combine advanced generative paradigms, large-scale datasets, massive parameters, and modern architectures to improve video quality.The passage identifies diffusion and autoregressive models as major paradigms underlying these foundation models.
  • Motivation: Text prompts alone often cannot express complex, multimodal, and fine-grained requirements, motivating control signals beyond text.Examples include camera trajectories and character actions.
  • Survey scope: The survey reviews controllable video generation’s theoretical foundations, practical applications, research advances, limitations, and future work.Its review begins with video generative-model concepts and follows methods into applications and downstream tasks.
  • Contributions: Its taxonomy classifies controllable generation methods by input control signals and accompanies this with foundations of GAN-, VAE-, Flow-, diffusion-, and autoregressive architectures.The taxonomy is intended to clarify existing methods and reveal core challenges.
  • Contributions: The survey also covers conditional-generation techniques, practical scenarios, shortcomings, and potential avenues for further exploration.The paper emphasizes the practical significance of conditional generation within the AIGC landscape.
  • Organization: The paper is organized around generative paradigms, representative models and taxonomy, control mechanisms, applications, and limitations.These topics are presented across Sections 2, 4, 5, 6, and 7.

2 Preliminaries

The preliminaries introduce major generative paradigms used for video generation and explain their core mechanisms. They cover GANs, VAEs, diffusion, flow-based, flow-matching, and autoregressive models.

  • GANs and VAEs: GANs generate samples through a minimax game between a generator and discriminator, while VAEs optimize an evidence lower bound using a variational posterior.GANs distinguish real from generated samples; VAEs approximate the true posterior.
  • GANs and VAEs: GANs and VAEs provide foundations for later diffusion and flow-based generative models.The passage explicitly positions both classical models as foundations for subsequent developments.
  • Diffusion Models: Diffusion models add Gaussian noise to data and learn the reverse denoising process for high-quality sample generation.DDPMs implement this process through a Markov chain.
  • Flow-based Models: Flow-based models learn invertible transformations that convert a simple base distribution into a complex target distribution.Continuous Normalizing Flows model this transformation as a continuous-time ODE governed by a vector field.
  • Flow Matching: Flow Matching learns a vector field from paired base and target samples using straight-line interpolation and squared velocity-matching error.The loss matches the direction and magnitude of transport between x0 and x1.
  • Autoregressive Models: Autoregressive models factorize sequence distributions by conditioning each element on preceding elements, though sequential generation affects efficiency.Their temporal correlations have renewed interest in video generation, including hybrid autoregressive-diffusion designs.

3 Method

The survey uses a PRISMA-guided, four-phase literature review and organizes controllable video generation through a multilayered framework centered on seven control modalities.

  • Literature Collection and Screening: The literature review follows four phases: identification, screening, eligibility assessment, and final inclusion, guided by PRISMA.The review targets controllable video generation research published from 2020 to 2025.
  • Literature Collection and Screening: 708 candidate papers were identified through venue-based searches and queries combining controllability, video generation, and generative-model terms.The search covered major computer vision and machine learning venues, high-impact arXiv preprints, and databases including ACM Digital Library and arXiv.
  • Literature Collection and Screening: After title-and-abstract screening, 388 papers remained, with two-author screening agreement reaching Cohen’s Kappa 0.87 on a random subset of 100 papers.Papers without controllable video generation or external-control mechanisms were excluded.
  • Literature Collection and Screening: Full-text eligibility assessment removed 142 papers, leaving 246 papers for final inclusion.Exclusions included papers failing the criteria upon detailed reading or contributing only limited relevance to controllable video generation.
  • Taxonomy: The taxonomy spans technical foundations, controllable modules, control modalities, creative support, technical tasks, and practical applications.Control modality is adopted as the primary dimension because it shapes architecture, conditioning mechanisms, technical challenges, and the connection between user intent and generated content.
  • Taxonomy: The survey divides controllable generation into seven modality classes and examines both single-condition and multi-condition settings.Examples include structure signals such as pose, depth, sketches, and bounding boxes, alongside identity and image controls; multi-condition methods must integrate interacting conditions consistently over time.

4 Video Generation Foundation Models

Video generation foundation models are mainly diffusion-based or autoregressive, with diffusion methods modeling complex spatial-temporal dependencies through iterative denoising.

  • Model Paradigms: Video generation foundation models are typically categorized into diffusion-based and autoregressive paradigms.These models have attracted attention for their ability to generate high-fidelity videos.
  • Diffusion Models: Diffusion-based methods model data distributions through iterative denoising, capturing complex spatial-temporal dependencies.The survey identifies diffusion models as a major foundation-model paradigm for video generation.
  • Diffusion Architectures: Diffusion video-generation models use either UNet-based frameworks or the newer Diffusion Transformer architecture.The survey introduces representative frameworks and provides detailed model information in supplementary material.

5 Controllable Video Generation Models with Various Conditions

The survey organizes controllable video generation by control signals, covering structural, identity, image, temporal, audio, landmark, and other modalities. These methods condition video synthesis on spatial, geometric, identity, or motion information, while challenges remain in physical awareness, consistency, and generalization.

  • Multimodal Control: The literature includes methods supporting multiple control modalities, with Venn-diagram overlaps counting works that combine control signals across seven categories.The categories are Structure, Image, Audio, Temporal, ID, Universal, and Other.
  • Taxonomy: The survey categorizes controllable video generation according to the primary control signal, including structure, ID, image, temporal, audio, other, and universal control.The taxonomy supports analysis of foundational techniques, architectural innovations, and category-specific challenges.
  • Structure Control: Structure control specifies spatial layout, articulated-object conformation, and scene geometry to guide physically plausible and semantically consistent video synthesis.Structural cues constrain scene arrangements and object configurations during generation.
  • Pose-Guided Generation: Pose-guided generation animates predominantly human subjects according to predefined pose sequences while preserving appearance across temporally coherent frames.Pose inputs may use 2D keypoints, 3D coordinates, or DensePose representations.
  • Pose-Guided Generation: Pose-guided methods remain challenged by self-occlusion, texture and identity consistency, natural transitions, and generalization to unseen poses, body shapes, and clothing.These difficulties can require robust inpainting and extensive, diverse datasets.
  • Depth-Guided Generation: Depth-guided generation uses depth-map sequences to control scene 3D structure, object layout, camera-relative distances, occlusions, and cross-frame geometry.Depth maps can be estimated from RGB videos, captured with sensors, or rendered from 3D models.
  • Depth-Guided Generation: Depth-guided systems face errors from inaccurate or temporally inconsistent depth maps, while rapidly changing depths and non-rigid deformations remain difficult.Input-depth noise can propagate into distorted generated geometry.
  • Landmark-Guided Control: Landmark-guided control uses spatial keypoints or landmark trajectories to constrain facial expressions, gestures, body poses, and regional deformations.Signals may be static, dynamic, or derived from high-level descriptions such as poses or emojis.

5.2 ID Control

ID control preserves or manipulates entities’ visual identity throughout generated videos using reference images or identity embeddings. The survey covers person- and subject-guided generation, identity-injection architectures, and the shift toward tuning-free and more compositional systems.

  • ID Control: ID control preserves and manipulates the visual identity of humans, animals, or objects across an entire generated video.Reference images or identity embeddings help maintain subject integrity across poses, motions, and viewpoints.
  • Person-Guided Generation: Person-guided generation animates a specific individual from one or more ID images using text, pose, or audio driving signals.ID images encode static identity, while driving signals control dynamics.
  • Identity Injection: Representative methods inject identity through face adapters and cross-attention, DiT anchor-concept prompts, or reward-based identity-motion alignment objectives.Examples include ID-Animator, Movie Weaver, PersonalVideo, and Vlogger.
  • Subject-Guided Generation: Subject-guided generation constrains which characters, objects, or scenes appear and how they co-occur, interact, and persist in context.Multi-subject methods address co-occurrence and temporal consistency using mechanisms such as attention control with object segmentation.
  • Architectures: DiT backbones are increasingly explored alongside dominant UNet architectures to leverage global attention for improved temporal consistency.This architectural shift accompanies the use of conditional encoders and general control models such as adapters.
  • Trends and Challenges: The ID-control roadmap is moving from slow fine-tuning toward tuning-free approaches, while multi-subject generation still needs better disentanglement and composition.The stated goal is to resolve conflicts between different controllable methods within unified creative-support systems.

5.3 Image Control

Image control generates videos conditioned on reference-image features to preserve visual alignment, stylistic fidelity, and semantic relevance over time. The survey reviews image injection, motion disentanglement, physical conditioning, and persistent challenges in prompt-image alignment and multi-object consistency.

  • Image-Guided Generation: Image-guided video generation conditions synthesis on a reference image to maintain visual consistency and coherence across temporal frames.Reference features support stylistic fidelity, visual alignment, and semantic relevance.
  • Alignment Challenges: Reference-image influence can weaken later in a video as prompts dominate, creating a prompt-image alignment challenge.DreamVideo’s Image Retention module maintains information from both the input image and prompt.
  • Combined Conditions: Image conditions can combine with bounding-box, motion, and depth controls, including methods that predict motion or disentangle motion vectors from image information.These combinations target higher-quality, memory-efficient, and temporally consistent generation.
  • Physical Conditioning: PhysGen conditions training-free video generation on input forces and images while modeling physical properties such as mass and elasticity.The method also considers environmental conditions and external forces.
  • Summary: Image control maps visual features into the backbone through conditional encoders, while UNet, DiT, and autoregressive foundations support personalization and temporal coherence.Persistent challenges include multi-object consistency and explaining how models balance conflicting image and prompt inputs.

5.4 Temporal Control

Temporal control guides motion and timing across frames through signals such as trajectories, optical flow, and camera movements. The survey reviews these control forms, their diffusion-based implementations, and challenges involving consistency, realism, and complex scenes.

  • Temporal control regulates motion evolution and timing to produce coherent, realistically paced, causally consistent frame transitions.
  • Flow-Guided Generation: Flow-guided methods use optical flow, motion fields, or trajectories to improve temporal consistency and object dynamics.Examples include Motion-I2V, MOFA-Video, I2VControl, and MCDiff.
  • Challenges: Key difficulties include inaccurate-flow artifacts, physically implausible trajectories, occlusions, interactions, long-range consistency, and 3D camera disocclusions.The survey also identifies smooth camera transitions and integrated spatial-temporal control as ongoing challenges.
  • Trajectory-Guided Generation: Trajectory-guided generation lets users specify object or region paths through coordinates, bounding-box tracks, sketches, text, or drag-based interfaces.These inputs define object extent, location, or movement over time.
  • Trajectory-Guided Generation: Diffusion-based methods support motion prompts, explicit path following, appearance changes along trajectories, and image-to-video motion control.Representative methods include MotionBooth, Motion-I2V, TrailBlazer, Direct-a-Video, Peekaboo, and MOFA-Video.
  • Camera-Guided Generation: Camera-guided generation controls viewpoint, camera motion, and sometimes intrinsic parameters through poses, transformations, or textual descriptions.Methods address camera trajectories, novel views, and joint human-motion and camera control.

5.5 Audio Control

Audio control uses speech, music, or sound to guide temporally aligned and semantically coherent video generation. The survey covers voice-driven talking portraits, sound-guided synthesis, synchronization methods, and unresolved conflicts between visual and audio conditions.

  • Audio-guided generation conditions video synthesis on speech, music, or general sound to align visual content with audio cues.Voice control focuses on talking faces and personalized portrait animation, while sound control covers broader audio inputs.
  • Voice-Guided Generation: Voice-guided systems combine audio and image conditions to generate talking portraits with synchronized lips, head poses, and facial expressions.The central technical issue is estimating facial expressions from varied voice inputs.
  • Voice-Guided Generation: Voice-generation methods use pretrained audio encoders, disentangled motion spaces, audio-to-motion modules, face-locating components, and reference-image features.Examples include wav2vec-based extraction, EDTalk, EMO, EchoMimic, and EchoMimicV2.
  • Challenges: Audio-video synchronization, control imbalance, style variation, efficiency, and long-term consistency remain major challenges.Visual features can suppress subtle audio cues, while fine-grained alignment is needed to prevent synchronization drift.
  • Sound-Guided Generation: Sound-guided methods extract texture, pitch, rhythm, and emotional properties using MFCCs, Mel spectrograms, BEATs, or CLAP.Diffusion approaches such as AV-Link and ASVA use temporally aligned activations and audio features for synchronization.
  • Sound-Guided Generation: Text-audio methods coordinate motion and emotion through multimodal conditioning, dual-branch alignment, or refined text-to-video pipelines.TA2V, MotionCraft, and DAA2V address semantic and temporal alignment in different ways.

5.6 Other Control

Other control modalities extend controllable video generation beyond identity, structure, and motion. The survey discusses text rendering, style transfer, point trajectories, and bird’s-eye-view conditioning, emphasizing fine-grained control and persistent consistency challenges.

  • Other control includes text rendering, style transfer, point manipulation, and bird’s-eye-view conditioning for semantically aligned video generation.
  • Text Rendering: Text-rendering methods target vivid videos with aligned text for applications such as advertisements and movies.Earlier CLIP- and 2D-VAE-based approaches failed to generate clear, aligned text.
  • Style Control: Style-guided generation transfers artistic characteristics from reference images or videos to target videos for creative expression and visual aesthetics.Recent work adds fine-grained stylization and frameworks for diverse artistic styles.
  • Point Control: Point-guided generation uses sparse user-defined trajectories to animate specific objects or regions, with stability across space and time as its main challenge.Later approaches commonly condition pretrained diffusion models on point inputs.
  • BEV Control: BEV-guided generation uses top-down semantic maps to specify scene layout and object dynamics in controllable autonomous-driving videos.Challenges include geometric consistency, realistic appearance, dynamic interactions, and the semantic gap between maps and visual details.
  • Summary: These modalities connect foundation models with creative and interactive tools through conditional encoders, adapters, and cross-attention mechanisms.The field is shifting toward DiT backbones, while intuitive controls and automatic suggestions remain open directions.

5.7 Universal Control

Universal control integrates text, spatial features, and temporal signals so users can jointly specify content, style, motion, and layout. The survey reviews encoder-based, attention-based, token-unification, and caption-mediated designs while noting condition-balancing and evaluation challenges.

  • Universal control integrates text, spatial conditions, and temporal information to customize object behavior, motion, camera angles, and scene layout.
  • Frameworks: Universal-guided frameworks accept inputs such as camera poses, depth maps, human poses, motion sequences, video clips, sketches, and text.VideoComposer and Any2Caption exemplify explicit multimodal conditioning and structured-caption mediation.
  • Challenges: Major challenges include balancing conditions, preserving temporal consistency, and establishing comprehensive standardized evaluation across modalities.The survey notes that one condition can overshadow others and that universal-control evaluation criteria remain under development.
  • Condition Fusion: Universal control combines signals like text, poses, depth, and motion through encoders, attention fusion, and token unification.This supports simultaneous customization of content, style, motion, and layout beyond single-condition models.
  • Architectural Trends: Architectures are shifting from UNet-with-adapters designs toward scalable DiT and autoregressive backbones for multi-condition fusion and longer or higher-resolution sequences.FullDiT and InfinityStar illustrate these newer directions, while Any2Caption uses caption-based control.

6 Application

Controllable video generation supports applications ranging from video repair and object insertion to video-to-4D generation and autonomous-driving simulation. These tasks use controllable models to preserve spatial and temporal consistency while adding or reconstructing dynamic content.

  • Video Completion and Inpainting: Video completion and inpainting fill missing regions or remove unwanted objects while maintaining spatial and temporal consistency.These techniques are used in film post-production and historical video restoration.
  • Video Composition: Video composition embeds target objects into existing videos while preserving spatial-temporal integration and object appearance details.Applications include advertising, AR/VR, virtual product placement, furniture try-on, and animated-character storytelling.
  • Video-to-4D Generation: Video-to-4D generation models spatial geometry and temporal dynamics, enabling dynamic assets from handheld videos and supporting simulation and robotic perception.4D-fy combines multiple pretrained diffusion models through mixed score distillation to generate realistic appearance, structure, and motion.
  • Autonomous Vehicle and Generative World Model: DriVerse generates driving-simulation videos from a single image and future trajectories using multimodal trajectory prompts and latent motion alignment.The method targets high-fidelity videos with temporally and spatially consistent dynamic objects.

7 Discussion and Future Work

The survey identifies compositional controllability, reasoning-generation alignment, long-video coherence, and computational efficiency as central challenges. It highlights constraint orchestration, multimodal agents, hybrid architectures, and efficient long-range modeling as promising directions.

  • Unified Multi-Condition Control and Constraint Orchestration: Existing methods often optimize only selected constraints, creating tradeoffs when structure, temporal consistency, motion, and style must be satisfied together.This limitation makes robust and scalable support for diverse controls challenging.
  • Unified Multi-Condition Control and Constraint Orchestration: Constraint orchestration could combine heterogeneous conditions through hierarchical or modular frameworks and adaptively re-weight them according to user intent and content.The survey cites VideoComposer and FullDiT as steps toward universal or multi-condition integration, while noting that competing constraints remain difficult to balance efficiently.
  • Unified Video Reasoning and Generation: Reasoning modules could translate high-level intent into concrete conditions and support iterative refinement, but alignment with video generators remains limited for complex instructions and long horizons.Phantom demonstrates progress toward semantically rich, prompt-driven generation.
  • Unified Video Reasoning and Generation: Multimodal LLM agents can provide a unified interface for planning, grounding, and refinement, but require reliable verification, safety mitigation, and efficient cross-modal components.The proposed interaction combines text with other signals while remaining natural and editable.
  • Hybrid and Scalable Autoregressive Methods: Long-video generation remains difficult because diffusion models can accumulate inconsistencies, whereas autoregressive models improve temporal coherence at higher latency and compute.The survey presents hybrid diffusion-autoregressive designs as a way to combine global latent consistency with sequential continuity.
  • Hybrid and Scalable Autoregressive Methods: Memory-efficient architectures and large-scale self-supervised pretraining may reduce long-range modeling cost and improve generalization.Examples include sparse attention, recurrent latent modules, WebVid-10M, and Panda-70M.

8 Conclusion

The survey synthesizes controllable video generation from theoretical foundations through conditioning methods, applications, and future directions. It organizes methods by control signals and connects foundational models with practical video-generation systems.

  • Conclusion: The survey reviews GAN, VAE, diffusion, flow-based, and autoregressive foundations alongside representative video generative models.It uses these foundations to frame the development of controllable video generation.
  • Conclusion: A structured taxonomy categorizes controllable generation methods by conditioning signals beyond text and reviews how those conditions enter video-generation pipelines.The survey also examines applications and synthesizes research progression from core principles to implementation strategies.
Loading 2507.16869v3…