Source-linked AI summary
SmartDirector: Keyframe-Conditioned Cinematic Video Generation with Narrative Pacing Control
Zhida Zhang, Jie Ma, Zhan Peng, Haoxue Wu, Yang Han, Jun Liang, Jie Cao, Jing Li
TL;DR
Existing video generators offer limited control over fine-grained narrative structure and temporal pacing because they typically use sparse conditioning. SmartDirector uses multiple keyframes in a two-stage generation framework and substantially improves objective quality and semantic fidelity across single-shot and multi-shot synthesis.
Problem
Existing video generators mainly use sparse conditioning, limiting precise control over fine-grained spatial-temporal content and narrative structure.
Method
SmartDirector combines keyframe-aligned Multi-Chunk VAE conditioning with Director-Gen synthesis and Director-SR refinement for single-shot, multi-shot, and video-extension generation.
Results
SmartDirector substantially outperforms the baseline, reducing Multi-Shot FVD from 251.83 to 65.65 and raising average semantic score from 59.32 to 88.48.
Takeaways & Limitations
SmartDirector provides flexible keyframe-conditioned video generation with fine-grained control over narrative structure and temporal pacing.
Takeaways & Limitations
The low-resolution first stage creates an information bottleneck that leaves a marginal fidelity gap despite super-resolution refinement.
Abstract
from arXiv · showhide
The narrative quality of a video fundamentally determines its perceptual value. Although existing video generation methods can produce visually appealing content, they predominantly rely on sparse conditioning signals such as text prompts or first/last frames, which limits precise control over narrative structure and temporal pacing. In this paper, we propose SmartDirector, a framework that enhances the narrative capacity of video generation models through multiple keyframes. SmartDirector supports flexible generation scenarios including single-shot generation, multi-shot narrative synthesis, and video extension. The framework operates in two stages: Director-Gen generates a low-resolution video conditioned on the provided keyframes, and Director-SR refines the output by exploiting high-resolution keyframes as semantic anchors to recover fine-grained details. To enable robust multi-keyframe training, we construct a data pipeline that curates single-shot and multi-shot sequences from movies. Extensive experiments demonstrate that SmartDirector substantially outperforms existing state-of-the-art approaches. We will release the code to facilitate further research.
1 Introduction
SmartDirector addresses the limited narrative and fine-grained control of sparse-conditioned video generation with arbitrary-keyframe conditioning across single-shot, multi-shot, and video-extension scenarios. Its two-stage design combines Multi-Chunk VAE-based generation with keyframe-conditioned super-resolution to preserve temporal continuity and recover high-resolution details.
- Motivation: Existing video generation methods rely mainly on text prompts or first/last frames, limiting precise control over fine-grained spatiotemporal structure.The paper situates this limitation amid a shift toward long, multi-shot narratives.
- Motivation: Generating adjacent-keyframe clips autoregressively causes abrupt boundary discontinuities and weak narrative consistency because global context is neglected.The paper identifies this as a naive alternative to keyframe-conditioned synthesis.
- SmartDirector: SmartDirector provides unified arbitrary-keyframe conditioning for single-shot synthesis, multi-shot synthesis, and video extension.The framework is described as flexible across these generation settings.
- Director-Gen: Multi-Chunk VAE places keyframes at arbitrary temporal positions while avoiding causal temporal-VAE constraints and supporting smooth, continuous generation.During training, videos are partitioned into chunks at keyframe positions.
- Director-SR: Director-SR upsamples low-resolution Director-Gen videos using high-resolution keyframes as semantic anchors to recover fine-grained details.The examples given are 480p generated videos and 1080p high-definition outputs.
2 Related Work
Prior video-generation research has progressed from short single-shot clips to long multi-shot narratives, but sparse conditioning limits fine-grained spatial-temporal and narrative control. Video super-resolution has likewise evolved from GAN-based methods toward diffusion models, while existing approaches primarily emphasize pixel-level enhancement.
- Video Generation: Video generation has advanced from short single-shot clips to long multi-shot narratives, yet text or first/last-frame conditioning limits fine-grained spatial-temporal and narrative control.Recent approaches incorporate multiple keyframes to provide more precise control.
- Video Super-Resolution: Video super-resolution research has shifted from early GAN-based approaches toward recent diffusion-model methods.SeedVR uses shifted window attention for long-sequence restoration, while DoVE introduces an efficient one-step diffusion model for real-world video super-resolution.
- Video Super-Resolution: Existing video super-resolution methods primarily focus on pixel-level enhancement.
3 Method
SmartDirector uses a two-stage keyframe-conditioned pipeline: Director-Gen generates videos with chunk-aware latent modeling, while Director-SR restores high-resolution detail using keyframes as semantic anchors. Its training data pipeline constructs coherent multi-shot videos with structured narrative, camera, and character captions.
- Director-Gen: Director-Gen splits videos at keyframe positions, encodes each chunk independently with a 3D causal VAE, and injects noise only into non-keyframe positions.Each keyframe becomes the first frame of its chunk, addressing the causal limitation of the 3D VAE.
- Director-Gen: Full spatio-temporal attention models dependencies across chunks, while MC-RoPE assigns fractional temporal indices to avoid discontinuities at keyframe boundaries.MC-RoPE is introduced because continuous or per-chunk-reset temporal indices can create boundary discontinuities.
- Director-SR: Director-SR replaces upsampled low-resolution latents at keyframe indices with corresponding high-resolution latents, enforcing keyframe conditioning during refinement.The stage is motivated by low-resolution outputs that lose facial and textual detail and by conventional VSR methods’ lack of frame-level control.
- Director-SR: Director-SR refines Director-Gen results and can also super-resolve arbitrary low-resolution videos independently.This provides an additional operating mode beyond refinement of generated outputs.
- Data Curation: The data pipeline segments cinematic videos into shots, aggregates consecutive clips with shared scenes and storylines, and produces structured captions spanning narrative, camera motion, and character identity.AutoShot performs segmentation, while VLM-based aggregation and hierarchical captioning preserve multi-shot semantic continuity.
4 Experiments
SmartDirector is evaluated on diverse single-shot and multi-shot video benchmarks using objective, semantic, human, and qualitative assessments. It consistently outperforms baselines, while ablations show that its Multi-Chunk keyframe conditioning preserves temporal coherence and avoids redundancy-induced stuttering.
- Benchmark: The benchmark contains 250 single-shot and 250 multi-shot videos from movies, TV series, and animations, spanning 3–15 seconds at 24 FPS and at least 1080p.Keyframes are randomly sampled as conditioning signals for each video.
- Objective Quality: 41.12 vs. 226.85 FVD in single-shot and 65.65 vs. 251.83 in multi-shot evaluation shows substantially closer distributional fidelity than the baseline.The multi-shot improvement is especially pronounced because scene transitions and camera cuts add temporal complexity.
- Semantic Fidelity: 91.30 vs. 83.87 average semantic score in single-shot and 88.48 vs. 59.32 in multi-shot evaluation demonstrates stronger high-level semantic fidelity.Narrative Coherence has the largest single-shot gain, at +12.56.
- Human Evaluation: SmartDirector consistently outperforms Dreamina in human evaluation, including a 54.73% win rate in multi-shot Overall Quality.The study includes 30 participants evaluating 500 video pairs across Identity Consistency, Narrative Pacing, Keyframe Adherence, and Overall Quality.
- Qualitative Results: Compared with Dreamina, SmartDirector avoids implausible intermediate-frame motion and identity drift in representative qualitative examples.The comparison highlights artifacts between keyframes, including physically inconsistent trajectories and gradual deviation in the cat’s appearance.
- Director-SR and Ablation: Director-SR matches SparkVSR in PSNR and SSIM while achieving lower LPIPS across all four benchmarks, and reconstructs faithful facial details and legible text from degraded inputs.Ablations further show that removing Multi-Chunk causes temporal drift and narrative collapse, while keyframe replication introduces noticeable stuttering from temporal redundancy.
5 Conclusion
SmartDirector is a two-stage keyframe-conditioned framework for cinematic video generation that provides fine-grained control over narrative structure and temporal pacing. Director-Gen creates a low-resolution video, while Director-SR uses high-resolution keyframes as semantic anchors to restore details and repair artifacts.
- 5 Conclusion: SmartDirector enables fine-grained control over narrative structure and temporal pacing through keyframe-conditioned cinematic video generation.It is presented as a two-stage framework.
- 5 Conclusion: Director-Gen synthesizes a low-resolution video from provided keyframes, and Director-SR refines it using high-resolution keyframes as semantic anchors.Director-SR recovers fine-grained details and repairs generative artifacts introduced during the first stage.
6 Appendix: LLM Evaluation Protocol
The appendix specifies a Gemini-based evaluation protocol that scores generated videos for prompt adherence, narrative coherence, and physical consistency using videos, text prompts, and reference keyframes. It defines structured analyses, dimension-specific criteria, and 0–5 scoring scales.
- Evaluation Protocol: The evaluator performs blind visual analysis and prompt-consistency checking, then returns dimension-level analysis, an overall summary, and a final 0–5 score.Inputs include a generated video, detailed text prompt, and reference keyframes.
- Prompt For Instruction-Following Assessment: Instruction-following assessment measures character, emotion, scene, and spatial consistency against the text prompt across the video sequence.Criteria include stable identity and attire, logically evolving emotions, consistent environments and props, and coherent object positioning and camera perspective.
- Prompt For Video Narrative Coherence Assessment: Narrative coherence assessment evaluates plot progression, completeness of story beats, causal relationships, physical realism, motion continuity, and plausible interactions.The protocol penalizes skipped transitions, teleportation, inconsistent movement direction, and physically impossible actions.
- Prompt For Video Physical Consistency Assessment: Physical consistency assessment checks gravity and weight, collision and clipping, object permanence and structural integrity, and realistic fluid or particle dynamics.Examples include preventing floating, inappropriate object penetration, spontaneous disappearance or morphing, and implausible water, fire, smoke, or hair motion.
7 Detailed Human Evaluation Protocol
SmartDirector was evaluated against Dreamina in a blind, double-blind human study using randomly sampled test videos, extracted keyframes, and LLM-generated prompts. Participants compared video pairs across single-shot and multi-shot scenarios using overall quality and optional narrative-focused criteria on a five-point scale.
- Evaluation Setup: The study compared SmartDirector with Dreamina using randomly sampled test videos, extracted keyframes, and LLM-generated prompts as shared inputs.The evaluation was designed to ensure a fair and objective comparison.
- Evaluation Setup: Each participant evaluated 10 video pairs comprising 5 Single-Shot and 5 Multi-Shot scenarios through a web-based interface.Backend-randomized A/B labels prevented participants and evaluators from knowing model identities.
- Evaluation Criteria: Participants rated Overall Quality and could optionally assess Identity Consistency, Narrative Pacing, and Keyframe Adherence.These dimensions covered temporal consistency, narrative rhythm and transitions, and reconstruction of reference-keyframe content, composition, and poses.
- Scoring: Responses used a 5-point Likert-style scale ranging from A significantly better than B to B significantly better than A, with a neutral midpoint.Ratings were collapsed into SmartDirector Better, Neutral, and Dreamina Better categories to calculate the GSB score.
- Results Reporting: Table 3 presents the detailed percentage breakdown of user preferences for all evaluated dimensions.The table uses ’S’ for SmartDirector and ’D’ for Dreamina.
8 Limitations
SmartDirector retains limitations despite its strong keyframe-conditioned video-generation capabilities. Its two-stage design trades some fine-grained conditioning fidelity and spatial clarity for lower computational overhead, while the temporal VAE constrains keyframe placement to bounded discretization.
- SmartDirector demonstrates strong keyframe-conditioned cinematic video generation, but several limitations remain.
- The two-stage pipeline reduces computational overhead versus direct native 1080p generation but may weaken fine-grained keyframe adherence and slightly reduce spatial clarity.The keyframe-conditioned super-resolution stage is used to mitigate this gap and restore high-frequency details.
- The temporal VAE requires each video chunk to follow a 4n + 1 frame structure, preventing strictly arbitrary frame placement while bounding misalignment within ±2 frames.The passage characterizes this bounded misalignment as perceptually negligible in practical applications.