PaperlayerView exact revision on arXiv

Source-linked AI summary

DramaDirector: Geometry-Guided Short Drama Generation

Hengji Zhou, Sijie Liu, Jianrun Chen, Xingchen Zou, Lianghao Xia, Liqiang Nie

arXiv:2606.24107v2cs.CVcs.AI

TL;DR

Plot-to-short-drama generation requires both narrative planning and cinematographic geometry that text-only pipelines underspecify. DramaDirector addresses this with geometry-grounded retrieval, decoupled storyboard conditions, schema-constrained training, and alignment-based optimization, while DramaBoard supplies a structured benchmark. Experiments report improved faithfulness, consistency, controllability, and generation quality over representative baselines.

  • Problem

    Text-based short-drama pipelines underspecify shot grammar and spatial geometry needed for faithful, coherent multi-shot video generation.

  • Method

    DramaDirector decouples each shot into static visual and dynamic narrative conditions, retrieves depth-pose references, and trains the planner with SFT and GRPO using a learned text-visual alignment reward.

  • Results

    DramaDirector improves storyboard coherence, controllability, and generation quality over representative baselines.

  • Takeaways & Limitations

    Geometry-guided planning and retrieval-grounded realization provide a controllable route for visually grounded short-drama generation.

  • Takeaways & Limitations

    Retrieval can be imperfect for complex or long-tail interactions when the reference gallery lacks comparable spatial layouts or body configurations.

Abstract

from arXiv · show

Short dramas, with their rapid shot rhythms, dialogue-driven focus shifts, and demanding cinematographic grounding, pose challenges that prompt-level or text-only video generation pipelines struggle to meet. We study plot-to-short-drama generation, where a global plot and local context are transformed into visually grounded multi-shot videos. We propose DramaDirector, a geometry-grounded framework that lets the planner borrow cinematographic geometry from a gallery of real short-drama shots indexed by depth and pose. DramaDirector decouples each shot into static visual and dynamic narrative conditions, trains the planner with schema-constrained SFT and GRPO under a learned text-visual alignment reward, and retrieves depth-pose references to guide first-frame generation and image-to-video synthesis. We also introduce DramaBoard, a benchmark built from 35 live-action dramas, 2.8K episodes, and 81K shots, with structured storyboards and multi-dimensional evaluation protocols. Experiments show that DramaDirector improves over representative multi-agent and video generation baselines on faithfulness, consistency, and controllability. Our code is released at: https://github.com/iLearn-Lab/DramaDirector

1 Introduction

DramaDirector addresses the gap between text-based storyboard planning and the visual geometry required for coherent short-drama generation. It introduces a geometry-grounded planner and DramaBoard benchmark, with experiments reporting improvements over representative baselines.

  • Challenges: Textual storyboards often miss short-drama shot grammar, including fast cuts, dialogue framing, stacked close-ups, and one-beat-per-shot emotional rhythm.LLMs tend to produce novelistic storyboards because they rarely encounter large-scale short-drama corpora during pretraining.
  • Challenges: Text descriptions also underspecify spatial geometry such as occlusion, speaker positions, gaze directions, and depth ordering, leaving downstream generators with ambiguous visual cues.Text-video alignment cannot recover visual information that the storyboard never specifies.
  • Geometry-grounded generation: DramaDirector converts cinematographic grounding from a text-prompting problem into retrieval and reward using real short-drama visual priors.The planner retrieves depth-pose references and is optimized with a learned text-visual alignment reward.
  • Benchmark: DramaBoard contains 35 live-action dramas, 2.8K episodes, and 81K shots with aligned keyframes, dialogue, metadata, structured storyboards, and multi-dimensional evaluation protocols.Its evaluations cover storyboard narrative quality, storyboard-to-video instruction following, and intrinsic video quality.
  • Method: DramaDirector decouples each shot into static visual and dynamic narrative conditions, using the former for depth-pose retrieval and first-frame generation and the latter for video synthesis.The planner is trained with schema-constrained multi-task SFT and GRPO so its storyboards are executable for synthesis.
  • Results: Experiments show that DramaDirector improves storyboard narrative quality, storyboard-to-video controllability, and intrinsic video quality over representative baselines.These results support the framework’s effectiveness across planning, controllability, and generated-video quality.

2 Methodology

DramaDirector converts plot and local context into storyboarded shots that separate static visual geometry from dynamic narrative conditions, then grounds synthesis with retrieved depth–pose references. It trains the planner with schema-aware objectives and reinforcement learning while using a learned geometry-focused text–visual alignment model for retrieval and rewards.

  • Geometry-Grounded Drama Generation: DramaDirector represents each shot with static visual condition q_j and dynamic narrative condition d_j, separating keyframe geometry from actions, dialogue, and emotion.The storyboard intermediate decomposes planning from realization, addressing the geometric underspecification of direct text-to-video synthesis.
  • Geometry-Grounded Drama Generation: Retrieved depth–pose references anchor first-frame generation with layout, pose, and depth, while dynamic conditions guide image-to-video rollout.The reference gallery preserves compositional geometry while discarding appearance and identity cues.
  • Storyboard Planning: The planner is adapted to the storyboard schema through SFT on continuation, single-shot, multi-field, ordering, and summary-guided reconstruction tasks.These objectives target local realization, cross-field dependencies, temporal coherence, and plan-to-shot grounding.
  • Reinforcement Learning: GRPO optimizes a combined retrieval, video-readiness, and schema-validity reward, with annealed weights shifting training toward the harder video-readiness objective.The retrieval reward favors target-episode visual matches over generic descriptions, while the LLM judge evaluates action concreteness, detail, transitions, emotion, and pacing.
  • Geometry-Aware Text-Visual Alignment: A dedicated alignment model matches static storyboard conditions with depth–pose evidence using modality fusion and bidirectional contrastive learning.Its vectors are shared by gallery retrieval and the retrieval reward, with negatives spanning non-adjacent shots, episodes, and dramas.

3 Evaluation

DramaDirector is evaluated through end-to-end generation, ablations, retrieval discriminability, first-frame quality, hyperparameter studies, and qualitative inspection. Across these evaluations, it improves storyboard grounding, controllability, and visual quality, while depth–pose retrieval and both SFT and GRPO contribute materially.

  • Overall Performance Comparison (RQ1): DramaDirector leads Relevance and Coherence, achieves the highest Similarity and Purity, and leads Aesthetics and DIS while ranking second on Consistency in the main comparison.UniVA is slightly better on Engagement and Dreamrunner leads Faithfulness and Consistency; the comparison excludes generator-swap variants.
  • Ablation Study (RQ2): Removing auxiliary storyboard tasks mainly harms coherence and temporal consistency, while removing the retrieval reward causes the largest coherence drop.SFT-only retains engagement but weakens faithfulness and purity, supporting the importance of both multi-task supervision and GRPO-based visual grounding.
  • Ablation Study (RQ2): Depth is the dominant retrieval cue, pose is also important, and the full depth–pose design provides the most balanced video-generation performance.Removing depth nearly collapses retrieval, while removing pose also causes a large drop; contrastive-loss and hard-negative ablations are milder.
  • Retrieval Discriminability (RQ3): DramaDirector achieves the best retrieval result, outperforming generic multimodal encoders and VidoRAG across every reported metric.The advantage is largest at the larger recall cutoff, while NDCG gaps remain narrow.
  • Quality of First-Frame Generation (RQ4): DramaDirector leads Aesthetics, Consistency, Position, and Action in retrieval-grounded first-frame generation, with the largest spatial-action margin over text-only generation.Depth constrains spatial layout, while pose governs character dynamics, giving the two cues complementary roles.
  • Hyperparameter Study (RQ5): The selected SFT checkpoint at step 426 and GRPO checkpoint at step 441 provide the strongest reported balance across training-step studies.SFT behavior is non-monotonic, while later training generally improves consistency and lowers DIS.

4 Related Works

Related work on narrative video generation spans script-level supervision, multi-agent orchestration, controllable element composition, and multimodal retrieval. DramaDirector differs by tailoring post-training to structured short-drama storyboard planning and geometry-based visual grounding.

  • Short-Drama Generation: Narrative video-generation systems use script-level supervision, multi-agent orchestration, or controllable element composition to inject narrative and cinematographic structure.Examples include SkyScript, MovieAgent, GENMAC, and SkyReels-A2.
  • Multimodal RAG: Other visual-creation systems apply SFT or GRPO to multi-step generation, while multimodal RAG systems retrieve across video, audio, and text evidence.These approaches provide adjacent foundations for visual creation and video understanding.
  • Short-Drama Generation: DramaDirector differs by emphasizing structured shot planning and visual grounding specifically for short-drama storyboard generation.Its distinction is framed against prior short-drama generation and multimodal retrieval systems.

5 Conclusion

The paper frames plot-to-short-drama generation as visually grounded storyboard planning and introduces DramaDirector alongside the DramaBoard benchmark. Experiments report improved coherence, controllability, and generation quality over representative baselines.

  • DramaDirector couples schema-constrained multi-shot planning with controllable video realization through multi-task supervision, a text-visual alignment reward, and depth–pose references.
  • DramaBoard provides aligned keyframes, dialogue, structured storyboards, and protocols spanning narrative, instruction-following, and video quality.
  • Experiments show improved storyboard coherence, controllability, and generation quality over representative baselines.The paper identifies broader genres, tighter audio-visual synchronization, and interactive storyboard editing as future work.

Limitations

DramaDirector’s retrieval-augmented depth–pose references improve visual grounding, but retrieval remains imperfect for complex or long-tail interactions absent from the reference gallery.

  • Retrieval can fail for complex or long-tail interactions when the gallery lacks comparable spatial layouts or body configurations.
  • The system treats retrieved pose and depth maps as reference cues rather than hard constraints during first-frame generation.Pose maps indicate action and body configuration, while depth maps indicate spatial layout and scene geometry.
  • When retrieved conditions conflict substantially with the storyboard, textual constraints take precedence.

Ethical Considerations

DramaBoard is built from legally authorized live-action short dramas, with distribution permissions covering its frames, transcripts, and structured annotations.

  • The dataset uses 35 live-action short dramas whose copyrights, usage rights, and distribution permissions are held by the authors.The released dataset is intended solely for academic research and is not licensed for commercial use.

A.1 Baseline Methods

The appendix organizes comparisons across short-drama generation, image generation, generative-agentic, and retrieval-augmented frameworks, adapting baselines to a common end-to-end workflow.

  • Short-Drama Generation Methods: Short-drama generation baselines include SkyScript, ShoulderShot, SkyReels-A2, and Dreamrunner, covering script formatting, dialogue consistency, element composition, and spatial-temporal control.
  • Short-Drama Generation Methods: SkyScript is adapted by serializing the input plot into a shooting-script-style shot plan and passing its shot descriptions through the same downstream image and video interfaces as other baselines.
  • Image Generation Frameworks: Image-generation comparisons include ComfyMind and Gen-Search, which use collaborative workflow planning or search-augmented visual grounding.
  • Generative Agentic Frameworks: Generative-agentic baselines include ComfyAgent, MovieAgent, UniVA, VideoAuteur, and GENMAC, using multi-agent or hierarchical planning for video generation.
  • Retrieval-Augmented Frameworks: Retrieval-augmented comparisons include Tongyi-Embedding-Vision, LightRAG, RAG-Anything, and VidoRAG, spanning cross-modal embedding, graph retrieval, and multimodal reasoning.

A.2 Generation Workflow Comparison

DramaDirector compares retrieval-augmented depth–pose guidance with image-to-video and text-to-video workflows, using short-drama-native references to improve visual control and alignment.

  • DramaDirector retrieves depth and skeleton conditions from a short-drama-native storyboard library to reduce the semantic gap between storyboard text and visual generation.
  • Figure 7 contrasts three representative-shot workflows: DramaDirector, image-to-video, and text-to-video.
  • Compared with text-to-video, the retrieval-augmented workflow provides more controllable, consistent, stable, and storyboard-aligned results.The workflow removes appearance noise from raw RGB references while preserving geometric guidance.

A.3 Hyperparameter Study

The study examines retrieval-alignment hyperparameters, especially depth–pose fusion, within a pipeline that preprocesses shots, builds geometry references, and evaluates multimodal generation quality. Results indicate that balanced fusion is competitive, while extreme weighting can degrade retrieval and lower depth weighting may compromise strict preservation during synthesis.

  • Fusion-weight study: A slightly pose-emphasized depth–pose fusion of 0.4/0.6 achieves the best overall retrieval metrics, but improves N@10 by only 0.24 over equal fusion at 0.5/0.5.Extreme weighting degrades retrieval, making the marginal gain over equal fusion practically small.
  • Fusion-weight study: Reducing the depth weight below 0.5 occasionally compromises strict preservation during downstream video synthesis.
  • Geometry preprocessing: The retrieval pipeline represents each shot with depth and pose controls extracted from representative keyframes, using normalized Depth Anything V2 maps and DWPose whole-body pose maps.These controls provide the geometric inputs used for retrieval and first-frame conditioning.
  • Gallery construction and split policy: The gallery excludes shots from each held-out test sample’s source episode, preventing episode-level leakage during depth–pose retrieval.
  • Storyboard construction: DramaDirector converts corrected transcripts, episode context, keyframes, and shot durations into structured storyboard objects that separate camera, subject, background, and narrative information.These structured conditions support later retrieval, generation, and evaluation stages.
  • Evaluation protocol: Evaluation separates image quality, storyboard narrative quality, instruction following, and intrinsic video quality, with DIS measuring temporal smoothness through masked SSIM distance after flow warping.Judge-scored metrics use a 0, 0.5, 1 scale, and character identity and story continuity are hard constraints.
Loading 2606.24107v2…