Source-linked AI summary
The Script is All You Need: An Agentic Framework for Long-Horizon Dialogue-to-Cinematic Video Generation
Chenyu Mu, Xin He, Qu Yang, Wanshun Chen, Jiadi Yao, Huang Liu, Zihao Yi, Bo Zhao, Xingyu Chen, Ruotian Ma, Fanghua Ye, Erkun Yang, Cheng Deng, Zhaopeng Tu, Xiaolong Li, Linus
TL;DR
Video generators can create realistic clips from text but struggle to convert sparse dialogue into long-form, coherent cinematic narratives. The paper proposes an agentic pipeline built around ScriptBench and ScripterAgent, with DirectorAgent executing scripts through continuous cross-scene generation. Experiments report improved script faithfulness and temporal fidelity across tested video models, alongside a trade-off between visual spectacle and script adherence.
Problem
Video generation models struggle to bridge the semantic gap between high-level dialogue and detailed, executable plans for coherent cinematic video.
Method
The framework combines ScriptBench, ScripterAgent, DirectorAgent, and CriticAgent to transform dialogue into scripts and orchestrate long-horizon video generation.
Results
The framework improves script faithfulness and temporal fidelity across tested video models, with Script Faithfulness gains up to +0.4 points and VSA gains over 7 points.
Takeaways & Limitations
The results support a script-centric paradigm for automated storytelling and reveal a trade-off between visual spectacle and narrative adherence in leading video models.
Takeaways & Limitations
Cross-scene continuous generation reduces identity drift and layout inconsistencies, but imperfect lip synchronization and fine-grained action misalignment remain.
Abstract
from arXiv · showhide
Recent advances in video generation have produced models capable of synthesizing stunning visual content from simple text prompts. However, these models struggle to generate long-form, coherent narratives from high-level concepts like dialogue, revealing a ``semantic gap'' between a creative idea and its cinematic execution. To bridge this gap, we introduce a novel, end-to-end agentic framework for dialogue-to-cinematic-video generation. Central to our framework is ScripterAgent, a model trained to translate coarse dialogue into a fine-grained, executable cinematic script. To enable this, we construct ScriptBench, a new large-scale benchmark with rich multimodal context, annotated via an expert-guided pipeline. The generated script then guides DirectorAgent, which orchestrates state-of-the-art video models using a cross-scene continuous generation strategy to ensure long-horizon coherence. Our comprehensive evaluation, featuring an AI-powered CriticAgent and a new Visual-Script Alignment (VSA) metric, shows our framework significantly improves script faithfulness and temporal fidelity across all tested video models. Furthermore, our analysis uncovers a crucial trade-off in current SOTA models between visual spectacle and strict script adherence, providing valuable insights for the future of automated filmmaking.
1 Introduction
The paper identifies a semantic gap between coarse dialogue and the detailed filmmaking plans needed for coherent cinematic video. It addresses this gap with a three-agent framework, ScriptBench, and evaluations targeting script quality, temporal coherence, and model trade-offs.
- Motivation: Current video generators produce realistic clips from text but struggle to turn high-level dialogue into detailed, executable cinematic plans.This limitation is framed as a semantic gap between narrative concepts and cinematic execution.
- Framework: The proposed framework uses ScripterAgent for script generation, DirectorAgent for long-horizon execution, and CriticAgent for evaluation.Together, the agents address contextual understanding, filmmaking knowledge, and creative reasoning challenges.
- Benchmark and training: ScriptBench provides a large-scale benchmark with dialogue, audio, and character-position context, annotated through an expert-guided pipeline.The benchmark uses multimodal context and multi-round error correction to support cinematic plausibility.
- Benchmark and training: ScripterAgent transforms coarse dialogue into structured cinematic scripts using supervised fine-tuning followed by reinforcement learning.The two-stage training aligns script structure and creative choices with expert aesthetics.
- Evaluation: The framework evaluates script quality and temporal coherence while identifying a trade-off between visual spectacle and script faithfulness.The evaluation introduces Visual-Script Alignment and compares framework outputs across tested video models.
2 ScripterAgent: Dataset and Model
ScripterAgent addresses dialogue-to-script generation by using ScriptBench and expert-guided annotation to produce structured cinematic plans. Its two-stage training combines structural supervision with reinforcement learning that balances technical correctness and cinematic preferences.
- ScriptBench: ScriptBench pairs coarse dialogue with multimodal context and expert-guided, multi-round validation to produce cinematic script annotations.The dataset uses dialogue, audio, and character positions, while its pipeline expands sparse dialogue into shot-level scripts under domain constraints.
- Annotation pipeline: The annotation pipeline reconstructs scene context, plans shots under continuity constraints, and applies adaptive error correction for structural and semantic fidelity.Its checks cover dialogue completeness, character appearance, scene coherence, and related validity requirements.
- ScripterAgent: ScripterAgent transforms coarse-grained dialogue into fine-grained, structured cinematic scripts for automated filmmaking.The model targets shot composition, pacing, visual continuity, and narrative structure.
- Stage One: SFT: SFT teaches the model cinematic-script syntax and structure by mapping multi-turn dialogue to structured JSON scripts and maximizing ground-truth conditional likelihood.Qwen-Omni-7B is used as the base model; training runs for 20 epochs with a 1 × 10^-5 learning rate and 8,192-token maximum sequence length.
- Stage Two: RL: GRPO refines ScripterAgent beyond structural validity by aligning outputs with expert directorial preferences for subjective cinematic quality.The hybrid reward combines rule-based structural correctness with a human-preference proxy trained from senior art-director judgments, using α = 0.4.
3 DirectorAgent: Long-Horizon Script-to-Video Execution
DirectorAgent executes generated scripts over long horizons through shot-aware segmentation and cross-scene continuity conditioning. Frame anchoring relays visual state between segments, reducing identity and layout inconsistencies while leaving some fine-grained synchronization challenges unresolved.
- Execution challenge: DirectorAgent orchestrates script-to-video execution to address video models’ restricted temporal capacity and the artifacts caused by independent clip generation.Naive segmentation can produce identity drift, inconsistent styling, and lost narrative continuity.
- Cross-Scene Continuous Generation: The Cross-Scene Continuous Generation Strategy combines cinematographic shot-aware segmentation with frame anchoring to preserve semantic and visual consistency.It partitions scripts at natural boundaries rather than arbitrary temporal cuts.
- Shot-based segmentation: Shot segmentation preserves complete camera takes, fits scenes within model generation windows, follows narrative breakpoints, and avoids complex camera-motion transitions.The four principles are shot integrity, duration adaptation, semantic coherence, and technical feasibility.
- Frame-anchored continuity: Frame anchoring conditions each scene’s first frame on the preceding scene’s final frame, preserving character appearance, attire, facial details, and spatial layout.This visual relay provides a continuity prior across generation cycles and reduces jarring scene changes.
- Effectiveness and limitations: The strategy significantly reduces identity drift and layout inconsistencies, but imperfect lip synchronization and fine-grained action misalignment remain.These residual issues define an important boundary of the continuity approach.
4 CriticAgent: The Evaluation Framework
The evaluation framework assesses both dialogue-to-script and script-to-video generation using objective metrics, CriticAgent scores, and human judgments. It evaluates structural correctness, artistic quality, script-video alignment, and overall video quality.
- Evaluation Framework: The framework evaluates both pipeline stages, combining objective metrics, CriticAgent scoring, and qualitative human assessment.Dialogue-to-script evaluation covers structural correctness and artistic quality; script-to-video evaluation covers alignment and video quality.
- Script Evaluation: CriticAgent assesses generated scripts using format compliance, shot division rationality, content completeness, and narrative coherence.These criteria measure structural validity, shot segmentation, actionable visual detail, and logical flow.
- Script Evaluation: Professional directors and screenwriters rate scripts on character portrayal consistency, dramatic tension and rhythm, and visual imagery and cinematic expressiveness.The panel uses a 0–5 scale to judge creative qualities associated with filming potential.
- Video Evaluation: CriticAgent evaluates generated videos across camera articulation, body language and blocking, visual fidelity, emotional progression, and narrative timing.The dimensions cover cinematic quality and faithfulness to the source script and reference audio.
- Video Evaluation: Video evaluation also measures script faithfulness, narrative coherence, character consistency, physical-law adherence, and general visual quality.These criteria assess adherence to scripted events, story continuity, identity stability, plausible dynamics, realism, aesthetics, and rendering stability.
- Automated Metrics: Visual-Script Alignment measures whether scripted visual events occur within their designated time intervals, complementing standard quality metrics and script-alignment measures.Given K shot units and intervals T_k, it uses visual-text similarity; higher scores indicate stronger temporal alignment between video structure and the script plan.
5 Experiment
The experiments show that ScripterAgent improves cinematic script quality and, when used to condition video models, strengthens script faithfulness, temporal-semantic alignment, and consistency. They also reveal a trade-off between visual spectacle and strict script adherence.
- Script Generation: ScripterAgent outperforms MovieAgent in structural and human-rated creative quality, including +0.4 points in three structural metrics, Visual Imagery 4.3 vs. 3.8, and Dramatic Tension 4.1 vs. 3.7.The reported structural gains are in Format Compliance, Content Completeness, and Narrative Coherence.
- Video Generation: Conditioning video models on ScripterAgent scripts raises mean AI ratings from 4.2 to 4.5 and mean human ratings from 3.7 to 4.2.Script Faithfulness improves from 3.2 to 4.0 for Wan2.6 and from 3.6 to 4.2 for Sora2-Pro; HYVideo1.5 reaches 4.6 overall fidelity.
- Model Trade-offs: Sora2-Pro leads visual spectacle, whereas HYVideo1.5 leads script adherence, character consistency, and narrative coherence.Sora2-Pro scores highest in Visual Appeal at 4.8 and Physical Law adherence at 4.5; HYVideo1.5 leads Script Faithfulness at 4.6.
- Automatic Metrics: ScripterAgent outputs improve Visual-Script Alignment, with Veo3.1 rising from 51.4 to 53.8 and Sora2-Pro from 48.6 to 50.6.HYVideo1.5 achieves the highest VSA score, 54.8, while average CLIP scores increase by 1.7.
- Automatic Metrics: Explicit action descriptions increase Dynamic Degree across models, while cross-scene generation improves subject and background consistency by mitigating identity drift.Sora2-Pro rises from 75.2 to 79.5 and Kling2.6 from 52.0 to 56.5 in Dynamic Degree.
6 Related Work
Prior work addresses story generation, multimodal narrative consistency, and video visualization, but existing systems still require manual high-level cinematographic planning. This paper introduces hierarchical multi-agent reasoning to automate long-form movie generation with temporal and narrative consistency.
- Video Generation: Existing video-generation systems rely on diffusion and language models but remain limited in managing long-form content.The related work positions long-horizon coherence as an unresolved limitation.
- LLMs for Film Production: LLM-based film-production systems automate scene and character tasks, yet MovieAgent and related models rely on manual narrative and cinematographic planning.This limitation motivates a more autonomous planning pipeline.
- Story Visualization: Story-visualization methods improve temporal consistency and motion but still lack automated high-level planning for cinematography, scene structure, and character interactions.These methods therefore continue to require manual guidance.
- Proposed Direction: The proposed multi-agent chain-of-thought framework uses hierarchical reasoning to automate long-form movie generation while targeting temporal consistency, narrative integrity, and visual appeal.Its stated goal is to maintain these properties over extended durations.
7 Conclusion
The paper presents a script-centric agentic framework that converts sparse dialogue into executable cinematic plans before video synthesis. Experiments report improved narrative coherence, temporal fidelity, and script faithfulness, while identifying a trade-off between visual spectacle and narrative adherence.
- Conclusion: The framework combines ScripterAgent, DirectorAgent, and CriticAgent to automate dialogue-to-video generation through detailed executable scripts.Its efficacy is attributed to ScriptBench and ScripterAgent’s two-stage SFT+RL training.
- Conclusion: Experiments validate universal improvements in narrative coherence, temporal fidelity, and script faithfulness across state-of-the-art video models.The VSA metric provides a quantitative measure of these improvements.
- Conclusion: The study identifies a trade-off between visual spectacle and narrative adherence and proposes future work on fine-grained control and adaptive cinematic styles.Suggested directions include precise lip synchronization and diverse cinematic-style generation.
A.2 Ablation Study on Scripting and Agentic Components
The ablation study separates the contributions of scripting, segmentation, and frame anchoring. Script conditioning improves visual description and body blocking, while the full agent pipeline performs best across evaluated metrics and backbones.
- Experimental Configuration: The ablation compares raw dialogue, script-only generation, script plus segmentation, and the full agent with frame anchoring.The full pipeline combines script conditioning, shot-aware segmentation, and visual-continuity frame anchoring.
- Quantitative Results Analysis: The Full Agent consistently outperforms baselines across five cinematic dimensions and both Wan2.6 and HYVideo1.5 backbones.Evaluation uses Gemini 2.5 Pro, Qwen 3 VL, and GLM 4.6 V as critics, with scores on a 0–5 scale.
- Quantitative Results Analysis: Introducing the script stage markedly improves Visual Descriptive Fidelity and Kinetic Body Language & Blocking.The ablation reports a functional separation between scripting-related visual and kinetic improvements and later temporal components.
- Quantitative Results Analysis: Shot-aware segmentation and frame anchoring drive the highest Narrative Pacing & Timing and Cinematic Camera Articulation scores.The reported critic agreement across three diverse models supports these component-level improvements.
B Details on Evaluation Metrics
Human evaluation scores generated scripts and videos on a 0–5 scale.
- Human evaluation uses a 0–5 scoring scale for generated scripts and videos.
B.1 Human Evaluation for Script Generation
Human evaluation of script generation uses 0–5 criteria covering dramatic structure, visual expressiveness, and character consistency.
- Character Portrayal Consistency: Character Portrayal Consistency evaluates whether each character’s personality, speaking style, and behavior remain coherent throughout the script.
- Character Portrayal Consistency: Character consistency ranges from invalid or completely inconsistent portrayals at 0–1 to stable identities with coherent speech and behavior at 5.
- Dramatic Tension & Rhythm: Dramatic Tension & Rhythm measures how scripts build, sustain, and release tension while maintaining natural, engaging pacing.
- Dramatic Tension & Rhythm: Scores range from invalid or absent story structure at 0 to strong, compelling tension with sustained viewer interest at 5.
- Visual Imagery & Cinematic Expressiveness: Visual Imagery & Cinematic Expressiveness assesses vivid visual information and the use of cinematic language for filming.
- Visual Imagery & Cinematic Expressiveness: Visual expressiveness ranges from unreadable or non-visual scripts at 0–1 to highly cinematic, filmable guidance at 5.
B.2 Human Evaluation for Video Generation
Human evaluation of video generation uses 0–5 criteria for script faithfulness, narrative coherence, visual appeal, character consistency, physical plausibility, and rendering quality.
- Annotators watch each generated video against its script or dialogue reference and assign 0–5 scores.
- Visual Appeal: Visual Appeal evaluates realism, aesthetic quality, and rendering stability.
- Script Faithfulness: Script Faithfulness measures how accurately videos follow intended scenes, actions, and plot progression.
- Narrative Coherence: Narrative Coherence measures logical storytelling, reasonable scene transitions, and pacing.
- Character Consistency: Character Consistency evaluates whether characters maintain stable identities and appearances across the video.
- Physical Law Adherence: Physical Law Adherence assesses whether motion and interactions roughly follow real-world physical laws.
B.3 AI-Based Rating Metrics
AI-based evaluators score scripts and videos on 0–5 scales aligned with film-production standards, using structured prompts and dimension-specific criteria.
- Gemini-2.5-Pro evaluates script structural and logical correctness and video perceptual and technical audio-visual quality.
- Script Generation Evaluation Prompt: Script evaluation scores format compliance, shot division rationality, content completeness, and narrative coherence on 0–5 scales.
- Script Generation Evaluation Prompt: Shot Division Rationality assesses whether shot breaks align with narrative beats and emotional shifts without excessive fragmentation or length.
- Video Generation Evaluation Prompt: Video evaluation scores cinematic camera articulation, kinetic body language and blocking, and visual descriptive fidelity on 0.0–5.0 scales.
- Video Generation Evaluation Prompt: The video rubric reserves high scores for sophisticated cinematic techniques and assigns low scores to simple dialogue videos with minimal movement.