AutoDirector: Online Auto-scheduling Agents for Multi-sensory Composition

1 Introduction

• Multi-sensory composition integrates sensory elements into a longer, more intricate process, and readily available clearly copyrighted resources for training are scarce.The production process includes scriptwriting, shooting, scoring, dubbing, and special effects.Source: §1 · p. 2
• Existing agents struggle to manage time-consuming movie-production tasks, while weak interactive approaches limit films’ expressive capabilities.The paper identifies scheduling and iterative communication with users as distinct challenges.Source: §1 · p. 2
• AutoDirector dynamically schedules production tasks in parallel and uses user feedback to modify or enhance them.The described tasks include scriptwriting, shooting, scoring, dubbing, and special effects.Source: §1 · p. 2
• AutoDirector outperforms the baseline in both efficiency and quality of multi-sensory composition.The paper also describes automatic scheduling and iterative task modification based on user feedback.Source: §1 · p. 2

2 Related Work

• Video-generation work includes prompt-based editing, coarse-to-fine long-video generation, script-directed production, and frame-by-frame movie creation [pik 2023; Yin et al. 2023; Hong et al. 2023a; Song et al. 2024].The related work also names Sora (sor 2024) and Vlogger (Zhuang et al. 2024).Source: §2 · pp. 2–3
• MovieLLM creates frame descriptions for frame-by-frame video, but the cited work is described as expensive and lacking smooth transitions, dubbing, and background music [Song et al. 2024; Achiam et al. 2023].Source: §2 · p. 3
• The paper presents detailed multi-turn interaction between users and the director as a way to generate more satisfying videos resembling natural scenes.Source: §2 · p. 3
• LLM-powered agent systems have been applied across domains, including API integration with ToolLLM and multi-agent conversations with Autogen [Qin et al. 2023; Wu et al. 2023].Source: §2 · p. 3
• Existing pipeline workflows cannot reasonably schedule tasks or interact with users in real time, motivating integration of both into pipelines.The paper highlights potential speedups and improved efficiency from that integration.Source: §2 · p. 3

3.1 Overall of AutoDirector

• Networked task dependencies and user feedback make film production a continuous scheduling process rather than a linear pipeline.AutoDirector plans, revokes, or reschedules tasks from progress reports and user feedback, and waits when no work is ready.Source: §3.1 · p. 3

3.2 AutoDirector with Cognitive Scheduling

• Film production is represented as events with dependencies, planned across time slices until all events are complete.The director uses the current progress report and any user feedback at each time segment to decide what to schedule or wait for.Source: §3.2 · p. 3
• Feedback can trigger task retries and revocations, while eligible events are scheduled together and none are scheduled when dependencies are unmet.The next progress report follows execution of the confirmed planned events.Source: §3.2 · p. 3

3.3 Film Crews

• The scriptwriter, art director, and actors create scenes, designs, and dialogue, while the action director, voice-over role, and post-production complete shots and sound.Shooting waits for art direction; dubbing waits for acting; post-production waits for dubbing, action, and art direction.Source: §3.3 · p. 4
• A human user can interact with the director during production and may choose not to comment in any round.The user role is distinct from the crew roles played by LLMs.Source: §3.3 · p. 4

3.4 Implementation Details

• AutoDirector is GPT-4-based and delegates scriptwriting while calling other tool modules through APIs for remaining steps.The agent runs on a CPU, although the visual synthesis tools require additional resources; the experiments used an Nvidia A100 80GB GPU.Source: §3.4 · p. 4
• Implementation assigns emotion labels to dialogue for voice synthesis and generates long shots by iteratively extending scene-conditioned frames to match voice-over length.Reverse frames are added after the first extension round to address error accumulation.Source: §3.4 · p. 4
• The system generates theme-based background music and combines it with voice-overs, while stitching scenes with cross-dissolve transitions.Source: §3.4 · p. 4

4.1 Experiment Setup

• Generated films were manually assessed for visual aesthetics, narrativity, and controllability across selected movie themes.Aesthetics covers frame quality and cross-frame coherence; narrativity covers story coherence, logic, and emotional expression; controllability measures user control.Source: §4.1 · p. 5
• The baseline uses the same synthesis tools as AutoDirector but lacks its planning and interaction capabilities.Source: §4.1 · p. 5
• NUWA-XL, Vlogger, OpenSora, and Pika serve as references, though they support video generation rather than multi-sensory movie generation.Source: §4.1 · p. 5

• AutoDirector achieves much higher aesthetic scores than the baseline by using user feedback to produce scenes that better align with human aesthetics.The paper describes fine-grained control over movie generation.Source: §4.2 · p. 5
• Interactive evaluation and revision help AutoDirector address narrative logic defects, which the paper identifies as a key reason for its higher overall score than the baseline.Source: §4.2 · p. 5
• Stable Video Diffusion was used as the video synthesis backend because its open-source implementation allows precise control over calls and timing for quantitative experiments.The paper says AutoDirector can theoretically support any video synthesis model as its backend.Source: §4.2 · p. 5
• About 40% acceleration is reported for AutoDirector over the serial baseline, without performance degradation, in an experiment assuming no user feedback.The paper attributes the speedup to parallelizable production steps and module scheduling.Source: §4.2 · p. 5

• Combining time scheduling and user interaction achieves the best state for both time and effect satisfaction.Source: §4.3 · p. 5
• Removing time scheduling significantly lowers satisfaction with time efficiency, although the effect does not change significantly.Source: §4.3 · p. 5
• Without user interaction, satisfaction with both time and effect decreases; the paper attributes this partly to perceived execution time and lack of communication about intermediate results.Source: §4.3 · p. 5

Comparison of Interaction Frequency

• Unlimited interaction makes final results more likely to satisfy users by interrupting error accumulation, while added interaction improves satisfaction with intermediate and final results.Low-frequency interaction improves intermediate-process satisfaction, but its effect on final-result satisfaction is limited.Source: Comparison of Interaction Frequency · p. 5, Table 4 · p. 6

Comparison of Interaction Types

• AutoDirector significantly outperforms other models in overall quality, with the table caption highlighting aesthetic images, high-quality scripts, and improved narrative logic.Source: Table 1 · p. 6
• The efficiency table reports that AutoDirector significantly enhances generation efficiency compared with the pipeline method.Source: Table 2 · p. 6
• The ablation table reports that time scheduling and user interaction both matter, with their combination yielding the highest user satisfaction.Source: Table 3 · p. 6
• Even Yes/No evaluations improve picture expressiveness over the original by enhancing background details.Source: Comparison of Interaction Types · p. 6
• Detailed Comments produce pictures emphasizing both prewar tension and the lovers’ reluctance, whereas Critical Comments highlight the cruelty of war.Source: Comparison of Interaction Types · p. 6

Agents Workflow

• Dialogue delivery can begin while art creation is underway once its prerequisites are complete; when the actor task remains ongoing, AutoDirector waits for completion.Source: Agents Workflow · pp. 6–7, Figure 5 · p. 7
• After a user requests changes to the fifth act, AutoDirector revokes and reruns the dialogue task, and the actor adjusts the dialogue to the new requirements.Source: Agents Workflow · p. 7

Multi-sensory Movie Generation Result

• “The General’s Wedding” is a 1-minute-18-second film about a general drafted on his wedding day, with coherent plots, shots, and multiple characters.Source: Multi-sensory Movie Generation Result · p. 7
• Tone descriptions support scene- and style-specific voiceovers, while lip-sync and melancholy, heroic background music contribute to the film’s human-made appearance and romantic-tragic atmosphere.Source: Multi-sensory Movie Generation Result · p. 7

5 Broader Impact and Limitations

• Human interaction and time scheduling improve generated-content quality and significantly increase efficiency.The passage also says this can reduce costs and enable more creative ideas.Source: §5 · p. 7
• Fine-grained control over movie generation opens new possibilities for content creation.The passage frames this as a broader impact of AutoDirector.Source: §5 · p. 7
• Computational-resource demands may limit AutoDirector’s applicability in low-resource settings.Source: §5 · p. 7

6 Conclusion

• AutoDirector addresses limited parallelism and online scheduling in production, alongside users’ diverse and evolving filmmaking needs.Source: §6 · p. 7
• Automatic scheduling and interactive modification improve multi-sensory film-production efficiency and support ongoing adjustment based on user feedback.Source: §6 · p. 7
• The framework broadens human-machine collaboration and demonstrates AI’s potential to assist people acting as film directors.The stated scope is creating complex multi-sensory films.Source: §6 · p. 7