Source-linked AI summary
AesopAgent: Agent-driven Evolutionary System on Story-to-Video Production
Jiuniu Wang, Zehua Du, Yuyuan Zhao, Bo Yuan, Kexiang Wang, Jian Liang, Yaxi Zhao, Yihen Lu, Gengliang Li, Junlong Gao, Xin Tu, Zhenyu Guo
TL;DR
Story-to-video production still faces gaps in image expressiveness, narrative quality, and user engagement. AesopAgent combines agent orchestration, RAG-based iteration, and specialized utilities, and reports strong visual-storytelling comparisons with prior systems.
Problem
Story-to-video production requires cooperation among modules, while gaps remain in image expressiveness, narrative quality, and user engagement.
Source: §1 · p. 2Method
AesopAgent combines a Horizontal Layer that uses expert experience and professional knowledge for iterative workflow, prompt, and utility optimization with a Utility Layer for consistent image generation and video assembly.
Source: §1 · pp. 2–3, §5 · p. 17Results
AesopAgent reports state-of-the-art visual storytelling versus ComicAI and Artflow, and leading image complexity and narrative depth versus NUWA-XL [64] and AutoStory [59].
Source: §1 · pp. 2–3Takeaways & Limitations
AesopAgent can integrate external software such as Runway to cater to specific user needs, underscoring its reported scalability.
Source: §1 · p. 3
Abstract
from arXiv · showhide
The Agent and AIGC (Artificial Intelligence Generated Content) technologies have recently made significant progress. We propose AesopAgent, an Agent-driven Evolutionary System on Story-to-Video Production. AesopAgent is a practical application of agent technology for multimodal content generation. The system integrates multiple generative capabilities within a unified framework, so that individual users can leverage these modules easily. This innovative system would convert user story proposals into scripts, images, and audio, and then integrate these multimodal contents into videos. Additionally, the animating units (e.g., Gen-2 and Sora) could make the videos more infectious. The AesopAgent system could orchestrate task workflow for video generation, ensuring that the generated video is both rich in content and coherent. This system mainly contains two layers, i.e., the Horizontal Layer and the Utility Layer. In the Horizontal Layer, we introduce a novel RAG-based evolutionary system that optimizes the whole video generation workflow and the steps within the workflow. It continuously evolves and iteratively optimizes workflow by accumulating expert experience and professional knowledge, including optimizing the LLM prompts and utilities usage. The Utility Layer provides multiple utilities, leading to consistent image generation that is visually coherent in terms of composition, characters, and style. Meanwhile, it provides audio and special effects, integrating them into expressive and logically arranged videos. Overall, our AesopAgent achieves state-of-the-art performance compared with many previous works in visual storytelling. Our AesopAgent is designed for convenient service for individual users, which is available on the following page: https://aesopai.github.io/.
1 Introduction
AesopAgent turns story proposals into videos through agent-orchestrated workflows and two complementary layers. It combines RAG-driven iteration with utilities for consistent imagery, and reports strong visual-storytelling comparisons.
1 Introduction
- The paper identifies remaining gaps in image expressiveness, narrative quality, and user engagement when video production requires cooperation among multiple modules.AesopAgent is proposed to address these gaps by harnessing visual and narrative generation technologies and identifying AI utilities suited to user requirements.Source: §1 · p. 2
- The Horizontal Layer uses expert experience and professional knowledge in E-RAG and K-RAG to iteratively optimize workflows, prompts, and utility usage.The databases are constructed and updated with expert guidance.Source: §1 · pp. 2–3
- AesopAgent reports state-of-the-art visual storytelling against ComicAI and Artflow, with leading image complexity and narrative depth versus NUWA-XL [64] and AutoStory [59].The evaluation also reports superiority to ComicAI in image-element restoration, rationality, and composition, and notes integration with external software such as Runway.Source: §1 · pp. 2–3
- Agent techniques automate conversion of story proposals into videos through script generation, image generation, and video assembly.The workflow is designed by the agent to orchestrate the video-generation tasks.Source: §1 · p. 3
- Specialized utilities support cohesive image composition, multiple-character consistency, and style consistency across shots.The system assembles images, audio, and narration into videos enhanced with special effects.Source: §1 · p. 3
2 Related Works
Related work surveys agent memory, planning, and model-enhancement approaches, then reviews generative models and story-visualization systems. The paper positions its focus around assembling story-relevant visual and audio elements into complete videos.
2.1 AI Agent
- Agent research spans shared or long-term and short-term memory, planning and utility-use strategies, and peripheral RAG enhancements or occasional LLM fine-tuning.The cited examples include shared memory pools in multi-agent systems, while single-agent systems use long- and short-term memory; planning ranges from prompts and chains to structured procedures.Source: §2.1 · pp. 3–4
2.2 Generative Models
- Generative models produce new text, images, video, or audio; the review covers language, diffusion, transformer-based vision, video synthesis, and multimodal systems.DALL-E 3 is described as generating high-resolution images and converting textual descriptions into vivid images.Source: §2.2 · p. 4
2.3 Story Visualization
- Story visualization generates visually consistent images or videos from stories, but the reviewed systems lack a complete presentation of keyframes, audio, and special effects in story-rich videos.Prior systems include proposal-to-video approaches such as Artflow and ComicAI.Source: §2.3 · p. 4
3.1 Overall Framework
AesopAgent converts a user story proposal into a video using a Horizontal Layer for iterative orchestration and optimization and a Utility Layer for task-specific tools.
3.1 Overall Framework
- The Horizontal Layer aggregates expert experience and professional knowledge into E-RAG and K-RAG, then uses them to enhance prompts and continuously optimize the workflow and utility usage.It performs high-level strategic management and optimization iteratively.Source: §3.1 · pp. 4–5
- The framework depicts a proposal-to-video workflow alongside its two components: Horizontal Layer optimization and Utility Layer support for image generation and assembly.The workflow accepts a story proposal as input and produces a video that conveys the story.Source: §3.1 · p. 4, Figure 2 · p. 5
- The Utility Layer provides task-specific utilities, primarily for image creation and video assembly.These utilities are intended to support effective execution of each workflow step.Source: §3.1 · p. 5
3.2 Horizontal Layer
The Horizontal Layer uses agents to create and optimize the video-generation workflow, with K-RAG professional knowledge and E-RAG accumulated experience informing planning, prompts, and utility use. The workflow runs from script generation through image generation to video assembly.
3.2 Horizontal Layer
- The Horizontal Layer combines accumulated expert experience and professional knowledge to support agent-based workflow creation, task planning, prompt execution, and database updates.The paper identifies K-RAG as storing expert knowledge and E-RAG as summarizing feedback to refine the workflow, prompts, and utility use.Source: §3.2 · p. 5
3.2.1 Task Workflow Orchestration Module
- The two-stage orchestration uses expert-evaluated plans and E-RAG experience to establish a feasible workflow, then incorporates execution feedback to refine its three steps.The authors say the workflow is not yet optimal and that continued experience is expected to make it more reasonable.Source: §3.2.1 · p. 6, Figure 3 · p. 6
3.2.2 Prompt Optimization Module
- K-RAG retrieves professional screenwriting and filming knowledge, while E-RAG retrieves experience to iteratively refine script prompts for narrative-rich, engaging scripts.The script process generates titles, selects characters, and produces action and shot information.Source: §3.2.2 · p. 7
3.2.3 Utilities Usage Module
- The Utility Layer groups utilities for composition, character and style consistency, and video assembly, while K-RAG selects tools and E-RAG iteratively refines image generation from feedback.The workflow assembles images, audio, and narration into dynamic video; the first three utility modules support consistent image generation.Source: Figure 6 · p. 8, §3.2.3 · p. 8
3.3 Utility Layer
The Utility Layer turns shot descriptions into composed, character-consistent, stylistically unified images, then assembles media into a dynamic video.
3.3.1 Image Composition Rationality
- Image composition uses trained “magic words” for framing and bounding boxes to place described objects at specified locations.Bounding-box conditioning can place different characters in designated locations to create a cohesive scene.Source: §3.3.1 · pp. 8–9
3.3.2 Multiple Characters Consistency
- Character consistency combines attached descriptions of attributes with separate descriptions for each main character, then inpaints their image regions.The diffusion-based inpainting unit uses DreamBooth and parameter fusion of multiple models for consistent character generation.Source: §3.3.2 · p. 9
3.3.3 Image Style Consistency
- Style consistency uses style-trained image synthesis or ControlNet style transfer to unify character-consistent images.Style training can use image-text pairs and UNet or LoRA parameters; ControlNet uses a depth map and λct controls editing intensity.Source: §3.3.3 · p. 9
3.3.4 Dynamic Video Assembly
- Dynamic video assembly selects and edits narrative-matched audio and video materials, then synchronizes them on a timeline.Audio includes background music, sound effects, and TTS voice; video editing can add camera movements and transitions.Source: §3.3.4 · p. 10
4.1 Main Results of Horizontal Layer
The Horizontal Layer iteratively uses expert experience and professional knowledge to refine workflows, prompts, scripts, and utility usage.
4.1.1 Task Workflow Orchestration
- E-RAG incorporates expert suggestions across iterations; after character instability emerged, character design and selection moved to the workflow’s first step.Experts initially suggested decomposing storyboards into actions and shots, and the agent recorded suggestions and prior results for later iterations.Source: §4.1.1 · pp. 10–11
4.1.2 Prompt Optimization and Script Evaluation
- Expert review identified mismatched shot counts for action complexity, prompting global shot-number planning during action division.E-RAG was used to implement this prompt optimization.Source: §4.1.2 · p. 11
- Compared with Artflow and ComicAI, AesopAgent scripts maintained the highest fidelity and logical coherence; without K-RAG, completeness, fidelity, and logical coherence declined, particularly fidelity.Professional screenwriters gave a high evaluation of the overall scripts and final video.Source: §4.1.2 · p. 12
4.1.3 Utilities Optimization
- For style consistency, the agent found DALL-E 3 insufficient and experts validated a proposed LoRA–ControlNet solution; expert and GPT-4V feedback also refined image-utility use.Adding background detail and framing restrictions improved problematic images, while avoiding certain terms reduced film-strip-like outputs.Source: §4.1.3 · pp. 12–13
4.2 Main Results of Utility Layer
The Utility Layer is evaluated through generated-image comparisons and an extension that animates keyframes into video clips.
4.2.1 Effectiveness of Utility Modules
- Across two stories, AesopAgent’s utilities showed enhanced control over style and multi-character consistency versus SDXL [41] and ComicAI; in Image 4, they preserved Auntie and her interaction with Epaminondas.Source: §4.2.1 · p. 13, Figure 11 · p. 14
4.2.2 The Extension of Utilities
- Runway Gen-2 [9] turns each generated keyframe into a 4-second clip, dynamically enriching actions such as Goldilocks eating porridge and the characters’ interaction.The authors state that other utilities, including Sora, could also serve as downstream applications.Source: §4.2.2 · p. 15, Figure 12 · p. 15
4.3 Comparison with Other Methods
AesopAgent is compared with ComicAI on manually scored image quality and with Human Design, NUWA-XL, and AutoStory on three stories. The reported results favor AesopAgent on key image-quality dimensions and qualitative storytelling comparisons, with a lower element-state score noted as a qualification.
4.3.1 User Study with ComicAI
- The manual evaluation scores over 200 shot images from 10 randomly selected stories, rated by five annotators across fidelity, rationality, and element state.Overall image quality uses a weighted sum: 50% fidelity, 30% rationality, and 20% element state.Source: §4.3.1 · p. 15
- 17.95 points in fidelity and 4.13 points in rationality above ComicAI; AesopAgent also outperforms it in overall score, despite slightly lower element state.The authors attribute the element-state result to the larger number of elements in AesopAgent images, which makes precise organization harder.Source: §4.3.1 · pp. 15–16
- Image composition, character-description refining, character inpainting, and style consistency modules help images reflect script requirements and maintain cohesive style.The paper credits these Utility Layer modules with the reported overall-score advantage over ComicAI.Source: §4.3.1 · p. 16
4.3.2 Qualitatively Comparison
- Across three stories, AesopAgent shows more detailed scenes than Human Design, more vivid action shots than NUWA-XL, and better captures the cat-and-dog interaction than AutoStory.Figure 13 presents the keyframes and labels each frame with its semantic meaning.Source: §4.3.2 · pp. 16–17, Figure 13 · p. 16
5 Conclusion
The conclusion presents AesopAgent as a two-layer system that turns user story proposals into videos through iterative workflow optimization and utility-supported image consistency. Reported comparisons favor its scripts, images, and narrative capability, while future work targets broader user requirements and faster task completion.
5 Conclusion
- AesopAgent combines a Horizontal Layer for iterative E-RAG and K-RAG workflow, prompt, and utility optimization with a Utility Layer for consistent image composition, characters, and style.It assembles images, audio, and narration into videos enhanced with special effects.Source: §5 · p. 17
- The conclusion reports higher script and image scores than ComicAI and Artflow, and stronger overall narrative capability than NUWA-XL and AutoStory.Source: §5 · p. 17
- The authors identify broader themes and genres, more styles, user-defined characters, and faster, more convenient handling of complex tasks as future goals.They also mention possible extension to movie generation.Source: §5 · p. 17