PaperlayerView exact revision on arXiv

Source-linked AI summary

StoryAgent: Customized Storytelling Video Generation via Multi-Agent Collaboration

Panwen Hu, Jin Jiang, Jianqi Chen, Mingfei Han, Shengcai Liao, Xiaojun Chang, Xiaodan Liang

arXiv:2411.04925v2cs.CVcs.AIcs.MA

TL;DR

Customized storytelling video generation remains challenging because existing methods struggle to preserve a designated subject across shots. StoryAgent coordinates specialized agents and uses storyboard redrawing and LoRA-BE customized animation. The paper reports improved consistency and stronger results than comparison methods across evaluations.

  • Problem

    Existing storytelling-video systems struggle to preserve designated protagonists across shots, and customized text-to-video methods also have difficulty maintaining reference-subject fidelity.

    Source: §1 · p. 2

  • Method

    StoryAgent coordinates five specialized agents and combines a subject-redrawing storyboard pipeline with LoRA-BE customized image-to-video generation.

    Source: §1 · p. 2

  • Results

    StoryAgent receives the highest scores across all five user-study aspects on two datasets, with especially strong inter-shot consistency.

    Source: §4.3 · p. 10

  • Takeaways & Limitations

    The authors conclude that storyboard generation and LoRA-BE improve consistency across shots and during animation, respectively.

    Source: §5 · pp. 10–11

    Customized human videos remain challenging because of constraints in the underlying video-generation model, and shot duration remains relatively short.

    Source: §5 · p. 11

Abstract

from arXiv · show

The advent of AI-Generated Content (AIGC) has spurred research into automated video generation to streamline conventional processes. However, automating storytelling video production, particularly for customized narratives, remains challenging due to the complexity of maintaining subject consistency across shots. While existing approaches like Mora and AesopAgent integrate multiple agents for Story-to-Video (S2V) generation, they fall short in preserving protagonist consistency and supporting Customized Storytelling Video Generation (CSVG). To address these limitations, we propose StoryAgent, a multi-agent framework designed for CSVG. StoryAgent decomposes CSVG into distinct subtasks assigned to specialized agents, mirroring the professional production process. Notably, our framework includes agents for story design, storyboard generation, video creation, agent coordination, and result evaluation. Leveraging the strengths of different models, StoryAgent enhances control over the generation process, significantly improving character consistency. Specifically, we introduce a customized Image-to-Video (I2V) method, LoRA-BE, to enhance intra-shot temporal consistency, while a novel storyboard generation pipeline is proposed to maintain subject consistency across shots. Extensive experiments demonstrate the effectiveness of our approach in synthesizing highly consistent storytelling videos, outperforming state-of-the-art methods. Our contributions include the introduction of StoryAgent, a versatile framework for video generation tasks, and novel techniques for preserving protagonist consistency.

1 INTRODUCTION

Customized storytelling video generation remains difficult because existing systems struggle to preserve a designated protagonist across shots. StoryAgent combines specialized agents with storyboard and animation methods intended to improve that consistency.

Motivation

  • Existing agent-driven S2V systems do not support designated protagonists, while customized text-to-video methods still struggle to preserve reference-subject fidelity across scenes.The paper reports that TI-AnimateDiff, DreamVideo, and Magic-Me fail to preserve the reference subject’s appearance.Source: §1 · p. 2

StoryAgent

  • StoryAgent assigns story design, storyboard generation, video creation, coordination, and review to five specialized agents for customized storytelling video generation.The framework’s agent functions can be replaced to support a range of video-generation tasks.Source: §1 · p. 2

Consistency methods

  • A generation-removal-redrawing storyboard pipeline targets inter-shot consistency, while LoRA-BE and background-agnostic augmentation target intra-shot consistency.The paper reports comparisons on customized and public datasets.Source: §1 · p. 2

Evaluation

  • Experiments on customized and public datasets show StoryAgent outperforming state-of-the-art customized video-generation approaches in generating consistent storytelling videos.The paper also presents a public-dataset evaluation protocol and collects new internet subjects for testing.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work addresses story visualization, image animation, and agent-based video production, but the cited approaches leave challenges in diverse story scenes and customized-subject consistency.

Story Visulization

  • Story-visualization methods use diffusion, autoregressive generation, or layout-conditioned image generation, but some rely on story-specific datasets and customizable-subject methods can lose detail consistency across sequences.StoryAgent’s related-work discussion says its storyboard generator uses removal and redrawing to address detail consistency.Source: §2 · p. 3

Image Animation

  • Image-animation research spans optical-flow and diffusion approaches, yet animating out-of-domain customized subjects remains challenging and can produce distorted subjects.LoRA-BE is presented as an approach to improve customized generation.Source: §2 · p. 3

AI Agent

  • AesopAgent uses an agent-driven system for story-to-video production, but generating storytelling videos for customized subjects remains a challenge.The passage describes script generation, image generation, and video assembly as its stages.Source: §2 · pp. 3–4

3 STORYAGENT

StoryAgent coordinates five agents through storyline generation, storyboard creation, and video generation. Its visual agents use reference-subject storyboard redrawing and customized image animation to address consistency across and within shots.

3 STORYAGENT

  • StoryAgent takes a prompt and reference-subject videos through storyline generation, storyboard creation, and video generation, with five agents collaborating on the workflow.Story descriptions and storyboards can be reviewed by the observer or user before the workflow advances.Source: §3 · p. 4

3.1 LLM-BASED AGENTS

  • The agent manager uses an LLM and completed-task context to select the next agent, while the story designer generates shot descriptions from narrative requirements.The designer can use GPT-4, Claude, or Gemini.Source: §3.1 · p. 5
  • The optional observer assesses agent outputs and signals whether to proceed or provide feedback, but current aesthetic-quality methods remain unreliable.Users may substitute human evaluation or omit the observer.Source: §3.1 · p. 5

3.2 VISUAL AGENTS

  • The storyboard generator creates initial images from shot descriptions, segments their subjects, then uses a reference subject and StoryAnyDoor to redraw the masked regions.The pipeline is designed to preserve character-detail consistency across shots.Source: §3.2 · pp. 5–6
  • LoRA-BE customizes image-to-video generation by tuning attention-block LoRA parameters and separate token embeddings for each cross-attention block using reference videos.The method adds a localization loss that uses subject masks to increase token-video similarity at subject locations.Source: §3.2 · pp. 6–7
  • In DynamiCrafter inference, a storyboard image and text and image embeddings condition a U-Net that denoises a sampled video latent for each shot.The storyboard image is encoded and concatenated with the noisy latent as U-Net input.Source: §3.2 · p. 6

4 EXPERIMENTS

The experiments evaluate StoryAgent on public storytelling datasets, open-domain subjects, user studies, and component ablations.

4 EXPERIMENTS

  • Storyboard training used 4–5 reference videos per subject and 20,000 steps; LoRA-BE training used 400 epochs on an NVIDIA V100 GPU.The storyboard model used AnyDoor; DynamiCrafter was the foundation for I2V training.Source: §4 · p. 7
  • Evaluation used PororoSV and FlintstonesSV, plus eight open-domain subjects; public-dataset metrics were reference-based, while open-domain metrics were non-reference.The open-domain set had no ground truth; it included IS, Clip-score, Clip-temp, warping error, and Flow-score.Source: §4 · p. 7

4.1 EVALUATION ON PUBLIC DATASETS

  • Figure 5 compares StoryAgent with TI-SparseCtrl and SVD, reporting that StoryAgent preserves both inter-shot and intra-shot consistency and more closely resembles ground truth.TI-SparseCtrl shows character variation across shots, while SVD changes within shots, especially in the second and third shots.Source: Figure 5 · p. 8, §4.1 · p. 8
  • StoryAgent outperformed SVD and TI-SparseCtrl on FVD and LPIPS and improved SSIM against ground-truth videos on PororoSV and FlintstonesSV.The evaluation paired the methods with a storyboard to address background discrepancies from direct text-to-video generation.Source: Table 1 · p. 8, §4.1 · p. 8

4.2 EVALUATION ON OPEN-DOMAIN SUBJECTS

  • On open-domain subjects, StoryAgent’s storyboard pipeline better preserved reference-subject details across shots, while its video creator helped maintain within-shot consistency.The cited comparison notes clothing details and describes other methods’ consistency problems.Source: Figure 6 · p. 9, §4.2 · p. 9
  • On the open-domain dataset, StoryAgent outperformed other methods on text-video alignment while achieving comparable IS and Clip-temp performance.The paper reports high consistency alongside comparable video quality.Source: Table 2 · p. 9, §4.2 · p. 9

4.3 USER STUDIES

  • With 14 valid responses, StoryAgent received the highest scores across all five user-rated aspects on both datasets, especially inter-shot and intra-shot consistency.Participants rated consistency, subject-background harmony, text alignment, and overall quality on a 1–5 scale.Source: §4.3 · pp. 9–10, Table 3 · p. 10, Table 4 · p. 10

4.4 ABLATION STUDIES

  • Without RoLA-BE, DynamiCrafter failed to preserve intra-shot consistency and its video-quality and human-perception scores decreased; with it, videos retained consistency and quality.The ablation compares the proposed component against a baseline that fine-tuned DynamiCrafter’s image-injection module.Source: §4.4 · p. 10, Table 5 · p. 10

5 CONCLUSION

StoryAgent divides customized storytelling video generation into story-description, storyboard, and animation subtasks, using storyboard generation and RoLA-BE to address consistency. The authors report superior consistency in qualitative and quantitative assessments, while noting several limitations.

5 CONCLUSION

  • StoryAgent separates story-description generation, storyboard creation, and animation, with storyboard generation and RoLA-BE targeting inter-shot and intra-shot consistency, respectively.Source: §5 · p. 10
  • Qualitative and quantitative assessments affirm superior consistency in StoryAgent’s generated results.The conclusion also notes that customized human videos, shot duration, and capture of text-specified details remain constrained.Source: §5 · p. 11
  • The method faces challenges with customized human videos, relatively short shots, and fully capturing text-specified details.The authors identify training more generalized base models on larger datasets and supporting multiple coherent subjects as future directions.Source: §5 · p. 11

A APPENDIX

The appendix outlines further details of agent scheduling and additional evaluations, visualizations, ablations, and user studies.

A APPENDIX

  • The appendix provides more detail on the agent scheduling process in the Agent Manager.Source: A APPENDIX · p. 15
  • Planned additional evaluations cover public datasets, open-domain subjects, storytelling-video visualizations, and public-dataset ablations.Source: A APPENDIX · p. 15
  • The appendix also covers the Observer agent’s performance and details of the user studies.Source: A APPENDIX · p. 15

A.1 MORE DETAILS OF THE AGENT SCHEDULING PROCESS IN AM

Figure 7 depicts the Agent Manager’s scheduling process, distinguishing agent selections from signals and showing prompts used by the Story Designer and AM.

A.1 MORE DETAILS OF THE AGENT SCHEDULING PROCESS IN AM

  • Solid arrows indicate AM selecting an agent after receiving a signal; dashed arrows indicate signals produced by the selected agent.Source: Figure 7 · p. 15
  • The figure also shows prompts used by the Story Designer and AM.Source: Figure 7 · p. 15

A.2 MORE EVALUATIONS ON PUBLIC DATASETS

On FlintstonesSV, StoryAgent with LoRA-BE is reported to produce results closer to ground truth and maintain subject temporal consistency better than other methods; the appendix also shows a Kitty visualization.

A.2 MORE EVALUATIONS ON PUBLIC DATASETS

  • StoryAgent with LoRA-BE produces FlintstonesSV results closer to ground truth and maintains subject temporal consistency better than other methods.Source: §A.2 · p. 15, Figure 8 · p. 16
  • The appendix includes a storytelling-video generation visualization on the FlintstonesSV dataset.Source: Figure 8 · p. 16
  • The appendix also includes a storytelling-video generation visualization for the open-domain subject Kitty.Source: Figure 9 · p. 16

A.3 MORE EVALUATIONS ON OPEN-DOMAIN SUBJECTS

Open-domain examples compare StoryAgent with other customization methods on birds and cartoon subjects. The reported comparisons emphasize subject identity across shots and distinguish storyboard-level consistency from consistency within a shot.

A.3 MORE EVALUATIONS ON OPEN-DOMAIN SUBJECTS

  • StoryAgent maintains detailed subject cues across open-domain sequences, including Kitty’s clothes and the bird’s appearance, while TI-SparseCtrl loses consistency when subjects change.The examples are reported for generated videos whose story descriptions come from the story designer agent.Source: §A.3 · p. 16, Figure 10 · p. 17
  • SVD maintains the real-world bird’s temporal consistency, but its movements follow the text less well than StoryAgent’s more vivid bird videos.Source: §A.3 · p. 16, Figure 10 · p. 17
  • For the elephant, StoryAgent preserves the reference subject’s identity in all shots, unlike TI-AnimatedDiff, DreamVideo, and Magic-Me.Figure 11 describes the other three methods as failing to generate a consistent subject with the reference videos.Source: Figure 11 · p. 17, §A.3 · p. 17
  • The storyboard generator supports inter-shot consistency, while stored subject information in the video creator supports intra-shot consistency.The paper attributes these roles to the respective agents in the elephant comparison.Source: §A.3 · p. 17

A.4 MORE ABLATION STUDIES

The additional ablations compare fine-tuned DynamiCrafter with and without RoLA-BE on PororoSV and FlintstonesSV. The reported examples favor the RoLA-BE configuration for subject fidelity and consistency.

A.4 MORE ABLATION STUDIES

  • On FlintstonesSV, simple fine-tuning leaves inconsistency, whereas the RoLA-BE strategy yields more consistent results with the ground truth.Source: Figure 13 · p. 18, §A.4 · p. 18
  • On PororoSV and FlintstonesSV, the RoLA-BE configuration produces closer-to-ground-truth, script-fitting results and better inter-shot and intra-shot consistency.The FlintstonesSV comparison also describes the RoLA-BE results as high quality.Source: §A.4 · p. 18, Figure 13 · p. 18, Figure 12 · p. 18
  • On PororoSV, fine-tuned DynamiCrafter still fails to generate customized subjects, while the RoLA-BE results are closer to ground truth and fit the script.Source: §A.4 · p. 18, Figure 12 · p. 18

A.5 THE PERFORMANCE OF OBSERVER

The observer evaluation reports limits in storyboard-quality scoring, alongside the user-study indicators and social-impact concerns discussed in the merged sections. The paper describes detection methods as possible mitigations for misuse-related risks.

A.5 THE PERFORMANCE OF OBSERVER

  • MLLMs do not reliably distinguish storyboard quality; the aesthetic predictor performs relatively better but remains imperfect, so the experiments bypass the observer.The authors leave improving aesthetic-quality assessment methods to future work.Source: §A.5 · p. 19

A.6 THE DETAILS OF USER STUDIES

  • The user questionnaire assesses inter-shot and intra-shot subject consistency, subject-background harmony, text alignment, and overall video quality.Inter-shot consistency considers reference images, whereas intra-shot consistency is judged within a shot without reference images.Source: §A.6 · p. 19

A.7 SOCIAL IMPACT

  • The paper identifies deepfake misuse and ownership or copyright infringement as risks of storytelling-video synthesis, and says forensic analysis and manipulation detection could alleviate negative effects.Source: §A.7 · p. 20
Loading 2411.04925v2…