Source-linked AI summary
AniMaker: Multi-Agent Animated Storytelling with MCTS-Driven Clip Generation
Haoyuan Shi, Yunxin Li, Xinyu Chen, Longyue Wang, Baotian Hu, Min Zhang
TL;DR
Multi-scene storytelling videos remain difficult to keep coherent, and fixed-length keyframe-to-clip pipelines can fragment transitions and pacing. AniMaker coordinates four agents, using MCTS-Gen for candidate search and AniEval for context-aware clip review. It ranked 1st in VBench, scored 14.6% higher in AniEval, and received better human ratings (3.22 versus 2.07).
Problem
Fixed-length keyframe-to-clip pipelines can fragment transitions and pacing, while generating and evaluating multiple candidates is costly and existing metrics miss cross-clip coherence.
Source: §1 · p. 2Method
AniMaker coordinates four specialized agents and uses MCTS-Gen to search clip candidates and AniEval to assess clips in surrounding context.
Source: §1 · p. 2, §3.2 · pp. 3–4Results
AniMaker ranked 1st in VBench, scored 14.6% higher in AniEval, and received better human ratings (3.22 versus 2.07).
Source: §1 · p. 3Takeaways & Limitations
The authors frame the results as bringing AI-generated storytelling videos closer to production-grade quality.
Source: §1 · p. 2AniMaker remains limited in narrative scope and complexity relative to professional filmmaking's cinematic grammar and narrative structures.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Despite rapid advancements in video generation models, generating coherent storytelling videos that span multiple scenes and characters remains challenging. Current methods often rigidly convert pre-generated keyframes into fixed-length clips, resulting in disjointed narratives and pacing issues. Furthermore, the inherent instability of video generation models means that even a single low-quality clip can significantly degrade the entire output animation's logical coherence and visual continuity. To overcome these obstacles, we introduce AniMaker, a multi-agent framework enabling efficient multi-candidate clip generation and storytelling-aware clip selection, thus creating globally consistent and story-coherent animation solely from text input. The framework is structured around specialized agents, including the Director Agent for storyboard generation, the Photography Agent for video clip generation, the Reviewer Agent for evaluation, and the Post-Production Agent for editing and voiceover. Central to AniMaker's approach are two key technical components: MCTS-Gen in Photography Agent, an efficient Monte Carlo Tree Search (MCTS)-inspired strategy that intelligently navigates the candidate space to generate high-potential clips while optimizing resource usage; and AniEval in Reviewer Agent, the first framework specifically designed for multi-shot animation evaluation, which assesses critical aspects such as story-level consistency, action completion, and animation-specific features by considering each clip in the context of its preceding and succeeding clips. Experiments demonstrate that AniMaker achieves superior quality as measured by popular metrics including VBench and our proposed AniEval framework, while significantly improving the efficiency of multi-candidate generation, pushing AI-generated storytelling animation closer to production standards.
1 Introduction
The introduction identifies fragmentation, costly candidate generation, and limited evaluation as barriers to coherent multi-shot storytelling. AniMaker combines specialized agents, MCTS-Gen, and AniEval; experiments report gains across evaluation frameworks and generation efficiency.
1 Introduction
- Fixed-length clips mapped from keyframes can fragment videos, producing disjointed transitions and unnatural pacing, while one flawed clip can degrade overall quality.Source: §1 · p. 2
- Best-of-N clip selection is overlooked because generating candidates is costly and existing metrics such as VBench assess clips individually rather than cross-clip coherence.Source: §1 · p. 2
- Four agents handle storyboards, clip generation, review, and post-production, with MCTS-Gen searching candidates and AniEval evaluating clips in context.Source: §1 · p. 2
- TinyStories experiments show gains across VBench, AniEval, human evaluation, and multi-candidate efficiency, while the current work targets simple storytelling.Source: §1 · p. 2
- AniMaker ranked 1st in VBench, scored 14.6% higher in AniEval, and received better human ratings (3.22 versus 2.07).Source: §1 · p. 3
2 Related Work
Related work addresses visual consistency in storyboards, video consistency across clips, and agent-based coordination of storytelling-video pipelines. The cited approaches leave challenges in detail preservation, long-video consistency, and manual pre- or post-processing.
Storyboard Visualization
- Storyboard methods use adapters or consistency-aware approaches, but preserving visual details across multiple characters or changing backgrounds remains challenging.Source: §2 · p. 3
Video Generation
- Video-generation models and storytelling methods improve animation, temporal modeling, or contextual understanding, yet multi-clip consistency and long-video repetition remain challenges.Source: §2 · p. 3
Storytelling Video Agent
- Storytelling-video agents commonly coordinate script-to-keyframe-to-clip pipelines, while scriptwriting, character design, voiceovers, and sound effects are often handled manually.Source: §2 · p. 3
3 Methodology
AniMaker turns textual input into animation through four specialized agents. Its methodology formalizes story scripts and storyboards, then uses MCTS-Gen for clip generation and AniEval for evaluation.
3.1 Task Formulation
- Scripts organize ordered shots with descriptions, characters, and backgrounds; storyboards add reference banks and keyframes at shot transitions.Clips at transitions use keyframe guidance, while other clips can continue from the previous clip’s last frame; the assembled sequence is post-processed using the script.Source: §3.1 · p. 3
3.2 Pipeline Overview
- Four agents collaborate: the Director creates storyboards, Photography generates candidate clips, Reviewer evaluates them, and Post-Production assembles the sequence with voiceovers and subtitles.The pipeline is presented as transforming textual input into animation.Source: §3.2 · pp. 3–4
3.3 Multi-Agent Framework: AniMaker
- The Director creates a raw script and validates its consistency and narrative flow, then builds character and background banks and generates keyframes.The resulting storyboard serves as the production blueprint for later phases.Source: §3.3.1 · p. 4
- MCTS-Gen treats clip sequences as tree paths, generates and scores candidate clips, expands nodes using UCT, propagates child scores upward, and selects high-AniEval paths.Its parameters include initial candidates per node, UCT-guided expansions per iteration, and an exploration–exploitation factor; the method aims to balance search quality and resource use.Source: §3.3.2 · pp. 4–5
- AniEval adapts EvalCrafter for automated action assessment and adds multi-shot measures for frame consistency, object appearance changes, and animated-face consistency.It comprises four domains and 14 fine-grained metrics, with contextual scoring based on preceding and succeeding content.Source: §3.3.3 · p. 5
- Post-Production scripts narration and dialogue, selects voice profiles, generates and verifies audio, then assembles the film with synchronized visuals and subtitles.Voice scripts specify emotional tones and desired voice timbres; audio checks include duration consistency and content accuracy.Source: §3.3.4 · p. 5
4 Experiments
AniMaker is evaluated on multi-scene storytelling narratives through qualitative analysis, benchmark metrics, human ratings, and ablations. The reported results show strong performance in clip consistency, action completion, and generation efficiency.
4.1 Settings
- Across 10 TinyStories narratives, the evaluation includes complex multi-character interactions and diverse backgrounds, and compares AniMaker with visual-narrative specialists and video generators.StoryDiffusion and StoryAdapter use external image-to-video models; the video-generator group uses built-in video modules.Source: §4.1.1 · p. 5, §4.1.2 · p. 5
4.1.2 Baseline.
- Keyframe evaluation uses CLIP coherence, I-I Sim, and T-I Sim, while video generation is assessed with VBench and AniEval.Source: §4.1.3 · p. 6
4.2 Qualitative Analysis
- Qualitative analysis reports visual consistency across scenes, more complete extended actions, smoother clip transitions, and robust multi-character, multi-scene narratives.The authors attribute consistency to storyboard references and transition quality to AniEval’s contextual scoring of adjacent clips.Source: §4.2 · p. 6
4.3 Quantitative Comparisons
- AniMaker’s keyframes score 0.31 on Text-to-Image Similarity, 19.2% above the best baseline; MCTS-Gen maintains quality with over 50% fewer generations than exhaustive search.For the same candidates, exhaustive search requires 9 generations per node; MCTS-Gen uses fewer while maintaining quality.Source: §4.3 · pp. 6–7, Figure 4 · p. 7
- AniMaker achieves an AniEval total score of 76.72, 14.6% above VideoGen-of-Thought’s 66.93, and the best VBench average rank of 2.50.The AniEval comparison is reported in Table 4; the VBench average rank is reported in Table 3.Source: Table 4 · p. 7, §4.3 · p. 7, Table 3 · p. 7
4.4 Human Rating
- In human ratings across five dimensions, AniMaker achieves superior performance across all metrics, especially Character Consistency.Ten participants rated 90 videos from nine models across 10 stories on a 1–5 scale.Source: §4.4 · p. 7, Table 5 · p. 8
4.5 Ablation Studies
- A single-candidate MCTS-Gen ablation reduces AniEval performance by 7.1%, while selecting with VBench yields 73.18, a 4.6% decrease from the raw method.The resulting videos also show degraded action expressiveness and cross-clip consistency.Source: §4.5 · pp. 7–8
5 Limitations
The authors identify limits in narrative scope and complexity, a gap from commercial film quality, and defects in current video-generation models. They describe their framework as modular and updateable as models improve.
5 Limitations
- The method remains short of commercial film-production quality and is constrained in the narrative scope and complexity it can handle.Source: §5 · p. 8
- Underlying video-generation models can produce artifacts, incoherence, hallucinations, and physically unrealistic interactions.Source: §5 · p. 8
- The framework’s modular, plug-and-play architecture allows model integration, and the authors plan to update it as more advanced models emerge.Source: §5 · p. 8
6 Conclusion
AniMaker combines multi-agent production with MCTS-Gen and AniEval to generate storytelling animation from text. The appendix also presents prompts spanning story, imagery, video, and voiceover tasks.
6 Conclusion
- AniMaker introduces MCTS-Gen for clip-generation exploration and resource use, alongside AniEval for multi-shot storytelling-animation evaluation.The conclusion describes the system as emulating professional workflows and transforming text input into coherent storytelling animation.Source: §6 · p. 8
- The system coordinates specialized agents across storyboarding, generation, evaluation, and post-production.The conclusion reports substantial gains in technical metrics and perceived quality, without specifying values in this passage.Source: §6 · p. 8
A LMM Prompts
- The LMM prompts cover narrative-to-script conversion and validation, scene imagery, video generation, and voiceover creation, modification, and verification.The section describes the prompt sequence in Figures 7–14.Source: A LMM Prompts · p. 12
B Details of AniEval
AniEval extends EvalCrafter with automated action assessment and additional metrics, organizing evaluation into four domains and 14 fine-grained metrics.
B Details of AniEval
- AniEval automates action-related assessment by comparing user prompts with actions identified in video clips.This is one of the two stated ways it enhances EvalCrafter.Source: B Details of AniEval · p. 12
- AniEval adds DreamSim, Count-Score, and Face Consistency to its evaluation metrics.These are the additional metrics named in the passage.Source: B Details of AniEval · p. 12
- AniEval groups its comprehensive assessment into four primary domains comprising 14 fine-grained metrics.Source: B Details of AniEval · p. 12
B.1 Overall Video Quality
Overall video quality is assessed with Dover aesthetic and technical scores, while MUSIQ provides a perceptual-quality index.
B.1 Overall Video Quality
- Dover provides an aesthetic score, VQA_A, focused on visual appeal, composition, color harmony, and artistic quality.Source: §B.1.1 · p. 12
- Dover’s technical score, VQA_T, evaluates common distortions such as noise and artifacts.Source: §B.1.1 · p. 12
- MUSIQ evaluates perceptual quality by averaging frame scores into an index covering composition, sharpness, and distortions.Source: §B.1.1 · p. 12
B.1.2 MusIQ.
AniEval combines frame-level perceptual and text-alignment measures with object, cross-clip, consistency, and motion assessments. Its metrics use model outputs and averages to quantify these aspects.
B.1.2 MusIQ.
- MUSIQ scores sampled frames individually and averages them into a clip-level index of composition, sharpness, blur, and compression artifacts.Source: §B.1.2 · p. 12
B.2 Text-Video Alignment
- CLIPScore averages per-frame cosine similarities between prompt and image embeddings from pretrained ViT-B/32 CLIP.Source: §B.2.1 · p. 12
B.2.2 Text-Story Consistency.
- Text-Story Consistency averages BLEU similarity between the prompt and five video captions generated by BLIP2.Source: §B.2.2 · p. 12
B.2.3 Detection-Score.
- Detection Score averages binary frame results indicating whether SAMTrack detects the prompt’s key object.Gemini identifies the key object and desired quantity for each prompt.Source: §B.2.3 · p. 12
B.2.4 Count-Score.
- Count Score averages binary frame results for whether detected object counts match the target quantity.Source: §B.2.4 · p. 12
B.3.1 DreamSim.
- DreamSim averages minimum frame distances between a clip and an adjacent clip; Face Consistency compares facial embeddings across reference and video frames or adjacent clips.Face embeddings are evaluated using L2 distance.Source: §B.3.1 · p. 12, §B.3.2 · p. 12
B.3.2 Face Consistency.
- Warp differences average pixel-wise errors after optical-flow-based frame warping, while semantic consistency averages CLIP embedding similarities between consecutive frames.Source: §B.3.3 · p. 12, §B.3.4 · p. 12
B.4.1 Action Recognition.
- Motion assessment includes Gemini-determined action and speed targets, confidence-weighted action-recognition similarities, and average optical flow as action strength.Action recognition uses the top five predictions from MMAction2 and VideoMAE V2; CLIP compares them with the target action.Source: §B.4 · p. 13, §B.4.1 · p. 13, §B.4.2 · p. 13
B.4.3 Motion AC-Score.
Motion alignment is scored using average optical flow, classified as fast or slow, then compared with the motion level required by the prompt.
B.4.3 Motion AC-Score.
- The motion-alignment score is the average optical-flow score.Source: §B.4.3 · p. 13
- A score above the empirical threshold ρ=5 is labeled fast; otherwise, it is labeled slow.Source: §B.4.3 · p. 13
- The final comparison checks whether the fast-or-slow label matches the motion level required by the prompt.Source: §B.4.3 · p. 13