PaperlayerView exact revision on arXiv

Source-linked AI summary

Manga2Animate: An Agentic Framework for Faithful Manga Animation with Script and Storyboard Planning

Sheng-Wei Zhang, Hsi-An Chen, Yan-Ting Liou, Chih-Hsuan Chen, Shih-Chen Tseng, Yu-Lun Liu

arXiv:2610.26851v1cs.CV

TL;DR

Existing manga-to-animation approaches can lose visual identity or misread manga structure, despite manga already specifying characters, framing, and plot. Manga2Animate uses source-checked scripts and storyboards with iterative agent review before video synthesis. The paper reports greater fidelity than prior methods and strong user-study preference, while noting variability and difficult source material as limitations.

  • Problem

    Text-first systems can discard manga’s visual appearance, while direct page-to-video methods can misread panel order and speech balloons, making faithful adaptation challenging and costly.

    Source: §1 · p. 2

  • Method

    Manga2Animate uses creator and reviewer agents to iteratively refine source-grounded animation scripts and visual storyboards before video generation.

    Source: §1 · pp. 2–3

  • Results

    Manga2Animate produces more faithful characters, art style, and plot than existing methods, including same-script baselines, and users selected its storyboards and videos as best overall in over 90% of responses.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Reviewed scripts and storyboards give video models a faithful plan that constrains generation while letting them focus on animating the story.

    Source: §1 · p. 2

    Outputs may vary across runs, API-based models may restrict copyrighted or policy-sensitive content, and complex, sketchy, cluttered, or context-dependent manga may be misinterpreted.

    Source: §5 · p. 9

Abstract

from arXiv · show

Adapting existing manga into coherent animation that stays faithful to the source is a valuable yet challenging task. Existing methods either access the manga mainly through text, losing its characters and art style, or feed manga pages directly into video generation models, which misinterpret the manga's structure and produce incorrect storylines. In this paper, we propose Manga2Animate, a multi-agent framework that grounds every stage of generation in the source manga. We decompose the task into two intermediate representations, an animation script and a visual storyboard. Each representation is produced by a creator agent and reviewed against the source panels by a paired reviewer agent. We tailor representation-specific selection and dual-level feedback to progressively refine both before synthesizing the final animation. Extensive experiments and a user study demonstrate that Manga2Animate produces animations more faithful to the source manga and more consistent across shots than state-of-the-art approaches. Project page: https://annchenn.github.io/Manga2Animate/

1 INTRODUCTION

Manga adaptation is costly, and existing approaches either lose visual identity by translating manga into text or misread page structure when feeding pages directly to video models. Manga2Animate inserts source-checked scripts and storyboards before video generation, and reports stronger fidelity than prior methods.

1 INTRODUCTION

  • Manga adaptation remains slow and costly, often requiring large teams months of storyboarding, key animation, in-betweening, and coloring.Source: §1 · p. 2
  • Text-based storytelling can lose the source art style and distort characters, while direct page-to-video generation can tell the wrong story or render manga panels as video frames.Source: §1 · p. 2, Figure 2 · p. 2
  • The framework turns manga into an animation script and storyboard, each built by agents and checked against the original panels before video generation.The script fixes plot, while the storyboard fixes characters, composition, and art style.Source: §1 · p. 2
  • Manga2Animate produces animations more faithful to characters, art style, and plot than existing methods, including methods given the same script.In a user study, participants selected its storyboards and videos as best overall in over 90% of responses.Source: §1 · p. 2
  • The paper presents a framework for faithful manga-to-animation generation and identifies source-verified, dual-feedback script-and-storyboard planning as key to preserving characters, style, and plot.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work spans manga understanding, story visualization, multi-shot video generation, and agentic video production, but these approaches generally generate from text or prompts rather than grounding outputs in manga panels. Manga2Animate differentiates itself by reviewing intermediate outputs against the source manga.

2 RELATED WORK

  • Manga understanding has advanced from detecting panels, characters, and text to panel ordering, dialogue attribution, character naming, and chapter narration, yet models still misread cross-panel narrative.Manga generation remains largely unexplored; Manga2Animate reviews intermediate outputs against source panels.Source: §2 · p. 3
  • Story-visualization methods increasingly use visual memory, identity conditioning, shared attention, and language-model layout planning, while Manga2Animate conditions and selects its storyboard against manga panels.Source: §2 · p. 3
  • Multi-shot video research pursues cross-shot consistency through methods such as long-context tuning, keyframe memory, storyboard anchoring, and reference-based identity injection; Manga2Animate instead obtains consistency from reviewed scripts and storyboards.Source: §2 · p. 3
  • Agentic video systems can generate scripts, storyboards, clips, and assemblies, but generally invent content from prompts and check outputs against their own scripts; Manga2Animate reviewers check against manga panels.Earlier manga-to-animation systems only moved a virtual camera over panels or added synthesized speech.Source: §2 · p. 3

3 METHODOLOGY

Manga2Animate converts ordered manga panels into corresponding script clips and storyboard panels, then reviews and refines those representations before video generation. Its stages use different selection strategies and provide feedback across representations to preserve source alignment.

3.1 TASK FORMULATION

  • The framework maps ordered manga panels, a transcript, and character references into a coherent animated video through script and storyboard representations.Each script clip and storyboard panel corresponds to the same source panel, enabling review against that source unit.Source: §3.1 · p. 4

3.2 AGENTS

  • Creators produce each stage’s intermediate output, while reviewers assess outputs, select the best, and provide refinement feedback.The Script Reviewer can select character and clip descriptions independently at each panel position and reassemble them into one script.Source: §3.2 · p. 5, Figure 4 · p. 5

3.3 PRE-PROCESSING

  • Pre-processing detects panels and their reading order, transcribes page text into a unified transcript, and derives stable character descriptions from reference images.Descriptions retain identity-defining traits while excluding variable attributes such as clothing and accessories.Source: §3.3 · p. 5

3.4 SCRIPT AND STORYBOARD REFINEMENT

  • Script selection is compositional at clip level, whereas storyboard selection treats the jointly generated image as a whole to preserve shared style, color, and lighting.Two rounds of refinement use reviewer feedback; script and storyboard feedback jointly guide targeted revisions, with manga panels resolving conflicting feedback.Source: §3.4 · pp. 5–6

3.5 VIDEO GENERATION PIPELINE

  • Video generation supports first-frame and reference-conditioned modes, with the main experiments using reference conditioning.First-frame mode selects clips greedily using the storyboard panel and script; reference-conditioned mode selects among complete multi-shot candidates generated from the full script and storyboard.Source: §3.5 · p. 6

4 EXPERIMENTS

The experiments evaluate manga-conditioned storyboards and multi-shot videos through benchmark comparisons, qualitative examples, a user study, and ablations. The reported results favor Manga2Animate on fidelity and consistency, with reviewers and feedback contributing to the full model’s performance.

4.1 DATASETS

  • The evaluation set contains eight stories, each with six ordered manga panels, and supports measures of panel-level fidelity and cross-clip consistency.All methods use the same stories.Source: §4.1 · p. 6

4.2 BASELINES

  • Baselines include storyboard and video generators plus direct manga-to-video systems, with script-based methods given the same refined script for fair comparison.Character references are also provided when a baseline accepts them.Source: §4.2 · p. 6

4.3 EVALUATION METRICS

  • MangaEval adapts metrics to assess storyboard consistency, alignment, aesthetics, and cost, and video alignment, quality, aesthetics, and cost.Video metrics include subject and background consistency, motion smoothness, and dynamic degree; DINOv3 replaces DINO for I2VS, and DD uses a continuous motion-magnitude score.Source: §4.3 · pp. 6–7

4.4 QUANTITATIVE COMPARISONS

  • Manga2Animate performs better than AniMaker, ViMax, and StoryMem on I2VS, I2VB, BC, Shadow-v2, and Waifu-v3; its FLUX.2 + MiniMax-H3 setting also performs well.The reported gains persist across image/video model combinations; higher DD in direct generation measures more movement, not more accurate movement.Source: Table 1 · p. 7, §4.4 · p. 7

4.5 QUALITATIVE ANALYSIS

  • Qualitative examples show Manga2Animate retaining source appearance and content, while reviewed storyboards and scripts together reduce incorrect video content despite a shared refined script.The storyboard results illustrate failures such as uncolored panels or changed styles; video baselines include unrelated content, style shifts, and a basketball-to-tennis error.Source: §4.5 · pp. 7–8, Figure 5 · p. 7, Figure 6 · p. 8

4.6 USER STUDY

  • Thirty-seven participants preferred Manga2Animate most on every storyboard and video criterion; human plot and identity preferences agree directionally, but not one-to-one, with MangaEval fidelity metrics.The study used eight storyboard and eight video cases with method names hidden and outputs randomized.Source: Figure 7 · p. 8, §4.6 · p. 9

4.7 ABLATION STUDIES

  • Removing any stage-specific reviewer lowers overall alignment and consistency, while removing the Script Reviewer causes the largest I2VS drop.Some reviewer-removed variants score higher on motion smoothness or Shadow-v2 but lower on alignment, consistency, or dynamic degree.Source: Table 2 · p. 9, §4.7 · p. 9
  • Clip-wise script selection improves alignment, consistency, and aesthetics with small I2VB and motion-smoothness decreases; removing either feedback level, refinement, or storyboard generation lowers reported fidelity or video quality.The two feedback levels are complementary; the framework supports both reference-conditioned and first-frame video generation modes.Source: §4.7 · p. 9

5 CONCLUSION

Manga2Animate uses source-grounded scripts and storyboards, creator/reviewer agents, and representation-specific selection and feedback. It outperforms state-of-the-art methods on manga-fidelity and consistency metrics, while remaining vulnerable to variable outputs and difficult or context-dependent manga.

5 CONCLUSION

  • Manga2Animate grounds an animation script and visual storyboard in the source manga, then uses iterative creator/reviewer refinement, clip-wise script composition, whole-storyboard selection, and storyboard feedback to the script.The conclusion reports that it outperforms state-of-the-art storyboard and video methods on manga-fidelity and consistency metrics, even when baselines receive the same refined script and character references.Source: §5 · p. 9
  • The pipeline’s outputs may vary across runs, API models may restrict copyrighted or policy-sensitive content, and difficult layouts or story-context-dependent scenes may be misinterpreted.The stated challenging inputs include highly sketchy, cluttered, or compositionally complex manga and scenes requiring context from elsewhere in the story.Source: §5 · p. 9

APPENDIX

The appendix collects implementation, evaluation, baseline-adaptation, reviewer, and prompt details, alongside supplementary videos and a diagram of the representation correspondence.

APPENDIX

  • The appendix documents model and hardware settings, storyboard and video generation, full evaluation and ablation details, baseline adaptations, reviewer criteria, and agent prompts.It also points to an HTML page of generated videos and Fig. A1 for correspondence among manga panels, script clips, storyboard panels, and video clips.Source: APPENDIX · p. 16

A IMPLEMENTATION DETAILS

The implementation pairs specific language, image, and video models with reviewer-based candidate selection and supports reference-conditioned or first-frame video generation.

A IMPLEMENTATION DETAILS

  • Figure A1 maps each manga panel to a structured script clip and storyboard panel, which conditions the corresponding video clip directly or as part of a complete storyboard.Script clips specify the panel’s environment, character positions, actions, and expressions.Source: Figure A1 · p. 16
  • Gemini models power the creator and reviewer agents, while MAGI handles panel detection, reading-order estimation, and OCR.The open-source storyboard creator uses FLUX.2-dev and generates three candidates per panel for reviewer selection; experiments use two refinement rounds.Source: A IMPLEMENTATION DETAILS · p. 16
  • Reference-conditioned video generation uses the full script and storyboard grid, whereas first-frame mode starts each clip from its cropped storyboard panel.The reported main quantitative comparisons use reference-conditioned generation; Table A5 compares the two modes.Source: A IMPLEMENTATION DETAILS · pp. 16–17
  • US$5.65 per story is the reported average closed-source pipeline cost, including US$4.55 for Seedance.The paper cautions that wall-clock times depend on hardware, parallelism, and API latency.Source: A IMPLEMENTATION DETAILS · p. 17

B OPEN-SOURCE STORYBOARD BACKEND

The open-source backend generates storyboard panels separately, then selects and assembles them into a 3×2 grid. Source panels and character references condition generation and review.

B OPEN-SOURCE STORYBOARD BACKEND

  • Because FLUX.2-dev did not reliably produce a complete, high-quality multi-panel storyboard in one pass, the backend generates panels individually before composing the final storyboard.Each prompt combines character descriptions, script content, and constraints from the source panel to preserve composition, camera viewpoint, and pose.Source: B OPEN-SOURCE STORYBOARD BACKEND · p. 17
  • FLUX.2-dev generates multiple candidates per panel, which a Gemini-based reviewer scores and the system retains by highest score.Review criteria are manga/script follow, character identity consistency, art style and colorization quality, and background fidelity, each scored from 0–4.Source: B OPEN-SOURCE STORYBOARD BACKEND · p. 17
  • Each candidate is conditioned on its source manga panel and available character references, and the selected panels form a 3×2 grid for unified evaluation.Shared source-panel and character-reference conditions are intended to help maintain cross-panel consistency.Source: B OPEN-SOURCE STORYBOARD BACKEND · p. 17

C STORYBOARD QUANTITATIVE EVALUATION

Manga2Animate’s storyboard backends jointly lead on all four manga-fidelity metrics, while aesthetic scores can favor unconstrained text-to-image baselines. The evaluation adapts existing metrics to manga and reports different cost trade-offs for the two backends.

C STORYBOARD QUANTITATIVE EVALUATION

  • The evaluation adapts CIDS, OCCM, and PA for manga, and uses anime-specific aesthetic models instead of ViStoryBench’s general-purpose predictor.PA uses GPT-4.1 to reduce self-preference bias; shared refined scripts make it measure content adherence under an identical specification.Source: C STORYBOARD QUANTITATIVE EVALUATION · p. 17
  • The two Manga2Animate configurations achieve the best results across all four manga-fidelity metrics: FLUX.2 leads CSD and OCCM, while Gemini leads CIDS and PA over the best baseline.All baselines receive the same refined script, and the PA advantage holds with both open and closed image backends.Source: C STORYBOARD QUANTITATIVE EVALUATION · p. 18, Table A1 · p. 18
  • StoryDiffusion and Story-Iter score higher on anime-specific aesthetics but lower on manga-fidelity metrics, separating illustration appeal from source-aligned adaptation.Their text-generated imagery is not constrained by the original characters and composition.Source: C STORYBOARD QUANTITATIVE EVALUATION · p. 18
  • FLUX.2 avoids external image-generation APIs but takes 1 h 59 min on local GPUs; Gemini takes 20 min 2 s with 6 image-generation calls.Both are slower than StoryDiffusion and AnyStory but cheaper than AniMaker’s 8 h 9 min; the choice depends on compute resources and API budget.Source: C STORYBOARD QUANTITATIVE EVALUATION · p. 18

D ADDITIONAL ABLATION RESULTS

The ablations examine script selection, dual-level feedback, refinement and storyboard generation, and video-generation mode. Their results show metric-specific trade-offs across these design choices.

D ADDITIONAL ABLATION RESULTS

  • Clip-wise selection raises Waifu-v3 from 4.423 to 5.090, Shadow-v2 from 0.473 to 0.518, and DD from 5.951 to 9.457 over script-wise selection.I2VB and Motion Smoothness are slightly lower, with gaps within 0.004; the authors conjecture that clip-wise selection preserves more complete, specific descriptions.Source: D ADDITIONAL ABLATION RESULTS · p. 18, Table A2 · p. 19
  • Removing either script-level or storyboard-level feedback lowers CSD, CIDS, and PA, while their effects differ on OCCM and aesthetic metrics.The results indicate that the two feedback sources are complementary, mainly improving character identity and content alignment.Source: D ADDITIONAL ABLATION RESULTS · pp. 18–19, Table A3 · p. 19
  • Skipping second-round feedback or replacing the generated storyboard with manga panels lowers I2VS, I2VB, subject consistency, and background consistency; removing storyboard generation causes the larger drop.Some variants score higher on motion smoothness or Shadow-v2 despite lower alignment, consistency, or DD.Source: D ADDITIONAL ABLATION RESULTS · p. 19, Table A4 · p. 19
  • Reference-conditioned mode scores higher on I2VS, background consistency, and both aesthetic metrics, while first-frame mode scores higher on I2VB and slightly higher on subject consistency and motion smoothness.The pipeline also supports first-frame generation when full-storyboard reference conditioning is unavailable; the main results use reference-conditioned mode.Source: D ADDITIONAL ABLATION RESULTS · pp. 19–20, Table A5 · p. 20

E BASELINE ADAPTATION

The baselines are adapted to use Manga2Animate’s refined script where needed, with character references supplied when supported. Their original generation pipelines are otherwise retained for comparison.

E BASELINE ADAPTATION

  • Storyboard baselines receive six ordered prompts derived from the refined script; AnyStory also receives manga-cropped character references, unlike StoryDiffusion and Story-Iter.The generated panels are composed in order into a 3×2 grid for evaluation.Source: E BASELINE ADAPTATION · p. 20
  • Video baselines receive the refined script in their native formats, with manga-cropped references supplied to AniMaker and ViMax and manga panels used as StoryMem’s initial visual references.Their original remaining stages are retained, without Manga2Animate’s storyboard or reviewer modules.Source: E BASELINE ADAPTATION · pp. 20–21

F EVALUATION DETAILS

The evaluation defines separate storyboard- and video-level measures for visual consistency, content or manga alignment, quality, motion, and computational cost.

F EVALUATION DETAILS

  • Storyboard evaluation measures visual consistency, content alignment, aesthetic quality, and computational cost.Visual consistency uses CSD and CIDS; content alignment uses PA and OCCM.Source: Table A6 · p. 21, F EVALUATION DETAILS · p. 22
  • Video evaluation measures manga alignment, consistency, motion smoothness, dynamic degree, aesthetic quality, and computational cost across six panel-matched segments.Dynamic Degree measures motion magnitude rather than motion correctness.Source: Table A7 · p. 21, F EVALUATION DETAILS · p. 22
  • The video metrics are computed per segment against the corresponding source panels and then averaged across segments.Videos are split into six segments corresponding to the six manga panels.Source: F EVALUATION DETAILS · p. 22

G REVIEWER CRITERIA

The appendix details how creators and reviewers use source-grounded prompts and feedback to refine scripts and storyboard candidates. It describes criteria for assessing manga and script fidelity, visual consistency, layout, and individual redraws, alongside candidate scoring and selection.

G REVIEWER CRITERIA

  • Reviewers compare candidates criterion by criterion, return structured scores or winners and an overall winner, and suggest improvements only for criteria where the winner remains observably weak.Those suggestions are passed to the next refinement round.Source: G REVIEWER CRITERIA · p. 22
  • Character scripts are judged on clothing, appearance, reference-image color, and asymmetric-feature specificity; clip scripts are judged on objects, expression, background, placement, body orientation, and dialogue attribution.Character ties prioritize physical appearance, then clothing, left/right specificity, and color; clip ties prioritize dialogue assignment, body orientation, placement, expression, objects, and background.Source: G REVIEWER CRITERIA · pp. 22–23
  • Storyboard review uses 13 criteria spanning identity, style, layout, narrative and script alignment, backgrounds, panel details, and left/right consistency; some criteria are independently pass/fail, while others compare candidates.From round two, each new storyboard is compared with the previous best, judged against its generating script.Source: G REVIEWER CRITERIA · p. 23
  • The open-source panel reviewer scores manga/script fidelity, identity, art and color quality, and background fidelity; video review weights realism, commonsense, and prompt alignment, adding prior-clip coherence for later first-frame-mode clips.Panel scores use a 0–4 scale; video candidates incur a 0.5 hard-constraint penalty, and near-ties keep the first candidate.Source: G REVIEWER CRITERIA · p. 23

H AGENT PROMPTS

  • Character-description prompts require an exact ordered template, specific visible details, empty sections for unseen features, and no guessing or merging characters across reference images.Descriptions must follow the supplied image order for separate character references and use sequential IDs for multiple characters in one image.Source: H AGENT PROMPTS · p. 24
  • Script refinement treats both script and storyboard feedback as directives, fixes their root causes, preserves unflagged entries and clip-to-panel order, and treats manga visuals as the source of truth.Prompts also require camera-relative spatial descriptions, exact object counts and relations, and silent treatment of panels without speech bubbles.Source: H AGENT PROMPTS · p. 26

G REVIEWER CRITERIA

  • Storyboard comparisons assess character identity and art style alongside narrative order, script alignment, background and position consistency, and panel-internal object layout.The object-layout criterion is limited to composition within corresponding panels, not overall page layout or storyboard framing.Source: H AGENT PROMPTS · p. 31
  • When comparing storyboards generated from different scripts, reviewers judge each against its own script while using the manga page and character references as shared sources.Script-related judgments check each storyboard’s scene progression, actions, emotional flow, and scene details against its corresponding script.Source: H AGENT PROMPTS · p. 32
  • For single-panel redraws, reviewers score criteria independently on a 0–4 scale and weight manga/script follow—the action, pose, composition, and framing—most heavily.Other criteria include character identity, cel-shaded colorization without line-art residue or dialogue boxes, and background/environment fidelity.Source: H AGENT PROMPTS · p. 34

Related papers

Loading 2610.26851v1…