PaperlayerView exact revision on arXiv

Source-linked AI summary

ViStoryBench: Comprehensive Benchmark Suite for Story Visualization

Cailin Zhuang, Ailin Huang, Yaoqi Hu, Jingwei Wu, Wei Cheng, Jiaqi Liao, Hongyuan Wang, Xinyao Liao, Weiwei Cai, Hengyuan Xu, Xuanyang Zhang, Xianfang Zeng, Zhewei Huang, Gang Yu, Chi Zhang

arXiv:2505.24862v5cs.CV

TL;DR

Existing story visualization benchmarks provide limited coverage of narrative complexity, character references, and evaluation dimensions. ViStoryBench introduces a human-validated, multi-shot benchmark with diverse scripts, references, and automated metrics, and finds that current methods struggle to jointly optimize semantic and visual quality.

  • Problem

    Existing benchmarks often use limited metrics and constrained scenarios that do not represent real-world story visualization involving narrative alignment and character consistency.

  • Method

    ViStoryBench combines 80 multi-shot stories across 10 visual styles with character references, script annotations, and 12 human-validated automated metrics.

  • Results

    Current methods show different strengths: multimodal large models excel in narrative alignment but lag in visual quality, while commercial tools perform strongly in aesthetics and style similarity but often lack fine-grained narrative control.

  • Takeaways & Limitations

    The evaluation supports task-specific model selection and motivates multi-shot data, unified multimodal architectures, and more holistic evaluation metrics.

  • Takeaways & Limitations

    Evaluation is limited to multi-image generation with emphasis on inter-frame consistency, and its hybrid metrics remain vulnerable to expert-model weaknesses and VLM hallucinations.

Abstract

from arXiv · show

Story visualization aims to generate coherent image sequences that faithfully represent a narrative and match given character references. Despite progress in generative models, existing benchmarks remain narrow in scope, often limited to short prompts, lacking character references, or single-image cases, failing to reflect real-world narrative complexity and obscuring true model performance.We introduce ViStoryBench, a comprehensive benchmark designed to evaluate story visualization models across varied narrative structures, visual styles, and character settings. It features richly annotated multi-shot scripts derived from curated stories spanning literature, film, and folklore. Large language models assist in story summarization and script generation, with all outputs verified by humans for coherence and fidelity. Character references are carefully curated to maintain consistency across different artistic styles. ViStoryBench proposes a suite of multi-dimensional automated metrics to evaluate character consistency, style similarity, prompt alignment, aesthetic quality, and artifacts like copy-paste behavior. These metrics are validated through human studies and used to assess a broad range of open-source and commercial models, enabling systematic analysis and encouraging advances in visual storytelling.

1. Introduction

ViStoryBench addresses the lack of comprehensive, realistic evaluation for story visualization by combining diverse narrative data, multifaceted metrics, and broad model testing.

  • Existing story visualization benchmarks use limited metrics and constrained scenarios that do not reflect real-world applicability.
  • The dataset covers 80 story segments, 344 characters, diverse genres and visual styles, single and multiple protagonists, complex plots, and intricate world-building.
  • The benchmark evaluates image quality, diversity, prompt alignment, style consistency, character actions and interactions, visual novelty, and scene character-count correctness through 12 automated metrics.
  • Experiments evaluate more than 30 methods and compare automated metrics with human evaluations to provide model insights.
  • The authors plan to release the benchmark, construction prompts, model results, and code for reproducibility.

2. Related Work

Related work spans image, video, and 3D story generation, but existing evaluation remains fragmented in scope, style coverage, and character-reference support. ViStoryBench targets these gaps with a reference-based, multi-image benchmark.

  • 2.1. Story Visualization: Story visualization methods have progressed from GANs to diffusion, planning with LLMs or VLMs, story continuation, next-shot generation, comics, and unified multimodal models.
  • 2.1. Story Visualization: Story video systems increasingly generate multi-shot narratives, while ViStoryBench evaluates their image-level outputs by extracting video keyframes as storyboards.
  • 2.1. Story Visualization: Despite progress, challenges remain in multi-image coherence, long-range dependencies, fine-grained control, and complex narrative alignment.
  • 2.2. Related Benchmarks: Table 1 compares related benchmarks by styles, evaluation metrics, and evaluated methods, with ViStoryBench offering the most comprehensive coverage.
  • 2.2. Related Benchmarks: Compared with prior benchmarks, ViStoryBench provides reference-based multi-image evaluation with broader style coverage and stronger support for character consistency.

3. ViStoryBench

ViStoryBench defines story visualization as generating storyboard images from scripts, character references, and shot descriptions, then evaluates outputs across character, style, alignment, aesthetic, and artifact dimensions.

  • 3.1. Problem Definition: The task provides character descriptions and reference images plus storyboard shots specifying settings, plot correspondence, onstage characters, static actions, and camera perspective.
  • 3.2. Source Data: The dataset contains 80 manually curated story segments, with LLM-assisted summaries and scripts reviewed by humans for coherence and fidelity.
  • 3.2. Source Data: The final collection includes 344 characters, 509 reference images, and 10 distinct visual styles, with references aligned to textual descriptions.
  • 3.3. Evaluation Metrics: CIDS detects and crops character regions, extracts feature vectors, matches characters bipartitely, and averages matched-pair cosine similarities to measure self- and cross-similarity.
  • 3.3. Evaluation Metrics: Prompt alignment evaluates scene correspondence, camera-shot consistency, group interactions, and individual actions against storyboard descriptions.
  • 3.3. Evaluation Metrics: OCCM measures whether generated images contain the expected number of characters, while Copy-Paste Detection measures excessive reuse of input character references.

4. Experiments and Analysis

ViStoryBench evaluates story visualization across diverse methods, human judgments, and automated metrics, with a Lite subset designed for efficient assessment. Results reveal distinct strengths and trade-offs across model families, while the benchmark remains focused on multi-image consistency and has metric limitations.

  • Experimental Setup: ViStoryBench-Lite uses 20 stories and 108 characters to preserve the full benchmark’s composition, with consistently high performance correlation validating cost-effective evaluation.The subset contains 36 animated, 41 realistic, and 43 non-human characters and was designed to align with the full dataset in text styles and character-reference distributions.
  • Experimental Setup: The evaluation covers diverse image and video generation methods, including open-source systems, commercial models, multimodal large models, and multi-shot video generators.Video methods without intermediate images are evaluated using shot keyframes, and most methods produce 1080p outputs.
  • Human Evaluation: Automated metrics are compared with human ratings across environment consistency, character identification consistency, and subjective aesthetics, with reported correlations intended to reflect human preferences.The study maps these human dimensions to Style Similarity, CIDS, and Aesthetic Quality, respectively.
  • Insights: Multi-modal large models excel in narrative alignment but lag in visual quality, commercial tools emphasize aesthetics and style similarity, and story image methods achieve character consistency with limited generalization.Story video methods face per-frame quality challenges, while multi-shot video models show strong self-consistency but weaker customization capability.
  • Insights: Current methods still struggle to jointly optimize semantic and visual qualities, exposing a trade-off between consistency and diversity and motivating task-specific model selection.The findings also point toward multi-shot data, unified multimodal architectures, and more holistic evaluation metrics.
  • Limitations: The benchmark is deliberately scoped to multi-image generation and omits background-reference evaluation, while its hybrid metrics trade expert-model stability against VLM semantic richness and hallucination risk.These limitations constrain coverage of scene-level references and broader temporal or audio-visual storytelling evaluation.

5. Conclusion

ViStoryBench provides a high-fidelity, multidimensional benchmark for story visualization, but its current scope reflects unresolved evaluation and technology constraints. The authors identify boundaries involving temporal modeling, background references, metric reliability, dataset provenance, language, supported formats, and video evaluation.

  • ViStoryBench evaluates 80 multi-shot stories across 10 visual styles with character references and 12 human-validated metrics, enabling large-scale multidimensional comparison.The metrics cover character similarity, style similarity, prompt alignment, aesthetics, artifacts, and copy-paste behavior.
  • The benchmark is deliberately limited to multi-image generation and inter-shot consistency rather than synchronized audio-visual storytelling or full temporal modeling.This scope targets visual narrative coherence without adding audio alignment and temporal-modeling complexity.
  • ViStoryBench omits background-reference images and scene-level image-to-image similarity because current open-source methods provide limited background-conditioned generation support.Scene consistency is instead assessed through text-prompt alignment, and future work will add scene references.
  • Its hybrid metrics trade expert-model stability and continuity against VLM semantic richness and narrative alignment, with neither approach universally optimal.Expert models may lack adaptability in complex cases, while VLMs can hallucinate or diverge from human perceptual judgments.
  • The dataset includes third-party visual content that raises copyright and metric-overfitting concerns, while language choices may introduce quality differences unrelated to model capability.The benchmark also lacks reliable evaluation for comic or manga multi-panel layouts, inference efficiency, and comprehensive video temporal coherence.
  • The authors caution that generative models may reproduce stereotypes, amplify data-driven biases, or create misleading content, requiring responsible development and deployment.They call for collaboration among researchers, platform providers, and policymakers to address these risks.

B. Overall Insights

Results across ViStoryBench show no universal winner: methods trade narrative alignment, consistency, diversity, aesthetics, and reference adherence. Multimodal LLMs lead semantic metrics, video models favor cross-shot consistency, and specialized or commercial systems excel in narrower visual dimensions.

  • GPT-4o [36] achieves best or second-best results on Alignment Score (3.673), OCCM (93.5), CIDS, and CSD, but lags on Inception Score (9.02) and Aesthetics Score (5.49).This combines strong narrative understanding and consistency with weaker diversity and low-level visual quality.
  • Commercial systems specialize unevenly: MorphicStudio reaches CSD Cross 0.653, Doubao performs well on alignment at 3.494, while MOKI combines aesthetics 5.79 with CIDS Cross 0.214.These results show strong visual or semantic performance can coexist with weak character consistency.
  • Image methods also excel in specific niches, with OmniGen2 [86] leading CIDS Self at 0.537 and OCCM at 90.8, while Story-Adapter variants reach CSD Cross 0.456.These gains are not uniform across evaluation dimensions.
  • Performance varies widely: SEED-Story [98] and TheaterGen [6] score below 2.00 on Alignment and below 0.35 on CIDS Cross, while reference images can improve consistency at the expense of creativity.The Copy-Paste Baseline also performs optimally on many metrics but has notably low alignment, illustrating why no single metric suffices.
  • Sora2 reaches CIDS Self 0.813 and CSD Self 0.713, outperforming many image methods on cross-shot consistency, but scores lower on reference adherence with CIDS Cross 0.738 and CSD Cross 0.515.The results indicate a trade-off between temporal consistency and preserving visual reference constraints.

C. Details of Dataset Collection and Statistics

ViStoryBench is built from 80 culturally diverse, multi-shot stories with structured scripts, curated character references, and broad stylistic coverage. An LLM-assisted, human-reviewed pipeline converts narratives into coherent, verifiable shot descriptions.

  • The benchmark spans 10 visual styles and includes realistic and unrealistic story categories, enabling stratified evaluation across distinct visual and semantic characteristics.Reference images range from photographic or cinematic depictions to animation, stylized art, and fantasy characters.
  • The dataset contains 80 stories spanning diverse cultural regions and genres, totaling 1,317 shots with 4–30 shots per story and an average of 16.5 shots.The collection includes folktales, romance, suspense, horror, historical, fantasy, science-fiction, war, social-life, survival, and fairy-tale stories.
  • Each shot is annotated with Setting Description, Plot Correspondence, Onstage Characters, Static Shot Description, and Shot Perspective Design, with prompts supplied in English and Chinese.The structured fields cover narrative, characters, scene, composition, and camera information.
  • Character references are manually curated from existing visual works, analogous media, or SDXL-generated stylized portraits for lesser-known or original stories.This supports consistent character depiction across varied artistic styles.
  • Character coverage includes 344 unique characters and 509 reference images, with each character associated with 1–10 images and 89 characters having multiple references.The characters include real humans, virtual humans, and non-human entities.
  • Stories are manually curated from film, television, literature, folklore, novels, and picture books, while LLMs assist summarization and script adaptation before human review for coherence and logical consistency.The construction pipeline uses LLM shot planning and structured prompt decomposition to generate controllable, visually grounded descriptions.

D. Qualitative Results

Qualitative and subset analyses show that ViStoryBench exposes differences in prompt alignment, character consistency, and visual quality across methods. ViStoryBench-Lite closely tracks the full benchmark, supporting representative and efficient evaluation.

  • Visual comparisons of the first five shots and full story sequences reveal differences among methods in prompt alignment, character consistency, visual quality, temporal coherence, persistence, and scene transitions.The Copy-Paste Baseline provides a reference for character presence in each frame.
  • ViStoryBench-Lite exhibits a category distribution highly similar to the full dataset, preserving narrative and visual diversity.The comparison evaluates whether the reduced subset remains representative of the complete benchmark.
  • Quantitative comparisons show minimal performance differences between Lite and full settings across style consistency, character consistency, generative quality, diversity, and prompt alignment.Only marginal discrepancies occur in some VLM-based scene-level consistency and Character Interaction metrics.
  • ViStoryBench-Lite enables more efficient and cost-effective evaluation for user studies and commercial-platform assessments without compromising result reliability.This is especially useful when large-scale human involvement or commercial API access is required.

E.2. Full Evaluation on ViStoryBench-Lite

ViStoryBench-Lite evaluates story visualization through standardized shot scripts that encode plot, settings, camera design, characters, and static actions. The evaluation compares methods using consistent prompt and composition requirements across multi-shot stories.

  • Evaluation Setup: The benchmark adapts each evaluated method to accept reference character images and lengthy shot-script prompts while standardizing feasible output conditions.The supplied evaluation materials include complete Lite-set results for comprehensive comparison.
  • Shot Script Design: Shot scripts represent each story as ordered scenes containing plot correspondence, setting, camera design, onstage characters, and static shot descriptions.Settings exclude characters, while static descriptions specify character positions, expressions, actions, and scene elements.
  • Shot Composition: Adjacent shots should preserve compositional coherence while varying framing, transitions, angles, and camera types to match narrative emotion and pacing.The guidelines recommend gradual tightening or loosening, avoiding repetitive compositions and abrupt shifts between wide and extreme close-up views.
  • Shot Script Design: Static shot descriptions correspond sequentially to story segments, ensuring that each shot captures fixed character and object states rather than introducing unsupported scene content.The instructions require scene locations and environmental elements to remain consistent with the original story.

A Sample output

The sample output demonstrates how a story segment is converted into a structured visual shot with setting, camera, character, and action details. The accompanying results compare methods across the benchmark’s multiple dimensions.

  • Sample Shot Output: A sample shot places Fern at a bell tower during a magical ritual, specifying twilight setting details, a medium eye-level view, and her precise pose and action.The output includes environmental elements such as arches, a glowing arrow, fallen leaves, pigeons, and soft golden light.
  • Character Descriptions: The character descriptions provide visual identities for Fern, Frieren, and Himmel, including hair, eyes, clothing, accessories, and carried objects.These descriptions supply the character-reference information used in the shot scripts.
  • Performance Analysis: Across ViStoryBench-Lite, GPT-4o [36] and Gemini-2.5 [9] show superior prompt alignment, while Sora2 achieves the best balance across character and style consistency metrics.The comparison covers cross- and self-consistency for both characters and styles.
  • Performance Analysis: AnimDirector and MovieAgent (SD-3) perform strongly across dimensions, especially aesthetics and diversity, whereas SEED-Story [98] has strong self-style consistency but weaker other metrics.The findings indicate a trade-off between visual fidelity and meaningful variation.
  • Performance Analysis: No single model excels uniformly across all criteria, and the quantitative findings agree with qualitative observations supporting the evaluation metrics’ reliability.Commercial systems such as Doubao also show strong prompt alignment, generation quality, character fidelity, and stylistic coherence.

F. Method Evaluation Detail on ViStoryBench

The detailed evaluation standardizes method inputs and documents adaptations for reference images, prompts, and model-specific constraints. It also uses baselines and multidimensional tables to compare story-visualization systems fairly.

  • Evaluation Protocol: Evaluated methods receive standardized reference-image and shot-script inputs, with 16:9 output where feasible and fixed random seeds for reproducibility.Adaptations also address lengthy prompts and continuous story-image generation.
  • Baselines: The Copy-Paste Baseline directly stitches onstage character images into each shot to test whether CIDS, CSD, copy-paste rate, and matched-character metrics behave correctly.Its outputs are intentionally simple and serve as a metric-validation reference.
  • Method Adaptations: StoryDiffusion [111] is adapted for ViStoryBench by handling its 77-token limit with grouped encoder sd embed and restricting each shot to the highest-priority character when multiple references are unsupported.Both image-reference and text-only results are reported separately.
  • Method Adaptations: Story-Adapter exploits iterative image referencing, while UNO [90] uses its many2many configuration with shot prompts and onstage-character images.Story-Adapter results are compared across text-only and image-reference modes and multiple iteration scales.
  • Method Adaptations: OmniGen2 and related consistency-oriented methods generate shots from textual scene and action descriptions plus character references, while omitting previous-shot images when tests showed degraded performance.CharacterConsist reuses masks from the first generated frame to preserve identity and background consistency.

G. Details of User Study

The user study evaluates character consistency, environment consistency, and aesthetics on ViStoryBench-Lite using structured annotations and aggregated scores. Human results and automated analyses support the reliability of the benchmark’s evaluation protocol.

  • Evaluation Dimensions: Human evaluation measures character identification consistency, environment consistency, and subjective aesthetics using 0–4 Likert scores with explicit qualitative criteria.Character consistency ranges from nearly absent identity continuity at 0 to consistent identification across most images at 4.
  • Annotation Process: Twenty trained annotators evaluate balanced subsets of model-story pairs through a web interface displaying image sequences, prompts, and scoring tables.Assignments limit fatigue while maintaining coverage, and scores are aggregated by model and story with means and standard deviations.
  • Automated Evaluation: GPT-4.1 scores prompt alignment across scene correctness, camera composition, and character actions, producing an averaged Alignment Score.AnimDirector leads Scene Score at 3.61 and Character Interaction at 3.24, while MovieAgent (SD3) leads Individual Action at 2.50.
  • Metric Validation: Automated evaluation correlates better with semantic and stylistic fidelity than CLIP-based similarity or VinaBench [17] scoring in the case study.This analysis informed the benchmark’s evaluation prompts and scoring rubric.
  • Metric Stability: Stability analysis finds very low variance across automated evaluation runs, with standard deviations an order of magnitude smaller than observed model-performance gaps.Reported examples include Scene Score 2.82 ± 0.03 and Character Interaction 2.57 ± 0.04.

H.7. Correlation Analysis of Qwen-base and GPT-based Evaluation

The analysis compares GPT-4.1 and Qwen3-VL-8B-Instruct on Prompt Alignment scores, finding strong agreement in both overall ratings and sub-item evaluations.

  • Correlation Analysis: The analysis uses paired Prompt Alignment average scores from GPT-based and Qwen-based evaluations, each scored on a 0–4 scale.The Prompt Alignment average aggregates Scene, Shot, character interaction, and individual action sub-items.
  • Correlation Analysis: Correlations for the Scene, Shot, character interaction, and individual action sub-items remain moderate to strong, with Pearson averages of approximately 0.85–0.90 and Spearman averages of 0.82–0.88.These sub-item results extend the agreement beyond the aggregate Prompt Alignment score.
  • Correlation Analysis: GPT-4.1 and Qwen3-VL therefore provide similar relative evaluations of different story visualization methods on Prompt Alignment.The conclusion follows from the strong overall and sub-item correlations reported in the analysis.

I.2. Impact of Reference Image Selection on Cross-CIDS Metric

This section examines how the choice of reference images affects Cross-CIDS and describes related style and copy-paste evaluation procedures.

  • Reference Selection: Table S8 compares Cross-CIDS computed with dataset references against scores computed with generated references on ViStoryBench-Lite.The two reference settings isolate the effect of reference-image selection in methods that synthesize their own references.
  • Reference Selection: Cross-CIDS can change substantially when methods use synthesized reference characters instead of the original ViStoryBench references.Synthesized stylized references may differ substantially from the original subjects, affecting the metric; main tables use dataset references and supplementary results use generated references.
  • Style Similarity: Style Similarity measures both within-sequence self-consistency and generated-to-reference cross-similarity using style embeddings and pairwise cosine similarity.CLIP features are passed through CSD layers to separate style from content before comparison.
  • Copy-Paste Detection: The Copy-Paste Rate uses a temperature-scaled Softmax over cosine similarities to test whether a generated image is disproportionately close to one target reference rather than other references.With τ = 0.01, scores near 1 indicate stronger evidence of reference-specific copy-paste behavior.
  • Copy-Paste Detection: A high Copy-Paste Rate indicates strong similarity to the specific target reference, whereas a lower score suggests similarity distributed across references or generalization beyond that appearance.The metric is averaged across generated samples.

M. Benchmark Evaluation Efficiency

The benchmark combines metrics with widely varying computational costs, from efficient per-image scoring to substantially slower language-model-based Prompt Alignment evaluation.

  • Experimental Setup: Experiments use high-performance GPU accelerators such as NVIDIA H800 GPUs with 80GB memory under consistent measurement conditions.The setup is intended to support reproducible runtime comparisons.
  • Computational Cost: Aesthetic scoring averages 0.026 seconds per image, while character similarity requires 0.450 seconds per image pair.Pairwise character similarity is more demanding because it uses deep feature extractors.
  • Computational Cost: Prompt Alignment is the most computationally intensive metric, requiring over 25 seconds per image because it relies on large language model inference.Its latency is attributed primarily to LLM inference.
  • Computational Cost: The Inception Score takes 8.057 seconds because it is computed over the entire dataset, but this computation is performed only once per dataset.Its dataset-level scope differs from per-image and pairwise metrics.
Loading 2505.24862v5…