PaperlayerView exact revision on arXiv

Source-linked AI summary

VinaBench: Benchmark for Faithful and Consistent Visual Narratives

Silin Gao, Sheryl Mathew, Li Mi, Sepideh Mamooler, Mengjie Zhao, Hiromi Wakaki, Yuki Mitsufuji, Syrielle Montariol, Antoine Bosselut

arXiv:2503.20871v3cs.CVcs.AIcs.CL

TL;DR

Visual narrative generation lacks explicit knowledge constraints for faithful text-to-image translation and consistency across scenes. VinaBench annotates commonsense and discourse constraints, then uses them for new alignment and consistency metrics. Across tested models, learning with these constraints improves narrative faithfulness and cohesion, but generated narratives still contain unfaithful or inconsistent content and remain below human-crafted references.

  • Problem

    Visual narrative generation must address narrative alignment and visual consistency because textual narratives leave visual manifestations under-specified and discourse elements persist across images.

    Source: §1 · p. 2

  • Method

    VinaBench augments approximately 25K visual-textual narrative pairs with commonsense links, discourse features, and constraint-grounded evaluation metrics.

    Source: §1 · p. 2

  • Results

    Across representative models, learning with VinaBench’s commonsense and discourse constraints consistently improves visual narrative consistency and alignment with the input textual narrative.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    VinaBench provides a foundation for learning and evaluating faithful, self-consistent visual narratives while highlighting substantial room for improvement toward human visual storytelling.

    Source: §7 · p. 9

    Generated narratives still contain unfaithful or inconsistent contents even with narrative-constraint augmentation.

    Source: Model Setting FID CLIP-I CLIP-T · p. 16

Abstract

from arXiv · show

Visual narrative generation transforms textual narratives into sequences of images illustrating the content of the text. However, generating visual narratives that are faithful to the input text and self-consistent across generated images remains an open challenge, due to the lack of knowledge constraints used for planning the stories. In this work, we propose a new benchmark, VinaBench, to address this challenge. Our benchmark annotates the underlying commonsense and discourse constraints in visual narrative samples, offering systematic scaffolds for learning the implicit strategies of visual storytelling. Based on the incorporated narrative constraints, we further propose novel metrics to closely evaluate the consistency of generated narrative images and the alignment of generations with the input textual narrative. Our results across three generative vision models demonstrate that learning with VinaBench's knowledge constraints effectively improves the faithfulness and cohesion of generated visual narratives.

1. Introduction

Visual narrative generation must bridge implicit commonsense meanings and maintain discourse consistency across image sequences. VinaBench addresses these challenges with annotated constraints, new evaluation metrics, and benchmark experiments showing improved alignment and consistency.

1. Introduction

  • Text-to-image narrative generation must infer unstated visual manifestations and preserve characters, locations, and other discourse elements across scenes.The paper identifies narrative alignment and visual consistency as the two fundamental challenges.Source: §1 · p. 2
  • Prior methods typically map text directly to visual narratives without explicitly modeling commonsense or discourse constraints, causing unfaithful content and inconsistent image sequences.Reported inconsistencies include character appearance, background location, and time period.Source: §1 · p. 2
  • VinaBench contains approximately 25K visual-textual narrative pairs augmented with commonsense links and global and scene-specific discourse features.The annotations connect visual entities to textual narrative entities and track characters, appearance style, time of day, and location across scenes.Source: §1 · p. 2
  • VinaBench introduces reference-free or candidate-ranking metrics that assess narrative-element consistency and alignment while reducing bias from a single reference image.These metrics target the fact that multiple visual manifestations can faithfully express the same narrative.Source: §1 · p. 2
  • Across representative visual narrative generation models, learning the annotated constraints consistently improves visual narrative consistency and textual alignment, although substantial gaps remain versus human references.The results motivate further work on more robust visual narrative generators.Source: §1 · p. 2

2. Background and Related Work

Prior visual narrative systems often underrepresent implicit commonsense, diverse narrative structure, and long-range consistency. VinaBench instead annotates these constraints in varied storytelling data to support more faithful and coherent generation.

2. Background and Related Work

  • Existing visual narrative methods often learn direct text-to-visual mappings while overlooking deeper commonsense alignment between textual and visual expressions.Earlier commonsense approaches are described as limited to simple physical knowledge or token-level semantic alignment.Source: §2 · pp. 2–3
  • Recent storytelling datasets provide more diverse visual narrative resources than photo collections or single-cartoon video shots.VinaBench builds on Visual Writing Prompts, Storyboard20K, and StorySalon.Source: §2 · pp. 2–3
  • Visual narratives encode discourse structure through image categories and segments whose dynamics track persistence and change in characters, time, and location.These discourse dynamics motivate explicit modeling of consistency across narrative images.Source: §2 · p. 3
  • Unlike short-time video consistency, the paper focuses on maintaining visual elements over the longer span of narrative discourse.This distinction concerns consistency throughout the narrative rather than only local spatial-temporal coherence.Source: §2 · p. 3

3. VinaBench Data Construction

VinaBench augments diverse visual-textual narratives with commonsense links and discourse features constructed using language and vision models. An expert study reports high acceptance and low disagreement for the resulting annotations.

3. VinaBench Data Construction

  • VinaBench samples narrative pairs from three visual storytelling datasets and augments them with constraints for visual-textual alignment and narrative-element consistency.The construction pipeline uses hybrid VLMs and LLMs to annotate commonsense and discourse information.Source: §3 · p. 3

3.1. Commonsense Constraints

  • Commonsense constraints link visual entities in dense image captions to associated noun or verb phrases in the textual narrative.Dense captions use the narrative description as context, entities are extracted, and unmatched visual entities receive a “no link” output.Source: §3.1 · pp. 3–4

3.2. Discourse Constraints

  • Discourse constraints represent global character profiles and appearance style alongside scene-specific characters, time of day, and location.Character attributes and repeated locations are expected to remain consistent, while time and location may change as the narrative progresses.Source: §3.2 · p. 4
  • Character identities are assigned by matching image content and narrative context to the global profile, while other scene labels and narrative-wide style are annotated with VLMs.The character procedure uses a two-step prompt and prior narrative context to resolve references such as “Her.”Source: §3.2 · p. 4

3.3. Expert Study

  • Twelve experts manually assessed 100 narrative samples, checking global and scene-specific discourse labels, image captions, and commonsense links.Each sample was checked by two experts, with acceptance and disagreement rates reported.Source: §3.3 · pp. 4–5, Table 1 · p. 5
  • The expert study found high acceptance rates and fairly low disagreement across constraint types, supporting the reliability of the model-assisted construction scheme.The authors present this reliability as reducing the labor required from human annotators.Source: §3.3 · p. 5

4. VinaBench Evaluation

VinaBench evaluates narrative alignment and consistency with ranking-based and constraint-grounded VQA metrics rather than relying only on single-reference comparisons. Its metrics cover fine-grained scene content and consistency of style, characters, and locations.

4. VinaBench Evaluation

  • Prior full-reference and image-wise similarity metrics overlook open-ended visual expression and do not evaluate consistency between narrative images.The paper motivates metrics that assess both alignment with textual constraints and inter-image consistency.Source: §4 · p. 5

4.1. Alignment Ranking

  • Alignment ranking samples the top 100 test-set images for each narrative and reports the generated image’s mean reciprocal rank using CLIP-T or VQA-based scores.The resulting metrics are named CLIP-T-MRR and VQA-MRR.Source: §4.1 · p. 5

4.2. Fine-Grained Alignment

  • Fine-grained alignment metrics assess non-character entities, character number and attributes, time of day, and location against VinaBench constraints.VLM judgments determine whether each generated image satisfies the corresponding scene-level condition.Source: §4.2 · p. 5

4.3. Consistency

  • Consistency metrics evaluate shared appearance style across a narrative, identity consistency for recurring characters, and location consistency across scenes.Each criterion is applied only where the corresponding global or scene-specific discourse feature supplies a relevant constraint.Source: §4.3 · p. 6
  • The metrics average zero-shot VLM Yes probabilities and are run with MiniCPM-V-2.6 and LLaVA-OneVision-72B, whose results are reported as aligned.Using two VLMs is intended to reduce dependence on a single model’s preferences.Source: §4.3 · p. 6

5. Experimental Methods

The experiments compare unconstrained generation with LLM-generated and gold narrative constraints across three vision models and multiple evaluation settings. Models generate images from textual narratives augmented by serialized captions, commonsense links, and discourse features.

Experimental setup

  • LLM Constraints concatenate each textual narrative with generated scene captions, commonsense links, and serialized discourse features before image generation.The serialized representation includes character, time-of-day, and location information.Source: §5 · p. 6
  • Three vision models—ARLDM, StoryGen, and MM-Interleaved—are evaluated under No Constraint, LLM Constraints, and Gold Constraints settings.LLM Constraints are generated by Llama3.1-70B-Instruct with LoRA, while Gold Constraints use annotated constraints at inference.Source: §5 · p. 6
  • Gold Constraints replace LLM-generated constraints with annotated constraints during inference, providing an oracle comparison for constraint augmentation.Both generated and gold constraints use the same preprocessing: commonsense links are merged into captions and discourse features are serialized.Source: §5 · p. 6

Evaluation

  • Evaluation combines VinaBench’s Alignment and Consistency metrics with FID, CLIP-I, and CLIP-T against reference images or narrative text.The proposed metrics assess fine-grained alignment and consistency, while the conventional metrics measure image and text similarity.Source: §5 · p. 6

6. Experimental Results

Across VWP, Storyboard20K, and StorySalon narratives, adding narrative constraints improves faithfulness and consistency, while the proposed VQA-based metrics better track human judgments than CLIP-based measures. Ablations and generalization analyses indicate that aligned commonsense and discourse constraints provide complementary benefits beyond simply adding text.

6.1. VWP and Storyboard20K Narratives

  • LLM Constraints significantly improve alignment and consistency over No Constraint across the three evaluated vision models on VWP narratives.The result supports middle-stage narrative planning as an intermediate scaffold for visual generation.Source: §6.1 · p. 6, Table 2 · p. 7
  • Gold references score below the maximum CLIP-T-MRR of 1.0, showing that textual narratives can have multiple feasible visual manifestations and limiting full-reference evaluation.Generated visual narratives still lag behind gold references across metrics, leaving substantial room for improvement.Source: §6.1 · pp. 6–7
  • Alignment and Consistency correlate better with human evaluations than CLIP-I and CLIP-T, supporting the reliability of the proposed VQA-based metrics.The correlation study uses 100 VWP testing samples and compares four methods plus gold references.Source: Figure 3 · p. 7, §6.1 · p. 8
  • Constraint-augmented models show smaller performance drops than No Constraint when generalizing from VWP training data to zero-shot Storyboard20K narratives.This pattern indicates more robust visual narrative capabilities on out-of-distribution samples.Source: §6.1 · p. 8, Table 4 · p. 8
  • Removing commonsense links or any discourse-feature subset degrades all metrics, with removing all discourse features generally harming performance more than removing commonsense links.Discourse constraints appear especially beneficial for location and non-character content.Source: §6.1 · pp. 8–9
  • Randomly shuffled constraints perform worse than No Constraint, indicating that gains depend on alignment between constraints and their narratives rather than added input tokens.The Random ablation groups samples by length before shuffling constraints.Source: §6.1 · p. 9
  • Constraint-text similarity and generated-image text similarity achieve approximately 0.4 Pearson correlation in both Gold and LLM settings.The paired result links better alignment between narratives and intermediate constraints with better alignment between narratives and generated images.Source: §6.1 · p. 9

6.2. StorySalon Narratives

  • On StorySalon narratives, incorporating constraints again improves faithfulness and self-consistency across animation-style visual narratives.The consistent outcome across datasets and image styles supports the broad applicability of the knowledge-augmentation framework.Source: §6.2 · p. 9, Table 5 · p. 9

7. Conclusion

VinaBench targets faithfulness and self-consistency in visual narrative generation by providing discourse and commonsense constraints and metrics for evaluating them. The results show substantial remaining room for generated narratives to approach human visual storytelling.

Conclusion

  • VinaBench provides a benchmark with discourse and commonsense constraints for learning faithful visual narratives.The benchmark is designed to address faithfulness and self-consistency challenges.Source: §7 · p. 9
  • VinaBench introduces metrics that assess generated narratives’ consistency and alignment with the input textual narrative.These metrics address shortcomings in existing visual narrative evaluation.Source: §7 · p. 9
  • Generated visual narratives remain considerably below the level of human visual storytelling, motivating more robust future generators.The conclusion identifies this gap as an area for future study.Source: §7 · p. 9

S1. VinaBench Data Construction Details

VinaBench constructs a multi-source visual-textual narrative benchmark and annotates its samples with commonsense and discourse constraints. The resulting dataset includes longer sequences and fine-grained annotations designed to support faithful and consistent visual narrative generation.

Data sources

  • VinaBench combines visual-textual narratives from VWP, Storyboard20K, and StorySalon, covering movie-based and animation-style sources.The sources include MovieNet, LSMDC, YouTube videos, and E-books, with StorySalon texts translated into English when eligible.Source: S1. VinaBench Data Construction Details · p. 12
  • StorySalon filtering retains approximately 2K samples after translation and quality screening of narrative texts.The filtering excludes samples that are not fully translated or contain more than 10% uninformative scene texts.Source: S1. VinaBench Data Construction Details · p. 12, Table S1 · p. 12

Constraint annotation

Dataset filtering and statistics

Dataset novelty

S2. VinaBench Evaluation Details

VinaBench evaluates fine-grained alignment and consistency through VQA-based scores aggregated at the entity, scene, or narrative level.

  • VinaBench implements all proposed VQA-based fine-grained alignment and consistency metrics using zero-shot prompting.VQA scores are averaged across labeled entities, scenes, or narrative samples according to the metric.Source: S2. VinaBench Evaluation Details · p. 13
  • Non-character alignment scores average across each non-character entity labeled in gold commonsense links.Source: S2. VinaBench Evaluation Details · p. 13
  • Character and location consistency scores average across gold entities shared by multiple scenes, while style consistency averages across each narrative.Source: S2. VinaBench Evaluation Details · p. 13

S3. Experimental Setup Details

The experiments train and evaluate narrative models with serialized commonsense and discourse constraints across multiple model architectures and benchmark metrics.

Experimental Setup Details

  • VinaBench serializes linked captions, scene features, and global character attributes into autoregressively generated narrative constraints.Linked textual phrases are inserted after caption entities, while scene features encode characters, time, location, and attributes.Source: S3. Experimental Setup Details · p. 13
  • The setup evaluates ARLDM, StoryGen, and MM-Interleaved, which use different autoregressive diffusion or vision-language architectures.Source: S3. Experimental Setup Details · pp. 13–14
  • Tables S5–S7 report ranking-based and fine-grained alignment, consistency, and full-reference evaluation results on VWP narratives.The full-reference metrics use FID, for which lower values are better.Source: Table S5 · p. 14, Table S6 · p. 14, Table S7 · p. 14

S4. Full Experimental Results

Across benchmark evaluations, learning with VinaBench constraints improves narrative alignment and consistency, while ranking metrics and correlation analyses characterize the task as open-ended and constraint-sensitive.

Full Experimental Results

  • VinaBench constraints significantly improve visual narrative consistency and alignment across the reported model generations.The conclusion is consistent across the full evaluation tables.Source: S4. Full Experimental Results · p. 14, Table S8 · p. 15, Table S9 · p. 15
  • CLIP-T-MRR and VQA-MRR scores remain far below 1.0 for generations and gold references, supporting the open-ended nature of visual narrative generation.Source: S4. Full Experimental Results · p. 14
  • Nearly zero VinaBench metric scores for Anole and Lumina-mGPT under zero-shot or few-shot settings indicate that supervised learning or fine-tuning is necessary for these models.MM-Interleaved also fails without fine-tuning.Source: S4. Full Experimental Results · p. 15
  • Approximately 0.4 Pearson correlation links constraint–text similarity with generated-image–text similarity for both gold and LLM-generated constraints.The analysis uses MM-Interleaved generations on VWP narratives.Source: S4. Full Experimental Results · p. 15, Figure S1 · p. 15
  • The study evaluates MM-Interleaved with varied narrative-constraint settings, including 4-shot prompting Llama3.1-70B-Instruct and fine-tuning Llama3.1-8B-Instruct, Gemma-7B, and Qwen2-7B, compared with an adopted fine-tuning setting.Source: S4. Full Experimental Results · p. 15

Model Setting FID CLIP-I CLIP-T

The reported settings combine model and metric comparisons with constraint-generation analyses and qualitative examples, while generated narratives still exhibit faithfulness and consistency failures.

Model Setting FID CLIP-I CLIP-T

  • The adopted Llama3.1-70B-Instruct fine-tuning setting with LoRA best augments MM-Interleaved on VWP narratives.Table S14 compares fine-tuned and few-shot LLM constraint-generation settings using alignment and consistency averages.Source: Model Setting FID CLIP-I CLIP-T · p. 16, Table S14 · p. 16
  • The deployed baseline generations remain unfaithful or inconsistent even with narrative constraints, leaving more robust generation methods for future work.Source: Model Setting FID CLIP-I CLIP-T · p. 16
  • Constraint-augmented generations improve alignment, style consistency, character consistency, or background consistency in qualitative examples, but residual outfit and facial inconsistencies remain.The examples include improved lab-setting alignment, map depiction, and character or location consistency.Source: Figure S2 · p. 17
  • The benchmark includes visual-entity extraction from dense captions, with a prompt directing output of people-related nouns or noun phrases, illustrated by examples identifying a woman, teacher, students, and family.Source: Model Setting FID CLIP-I CLIP-T · p. 19

Related papers

Loading 2503.20871v3…