PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generation

Yiping Wang, Xuehai He, Kuan Wang, Luyao Ma, Jianwei Yang, Shuohang Wang, Simon Shaolei Du, Yelong Shen

arXiv:2412.16211v1cs.CVcs.CLcs.GR

TL;DR

Fine-grained video-quality benchmarks do not assess whether generated videos present consecutive events in prompted stories, a requirement for long-form generation. StoryEval evaluates event completion in short, multi-event stories with VLM verifiers and unanimous voting; none of the evaluated models averages 50% story completion.

  • Problem

    Existing benchmarks emphasize fine-grained video quality, while long-video generation also requires measuring whether prompted consecutive events are presented.

    Source: §1 · pp. 2–3

  • Method

    StoryEval evaluates 423 short stories with 2–4 consecutive events across seven classes using VLM event verification and unanimous voting.

    Source: §1 · pp. 3–4, §2.2.3 · p. 6

  • Results

    No evaluated model averages 50% story completion on StoryEval, and no model exceeds 20% completion on the Hard subset.

    Source: §3.1 · p. 7

  • Takeaways & Limitations

    StoryEval provides a complementary metric for assessing story presentation alongside fine-grained video-quality measures.

    Source: §1 · pp. 3–4

    The prompt suite targets stories that can fit within videos of at most 10 seconds, mostly involving one or two subjects.

    Source: §2.1.1 · p. 4

Abstract

from arXiv · show

The current state-of-the-art video generative models can produce commercial-grade videos with highly realistic details. However, they still struggle to coherently present multiple sequential events in the stories specified by the prompts, which is foreseeable an essential capability for future long video generation scenarios. For example, top T2V generative models still fail to generate a video of the short simple story 'how to put an elephant into a refrigerator.' While existing detail-oriented benchmarks primarily focus on fine-grained metrics like aesthetic quality and spatial-temporal consistency, they fall short of evaluating models' abilities to handle event-level story presentation. To address this gap, we introduce StoryEval, a story-oriented benchmark specifically designed to assess text-to-video (T2V) models' story-completion capabilities. StoryEval features 423 prompts spanning 7 classes, each representing short stories composed of 2-4 consecutive events. We employ advanced vision-language models, such as GPT-4V and LLaVA-OV-Chat-72B, to verify the completion of each event in the generated videos, applying a unanimous voting method to enhance reliability. Our methods ensure high alignment with human evaluations, and the evaluation of 11 models reveals its challenge, with none exceeding an average story-completion rate of 50%. StoryEval provides a new benchmark for advancing T2V models and highlights the challenges and opportunities in developing next-generation solutions for coherent story-driven video generation.

1. Introduction

Current T2V benchmarks emphasize fine-grained video quality, while longer, story-driven generation needs evaluation of whether consecutive prompted events are presented. StoryEval introduces such a benchmark and finds substantial event-completion failures among current models.

1. Introduction

  • Even the best evaluated model averages below 50% story completion, and top systems miss required actions in the refrigerator example.Figure 2 reports the average completion rate; the example illustrates missed actions despite depicting the refrigerator or elephant.Source: Figure 2 · p. 2, §1 · p. 3
  • EvalCrafter [25] and VBench [15] emphasize fine-grained spatial, temporal, and semantic quality metrics, which the paper says are beginning to saturate.The paper presents these measures as fundamental quality indicators, particularly for short footage.Source: §1 · p. 2
  • Story completion rate—the ratio of correctly depicted events—captures whether a video presents the consecutive events required by a longer story prompt.The paper frames this as a coarse-grained but essential metric for evaluating long-video generation.Source: §1 · p. 3
  • StoryEval uses 423 filtered stories with 2–4 events across seven categories, constructed through video captioning, brainstorming, and GPT-4o-guided generation.The prompt suite filters examples with weak human alignment or frequent GPT-4o refusals.Source: §1 · p. 3
  • GPT-4o and LLaVA-OV-Chat-72B verifier responses are combined through unanimous voting, and the paper reports high alignment with human evaluations.The benchmark is proposed as a complement to previous detail-oriented metrics.Source: §1 · p. 4

2. StoryEval

This section introduces StoryEval; related work on T2V models and prior benchmarks appears in the Appendix.

2. StoryEval

  • The section introduces StoryEval, while related work on T2V models and prior benchmarks is deferred to the Appendix.Source: §2 · p. 4

2.1. Prompt Suite

StoryEval’s prompt suite contains 423 short stories designed around a small number of consecutive events and filtered for evaluation quality. Prompts are drawn from real-world retrieval, human brainstorming, and guided GPT-4o generation, then organized into seven categories.

2.1.1. How to Create the Story-Oriented Prompt Suite?

  • Stories are designed for 10-second-level videos, with 2–4 consecutive events, few objects, and usually one or two subjects; longer movie-like stories are excluded.The general conditions target stories that can occur in a relatively short time span.Source: §2.1.1 · p. 4

Approach 1: Retrieving and captioning the real-world

  • About 70 prompts come from manually captioning suitable short videos retrieved from YouTube, Pexels, and Panda-70M.The authors note that these retrieval-based videos achieve 100% completion in human verification.Source: Approach 1: Retrieving and captioning… · p. 4
  • Human brainstorming adds about 40 prompts, including logically plausible but improbable stories and scenarios that are impossible in reality but imaginable in animation.The elephant-in-a-refrigerator story is one example of the latter kind.Source: Approach 1: Retrieving and captioning… · p. 4
  • Guided GPT-4o generation uses earlier prompts and the general conditions to produce diverse real-world and creative candidates, bringing the raw total to about 500.Candidates are selected for topic and action diversity and fit to the general conditions.Source: Approach 1: Retrieving and captioning… · p. 4

2.1.2. Post-Processing Filtration

  • Prompts whose GPT-4o judgments align relatively weakly with human feedback are filtered, including stories with fast movements or subtle changes that are hard to judge.The suite also excludes prompts that frequently trigger GPT-4o refusals or video-generation failures across models.Source: §2.1.2 · p. 5
  • Filtering yields a final suite of 423 prompts, excluding facial-expression events, prompts with verifier failures on more than two models, and stories with weak human alignment.The authors also count some other model failures as zero completion rather than filtering them.Source: §2.1.2 · p. 5

2.1.3. Prompt Suite Composition

  • The seven categories distinguish participants (Human, Animal, Object), source or style (Retrieval, Creative), and difficulty (Easy, Hard).Easy and Hard are assigned using performance across the top eight models, with the highest- and lowest-performing 100 prompts in each group.Source: §2.1.3 · p. 5
  • Creative prompts are disproportionately represented among Hard examples: 44 of 110 Creative prompts are Hard (40%), versus 100 of 423 prompts overall (21%).The UpSet plot also shows 14 prompts at the Hard–Creative–Object intersection.Source: Figure 4 · p. 5, §2.1.3 · p. 6

2.2. Evaluation Process

StoryEval evaluates generated videos with GPT-4o and LLaVA-OV-Chat-72B using sampled key frames. Its two-step event judging and repeated-response voting are designed to make completion judgments more stable and aligned with human feedback.

2.2.1. Evaluation model

  • GPT-4o supplies the main results, with LLaVA-OV-Chat-72B as an additional verifier; both assess sampled key frames from generated videos.The paper reports that the two-step querying process is more stable than one-step querying, especially for LLaVA-OV-Chat-72B.Source: §2.2.1 · p. 6, §2.2.2 · p. 6

2.2.2. Two-Step Quering

  • The two-step process first elicits detailed key-frame descriptions, then asks whether each listed event is complete, using the prompt and event list to score it.The event list makes the number of events explicit; the verifier also checks that subjects or objects remain consistent across events when the prompt implies they should.Source: §2.2.2 · p. 6

2.2.3. Unanimous Voting

  • Repeated verifier responses are aggregated by k-over-N voting: an event is complete when at least k of N judgments mark it complete.The evaluation uses N = 3 for GPT-4o and N = 2 for LLaVA-OV-Chat-72B.Source: §2.2.3 · p. 6
  • N-over-N voting, called unanimous voting, gives the best alignment with human feedback among the tested voting thresholds.The authors motivate repeated judgments by verifier overestimation on blurry or unclear generated videos.Source: §2.2.3 · p. 6

2.3. Human Annotation

Human annotations are used to validate GPT-4o’s evaluations and to filter prompts with weak human alignment. The annotation sample is drawn from anonymized video-prompt pairs, while annotators judge event completion directly.

2.3. Human Annotation

  • Human annotations are used to assess whether StoryEval’s GPT-4o evaluations align with human judgments across classes and the full prompt set.The annotations also inform post-processing filtration of prompts.Source: §2.3 · p. 6

2.3.1. Data Preparation

  • For each prompt, two of the 11 models are randomly sampled, yielding about 900 video-prompt examples, with each model assigned about 80 prompts.Model identities are anonymized and pairs shuffled; because models receive different prompts, their human-annotation performance may differ from benchmark performance on the shared suite.Source: §2.3.1 · p. 6

2.3.2. Labeling Process

  • None of the 11 models exceeds 50% average completion; for Retrieval prompts, the human upper bound is 100% when using the retrieved videos as a baseline.The table reports GPT-4o-verifier completion rates, with higher values better.Source: Table 1 · p. 7
  • Annotators directly score whether each story event is complete, using one-step querying and precautions similar to the VLM verifier, without an instruction to score strictly.They complete about 20 pre-questions and receive follow-up communication before the large-scale annotation.Source: §2.3.2 · pp. 6–7

3. Experiment

Across 11 video models, StoryEval finds weak completion of consecutive story events, while its scores align with human annotations and remain similarly ranked with a second VLM verifier. Comparisons with VBench and a voting ablation examine how StoryEval complements detail-oriented metrics and how verifier responses are aggregated.

3. Experiment

  • StoryEval evaluates three closed-source and eight open-source models using Kendall’s τ and Spearman’s ρ for human-alignment rankings.The models include Kling-1.5, Hailuo, and Pika-1.5 among the closed-source systems.Source: §3 · p. 7

3.1. Results

  • No evaluated model reaches 50% average completion, and none exceeds 20% on the Hard subset; Vchitect-2.0 (21.7%) also exceeds Pika-1.5 (19.4%).Even on Retrieval, where retrieved videos provide a 100% completion baseline, no evaluated model matches it.Source: §3.1 · p. 7

3.1.1. Comparison StoryEval with VBench

  • StoryEval and VBench temporal consistency can rank models differently: Vchitect-2.0 does well on StoryEval but poorly on temporal consistency, while Open-Sora-Plan shows the reverse.The authors suggest there may be a trade-off between temporal consistency and story completion.Source: Table 2 · p. 7, §3.1.1 · p. 8

3.1.2. Human Alignment

  • GPT-4o completion rates are highly correlated with human annotations, supporting GPT-4o’s use as a VLM verifier.Figure 6 compares average completion rates annotated by humans on the x-axis with GPT-4o rates on the y-axis, across seven classes and all data.Source: §3.1.2 · p. 8, Figure 6 · p. 8

3.1.3. Using LLaVA-OV-Chat-72B

  • LLaVA-OV-Chat-72B produces ranking correlations close to GPT-4o’s, but assigns higher average completion rates.Table 3 reports comparisons between the completion rates from the two verifiers.Source: §3.1.3 · p. 8, Table 3 · p. 8

3.1.4. Ablation Study of Unanimous Voting

  • Unanimous voting for GPT-4o yields the best human-alignment result among the tested strategies.The ablation compares processing three independent GPT-4o scores per event using averaging or k-over-3 voting against human annotations.Source: §3.1.4 · p. 8, Table 4 · p. 9

4. Conclusion

StoryEval evaluations show that current video generative models fall short of coherent story presentation across sequential events. Their performance is particularly weak on creative scenarios, and VLM-based scoring aligns with human annotations.

4. Conclusion

  • Current video generative models struggle to complete even half of prompted events, especially in creative scenarios.The conclusion describes a substantial gap between current models and the demands of coherent story presentation across sequential events.Source: §4 · p. 9
  • StoryEval’s automated VLM-based scoring is well-aligned with human annotations, supporting its use to assess these limitations.Source: §4 · p. 9
  • The paper identifies storytelling and event continuity as challenges for next-generation models intended for long-format, story-driven video content.Source: §4 · p. 9

5. Related Works

Related work describes rapid progress in diffusion and autoregressive video generation alongside persistent challenges in long-term continuity and narrative progression. Existing video benchmarks cover multiple quality dimensions, but the paper identifies coherent, story-driven long-video generation as an evaluation gap.

5.1. Text-to-Video Generative Models

  • Diffusion models dominate recent vision generation, while autoregressive video generation predicts tokens sequentially; long-term event continuity and logical narrative progression remain challenging.The paper notes contemporaneous work on alleviating these problems, but says there is no standard way to quantify story-presentation capacity.Source: §5.1 · p. 12

5.2. Video Generation Metrics and Benchmarks

  • Benchmarks such as VBench and EvalCrafter assess dynamic attributes, while specialized benchmarks address composition and physical realism; the paper says coherent story-driven long videos remain insufficiently addressed.Source: §5.2 · p. 12
  • StoryEval is presented as a holistic framework for assessing narrative completion and accurate depiction of sequential events in long-video generation.Source: §5.2 · p. 12

6. Supplementary of StoryEval Benchmark

The supplementary material describes prompt filtering and verifier selection, then details a two-step VLM evaluation process that scores each prompted event under strict completion and consistency criteria.

6.1. More Details about Prompt Suite

  • Retrieved prompts use real-world videos generally 5–20 seconds long, with longer examples retained when their events are simple enough for 10-second completion.The prompt suite also filters cases where GPT-4o rejects evaluation for security concerns; remaining cases can still produce zero completion rates and contribute to model non-response rates.Source: §6.1 · p. 12

6.2. More Details about Evaluation Process

  • The authors choose LLaVA-OV-Chat-72B over its 7B version because it performs better on noisier videos from open-source models.The two versions reportedly have similar results on high-quality videos.Source: §6.2 · pp. 12–13
  • The two-step query first asks a VLM to describe sampled key frames in temporal order, then provides the description, frames, prompt, and event list for scoring.The query illustration includes a bear near a waterfall as an example video.Source: §6.2 · p. 13

Please judge whether the event are completed very

7. Supplementary of Experiments

The supplement describes experimental settings, reports additional StoryEval results, and provides example videos, including some rejected by GPT-4o for security concerns.

7.1. Model details about experimental settings.

  • Open-source generation was extended toward 10 seconds when stable, with Pyramid-Flow limited to about 7 seconds; prompts averaged 83.4 characters and key-frame sampling depended on K.The supplement says longer videos or more frames do not necessarily ensure better StoryEval performance.Source: §7.1 · pp. 13–14, Table 5 · p. 14

7.2. Detailed Results

  • The detailed tables report 11-model GPT-4o results including non-response rates; LLaVA-OV-Chat-72B rankings mostly match GPT-4o, but VBench and StoryEval differ on Pyramid-Flow.The ranking difference supports StoryEval as a complementary metric to detail-oriented evaluation.Source: §7.2 · p. 14, Table 6 · p. 15, Table 8 · p. 15

7.3. Examples

  • GPT-4o flags some examples from three open-source models for potential self-harm, violence, or sexual content, though humans may disagree.The supplement also provides evaluation results and video demos.Source: §7.3 · p. 14, Figure 7 · p. 16
Loading 2412.16211v1…