PaperlayerView exact revision on arXiv

Source-linked AI summary

StoryBench: A Multifaceted Benchmark for Continuous Story Visualization

Emanuele Bugliarello, Hernan Moraldo, Ruben Villegas, Mohammad Babaeizadeh, Mohammad Taghi Saffar, Han Zhang, Dumitru Erhan, Vittorio Ferrari, Pieter-Jan Kindermans, Paul Voigtlaender

arXiv:2308.11606v2cs.CVcs.CL

TL;DR

Text-to-video story generation requires coherent, realistic videos that follow evolving prompts, while annotation-rich data is costly to create. STORYBENCH gathers human annotations and benchmarks three increasingly difficult generation tasks; the authors report benefits from story-like training data and discrepancies between human and automatic evaluation. The benchmark is limited to relatively short, mostly single-shot, user-generated real-world videos, and its annotation framework does not scale to long videos.

  • Problem

    Text-to-video models must produce visually high-quality, temporally consistent videos that realistically follow sequences of prompts, while high-quality story-like training annotations are expensive to collect.

    Source: §1 · p. 2, §4 · p. 6

  • Method

    STORYBENCH annotates three existing video datasets and evaluates action execution, story continuation, and story generation, alongside story-like training data and human and automatic evaluation.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Fine-tuning Phenaki for continuation improves consistency and action synthesis, and training on story-like data benefits video continuation, though human and automatic evaluations can disagree.

    Source: §1 · p. 3, §8 · p. 10

  • Takeaways & Limitations

    STORYBENCH provides a multi-task benchmark and evaluation resource for studying text-to-video generation from evolving prompts.

    Source: §1 · p. 2

    The videos are not very long, mostly single-shot and user-generated, and the annotation framework would not scale to long videos.

    Source: §7 · pp. 9–10

Abstract

from arXiv · show

Generating video stories from text prompts is a complex task. In addition to having high visual quality, videos need to realistically adhere to a sequence of text prompts whilst being consistent throughout the frames. Creating a benchmark for video generation requires data annotated over time, which contrasts with the single caption used often in video datasets. To fill this gap, we collect comprehensive human annotations on three existing datasets, and introduce StoryBench: a new, challenging multi-task benchmark to reliably evaluate forthcoming text-to-video models. Our benchmark includes three video generation tasks of increasing difficulty: action execution, where the next action must be generated starting from a conditioning video; story continuation, where a sequence of actions must be executed starting from a conditioning video; and story generation, where a video must be generated from only text prompts. We evaluate small yet strong text-to-video baselines, and show the benefits of training on story-like data algorithmically generated from existing video captions. Finally, we establish guidelines for human evaluation of video stories, and reaffirm the need of better automatic metrics for video generation. StoryBench aims at encouraging future research efforts in this exciting new area.

1 Introduction

Text-to-video generation must coordinate visual quality, action adherence, and temporal consistency despite expensive training and comparatively small video–text datasets. STORYBENCH introduces a human-annotated benchmark for evolving text prompts and evaluates task setups, story-like training data, and evaluation methods.

1 Introduction

  • Video generation remains difficult because models must maintain temporal coherence and reflect prompted actions, while training is computationally expensive and video–text datasets are much smaller than image–text datasets.The paper contrasts video–text corpora with image–text corpora as orders of magnitude smaller.Source: §1 · p. 2
  • STORYBENCH collects dense annotations for 6K videos from DiDeMo [17], Oops [18], and UVO [19], including action captions, timestamps, and segment labels.The 18K segments are tagged with 34 labels across six categories; the pipeline also transforms grounded captions into structured training stories.Source: §1 · p. 2
  • Its three tasks increase in difficulty from generating one action conditioned on video, to continuing a sequence, to generating a story from text without ground-truth visual conditioning.Story generation begins from a model-generated video based on a textual scene-background description.Source: §1 · p. 2
  • The study trains a 345M-parameter Phenaki model and evaluates zero-shot, dataset-specific fine-tuning, and multi-task fine-tuning setups.It also establishes human-evaluation guidelines and reports five automatic metrics, which the authors say do not align well with human judgment.Source: §1 · p. 2
  • Training text-to-video models on story-like data benefits video continuation, while human and automatic evaluations can disagree.The benchmark also provides evaluation guidelines and metrics for reproducible comparisons.Source: §1 · pp. 2–3

2 Related Work

Prior work advances text-to-video generation and story visualization, but STORYBENCH focuses on continuous videos from natural data rather than one key-frame per prompt or synthetic and cartoon stories.

2 Related Work

  • Text-to-video systems use diverse approaches, including autoregressive attention, diffusion, temporal modules added to image models, and cascaded video diffusion.The cited work includes GODIVA [30], NÜWA, NUWA-Infinity, NÜWA-XL [33], CogVideo [34], Make-a-Video [35], Video Latent Diffusion Models [36], and Imagen Video [37].Source: §2 · p. 3
  • Earlier story visualization generates one image per sentence, whereas STORYBENCH evaluates continuous videos on natural data and uses human-timed descriptions rather than automatically matched key-frames.Prior approaches include sequential conditional GANs [40], dual learning [43], structure and commonsense methods [44], StoryDALL-E [45], and a visual-memory diffusion framework.Source: §2 · p. 3

3 StoryBench

STORYBENCH evaluates continuous story visualization on real-world videos through multiple datasets, tasks, and evaluation setups. Its annotations and evaluation procedures are designed to expose model capabilities and failure modes.

3 StoryBench

  • STORYBENCH is a real-world benchmark for continuous videos from sequential text prompts, spanning three datasets, three difficulty levels, and three evaluation setups.It targets coherent storylines, visual realism, and control over generated content through text descriptions and video duration.Source: §3 · p. 3

3.1 Datasets

  • DiDeMo-CSV adds human-written action descriptions, timestamps, background descriptions, and counts of important actors to videos whose original temporal-localization queries may lack story context.Oops-CSV adapts VidLN captions into action sequences, which annotators refine and supplement with timestamps and context; UVO-CSV follows the same process.Source: §3.1 · p. 4
  • Segment annotations use 34 labels in six categories, with annotators recording whether each label appears in the video and/or its caption to support failure diagnosis.Categories include camera movement, entities, actions, interactions, and transitions.Source: §3.1 · pp. 4–5

3.2 Tasks

  • Action execution generates the next action from preceding ground-truth video; story continuation generates a prompt sequence from conditioning video; story generation instead starts from a model-generated background video.For the first action, action execution and story continuation use the first 0.5 seconds of ground-truth video as context.Source: §3.2 · p. 5

3.3 Human Evaluation

  • Human side-by-side ratings assess visual quality, text adherence, entity and background consistency, and action realism across paired outputs.Raters see the full story and conditioning video when applicable; three raters judge each pair across 100 randomly sampled stories.Source: §3.3 · p. 5

3.4 Automatic Metrics

  • Automatic evaluation includes image and video distribution distances, video–text matching, perceptual quality assessment, and frame similarity for conditioned tasks.FID compares image features, FVD compares video features, VTM measures feature similarity between video and text, PQA predicts subjective quality, and SIM compares generated frames with ground-truth matches.Source: §3.4 · p. 6
  • Automatic metrics are computed on generated videos, excluding conditioning footage; story-generation background frames are omitted and SIM is not reported for that task.The authors release evaluation code to support reproducible and comparable results.Source: §3.4 · p. 6

4 Training Data Challenge

Story-like training data is costly to annotate, and existing VidLN captions lack explicit action timestamps. The section describes an automatic pipeline that splits VidLN captions into action-level sentences and assigns approximate timestamps using key-frame annotations.

4 Training Data Challenge

  • The pipeline addresses training-data curation as a challenge because manual story-like video annotation is expensive and time-consuming.The paper presents this as a first approach to improving STORYBENCH performance.Source: §4 · p. 6
  • The pipeline turns original VidLN captions for Oops and UVO into multiple action-level sentences and determines their timestamps.The timestamps are approximate because the captions lack explicit action times; the pipeline derives them from key-frame associations.Source: §4 · p. 6
  • Flan-U-PaLM 540B [3] [56] splits each description into sentences that describe single actions while making minimal changes to the original words.The pipeline maps sentence words to associated key-frames using lemmas when the LLM modifies words.Source: §4 · pp. 6–7
  • Figure 2 summarizes the pipeline from video, caption, and annotated key-frames through sentence splitting, frame assignment, merging, and video segmentation.Source: Figure 2 · p. 7
  • Each sentence is assigned one key-frame, chosen most often from those referenced by its verbs, or otherwise across all its words.Consecutive sentences assigned to the same key-frame are concatenated, then mapped to time intervals using that frame's timestamp.Source: §4 · p. 7

5 Experimental Setup

The experiments use a compact Phenaki-based generator, PHENAKI-GEN, with a frozen video encoder and jointly trained text encoder and video generator. They also test a continuation-focused variant that predicts only the final frames from ground-truth context.

5 Experimental Setup

  • PHENAKI-GEN is a 345M-parameter Phenaki baseline composed of a C-ViViT encoder, text encoder, and MaskGiT model.Source: §5 · p. 7
  • C-ViViT compresses video in time and space while remaining autoregressive in time, and is pretrained on 27M web-scraped videos at 160×96 resolution before being frozen.Source: §5 · p. 7
  • MaskGiT predicts multiple masked video tokens conditioned on caption features, while the text encoder is jointly trained with the generator on 27M web-collected video captions.The video model is 345M parameters and is pretrained for 3M steps; after pretraining it generates long videos by conditioning each next six frames on the last five.Source: §5 · p. 7
  • PHENAKI-GEN is fine-tuned on STORYBENCH training data for 500K steps using batch size 64 and the pretraining learning rate and optimizer.Source: §5 · p. 7
  • PHENAKI-CONT trains for continuation by masking only the last six frames and conditioning on five ground-truth frames.The authors hypothesize that this setup would improve transitions and consistency.Source: §5 · p. 7

6 Results

Human raters favor continuation-mode fine-tuning for story continuation, while automatic metrics favor it on action execution and continuation but do not consistently track human judgments. Results also show benefits from story-like training data and limits in long-term consistency and multi-task fine-tuning.

6.1 Human Evaluation

  • Human raters vastly prefer PHENAKI-CONT-ST over PHENAKI-GEN-ZS and consistently prefer it over PHENAKI-GEN-ST when both use the same data.The comparison covers 100 randomly selected stories with three raters and majority votes; the videos can execute key actions while backgrounds drift over longer timescales.Source: §6.1 · p. 8, Figure 4 · p. 8, Table 3 · p. 9
  • Joint fine-tuning across all datasets produces videos of quality comparable to the zero-shot model; a larger model and longer training could improve performance.Source: §6.1 · p. 8
  • PHENAKI-CONT-ST-orig outperforms the zero-shot model except in visual quality, but its margins are smaller than those of PHENAKI-CONT-ST.The comparison supports the benefit of the training-data transformation pipeline.Source: §6.1 · p. 9
  • For action recognition, human ratings match story continuation results, whereas PHENAKI-GEN-ZS is generally preferred for story generation.The authors hypothesize continuation fine-tuning may reduce capability for generating from scratch.Source: §6.1 · p. 9

6.2 Automatic Evaluation

  • PHENAKI-CONT-ST performs best on FID, FVD, and SIM for action execution and story continuation, but story-generation FID improves while FVD does not.For story generation, this contrasts with human raters, who preferred PHENAKI-GEN-ZS across metrics.Source: §6.2 · p. 9, Table 4 · p. 9
  • Across datasets and tasks, PHENAKI-CONT-MT achieves strong overall results, but SIM, PQA, and VTM do not reflect human preferences reliably.SIM varies by less than one percentage point across models, and human raters vastly prefer PHENAKI-CONT-ST over PHENAKI-GEN-ZS.Source: §6.2 · p. 9

7 Limitations

StoryBench evaluates multi-sentence video generation, but its videos are short, mostly single-shot, user-generated examples rather than realistic filmmaking. The authors also note small baselines and imperfect alignment between automatic metrics and human ratings.

7 Limitations

  • The annotated videos are not very long, are mostly single-shot and user-generated, and exclude artistic content unseen in the real world.The authors say the annotation framework would not scale to long videos and that these video characteristics differ from realistic filmmaking applications.Source: §7 · pp. 9–10
  • Tables 5–7 report automatic metrics for action execution, story continuation, and story generation, respectively.The captions specify the feature backbones used for FID, SIM, FVD, PQA, and VTM.Source: Table 5 · p. 10, Table 6 · p. 10, Table 7 · p. 10
  • Automatic metrics do not fully align with human ratings, and the authors cannot exclude better metric performance on higher-resolution videos.They identify the need for an in-depth study of human versus automated evaluation.Source: §7 · p. 10
  • The baselines are small at 345M parameters compared with current 20B text-to-image models, leaving effective training of large text-to-video models open.The comparison is between the paper’s text-to-video baselines and current text-to-image models.Source: §7 · p. 10

8 Conclusion

The conclusion presents STORYBENCH as a benchmark for evaluating text-to-video models on evolving prompts and specified durations. The authors also state the release licenses for its datasets and code.

8 Conclusion

  • STORYBENCH combines timed text prompts and diagnostic segment labels to evaluate text-to-video models across three tasks, three datasets, and three setups.The benchmark targets challenges including temporal consistency and realistic action synthesis; the authors report that fine-tuning Phenaki for continuation improves these aspects and that human and automatic ratings diverge.Source: §8 · p. 10
  • The authors identify longer videos, richer scene variety, and more efficient annotation protocols as directions for extending the benchmark.They mention camera jumps and scene changes as examples of richer video variety.Source: §8 · p. 10

A Author Statement

  • The datasets are released under CC-BY-4.0, and the code under an Apache license.Data, code, and annotation guidelines are hosted on GitHub.Source: A Author Statement · p. 16

B Ethics Statement

The ethics statement highlights costs and risks associated with training and using text-to-video models. It frames STORYBENCH as a research benchmark and cautions against sensitive applications.

B Ethics Statement

  • Training current strong text-to-video models is computationally expensive, with financial and environmental costs from hardware and electricity use.The authors specifically connect environmental cost to the carbon footprint of tensor-processing hardware.Source: B Ethics Statement · p. 16
  • Web-harvested training data may reflect stereotypes, oppressive viewpoints, harmful associations, and English- and Western-cultural biases.The authors caution developers to assess model limitations and encourage benchmark evaluations of social and cultural biases.Source: B Ethics Statement · p. 16
  • Realistic generated videos could support surveillance, harmful content, and disinformation, including convincing deepfakes.The authors mention watermarking generated videos as a possible way to identify outputs from a particular model.Source: B Ethics Statement · p. 16
  • The authors say STORYBENCH is intended to measure research progress, do not release their baselines publicly, and restrict dataset use to creative goals.They state that the data should not be extended for sensitive domains.Source: B Ethics Statement · p. 16
  • The authors report that annotation contractors received their standard contracted wage, above the living wage in their country of employment.They do not disclose hourly wage estimates or total participant compensation, citing privacy concerns.Source: B Ethics Statement · p. 16

C Datasheet

The datasheet describes STORYBENCH’s motivation, annotated data, collection and processing, intended uses, release, and maintenance. Its annotations add timed action descriptions and labels to existing video datasets for evaluating continuous story visualization.

Motivation

  • Existing caption datasets provide limited support for generating video stories, while earlier story visualization produced one key frame per caption rather than continuous video.STORYBENCH maps separately described actions to timestamp intervals to evaluate stories of varying duration.Source: Motivation · pp. 16–17

Composition

  • STORYBENCH annotates subsets of DiDeMo, Oops, and UVO evaluation videos selected for availability, appropriateness, annotation quality, and preprocessing criteria.The annotations include timed story descriptions and 35 diagnostic labels across six categories; multiple stories can correspond to one video.Source: Composition · pp. 17–18
  • The benchmark relies on externally hosted source videos and does not provide archival copies of the complete underlying datasets.The source video resources are publicly available from their official websites; the recommended original train/dev/test splits help avoid leakage.Source: Composition · p. 18
  • The underlying videos may show identifiable people or sensitive attributes, and the authors cannot rule out a small number of inappropriate samples despite screening.The authors say they do not explicitly collect subpopulation annotations, though such information may be deducible from videos or descriptions.Source: Composition · p. 18

Collection Process

  • Human annotations were collected through web interfaces and manually inspected; a random sample was checked by other humans.Contractors were hired through a third-party company, and the authors report that they received standard contracted wages above their country’s living wage.Source: Collection Process · p. 19
  • No institutional review board reviewed the work because it used unchanged videos from publicly available datasets; people shown were not notified about the annotation collection.The authors say users can request removal of corresponding annotations, and authorize the dataset for academic endeavors given potential impacts and opportunities.Source: Collection Process · pp. 19–20

Preprocessing, Cleaning, and/or Labeling

  • Preprocessing asks annotators to select relevant segment labels and removes videos with a first action under 1.5 seconds or action gaps over 0.5 seconds.The authors also state that raw data were not retained under organizational data-retention policies and that preprocessing scripts are released.Source: Preprocessing, Cleaning, and/or Labeling · p. 20

Uses

  • The annotations may reflect imbalances in their source videos, and the authors encourage future datasets that increase fairness and reduce generative-model bias.The authors state that benchmark models and datasets should be used for research, not deepfakes, disinformation, harm, military, or surveillance tasks.Source: Uses · p. 21

D.1 Data Collection

The authors collect timestamped action stories and diagnostic labels for three video datasets, then preprocess the data for the benchmark tasks. They also describe human and automatic evaluation procedures.

D.1 Data Collection

  • Annotators built timestamped action stories; Oops and UVO captions were split and enriched with other-actor context, while DiDeMo queries were refined into coherent stories.For Oops and UVO, annotators also flagged multiple-story videos and unimportant actors; DiDeMo videos were trimmed to at most 30 seconds.Source: §D.1 · p. 22
  • The annotation process added 34 diagnostic labels, and managers found 97% of manually checked samples accurately reflected video narratives.Managers checked 25% of samples after annotators completed training.Source: §D.1 · p. 25

D.2 Data Preprocessing

  • Stories were filtered to require a first action of at least 1.5 seconds and no gap above 0.5 seconds between consecutive actions.The first-action threshold accommodates the 0.5-second conditioning video used for story continuation.Source: §D.2 · p. 25

D.3 Human Evaluation

  • Human evaluation uses side-by-side model comparisons, with raters choosing which model better meets five defined criteria.The interface is shown in Figure 15.Source: §D.3 · p. 25

D.4 Automatic Evaluation

  • The authors map FID, FVD, SIM, VTM, and PQA to different intended human-evaluation qualities, including visual quality, consistency, realism, and text adherence.These mappings are expectations based on what each metric compares or predicts.Source: §D.4 · p. 25

D.5 Robustness of Automatic Pipeline

The automatic story pipeline produces stories with substantial caption overlap with human references, while differing in sentence splitting and segment duration. The additional results section specifies the baseline evaluation setup.

D.5 Robustness of Automatic Pipeline

  • Figure 11 shows similar sentence-count distributions, although the pipeline splits captions into two or three segments more often than human annotators.Human annotators more often leave captions unsplit.Source: Figure 11 · p. 26, §D.5 · p. 27
  • 63.6% BLEU4 was observed for algorithmic captions mapped one-to-one to human captions with the same sentence count.The pipeline does not explicitly add the contextual information about other actors that human annotators were asked to include.Source: §D.5 · p. 27
  • Algorithmically generated video segments are slightly longer than human-annotated timestamps.The comparison uses the Oops Dev set of 1,578 stories.Source: §D.5 · pp. 25, 27, Figure 11 · p. 26

E Additional Results

  • Baselines were fine-tuned for 500K steps, and each generated four videos at 8 fps and 160×96 resolution per story.One output was sampled for human evaluation; automatic metrics report means and standard deviations.Source: E Additional Results · p. 28

E.1 Human Evaluation

Human raters generally prefer continuation models for story continuation, but no model clearly leads in story generation. Alternative automatic-metric backbones produce similar patterns, with a noted preference for zero-shot videos from InternVideo.

E.1 Human Evaluation

  • Three raters compare anonymized model pairs for each sampled story, with the majority vote reported as the result.Figure 17 displays wins and ties.Source: §E.1 · p. 28, Figure 17 · p. 30
  • On Oops action execution, PHENAKI-CONT-ST is competitive with PHENAKI-GEN-ZS and rates better for text adherence, background consistency, and action realism; no model clearly leads in story generation.Annotators largely prefer PHENAKI-CONT-ST to PHENAKI-GEN-ST across all criteria for Oops action execution.Source: §E.1 · p. 29, Figure 17 · p. 30
  • PHENAKI-CONT-ST typically outperforms both generation baselines in story continuation, especially on Oops; across UVO and DiDeMo it beats PHENAKI-GEN-ST except for consistency judgments, while PHENAKI-GEN-ZS retains a visual-quality advantage.PHENAKI-CONT-MT is always preferred to PHENAKI-GEN-MT.Source: §E.1 · p. 29, Figure 17 · p. 30

E.2 Automatic Evaluation

  • Alternative automatic-metric backbones show similar patterns, but InternVideo favors zero-shot PHENAKI-GEN videos on FVD and VTM.The reported alternative metrics use CLIP for FID and SIM, InternVideo for FVD and VTM, and DOVER for PQA.Source: §E.2 · p. 30, Table 10 · p. 31, Table 8 · p. 31

Related papers

Loading 2308.11606v2…