PaperlayerView exact revision on arXiv

Source-linked AI summary

StoryEngine: A State-Grounded Agentic Framework for Video Storytelling

Yingrui Wang, Zeqing Wang, Yeying Jin

arXiv:2609.33627v1cs.CVcs.AI

TL;DR

Long-form storytelling needs cross-shot event consequences to persist, but existing pipelines lack explicit state propagation and can carry visual errors into later shots. StoryEngine separates an authoritative semantic plan from fallible observations, propagates intended states, and grounds render plans. On a 60-story benchmark, it consistently outperforms representative baselines across narrative realization, cross-shot coherence, and visual consistency.

  • Problem

    Long-form storytelling requires event consequences to persist across shots, but existing pipelines lack explicit state propagation and may carry visual errors into later shots.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    StoryEngine separates a fixed authoritative semantic plan from fallible visual observations, propagates event-induced states, and translates them into grounded render plans with bounded local repair.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    On a new 60-story benchmark, StoryEngine consistently outperforms representative baselines in narrative realization, cross-shot coherence, and visual consistency, with the largest gains in anchor persistence and lighting coherence.

    Source: §5 · p. 10

  • Takeaways & Limitations

    StoryEngine preserves causal progression and prevents visual errors from becoming narrative facts across shots.

    Source: §5 · p. 10

    Incorrect initial entity catalogs, effect order, or placement graphs can persist because later visual evidence cannot autonomously correct planning errors.

    Source: E LIMITATIONS · p. 23

Abstract

from arXiv · show

Despite recent progress in agentic multi-shot video generation, producing coherent and consistent long-form stories remains challenging. Existing agentic pipelines typically rely on textual shot plans or previously generated pixels, yet lack an explicit mechanism for propagating the consequences of story events and maintaining the video world state across shots. As a result, missing visual details may be reconstructed inaccurately, while visual drift may propagate across subsequent shots, undermining both narrative coherence and visual consistency. To address these challenges, we propose StoryEngine, a state-grounded agentic framework for video storytelling. StoryEngine establishes a separation between authoritative semantic plans and unreliable visual observations. Specifically, StoryEngine maintains a structured representation of entity placement and story-relevant states, and propagates event-induced changes to define the intended start and end states of each shot. To visually realize these states, StoryEngine constructs canonical references for recurring entities and environments, and compiles state and visual constraints into executable render plans. Meanwhile, to realize these states correctly, a bounded evaluation-guided repair loop further corrects local state inconsistencies. Together, these mechanisms preserve causal story progression and prevent local visual errors from propagating across shots. To comprehensively evaluate long-form storytelling, we construct a benchmark across diverse scenarios and visual styles, with metrics assessing storytelling quality, narrative coherence, and visual consistency. Experimental results demonstrate that StoryEngine consistently outperforms state-of-the-art methods across all evaluation dimensions, validating its effectiveness for coherent and consistent video storytelling.

1 INTRODUCTION

Long-form storytelling requires coordinated shots that advance a coherent, consistent narrative, yet existing agentic pipelines do not explicitly track event-driven state changes across shots. StoryEngine separates semantic plans from fallible visual observations and combines state propagation, grounded rendering, and local repair; it outperforms representative baselines across the reported metrics.

1 INTRODUCTION

  • Coordinating and supervising individually plausible shots into an interconnected long-form story requires substantial human effort.The motivating example requires the backpack to stay in the locker and the notebook to remain on the desk until retrieved, while recurring settings and objects retain their visual identity.Source: §1 · pp. 1–2
  • Existing pipelines often use loosely structured shot descriptions and generated frames, without explicitly tracking event-driven changes to entity placements and story-relevant attributes.Reusing generated pixels can let misplaced objects or corrupted details become narrative facts, contributing to global incoherence and cross-shot visual inconsistency.Source: §1 · p. 2
  • StoryEngine propagates event-induced state changes to define each shot’s intended start and end states, then compiles state and visual constraints into executable render plans.Grounded render planning uses canonical references for recurring entities and scenes and selects action-relevant camera views.Source: §1 · p. 2
  • A bounded evaluation-guided repair loop corrects local state inconsistencies while keeping the authoritative semantic plan fixed.Compatible visual evidence may support continuity, while omitted actions and misplaced objects remain local errors rather than becoming narrative facts.Source: §1 · p. 3
  • Across a 60-story benchmark and two video backbones, StoryEngine consistently outperforms state-of-the-art agentic systems on narrative realization, cross-shot coherence, and visual consistency.The introduction also reports that it outperforms representative baselines on all metrics.Source: §1 · p. 3

2 RELATED WORK

Related work spans multi-shot video generation and agentic pipelines that divide production among specialized stages. These approaches still leave event consequences and the distinction between intended story facts and pixel-level errors insufficiently explicit.

2 RELATED WORK

  • Multi-shot methods extend video generation through shared visual context, longer temporal modeling, or reference frames, but per-shot quality and cross-shot narrative coherence and visual consistency remain challenging.The cited approaches are Kara et al. (2025), Guo et al. (2025), Zhang et al. (2026), and Wang et al. (2026).Source: §2 · p. 3
  • Agentic systems reduce manual production effort by dividing work into stages, including planning, storyboarding, generation, and evaluation (Hu et al., 2024; Wu et al., 2025).Recent systems also combine global planning with iterative refinement or propagate visual anchors across shots (Song et al., 2026; Huang et al., 2026).Source: §2 · p. 3
  • Unlike pipelines relying on textual descriptions, retrieved context, or generated images, StoryEngine explicitly propagates event consequences and separates story facts from errors in generated pixels.The related-work discussion notes that omitted actions, misplaced objects, and corrupted details may otherwise be carried into later shots.Source: §2 · p. 3

3 METHOD

StoryEngine separates the fixed semantic story plan from visual observations, which can help realize but cannot revise that plan. Its pipeline propagates story states, grounds render plans, and generates shots with continuity checks and local repair.

3.1 OVERVIEW

  • Story-state propagation creates the semantic plan, grounded render planning turns it into shot-level render contracts, and continuity-aware generation checks and repairs shots before concatenation.The three stages respectively define story states, compile visual constraints, and use compatible prior evidence without letting it revise the plan.Source: Figure 2 · p. 4, §3.1 · p. 4

3.2 STORY-STATE PROPAGATION

  • For each shot, the plan tracks recurring entities, environments, ordered events, spatial intent, and world-state placements and attributes; a reducer applies event effects to determine the next state.The intended next state follows the planned events, not generated pixels, so missed or corrupted details remain execution errors rather than becoming narrative facts.Source: §3.2 · pp. 4–5

3.3 GROUNDED RENDER PLANNING

  • Canonical entity and environment references, action-relevant views, and camera specifications are compiled with propagated states into shot contracts specifying opening facts, required motion, and end-state evidence.References anchor visual identity and layout, while phase-tagged criteria make semantic and visual constraints executable.Source: §3.3 · p. 5

3.4 CONTINUITY-AWARE GENERATION

  • A compatibility gate admits prior-shot evidence in reuse, reference, or fresh mode, rejecting incompatible content and checking the proposed opening frame against the next shot’s start criteria.An incompatible or unavailable tail is discarded, while compatible content may be reused or selectively retained.Source: §3.4 · pp. 5–6
  • A bounded evaluator-guided repair loop targets failed or unknown criteria while keeping the render plan and semantic plan fixed, returning a passing candidate or the best valid candidate if its budget runs out.Any remaining violation is recorded as a local degradation; generated video never feeds back into the authoritative story state.Source: §3.4 · p. 6

4 EXPERIMENTS

Experiments evaluate StoryEngine on a 60-story benchmark using diagnostic metrics for narrative realization, cross-shot coherence, and visual consistency. The results show consistent gains over baselines, while ablations examine the framework’s components and planner sensitivity.

4.1 BENCHMARK

  • The benchmark contains 60 stories in three 20-story diagnostic suites, with five setting categories represented in each suite.Each story requests 10 shots over two recurring locations, and its annotations are hidden from methods and frozen before generation.Source: §4.1 · p. 6, Figure 3 · p. 6

4.2 EVALUATION METRICS

  • Eight complementary metrics assess narrative realization, cross-shot coherence, and visual consistency across the benchmark’s three suites.The metrics are grouped by suite, with MIR computed across all 60 stories.Source: §4.2 · p. 7

4.3 EXPERIMENTAL SETUP

  • Comparisons include ViMax, MovieAgent, and planner-free Direct I2V, with all methods run using Veo 3.1 and Wan2.2-TI2V-5B.StoryEngine’s in-loop evaluator uses a repair budget of T = 3.Source: §4.3 · pp. 7–8

4.4 MAIN RESULTS

  • StoryEngine achieves the best scores on all seven video-based metrics with both backbones and the highest PEC among methods that produce a plan.Its Avg is 0.7690 with Veo 3.1 and 0.7372 with Wan2.2-TI2V-5B, exceeding the strongest baseline by 0.1416 and 0.1304, respectively.Source: Table 1 · p. 8, §4.4 · p. 8
  • Across backbones, StoryEngine leads in ECS, SPS, GPC, and MIR; the story example preserves object placement and recurring character and environment identity.SPS remains below 0.67 for all methods, leaving irreversible state progression an open challenge.Source: §4.4 · p. 8

4.5 ABLATION STUDY

  • Removing any component lowers Avg; w/o GRP drops it from 0.7690 to 0.7119, while qualitative ablations show distinct appearance and placement errors.w/o SCVC has the lowest ECS, APR, and MIR among component ablations; w/o SSP has the smallest Avg drop.Source: Table 2 · p. 9, §4.5 · p. 9, Figure 4 · p. 9
  • With the Gemini-3.5-Flash planner, StoryEngine’s Avg is 0.7357, still exceeding all Veo 3.1 baselines despite a decrease from its GPT-5.5-planner result.PEC and MIR are 0.8872 and 0.4179, respectively, with that planner.Source: Table 2 · p. 9, §4.5 · p. 9

5 CONCLUSION

The conclusion presents StoryEngine as a state-grounded framework that separates semantic plans from visual observations and reports consistent gains across three evaluation dimensions. It also identifies irreversible state progression and longer stories with more locations and characters as remaining challenges.

5 CONCLUSION

  • StoryEngine separates authoritative semantic plans from fallible visual observations and combines state propagation, grounded render planning, and repair of local inconsistencies.The conclusion says these mechanisms preserve causal progression and prevent visual errors from becoming narrative facts.Source: §5 · p. 10
  • On a new 60-story benchmark, StoryEngine consistently outperforms representative baselines in narrative realization, cross-shot coherence, and visual consistency.The conclusion notes the largest gains in anchor persistence and lighting coherence.Source: §5 · p. 10
  • Remaining challenges include irreversible state progression and longer stories with more locations and characters.The appendix overview points to Appendix E for discussion of method and evaluation limitations.Source: §5 · p. 10, A APPENDIX OVERVIEW · p. 11

B COMPLETE METHOD SPECIFICATION

StoryEngine keeps the semantic story plan authoritative while compiling its states into shot-level render contracts and using visual evidence only for continuity-aware generation. Its method combines explicit state transitions, canonical visual grounding, and bounded local repair.

B.1 END-TO-END PROCEDURE

  • The end-to-end procedure fixes the semantic plan and executable render plan before synthesis, then renders shots in order with a continuity gate and at most three repair attempts.Accepted shots are concatenated in planned order.Source: §B.1 · p. 12

B.4 CONTINUITY GATE AND LOCAL REPAIR

  • The continuity gate chooses reuse, reference, or fresh mode based on tail-frame compatibility, and bounded repair targets failed or unknown criteria without replanning the story.If no candidate meets every required criterion within the budget, selection prioritizes fewer failed requirements and unknown judgments before continuity, preference, and technical losses.Source: §B.1 · p. 12, §B.4 · pp. 13–14

B.2 AUTHORITATIVE SEMANTIC PLAN

  • Stable entity and environment catalogs, typed placements, ordered events, and validated state transitions define each shot’s authoritative semantic contract.After every nonempty event update, the system validates entity references, attribute domains, placement types, and acyclicity; later supplied states are assertions checked against the trajectory.Source: §B.2 · p. 12

B.3 GROUNDING AND RENDER-CONTRACT COMPILATION

  • Grounded render planning binds canonical entity and environment references to action-relevant views, then converts shot states and criteria into structured generator instructions.Criteria distinguish start, motion, end, and always-on requirements, with priorities separating requirements from preferences.Source: §B.3 · p. 13

B.5 PROMPT INTERFACES

  • The prompt interfaces present condensed contracts for semantic planning, opening-image and shot-video generation, and criterion evaluation.Descriptive fields and story-specific values are inserted as structured JSON.Source: §B.5 · p. 14

B.6 IMPLEMENTATION AND GENERATION DETAILS

The implementation fixes generation settings within each comparison and applies technical validity checks before semantic evaluation. Condensed prompt interfaces specify the semantic plan, image and video tasks, and criterion judgments.

B.6 IMPLEMENTATION AND GENERATION DETAILS

  • Veo 3.1 renders 1280×720 shots at 24 fps for 6 seconds, while Wan2.2-TI2V-5B renders 1280×704 shots at 24 fps for approximately 6.04 seconds.Wan uses 145 frames because it requires 4n+1 frames; Veo uses 144 frames.Source: §B.6 · p. 14
  • Candidates must decode, contain frames, match expected dimensions, and have compatible duration; black, white, frozen-from-decoding-failure, or missing-path outputs are rejected.Low optical flow alone does not invalidate a shot, and no reported metric uses audio.Source: §B.6 · pp. 14–15
  • The condensed prompt contracts direct planning to produce stable IDs, initial placements, state-changing events, and testable criteria, while generation follows the fixed render contract and evaluator returns criterion labels.The shot-video interface also specifies repair targets drawn from failed or unknown criteria.Source: §B.6 · p. 15

B.7 BASELINE AND ABLATION PROTOCOLS

The protocols standardize inputs and model backends within comparisons, define representative baseline pipelines, and construct ablations by removing one StoryEngine stage at a time.

B.7 BASELINE AND ABLATION PROTOCOLS

  • Within each comparison, methods share the benchmark brief, image generator, and video backbone; all LLM-based methods use GPT-5.5, and metric judges do not see render prompts or method names.Except for PEC, metrics use exported videos and frozen annotations.Source: §B.7 · p. 15
  • Direct I2V independently generates each shot’s opening keyframe without planning, previous tails, canonical references, state tables, or evaluator-guided repair; ViMax and MovieAgent retain their respective orchestration pipelines with common backends.PEC does not apply to Direct I2V because it produces no plan.Source: §B.7 · p. 15
  • Each ablation removes one stage: SSP eliminates shared propagated states, GRP removes visual grounding, and SCVC removes tail reuse and feedback-triggered regeneration.A separate planner-sensitivity variant replaces GPT-5.5 with Gemini-3.5-Flash while keeping other backends fixed.Source: §B.7 · p. 16

C BENCHMARK DETAILS

The benchmark organizes 60 stories into three diagnostic suites, with paired briefs and hidden annotation sheets. The suites target narrative realization, cross-shot coherence, and visual consistency.

C BENCHMARK DETAILS

  • The benchmark contains 60 stories in three 20-story suites—N20, T20, and C20—with five setting categories represented by four stories each.Ten settings appear in both N20 and T20, with different briefs and annotations; MIR is computed on all 60 stories.Source: §C.1 · p. 16, Table 4 · p. 17
  • Each brief is the method’s only input and specifies a 10-shot story, visual style, five recurring entities, forbidden content, and delivery requirements.The brief also states the story idea and shot-to-location plan.Source: §C.1 · p. 16
  • The suites impose different content constraints, including T20 prohibitions on extreme close-ups and C20 prohibitions on any second person.The listed restrictions supplement constraints shared across briefs.Source: §C.1 · p. 17
  • Annotation sheets are hidden from methods, contain typed blocks for locations, cast, props, and suite-specific criteria, and are frozen under content hashes.The sheets were authored with the briefs without consulting system outputs.Source: §C.1 · p. 17

C.2 SUITE DESIGN

The benchmark separates narrative realization, cross-shot coherence, and visual consistency into three suites; its example briefs show the kinds of events, state changes, and visual continuity each suite requires.

C.2 SUITE DESIGN

  • N20 tests ten required on-screen event changes and uses distractor events to measure false-positive judgments.Stories have two named characters, four key props, and five or six location changes.Source: §C.2 · p. 17
  • T20 tests anchor visibility across cuts and irreversible state changes that must not be undone.The suite records anchor presence across all 180 cuts and includes 60 monotone state changes.Source: §C.2 · p. 17
  • C20 tests whether revisited rooms retain their appearance and the character’s face and clothing remain legible.Its plans include 415 same-location shot pairs, 340 non-adjacent; lighting annotations specify either directional contrast or one consistent look.Source: §C.2 · p. 17

C.3 EXAMPLE BRIEFS

  • Example briefs illustrate the suites through required bakehouse events, cross-cut anchors and one-way changes, and a recurring character in rooms that must look consistent.The N20 brief requires visible changes; the T20 brief allows jobs to be ordered while preserving anchors and irreversible changes; the C20 brief requires the character and returning rooms to remain visually legible and consistent.Source: §C.3 · p. 18

D COMPLETE EVALUATION PROTOCOL

The protocol evaluates plans and exported videos against frozen story annotations, using distinct metrics for event coverage, state and location continuity, place consistency, identity retention, and lighting. It specifies evidence handling, aggregation, and boundaries for interpreting each metric.

D.1 COMMON PROTOCOL

  • Video metrics use exported shots in planned order, while PEC reads per-shot plan text; stories are macro-averaged or pooled by metric, with explicit rules for missing shots and repeated runs.PEC, ECS, and GPC use three independent evaluation runs; Avg is the unweighted mean of seven video metrics and excludes PEC.Source: §D.1 · p. 19

D.2 PLAN EVENT COVERAGE (PEC) / D.3 EVENT COMPLETION SCORE (ECS)

  • PEC scores plan-text coverage of required events against distractors, while ECS checks whether required events visibly occur across sampled frames and penalizes distractor false positives.PEC judges per-shot descriptions and excludes structured plan fields; ECS requires evidence of change across two different frames, rather than a static finished state.Source: §D.2 · pp. 19–20, §D.3 · p. 20

D.4 ANCHOR PERSISTENCE RATE (APR)

  • APR measures whether a named anchor is visible at both sides of each cut, counting violated cuts as failures and unclear cuts as undetermined.Undetermined cuts remain in the denominator.Source: §D.4 · p. 20

D.5 STATE PROGRESSION SCORE (SPS)

  • SPS scores irreversible state progression from ordered shot-level probes, comparing before, transition, and after evidence against a single forward change.The probe sequence is scored per state and averaged within stories; probes with absent or unclear evidence are dropped.Source: §D.5 · pp. 20–21

D.6 LOCATION ADHERENCE RATE (LAR)

  • LAR compares each readable shot’s judged location with its planned location, treating “other” as a readable mismatch.The judge can use “cannot tell,” in which case the shot is excluded.Source: §D.6 · p. 21

D.7 GEOMETRIC PLACE CONSISTENCY (GPC)

  • GPC assesses same-location shot pairs only when both locations agree and background geometry supports a shared place rather than a shared textured plane.A pair is excluded as degenerate when a frame combination has at least eight fundamental-matrix inliers but no admissible count.Source: §D.7 · pp. 21–22

D.8 MINIMUM IDENTITY RETENTION (MIR)

  • MIR tracks face-embedding consistency across shots and takes the minimum across qualifying identity clusters.It scores zero if fewer than two faces are detected or no cluster spans two shots.Source: §D.8 · p. 22

D.9 LIGHTING COHERENCE SCORE (LCS)

  • LCS compares lighting similarity across same-condition and different-condition shot pairs for stories that specify a lighting contrast.It is defined for 13 applicable stories; a single frozen look or changes that ignore the plan yield values near zero.Source: §D.9 · pp. 22–23

D.10 SCOPE OF METRIC INTERPRETATION

The authors caution that the eight metrics capture different failure modes and should be interpreted jointly, not through Avg alone.

D.10 SCOPE OF METRIC INTERPRETATION

  • The metrics have distinct blind spots: APR checks anchor presence rather than physical continuity, SPS covers only annotated irreversible states, and LAR identifies location without checking layout.GPC favors textured, approximately rigid backgrounds; MIR depends on face visibility and detector behavior; LCS applies only to 13 stories and does not check the annotated direction of lighting contrast.Source: §D.10 · p. 23
  • VLM-judged metrics may favor StoryEngine because the judge overlaps with its in-loop evaluator, although the evaluator and judge receive different inputs.MIR and LCS use no VLM and are unaffected by this overlap.Source: §D.10 · p. 23
  • PEC evaluates plan text rather than video, so its scores depend on how systems phrase per-shot descriptions.Avg is an equal-weight summary, not a calibrated preference score.Source: §D.10 · p. 23

E LIMITATIONS

The limitations identify risks from authoritative planning errors, finite visual evidence, and a benchmark with restricted narrative and media coverage.

E LIMITATIONS

  • Incorrect entity catalogs, effect order, or placement graphs can make planning mistakes persistent because later visual evidence cannot correct them autonomously.Schema checks do not catch every commonsense or cultural error; high-stakes use requires human inspection or separately validated replanning.Source: E LIMITATIONS · p. 23
  • Canonical references and panoramas are not true 3D scenes, leaving large viewpoint changes, severe occlusion, mirrors, transparent or deformable objects, crowds, and fine hand–object contact difficult.Compatible endpoints do not prove physically valid intervening motion, and an exhausted repair budget may leave criteria violated.Source: E LIMITATIONS · p. 23
  • The English, fictional, visually oriented benchmark covers 60 ten-shot stories over two locations, with one or two named characters and no audio.It excludes long narratives with many locations, crowded interaction, dialogue or audio continuity, lip synchronization, on-screen text, and interactive editing.Source: E LIMITATIONS · p. 23

Related papers

Loading 2609.33627v1…