PaperlayerView exact revision on arXiv

Source-linked AI summary

VIS-Ground: Video Interactive Storytelling with Contextual Grounding

Bingxuan Li, Yiwen Song, Xueqing Wu, Yanzhou Pan, Yang Li, Kuang Su, Jingyun Liu, Sebastian Ko, Huan Zhang, Tong Zhang, Nanyun Peng, Tomas Pfister, Yale Song

arXiv:2610.09326v1cs.CV

TL;DR

Interactive continuation must reconcile the grounding source, rendered video prefix, and current viewer request, whose dependencies may emerge only jointly. VIS-Ground compiles these inputs into grounded dependencies and candidate-specific constraints, then uses planning, verification, revision, and rendering to enforce them. It achieves the highest overall composite score with each of three video backbones.

  • Problem

    Interactive continuation requires reconciling what the source supports, what the viewer has seen, and the current request, but their dependencies may be implicit when inputs are considered separately.

    Source: §1 · p. 2

  • Method

    VIS-Ground aligns source, video, and story states, derives cross-context dependencies, and activates relevant candidate-specific constraints to guide planning and video repair.

    Source: §1 · p. 2, §3.1 · p. 5, §3.3 · p. 6

  • Results

    VIS-Ground achieves the highest overall composite score with each of three backbones, beating the strongest corresponding baselines by 12.3 points for Omni, 15.3 for Veo, and 3.4 for MiniMax.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Across three backbones, VIS-Ground improves performance by grounding continuation in aligned source, prefix, and request context and inducing constraints for generation.

    Source: §6 · p. 12

    VIS-Ground is not best on every individual metric, and visual quality remains largely bounded by the underlying video generator.

    Source: §5.3 · p. 11

Abstract

from arXiv · show

Video interactive storytelling enables viewers to actively steer how a video unfolds. However, once we allow viewers to intervene during generation, a new challenge arises: The viewer's request can have latent dependencies on both the grounding source and the current rendered video state. These dependencies may not be explicitly stated in any individual input, but emerge only when the source, rendered history, and new viewer intent are considered jointly. Existing interactive video generation systems primarily emphasize following viewer instructions, while source-grounded video generation methods focus on aligning generated content with an external narrative or knowledge source. This leaves a fundamental question underexplored: What context should a generation model ground on during interactive continuation, and how can heterogeneous, unstructured inputs be transformed into such grounding context? In this work, we formulate contextual grounding as the process of transforming heterogeneous input context into an executable constraint model for video generation. To address this challenge, we introduce VIS-Ground, which performs Structured Context Abstraction to recover grounded states and cross-context dependencies, Generation Constraints Induction to project relevant dependencies into candidate-specific constraints, and Constrained Video Generation to enforce these constraints through planning, verification, revision, and rendering. Across three video generation backbones, VIS-Ground consistently achieves the highest overall composite score, reaching an average absolute improvement of 10.3 points over the strongest per-backbone baselines. Detailed analysis further shows gains across both narrative and knowledge grounding, and reveals remaining challenges in dependency extraction, and faithful realization during video rendering.

1. Introduction

Interactive continuation must jointly account for the grounding source, rendered prefix, and new request, whose relationships may be implicit. VIS-Ground makes those relationships explicit and uses them to constrain generation; it achieves the highest overall composite score with each of three video backbones.

1. Introduction

  • Mean pre-to-post learning gains were 47.9 percentage points with interaction and 32.1 without interaction in the formative study.The study’s observations motivate further study of interaction as a way to support viewers’ understanding.Source: §1 · p. 2
  • A continuation must reconcile what the source supports, what the viewer has seen, and how the story has evolved, rather than interpreting the latest request alone.Directly combining source, prefix, and request leaves relationships implicit and risks incompatible state changes or unsupported content.Source: §1 · p. 2
  • VIS-Ground aligns video, story, and source states, derives instance-specific constraints from their dependencies, and uses them to guide planning, verification, and revision.It verifies both before and after rendering because a valid script does not guarantee faithful video realization.Source: §1 · pp. 2–3
  • VIS-Ground achieves the highest overall composite score with each of three backbones, beating the strongest corresponding baselines by 12.3 points for Omni, 15.3 for Veo, and 3.4 for MiniMax.The evaluated backbones are Omni-1.1-Flash, Veo 3.1, and MiniMax-H3.Source: §1 · p. 3

2. Video Interactive Storytelling with Contextual Grounding

The formative study reports higher mean learning gains with interaction among participants who passed the attention test, while task formulation treats continuation as conditioned on source, prefix, and current request. The prefix serves as the record of earlier instructions.

2.1. Formative Study

  • Among the seven attention-test passers, mean learning gains were higher with interaction; experience ratings from four respondents clustered toward the positive end of the five-point scale.These are exploratory observations, not a learning-gain estimate for all nine volunteers.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2. Task Formulation

  • A continuation is generated from the grounding source, previously rendered video prefix, and current viewer request; the prefix supplies context for the continuation.The formulation treats the prefix as a sufficient record of earlier instructions instead of explicitly conditioning on all prior viewer instructions.Source: §2.2 · pp. 3–4

3. Method

VIS-Ground compiles heterogeneous context into grounded dependencies and activates only those relevant to a candidate continuation. The resulting constraints guide script repair and verification of the rendered video.

3. Method

  • VIS-Ground compiles heterogeneous context into aligned states and dependencies, then projects relevant dependencies onto candidate continuations as constraints that guide planning, repair, and rendering.The overview depicts Structured Context Abstraction, Generation Constraints Induction, and Constrained Video Generation as successive stages.Source: §3 · p. 4, Figure 3 · p. 4

3.1. Compiling Context into Grounded Dependencies

  • Video, story, and source predicates form aligned states in M; derived dependencies R capture cross-context temporal, causal, persistence, compatibility, and source-supported relations.These dependencies can emerge only when the inputs are jointly grounded, rather than being explicit in any one input.Source: §3.1 · pp. 4–5

3.2. Projecting Dependencies into Generation Constraints

  • Only dependencies relevant to a candidate script activate as evidence-linked constraints; evaluation returns pass, fail, or unknown for compatibility, conflict, or insufficient evidence.Candidate relevance depends on whether the proposed continuation introduces an event, relation, or state transition that makes the dependency applicable.Source: §3.2 · p. 5

3.3. Constrained Video Generation

  • When script constraints fail, targeted feedback identifies conflicting script content and the violated dependency, avoiding regeneration from scratch.Constraint activation is recomputed after each update because revisions may introduce new events or state transitions.Source: §3.3 · p. 6
  • A valid script may still render incorrectly, so VIS-Ground checks realized video relations and uses violations to repair the script or clarify missing dependencies before rerendering.The process ends when applicable dependencies are satisfied or the computation budget is exhausted.Source: §3.3 · p. 6

4. Experiments

VIS-Bench evaluates interactive continuation in narrative- and knowledge-grounded settings across six semantic and perceptual dimensions. VIS-Ground achieves the highest composite score with every backbone, with gains across grounding settings and reduced cross-backbone variation.

4.1. VIS-Bench: Video Interactive Storytelling Benchmark

  • VIS-Bench pairs grounding sources, observed video prefixes, and viewer interventions in narrative- and knowledge-grounded settings.Narrative sources specify story characters, events, relationships, and progression; knowledge sources provide factual content to communicate faithfully.Source: §4.1 · pp. 6–7

4.2. Evaluation Metrics

  • Evaluation covers six dimensions: SF, SC, IF, JCS, VC, and VQ, with higher normalized scores indicating better performance.JCS is the equal-weight average of SF, SC, and IF; VQ assesses sampled-frame integrity and composition, not motion or audio.Source: §4.2 · p. 7

4.3. Evaluation Results

  • Direct Generation’s composite score spans more than 30 points across backbones, while VIS-Ground’s range is below 6 points, an approximately 82% reduction in variation.Source: Figure 5 · p. 8, §4.3 · p. 10

4.2.1. Evaluation Protocol

  • The protocol uses four complementary baselines and evaluates each method with Omni-1.1-Flash, Veo 3.1, and MiniMax-H3.The baseline set includes Direct Generation, MM-StoryAgent, MovieAgent, and VideoGen-of-Thought; the backbones separate interaction strategy from generator capability.Source: §4.2.1 · p. 8

4.3. Evaluation Results

  • Across SF, SC, IF, and JCS, VIS-Ground improves over the strongest competing result by roughly 10% on average across backbones and settings.Source: §4.3 · p. 8, Figure 6 · p. 9
  • VIS-Ground achieves the highest composite score with every video-generation backbone.Table 1 compares methods under narrative and knowledge grounding across three backbones and six metrics.Source: §4.3 · p. 8, Table 1 · p. 9
  • VIS-Ground retains a similar advantage in narrative and knowledge grounding, which emphasize story progression and external factual accuracy, respectively.Source: §4.3 · p. 9, Figure 6 · p. 9

5. Discussion

VIS-Ground shows a balanced overall composite advantage while case-study and ablation analyses expose how it handles evidence and dependencies. The failure analysis also identifies metric-level tradeoffs and remaining gaps in completed outputs.

5.1. Case Study

  • In the Apollo example, VIS-Ground preserves the museum handoff, includes the Apollo 1 fire and preceding test missions, and maintains a more coherent progression than the described baselines.MM-StoryAgent loses the museum handoff, MovieAgent introduces Apollo 13 too early, and VideoGen-of-Thought has visual discontinuity from the prefix.Source: §5.1 · pp. 10–11, Figure 7 · p. 10

5.2. Ablation Study

  • In narrative grounding, One-Pass Planning and Unstructured State reduce mean SF, SC, and IF by 4.36 and 7.60 percentage points, respectively, versus the full method.The observed reductions are consistent with complementary roles for structured context representation and dependency-guided planning, but do not independently isolate representation, repair, and reasoning-budget effects.Source: §5.2 · p. 11

5.3. Limitation and Failure Analysis

  • VIS-Ground achieves the strongest and most balanced overall composite performance across backbones, though it is not best on every individual IF, VC, and VQ comparison.The method may favor compatibility with prior evidence over literal fulfillment of an ambiguous, aggressive, or conflicting request; visual quality remains bounded by the underlying generator.Source: §5.3 · p. 11
  • In the 72-output failure-analysis sample, 67 score at least 75 on Source Faithfulness and 64 on each of Story Continuity and Interaction Fulfillment, while only 8 and 11 achieve perfect scores on the latter two.This is an analysis subset, not the total benchmark evaluation set.Source: §5.3 · p. 12

6. Conclusion

The conclusion presents VIS-Ground and VIS-Bench for video interactive storytelling, with VIS-Ground abstracting source, video prefix, and request into aligned states and generation constraints. The related-work discussion distinguishes this setting from initial-brief storytelling, prompt-responsive generation, and other forms of constrained generation and benchmarking.

6. Conclusion

  • VIS-Ground abstracts the source, video prefix, and viewer request into aligned states, then induces generation constraints; the conclusion reports improvement across three video backbones.The paper also introduces VIS-Bench for video interactive storytelling.Source: §6 · p. 12

A. Related Work

  • Unlike systems organized around an initial brief, script, or narrative source, VIS-Ground addresses requests made after video generation begins while retaining source faithfulness and prefix consistency.The related work names MM-StoryAgent (Xu et al., 2025), MovieAgent (Wu et al., 2025b), VideoGen-of-Thought (Zheng et al., 2024), Co-Director (Song et al., 2026), and ViMax (Huang et al., 2026) in the initial-brief group.Source: A. Related Work · p. 17
  • Interactive video generation and world models support evolving prompts or actions, while this task additionally uses an external grounding source and the rendered story prefix to plan each request.Examples include CausVid (Yin et al., 2025), StreamDiT (Kodaira et al., 2026), and LongLive (Yang et al., 2025).Source: A. Related Work · p. 17
  • VIS-Ground differs from lexical, musical, and acoustic control methods by deriving constraints from dependencies jointly emerging across the source, rendered prefix, and viewer request.The related-work discussion also connects its repair loop to agentic critique and revision methods, including METAL (Li et al., 2025), VISCO (Wu et al., 2025c), and AgentDebug (Zhu et al., 2025).Source: A. Related Work · p. 17
  • VIS-Bench targets joint source faithfulness, story continuity, and interaction fulfillment in grounded continuations, complementing benchmarks focused on other aspects of video generation.Table 2 compares related methods and benchmarks by whether they explicitly address listed capabilities, using check marks, partial coverage, or dashes.Source: A. Related Work · p. 18, Table 2 · p. 18

B. Benchmark Construction and Data Interface

VIS-Bench pairs grounding passages with interruption-point instructions and visual styles to study narrative- and knowledge-grounded continuations. Its full construction is larger than the fixed subset rendered for the reported evaluation.

B. Benchmark Construction and Data Interface

  • The continuation setting uses two 24-second scenes, each comprising three eight-second shots; illustrative examples are distinguished from empirical results.The appendix describes benchmark construction, grounded continuation, evaluation, and study materials.Source: B. Benchmark Construction and Data… · p. 18

B.1. Source collection and instance counts

  • VIS-Bench draws 250 CoQA source passages: 132 narrative-grounded and 118 knowledge-grounded.Narrative grounding preserves story characters, events, and relations; knowledge grounding preserves source facts and their qualifications.Source: §B.1 · p. 19
  • The benchmark construction includes 1,004 interaction instructions, yielding 2,008 planned instruction–style instances when each instruction is paired with both visual styles.A passage may support multiple instructions, and an instruction may appear in both styles.Source: §B.1 · p. 19
  • Because each configuration requires rendering a 48-second continuation, evaluation uses 16 instances per backbone and five methods, for 240 continuations rather than the full 2,008 instances.For each of three backbones, the sample includes eight narrative- and eight knowledge-grounded instances from distinct source passages.Source: §B.1 · p. 19

B.2. Evaluation setting and inputs

The evaluation instance combines a persistent grounding source, an observed video prefix, and a current instruction, with target requirements reserved for evaluation. The accompanying method material represents context and dependencies, activates candidate-specific constraints, and repairs both scripts and rendered outputs, while noting that the procedure does not guarantee success.

B.2. Evaluation setting and inputs

  • A continuation instance combines a grounding source, observed video prefix, and current instruction; the prefix carries relevant observable history rather than conditioning separately on all earlier instructions.Target facts and continuation requirements are reserved for evaluation, not given to the generator as answer annotations or a reference checklist.Source: §B.2 · p. 19

C.1. Representation and evidence provenance

  • The aligned state model M = (V, H, S, A) represents video, story, source, and cross-context alignments, while grounded predicates retain evidence, temporal scope, and qualifications.The representation distinguishes inferred dependencies from directly observed facts and connects verification decisions to supporting evidence.Source: §C.1 · pp. 19–20

C.2. Candidate-conditioned activation

  • Dependencies become active constraints when applicable to a candidate script; each evaluation is pass, fail, or unknown, and repairs can require recomputing activation.Unknown denotes insufficient evidence, not verified satisfaction.Source: §C.2 · p. 20

C.3. Illustrative dependency trace

  • In the illustrative key-transfer trace, the source supplies the opening prerequisite and the prefix establishes that another character holds the key, so the candidate may need a return event before opening the cabinet.Output verification also checks whether the rendered video actually shows the planned prerequisite and can send missing transitions back for repair.Source: §C.3 · p. 20

C.4. Repair loop and termination

  • The repair loop extracts and aligns context, plans a script, activates and checks constraints, revises, renders, and checks realized relations until verification succeeds or the computation budget is exhausted.The procedure is not a proof of convergence or correctness: extraction may miss evidence, dependencies may be wrong, rendering may fail, and an incompatible request may lack a fully satisfying continuation.Source: §C.4 · p. 21

C.5. Two-scene implementation

The implementation extracts and aligns source facts, prefix observations, and request requirements, then uses bounded plan review and repair. Generation-time quality checks remain separate from final benchmark evaluation.

C.5. Two-scene implementation

  • Source facts, prefix states, and request requirements are extracted and aligned; uncertain observations and unsupported requests are recorded separately.The script validator runs after the initial proposal and each repair, with at most five repairs; reaching that limit does not establish validity.Source: §C.5 · p. 21
  • Plan review supplies the actual prefix and candidate plan, accepts only when required defect lists are empty, and allows at most three reviews and two repairs.The final plan may still be returned with a false pass flag after the review limit.Source: §C.5 · p. 21
  • Shot boundaries and audio are constrained: the first shot continues the prefix, later transitions use continuation or motivated cuts, and speech is limited by scene-level rules.The optional speech audit permits at most three calls; shared configuration caps are upper limits, not measured average usage.Source: §C.5 · p. 21

D.1. Internal quality control and benchmark separation

  • Generation-time review uses public task context and internal checks, not benchmark-private annotations, reference checklists, or final evaluator scores.These internal flags belong to generation; final benchmark assessment is a separate evaluation under a fixed rubric.Source: §D.1 · p. 22

D.2. Metric-specific evidence

The evaluation separates source faithfulness, story continuity, and interaction fulfillment from visual consistency and video quality. It normalizes semantic ratings over fixed applicable checks and reports a joint score alongside its constituent metrics.

D.2. Metric-specific evidence

  • Source faithfulness, story continuity, and interaction fulfillment use distinct checklists, while visual consistency and video quality assess perceptual properties.Rendered-output evidence is required for visible actions; scripts or plans alone do not establish that an action appeared in the video.Source: §D.2 · pp. 22–23, Table 4 · p. 22

D.3. Normalization and judge aggregation

  • Semantic checks use ratings from 0 to 4 and normalize complete ratings over fixed applicable checks; unresolved ratings yield a missing complete score, with bounds available.Two judges rate independently, and their metric scores are averaged; unknown evidence is not treated as zero or a midpoint.Source: §D.3 · p. 23

D.4. Joint score and overall composite

  • The overall composite equally averages setting-level mean JCS, while visual consistency and video quality are reported separately.The arithmetic-mean JCS can let strength in one semantic dimension compensate for weakness in another, so individual semantic metrics remain necessary; method rankings are unchanged under the reported robustness checks.Source: §D.4 · p. 23

E. Baseline Adaptations and Implementation Reporting

The baselines adapt several planning strategies to interactive continuation through a shared rendering interface. Their comparisons and ablations support conclusions about evaluated configurations, with limits on what can be attributed to individual components or computation.

E. Baseline Adaptations and Implementation Reporting

  • The four baseline families span direct generation, coordinated story production, hierarchical planning, and intermediate reasoning, adapted to resume a partially rendered story after intervention.Each adaptation specifies how source, observed prefix, and request enter its pipeline and how continuation attaches to the prefix.Source: E. Baseline Adaptations and… · p. 24
  • Direct Generation inserts context and the requested action into mechanical shot prompts; MM-StoryAgent uses writer–expert exchanges, while MovieAgent and VideoGen-of-Thought use staged planning and refinement.All are translated to a common renderer; some native components are not reproduced.Source: E. Baseline Adaptations and… · p. 24

E.1. Comparison design and interpretation

  • The comparison tests source-grounded, multi-shot continuation from a supplied prefix, not streaming throughput or maximum duration.The compared strategies use the same source, prefix, and request under a shared video-generation interface.Source: §E.1 · p. 24
  • Because adaptations omit some native renderer-specific features, results compare the adapted pipelines rather than original systems with every native component intact.The shared interface separates planning strategy from synthesis backbone.Source: §E.1 · p. 24
  • Unstructured State and One-Pass Planning probe explicit structure and the combined role of candidate-specific constraints and planning, but do not isolate every component’s causal effect.The One-Pass Planning comparison is not compute-matched to test constraint induction alone.Source: §E.1 · pp. 24–25
  • Shared inputs and backbones do not equalize realized token use, calls, retries, or latency, so the results establish neither an efficiency advantage nor a computation-independent gain.The comparisons support claims about evaluated end-to-end configurations.Source: §E.1 · p. 25

F. Formative Study and Human Evaluation

The appendix reports a small formative learning study, a human validation of automatic evaluation, and implementation prompts for context extraction, planning, and review. The study results apply to the analyzed participants and sampled outputs described here.

F.1. Formative study

  • The formative study enrolled nine volunteers; after two attention-test exclusions, seven participants had mean learning gains of 47.9 points with interaction and 32.1 without, a 15.8-point difference.Gains are percentage-point differences; these summaries are not estimates for all nine volunteers, and the experience-rating panel had four respondents.Source: §F.1 · p. 25

F.2. Formative-study interface

  • Figures 10–12 show the pre-test, noninteractive viewing, and interactive question-entry screens; participants could submit questions, and playback paused while they typed.The pre-test offered ten questions without correctness feedback and included an “I don’t know” option.Source: §F.2 · p. 25, Figure 10 · p. 26, Figure 11 · p. 27

F.3. Human validation of automatic evaluation

  • In a validation sample of 50 outputs, human inter-rater agreement was α=0.78–0.86; Gemini agreement was α=0.75–0.83 and Claude agreement was α=0.64–0.76.Three annotators rated each output using the automatic evaluation’s metric definitions and ordinal scale.Source: §F.3 · p. 28, Table 5 · p. 29

F.4. Prompt and implementation availability

  • The prompt manifest distinguishes shared extraction, two-scene planning and review, one-scene implementation, optional audio grounding, and final evaluation; Appendix G reproduces extraction and planning templates.Template instructions specify intended behavior, not guarantees of generated-output correctness.Source: §F.4 · p. 29, G. Prompt Templates · p. 29

G.1–G.4. Context extraction and alignment

  • Extraction prompts build ordered source facts with exact quotations, a conservative prefix ledger, source–prefix alignment, and atomic request requirements mapped to relevant unseen or partially shown facts.Unsupported requests and uncertain observations are separately recorded; facts are partitioned as shown, partially shown, unseen, or uncertain.Source: §G.1 · p. 29, §G.2 · p. 30, §G.3 · p. 30

G.5. Two-scene planning

  • Two-scene planning requires a 48-second continuation in two scenes, with supported facts, immediate prefix continuity, preserved persistent state, and explicit scene boundaries.Each scene is later divided into three 8-second shots; speech must be source-supported and concise.Source: §G.5 · pp. 30–31

G.6. Six-shot expansion

  • Six-shot expansion assigns three 8-second shots to each scene and requires transitions to preserve identities, state, and scene-boundary conditions.The first shot continues the prefix, and each scene’s last shot must reach its specified final state.Source: §G.6 · p. 31

G.7–G.8. Plan review and repair

  • Plan review checks visible actions, transitions, identity and style consistency, and speech fit; repair preserves the fixed six-shot, two-scene structure and listed evidence.Review accepts only when no material defect is found; repairs prohibit invented facts and require executable focal actions.Source: §G.7 · p. 32, §G.8 · p. 32
Loading 2610.09326v1…