PaperlayerView exact revision on arXiv

Source-linked AI summary

Complementary Retrieval-Augmented Prompting for Consistent Long-Form Video Generation

Xianghan Wei, Xiaoda Yang, Zhi Wang, An Pan, Daoan Zhang, Huayi Zhang, Yan Zhang, Wei Xu, Zishun Liao, Jianwen Lou

arXiv:2609.37407v1cs.CV

TL;DR

Long-form video generation must keep recurring characters, objects, and scenes consistent across independently generated shots, but selecting useful historical references and constructing prompts remains challenging. Complementary Retrieval-Augmented Prompting organizes visual elements and keyframes to retrieve complementary references, and EntityBench evaluations report improved cross-shot consistency over training-based methods and explicit workflow baselines.

  • Problem

    Selecting useful historical references for independently generated shots remains challenging because references can be redundant or fail to cover all target elements.

    Source: Introduction · p. 1, Element-Aware Complementary Retrieval · p. 4

  • Method

    The framework tracks script-derived visual elements and annotated keyframes, retrieves compact complementary references for target elements, and grounds their roles in a structured prompt.

    Source: Introduction · pp. 1–2

  • Results

    On EntityBench, the method achieved the best Overall score of 0.8670 and led all three LLM-judged categories: character consistency (0.8930), object consistency (0.8780), and scene consistency (0.8510).

    Source: Results · p. 7

  • Takeaways & Limitations

    Explicit intermediate states make generation failures diagnosable and support pre-generation verification and human-in-the-loop editing.

    Source: System Properties: Interpretability… · p. 6

    Optional external reference assets, such as initial character sheets, are not used in the algorithm or experiments.

    Source: §3 · p. 3

Abstract

from arXiv · show

While recent video foundation models excel at generating high-quality short videos, long-form video generation remains a critical challenge, where a major bottleneck lies in conditioning independently generated shots to preserve consistent characters, scenes, and objects throughout a story. Existing training-free approaches typically condition target shots using retrieved historical visuals. However, these references often suffer from severe informational mismatch, either introducing irrelevant contextual redundancy or failing to provide the full combination of required elements for the target shot. To resolve this, we present Complementary Retrieval-Augmented Prompting, an agentic framework that strategically aggregates a compact set of mutually supportive historical references to achieve complete and targeted conditioning for long-form video generation without retraining or modifying the underlying generator. Specifically, our framework explicitly models the visual elements required by each target shot by parsing the narrative script into a text-grounded visual element registry that tracks characters, objects, scenes, and their shot-level states. A VLM-annotated keyframe library further maps these elements to past visual observations. Guided by the required elements, our agent retrieves complementary references that maximize target-element coverage while minimizing historical noise. Finally, the retrieved references, structured element states, and grounding instructions are assembled into a unified prompt for the frozen video generator. This element-aware process provides comprehensive conditioning while remaining fully interpretable. Quantitative and qualitative evaluations on multi-shot story generation demonstrate that our method consistently outperforms recent-frame, memory-based, and entity-level retrieval baselines in cross-shot consistency and text-controllability.

Introduction

The paper frames long-form video generation as an interpretable retrieval and prompting problem: independently generated shots need references that preserve required visual elements while suppressing stale or conflicting content. It proposes an online, training-free framework that updates structured visual evidence as generation proceeds.

Introduction

  • Long-form shot-by-shot generation can cause recurring characters, objects, and scene layouts to drift, while selecting useful references and constructing prompts remains largely manual.The paper notes that training-based consistency methods can incur computational overhead and lack scalability, while existing agentic or explicit-memory pipelines impose design constraints.Source: Introduction · p. 1
  • Unlike approaches requiring full-script analysis, canonical entity references, or manually designed prompt layouts, the framework supports incremental and interactive story creation.Its registry and keyframe library update after each generated shot, without requiring a complete script or fixed generation schedule in advance.Source: Introduction · pp. 1–2
  • The framework maintains a script-grounded registry of shot-level visual element states and a VLM-annotated keyframe library, then retrieves compact references that jointly cover target elements while penalizing conflicting content.It assembles the selected frames into a structured multimodal prompt explaining how each image should be used or avoided by the frozen generator.Source: Introduction · pp. 1–2
  • Each selected frame is explainable through the elements it covers and the risks it may introduce; historical retrieval handles long-range consistency, while proximal references and boundary smoothing can handle local continuity.The framework grounds frames with both element-level and holistic guidance in a structured multimodal prompt.Source: Introduction · p. 2
  • On EntityBench, the framework improves cross-shot consistency over training-based methods and explicit workflow baselines.The introduction describes the approach as training-free and online, maintaining editable visual evidence without requiring the full script in advance.Source: Introduction · p. 2

Related Work

Prior work addresses long-form video consistency through trained or model-dependent mechanisms and through agentic, entity-centric workflows, but these approaches can constrain portability, interpretability, or incremental creation. The proposed framework instead uses annotated keyframes and complementary retrieval to provide interpretable, model-agnostic conditioning for frozen generators.

Training-based Long-Form Video Generation

Explicit and Agentic Long-Form Generation

  • Agentic pipelines decompose scripts, plan shots, allocate context, and use visual feedback; entity-centric systems track recurring characters, objects, and scenes.Examples include agentic pipelines (Lin et al. 2024; Zheng et al. 2024; Wu, Zhu, and Shou 2025; Huang et al. 2026; Liu et al. 2026) and entity-centric systems (Zhou et al. 2026; Yin et al. 2026).Source: Explicit and Agentic Long-Form… · p. 2
  • Seedance, Kling-Omni, Step-Video, and Vidu support multimodal prompting, but long-form workflows still rely heavily on manual reference selection and prompting.GroundShot (Lai et al. 2026) analyzes the full script and schedules entity-reference shots first, but requires a predefined complete generation task.Source: Explicit and Agentic Long-Form… · p. 2
  • The proposed framework supports incremental creation without a complete script or predefined order, retrieving complementary VLM-annotated keyframes under a limited reference budget.It explicitly grounds each frame's use while suppressing obsolete or conflicting content, providing an interpretable, model-agnostic interface for frozen multimodal video generators.Source: Explicit and Agentic Long-Form… · p. 2

Method

Complementary Retrieval-Augmented Prompting uses inference-stage historical conditioning to help independently generated shots preserve consistency across characters, objects, and scenes. It selects a compact set of references to maximize coverage of target-shot elements while minimizing redundancy.

Overview

  • The method targets long-form generation in which each shot is generated independently by a frozen short-video generator with no memory across shots.The narrative script is decomposed into a sequence of T shots.Source: Overview · p. 2
  • Complementary Retrieval-Augmented Prompting orchestrates historical conditioning through an agent that tracks structured visual evidence and retrieves complementary references.The framework is designed to operate at inference time rather than relying on passive, recency-biased retrieval.Source: Overview · p. 2
  • Reference selection is formulated to choose historical frames that collectively maximize coverage of required target-shot elements while minimizing informational redundancy.The framework is described as comprising four sequential modules, whose specifics are not included in the supplied passage.Source: Overview · p. 2

1. Narrative-Grounded Element Registry:

The framework parses the narrative script into a structured registry of required visual elements and their shot-level states.

1. Narrative-Grounded Element Registry:

  • The registry tracks required elements and their states for each shot by parsing the narrative script.Source: §1 · p. 2

2. Element-Linked Keyframe Library:

The keyframe library uses VLM annotations to link historical visual observations with narrative elements, creating a diverse pool of candidate references.

2. Element-Linked Keyframe Library:

  • A VLM annotates and indexes representative historical keyframes, establishing explicit cross-modal mappings for a high-diversity candidate reference pool.Source: §2 · p. 2

3. Element-Aware Complementary Retrieval:

The framework converts each shot’s narrative requirements into explicit element constraints, retrieves complementary historical keyframes, and combines them with grounded instructions for a frozen video generator. Its structured intermediate artifacts support diagnosis, verification, and human editing.

3. Element-Aware Complementary Retrieval:

  • For each shot, the specification includes a text prompt and duration, while a cut indicator distinguishes new scenes from shots continuing the preceding shot.Optional external reference assets can anchor identity, but the paper states they are not used in its algorithm or experiments.Source: §3 · p. 3

Narrative-Grounded Element Registry

  • A narrative-grounded registry tracks reusable scenes, characters, and objects, assigning each tracked element a shot-level state of REF, EXC, OPT, or NEW.REF indicates an element should appear, EXC that it should be absent, and OPT that its status is uncertain; newly introduced elements are marked NEW and bypass retrieval because they have no prior observations.Source: Narrative-Grounded Element Registry · p. 3

Element-Linked Keyframe Library

  • The element-linked keyframe library maps historical frames to tracked elements, preserving a diverse candidate pool for complementary matching.It filters degraded frames with HPSv3, retains locally distinct frames using CLIP image-feature similarity, and has a VLM annotate element presence, location, reference fidelity, and holistic frame description.Source: Element-Linked Keyframe Library · p. 4

Element-Aware Complementary Retrieval

  • Complementary retrieval selects a compact set of references that covers required elements while penalizing excluded elements and reducing redundant context.Greedy rescoring prioritizes uncovered required elements; only fully identifiable elements count as covered, allowing later selections to supply clearer references. An LLM then generates guidance for interpreting each selected frame.Source: Element-Aware Complementary Retrieval · pp. 4–5

Unified Prompt Assembly and Video Generation

  • The unified prompt combines shot-level element states and per-reference guidance to preserve relevant visual content while suppressing obsolete or conflicting context.Its five blocks specify the current task, prefix story context, element states, reference guidance, and global constraints; continuous shots also use the preceding clip as a video prefix, with RIFE-interpolated boundary frames to reduce flicker.Source: Unified Prompt Assembly and Video… · pp. 5–6

System Properties: Interpretability and Editability

  • The framework exposes element states, retrieved references, coverage, prompts, and annotations as interpretable artifacts for diagnosis, pre-generation checking, and human editing.Users can modify the registry, constraint vectors, or retrieved references within an iteratively refinable generation loop.Source: System Properties: Interpretability… · p. 6

Experiments

The experiments evaluate cross-shot consistency on EntityBench and find that the proposed method leads the benchmark and its ablations. Ablations and qualitative examples show the roles of complementary retrieval and grounded prompting in preserving recurring visual elements.

Benchmark and Metrics

  • Cross-Shot Consistency evaluates recurring characters, objects, and scenes across shots with 21 DINOv2- and LLM-based metrics; its 35-episode test set uses 480P, five-second shots.The test set comprises 20 Easy, 10 Mid, and 5 Hard episodes. The StoryMem reproduction preserves its memory-construction and keyframe-selection procedures while replacing the video generator with Seedance 2.0.Source: Benchmark and Metrics · pp. 6–7

Results

  • The method achieves the best Overall score, 0.8670, and leads LLM-judged character (0.8930), object (0.8780), and scene (0.8510) consistency.Across methods, training-based approaches are competitive on DINOv2 similarity, while explicit workflows generally score better on fine-grained LLM judgments; with the same Seedance 2.0 backend, historical-keyframe approaches outperform generated-reference approaches across all four categories.Source: Table 1 · p. 7, Results · p. 7

Ablation Study

  • Full ranks first across all four criteria; removing coverage, prompting, or element-aware retrieval lowers Overall by 0.0188, 0.0179, and 0.0214, respectively.Removing prompting also lowers character consistency by 0.0417, while No Reference reduces Overall to 0.5456; CLIP Retrieval removes element planning, complementary coverage, and reference grounding together.Source: Table 2 · p. 7, Ablation Study · p. 7

Qualitative Analysis

  • Qualitative examples show that complementary retrieval and grounded prompting reduce redundant references, suppress obsolete scene content, and preserve character, outfit, and scene states.Figures 2–3 illustrate coverage and grounding; Figures 4–5 show outfit inconsistency, character confusion, and clothing drift in comparison methods or ablations, while the full method maintains the intended states.Source: Qualitative Analysis · pp. 7–8

Conclusion

Complementary Retrieval-Augmented Prompting uses structured visual evidence and complementary references to condition frozen short-video generators for consistent long-form video generation. EntityBench experiments show superior cross-shot consistency, while explicit representations support interpretable, editable workflows.

Conclusion

  • The framework combines a text-grounded visual element registry and VLM-annotated keyframe library with coverage-aware retrieval and grounded prompts that specify frame contributions and suppress stale content.It turns generated history into structured, queryable evidence rather than implicit memory or manually curated references.Source: Conclusion · p. 8
  • Explicit intermediate representations make reference decisions interpretable and editable, support interactive creation without a complete script, and enable backend portability and stage-level failure tracing.Source: Conclusion · p. 8
  • Limitations include element-planning and visual-annotation errors, imperfect generator prompt adherence, and commercial API constraints; future work targets verification, longer narratives, and transfer across backends and interactive production.Source: Conclusion · p. 8
  • On EntityBench Hard_5, the method preserves marked character, outfit, and background states where VideoMemory, ViMax, and StoryMem show inconsistencies.The comparison uses Seedance 2.0 for all methods.Source: Conclusion · p. 8, Figure 4 · p. 9
  • Ablations show that removing complementary coverage or structured prompting, or using CLIP retrieval, causes character or outfit inconsistencies that the complete pipeline avoids.On EntityBench Easy_2, the complete pipeline maintains Priya’s intended character and Old Quarter scene states.Source: Conclusion · p. 8, Figure 5 · p. 9

Related papers

Loading 2609.37407v1…