PaperlayerView exact revision on arXiv

Source-linked AI summary

More Memory, Better Videos? A Controlled Study of Visual Memory in Long-Video Generation

Yuzhuo Chen, Huafeng Shi, Xinyu Wang, Yulong Xu, Yucheng Wang, Haoqin Hong, Guoxin Zhang, Zehua Ma, Weiming Zhang

arXiv:2610.35001v1cs.CV

TL;DR

Visual memory can preserve the past while conflicting with a story’s present, and existing consistency measures do not resolve when retrieval helps or interferes. VMem-Bench evaluates selected evidence and generated outcomes in two tracks. Retrieval improves recall and historical resemblance but also brings correctness trade-offs, while prompt details can support long-gap success without memory.

  • Problem

    It remains unclear when access to visual history helps generation and when it interferes, including through obsolete appearances, prompt-supplied looks, or forbidden entities.

    Source: §1 · p. 2

  • Method

    VMem-Bench measures selected references on 92 real films and generated outcomes on 30 authored stories using controlled retrieval comparisons and no-memory controls.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Retrieval improves recall and resemblance to earlier appearances but can reduce current-story correctness and entity exclusion, while prompt details can support long-gap success without memory.

    Source: §1 · p. 2, §5 · pp. 8–9

  • Takeaways & Limitations

    Assess what history adds, which temporal state is valid, and what unwanted content accompanies useful evidence; recall or historical resemblance alone does not establish memory utility.

    Source: §5 · p. 9

    The authored Track-B probes do not represent the distribution of movie genres.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Long-video generators increasingly retrieve past visual observations to prevent identity drift. The intuition is compelling: more access to history should make future video more consistent. But an image can be faithful to the past and wrong for the present. Does visual memory always improve the generated story? We investigate this question through VMem-Bench, a two-track diagnostic study. Track-A measures which references are selected from 92 real films (16,548 segments), before generation and without generator confounds. Track-B compares four retrieval policies with no-memory and gold-informed last-seen controls on 30 causally generated stories (2,824 segments) using a shared generator. Retrieval increases entity recall from 0.559 to 0.628-0.693 and improves similarity to past appearances, but decreases precision from 0.977 to 0.915-0.938 and forbidden-entity avoidance from 0.968 to 0.845-0.898. It can also reinforce obsolete appearances after the story changes them. Conversely, high long-gap recall can arise without memory when the prompt itself supplies the appearance; an appearance-stripping analysis exposes this confound. These results distinguish historical fidelity from current-story correctness and show why retrieval relevance or final-video consistency alone does not establish memory utility.

1. Introduction

Visual memory may preserve historical appearances while conflicting with a story’s current state or introducing forbidden entities. VMem-Bench separates reference selection from generated outcomes, finding retrieval benefits alongside costs and prompt-driven long-gap success without memory.

1. Introduction

  • Historical retrieval can anchor returning characters, but a faithful past image may be wrong when the story changes an entity’s state or presence.The motivating cases include a new outfit, a destroyed object, or a deceased person mentioned but not present.Source: §1 · p. 2
  • Existing consistency evaluations leave open when visual history helps generation and when it interferes, including through prompt-supplied looks and now-forbidden entities.The study measures current-state correctness and entity exclusion alongside historical resemblance.Source: §1 · p. 2
  • Retrieval increases recall and historical appearance similarity, but can reduce precision and forbidden-entity avoidance or restore outdated appearances.Long-gap success can also reflect appearance information already supplied by the prompt.Source: §1 · p. 2
  • VMem-Bench tests both selected references and generated story outcomes, combining 92 films with 30 authored stories and controlled retrieval comparisons.Its two tracks measure evidence before generation and outcomes afterward.Source: §1 · p. 2

2. Related Work

Prior work explores visual histories, memory-conditioned generation, and video consistency benchmarks. This study distinguishes selected evidence from generated-story correctness, including temporal validity, forbidden appearances, and prompt-information sensitivity.

2. Related Work

  • Visual-history systems use compressed frame or KV context, retrieval, or entity-indexed state; OneStory and StoryMem learn memory-conditioned generation approaches.The paper holds the generator fixed across retrieval policies to test whether relevance and historical resemblance predict current-story correctness.Source: §2 · p. 2
  • Existing benchmarks assess perceptual quality, movie-level consistency, memory, or narrative consistency, while SubjectAnchor combines subject-aware memory with a trained model and inclusion/exclusion tests.The paper does not establish superiority over SubjectAnchor’s trained architecture.Source: §2 · p. 2
  • The study jointly tests selected evidence, updated states, forbidden appearances, and prompt sensitivity, using no-memory controls and complete-policy comparisons.Because each policy evolves its own history, these comparisons have explicit limits on causal attribution.Source: §2 · pp. 2–3

3. Study Design: Evidence and Outcomes

VMem-Bench measures visual-history utility at two points: selected references before generation and story outcomes afterward. Its causal protocols, complementary datasets, and separate metrics distinguish historical evidence from generated correctness.

3. Study Design: Evidence and Outcomes

  • Track-A scores selected references before the real next segment is revealed, whereas Track-B generates and scores a clip before updating from its own output.The tracks measure complementary boundaries rather than matched instances.Source: §3 · p. 3, Figure 2 · p. 4

3.1. Causal Evaluation Contract

  • Both tracks process stories chronologically without access to future segments, but they differ in what is scored.Track-A scores references; Track-B scores generated clips.Source: §3.1 · p. 3

3.2. Data, Gold, and Memory Stresses

  • Track-A covers 92 films and 16,548 segments; Track-B uses 30 authored stories and 2,824 five-second segments, with memory stresses tagged per segment.Track-B instructions usually omit an entity’s full appearance after its first appearance or a state change, though many props encode looks in their names.Source: §3.2 · p. 3

3.3. Metrics

  • Track-A measures reference precision, recall, repetition, and selection efficiency, while Track-B scores required appearances, precision, forbidden-entity avoidance, and identity across returns.Track-B uses a blinded entity list and does not judge identity by comparing a generated entity with earlier renderings.Source: §3.3 · pp. 3–4
  • Policy-level differences compare each memory policy with no memory across recall, precision, and avoidance; they compare whole policies because each writes its own generated history.Figure 3 shows policy-minus-no-memory effects and paired story-bootstrap intervals; Table 3 reports absolute rates by stress, with a separate different-generator block.Source: §3.3 · p. 4, Figure 3 · p. 5, Table 3 · p. 5

4. Experimental Findings

Across the controlled comparisons, retrieval improves entity recall and historical appearance similarity, but can reduce precision and forbidden-entity avoidance or reinforce outdated states. Track-A measures selected references before generation, while Track-B measures generated outcomes under a shared pipeline.

4.1. Controlled Comparison

  • The study compares four retrieval policies with no-memory and gold-informed last-seen controls, using separate real-film reference scoring and authored-story generation tracks.The 30 authored stories cover all nine stress families but do not represent movie-genre distributions.Source: §4.1 · p. 5

4.3. Finding II: Past Fidelity Can Harm the Present

  • Retrieval raises pooled character/prop recall from 0.559 to 0.628–0.693, while pooled Precision falls from 0.977 to 0.915–0.938 and Avoid-OK from 0.968 to 0.845–0.898.For Seg+DINO-key, the Recall gain is +0.133, alongside a −0.057 Precision change and a −0.114 Avoid-OK change.Source: §4.2 · p. 6, §4.3 · p. 6

4.2. Finding I: Recall and Continuity Improve

  • Retrieval improves short-gap character recall and similarity to prior appearances, but prompt descriptions can explain some long-gap success.For 62 long-gap character returns, removing appearance descriptions lowers no-memory presence from 0.984 to 0.565; Seg+DINO-key then differs from no memory by only +0.016, with a confidence interval spanning zero. Historical crop similarity is 0.35 without memory and 0.49–0.57 with retrieved references, conditional on entities localized in both appearances.Source: §4.2 · p. 6

4.3. Finding II: Past Fidelity Can Harm the Present

  • On 61 changed returns, retrieval moves generated appearances closer to obsolete looks: DINOv3 similarity rises from 0.19 by +0.15–+0.20.The gold-informed last-seen control has pooled Recall/Precision/Avoid-OK of 0.639/0.964/0.941, but is not budget matched; the internal audit traces selected errors across history, retrieval, synthesis, and judging.Source: §4.3 · pp. 6–7, Figure 4 · p. 7

4.4. Finding III: Relevance Is Not Utility

  • Track-A’s Seg+DINO-key has the highest reference recall, but Text-frame has the best F1 on both datasets.Seg+DINO-key recall is 0.923/0.849 and precision 0.579/0.484 on BlenderOpenMovies/LSMDC; Text-frame F1 is 0.817/0.670. F1 rises with diminishing returns as reference caps grow, with Text-frame leading at every budget.Source: §4.4 · p. 7, Figure 5 · p. 7, Table 7 · p. 8
  • Across 425 returns, retrieval raises whole-frame MegaLoc similarity by +0.15 to +0.21, but that embedding measures scene similarity rather than location correctness.Geometry-aware location memory remains an untested hypothesis.Source: §4.4 · p. 7

4.5. Clip Quality Is Not Entity Correctness

  • No quality metric correlates positively with per-clip entity recall above ρ = 0.17 in any condition.No memory has higher subject and background consistency despite lower entity recall, while Seg+DINO-key has higher imaging and aesthetic scores.Source: §4.5 · pp. 7–8, Table 8 · p. 8

4.6. Sensitivity to the Judge and Video Backend

  • Both judges support recall gains and Avoid-OK losses for all four retrieval policies, while precision losses are supported for three; removing unparsable judgments preserves all four policies’ trade-offs.Absolute no-memory Recall is judge-dependent: 0.559 under Qwen3-VL and 0.361 under InternVL3.5. The missing-judgment check excludes 4.6–5.1% of segments per comparison and does not rule out systematic judging errors or nonrandom missingness.Source: §4.6 · p. 8, Table 9 · p. 8

5. Conclusion

The conclusion cautions that more visual memory does not necessarily improve generated videos: history can aid recall while also reintroducing retired entities or outdated states. It recommends assessing what history adds, which state is valid, and what unwanted content accompanies useful evidence.

5. Conclusion

  • Retrieved history improves entity recall and resemblance to earlier appearances, yet can reintroduce retired entities and reinforce obsolete states.Detailed prompts can also make long-gap generation succeed without visual memory.Source: §5 · pp. 8–9
  • Assess what information history adds, which temporal state is valid, and what unwanted content accompanies useful evidence.Recall or historical resemblance alone does not establish memory utility; the paper presents these as assessment principles, not a validated controller or universal memory-capacity law.Source: §5 · p. 9
  • Report memory-related assessments alongside perceptual quality rather than treating final-video consistency alone as evidence of utility.VMem-Bench provides evidence- and generation-stage measurements and an evaluation checklist.Source: §5 · p. 9

LLM Usage Statement

The authors disclose using OpenAI Codex for manuscript and code work, while taking responsibility for the final work and reported results.

LLM Usage Statement

  • OpenAI Codex assisted with manuscript writing and editing, code writing and modification, data summarization, reproducibility checks, and figure and table preparation.The authors take full responsibility for the final manuscript, code, analyses, and reported results.Source: LLM Usage Statement · p. 10

Long-Video Generation

The figure describes how the two tracks’ data and protocols are built and frozen. The appendix roadmap lists where additional materials are located.

Long-Video Generation

  • Track-A gold begins with a VLM draft of each real film, followed by normalization, VLM review, blocking human review, validation, and freezing.Source: Figure 7 · p. 12
  • Track-B stories are frozen only when the prompt stream and memory labels rebuild byte-identically from the human-written semantic skeleton.Source: Figure 7 · p. 12

Appendix

  • The appendix roadmap directs readers to cohort coverage, systems and controls, scoring, additional results, the author audit, and design implications and limitations.Source: Appendix · p. 12

A. Evaluation Lessons and Reader Checklist

Memory evaluation must distinguish what references are selected from what appears in generated clips, and interpret scores against the requested temporal state. The reader checklist emphasizes information contribution, valid appearances, unwanted content, and careful reconstruction.

A. Evaluation Lessons and Reader Checklist

  • Evaluate whether history adds information, matches the requested temporal state, and carries unwanted entities; an old appearance may still be valid for an explicit flashback.A prompt’s textual detail does not by itself establish that visual history is redundant.Source: A. Evaluation Lessons and Reader… · p. 12, Table 10 · p. 13
  • Track-A measures selected references and Track-B rendered appearances; because their stories differ, their absolute scores and story-level associations are not a paired retrieval-to-generation analysis.Studying that relationship requires matched inputs and both reference and output measurements.Source: A. Evaluation Lessons and Reader… · p. 12
  • Similarity to a previous appearance is conditional on detection in both appearances, so missing returns and visibility matter before treating it as unconditional success.After a requested state change, high similarity to an old appearance is ambiguous.Source: A. Evaluation Lessons and Reader… · p. 12
  • For shared-generator retrieval policies, compare against no memory using recall, precision, and Avoid-OK; Last-seen is privileged and variable-budget, while native-generator rows compare whole systems.Higher old-appearance similarity can be misleading after a requested state change.Source: A. Evaluation Lessons and Reader… · p. 12
  • Reconstruction uses frozen analysis data or saved segment scores, while a new rollout is a new trajectory rather than a byte-identical replay; proposed history filters remain hypotheses.The reproduction checklist specifies cohort, indexing, scoring, and snapshot checks.Source: A. Evaluation Lessons and Reader… · pp. 12–13

B. Data Construction and Cohort Coverage

The benchmark combines frozen, human-adjudicated real-film gold with authored causal stories and a common interface for comparing memory systems. Its evaluated cohort and realization setup define important boundaries on comparison and interpretation.

B.1. Benchmark Construction Pipeline (S1–S4)

  • Track-A’s four-stage pipeline normalizes and checks model-proposed annotations, routes blocks and sampled cases to human review, then freezes an accepted gold registry and layout hash.Changing the layout or exclusion set creates a new data version.Source: §B.1 · p. 13

B.2. Anatomy of a Frozen Gold Instance

  • A frozen Track-A instance pairs prompt-complete intents with harness-side typed entity and lifecycle records, while identities and required or forbidden targets remain scoring-only.Its diagnostic tags cover reappearance, multiple instances, scene returns, and state changes.Source: §B.2 · p. 13

B.3. Authored Stories and Cohort Coverage

  • The 30 evaluated authored stories contain 2,824 segments and 181 entities with multiple appearance states; all include nine memory stresses, but tag counts are opportunities, not outcome denominators.The cohort’s varied settings do not establish representativeness of real films.Source: §B.3 · pp. 13–14, Table 11 · p. 15

C.1. Comparison Scope and Neighboring Work

  • Neighboring systems are compared by reported design and evaluation scope, not shared-protocol performance; SubjectAnchor’s matched empirical comparison is not currently reproducible.The study does not claim its observed trade-off applies to every memory architecture.Source: §C.1 · p. 14, Table 12 · p. 15

C.2. Causal Adapter Contract

  • Adapters share frozen gold, causal guards, materialization, and scoring, mapping otherwise incomparable internal memory states to pre-generation references and post-generation outputs.Methods needing future story information or modalities absent from passive film are not assigned surrogate quantitative scores.Source: §C.2 · p. 14, Table 13 · p. 16

C.3. The Four Retrieval Memories

  • Four retrieval memories vary in segment and frame ranking or sampling; Track-B uses each policy’s own generated history in a shared FLUX-to-Wan keyframe-first stack.That realization can pull memory intended for later in a segment into the first frame; the reported retrieval rows did not use the listed alternative Wan backends.Source: §C.3 · pp. 15–16

C.4. Last-Seen Reference Control

  • The Last-seen control selects up to three previously seen required characters or props by longest gap and supplies frames from their latest appearances in that rollout.It is privileged but not an information upper bound: it gives fewer references than retrieval, and its frame may show an earlier incorrect rendering.Source: §C.4 · p. 16

C.5. Released Causal-Memory Systems

  • Released systems retain distinct native mechanisms: LongLive-RAG retrieves latents, MemFlow combines recent and compressed KV memory, and IAMFlow’s Track-A results are only a scalability diagnostic.IAMFlow’s native per-frame archive exceeds host memory on long films.Source: §C.5 · p. 16

C.6. IAMFlow Track-A Scalability

IAMFlow’s native Track-A archive scales with observed video duration and reaches a practical host-memory horizon. The reproduction materials distinguish frozen-artifact checks from regenerating video.

C.6. IAMFlow Track-A Scalability

  • 15-second segments add about 5.3 GiB of host memory; an 800–900 GiB budget gives the native carrier a horizon of roughly 3,000 archived KV frames.The study does not substitute a bounded-KV variant because that would change the memory carrier used for recall.Source: §C.6 · pp. 16–17
  • Causal memory systems do not evict or withhold entries because a story changed its state; their reads and forgetting instead follow relevance, budget eviction, or age-based compression.Explicit entity records are kept only by IAMFlow and non-causal VideoMemory, addressed through entities named in the instruction or shot.Source: Table 14 · p. 17

C.7. Reproduction Checklist

  • Reproduction requires the exact evaluated cohort, fixed conditions, correct gold-to-production indexing, pinned judge settings, retained scores, and the specified paired story bootstrap.Tables should be regenerated from frozen analysis JSON rather than newly scored or partially completed runs.Source: §C.7 · p. 17
  • A metadata audit found 180 manifests and score snapshots covering all 2,824 segments per condition, but it did not regenerate pixels or verify external download availability.The current cohort gold and prompts match the pinned generation checkout byte for byte.Source: §C.7 · p. 17

D.1. Entity Scoring and Statistical Uncertainty

The supplementary analyses clarify how memory selection, identity scoring, prompt information, and evaluation choices shape the measured trade-offs. Retrieval can provide diverse references and improve historical continuity, but reference diversity does not ensure correctness, and long-gap recall can reflect prompt information rather than memory.

D.1. Entity Scoring and Statistical Uncertainty

  • Track-B uses one deterministic judge call per segment, and unparseable outputs count required entities as missed and forbidden entities as absent; this occurs more often for retrieval than no memory.The differences are 91–115 of 2,824 segments for retrieval memories versus 55 for no memory, slightly lowering retrieval recall relative to the control.Source: §D.1 · p. 18

D.2. Identity Measurement and Calibration

  • The presence judge checks each entity against its gold description, so it cannot distinguish a returning character from a description-matching stranger.Two versioned VMem-Bench scorers measure identity directly; the VLM judge can overrate sameness for visible look-alikes, making DINOv3 crop similarity the primary identity measure.Source: §D.2 · p. 19

E.1. Coverage, Precision, and Redundancy

  • Seg+DINO-key has the best selection efficiency (0.431/0.393), lowest appearance similarity RS (0.488/0.393), and near-lowest view repetition RV (0.225/0.143), yet is the least precise retrieval memory.Diverse views are not necessarily correct views.Source: §E.1 · p. 19

E.2. Reference Budget

  • Each retrieval policy hands over more of its own top-ranked references as the budget grows; F1 rises with diminishing returns, and memory ordering holds at every tested budget.This sweep measures reference evidence, not generation utility.Source: §E.2 · pp. 19–20

E.3. Growing History

  • Retrieval stays nearly flat in F1 as archived films grow, while released memories lose F1; that decline mixes lossy storage with reading because they evict or compress entries.The testbed re-ranks the full history; the far bins contain few films.Source: §E.3 · p. 20, Figure 13 · p. 20

F.1. Qualitative Rollouts

  • On the same prompt stream, retrieval restores a recently seen character that no memory replaces with a stranger, but also returns a dead character that no memory keeps out.The probe illustrates both useful recall and unwanted reappearance.Source: §F.1 · p. 20, Figure 14 · p. 21

F.2. Return Gaps and Prompt Information

  • Every condition, including no memory, recalls more entities at long gaps because instructions name returning entities and often describe them after long absences.Memory gains concentrate on recent character returns; references add nothing for described props and hurt for changed characters.Source: §F.2 · p. 20, Table 17 · p. 22

F.4. Effects across Memory Stresses

  • Memory helps recent returns and persistent states but hurts when entities must stay out; per-clip quality differences among retrieval memories are small and do not correlate strongly with recall.The robustness checks retain lower precision and avoidance, while reference-count strata are unmatched and do not isolate a count effect.Source: §F.4 · p. 20, §F.5 · p. 20, §F.6.1 · pp. 21–22

F.6.2. Independent Judge

The robustness analyses test whether the reported trade-offs depend on the judge or video backend, and an audit assigns sampled failures to stages without treating those assignments as causal. The comparisons remain complete-policy evaluations, with distinct histories and limited rollout and story coverage.

F.6.2. Independent Judge

  • Under an independent judge, every retrieval memory retains a positive recall gain (+0.060 to +0.128) and similarity-retrieval costs remain negative for Avoid-OK (−0.015 to −0.054).Precision costs remain negative for three of four memories; smaller last-seen costs vanish, and some per-stress effects are not significant under the second judge.Source: §F.6.2 · p. 22, Table 21 · p. 24

F.6.3. Alternative Video Backends

  • With MiniMax-H3, the FL2VA realization reproduces every main effect’s sign and similar size, while removing the keyframe reproduces similarity-retrieval costs but not the last-seen recall gain.The no-keyframe H3 partly copies references, especially last-seen frames; eight stories are too few to test stale-look effects.Source: §F.6.3 · p. 23, Table 22 · p. 24

G.1. Findings and Interpretation

  • Among sampled required-entity failures, the audit attributes 36% to selection, 29% to writing, 7% to synthesis, and 27% to judge misses.Among 37 resolved forbidden-render units, 26 showed the entity in both references and output; this co-occurrence does not establish that references caused the render.Source: §G.1 · p. 25, Table 23 · p. 26

G.2. Audit Protocol

  • The audit inspected 224 units from an earlier rollout, with a single author; its sample was drawn from judge-flagged failures and controls rather than all generated outputs.The four shared-generator retrieval memories were inspected; released systems expose only final pixels for this kind of attribution.Source: §G.2 · p. 25

G.3. Reliability and Measurement Scope

  • Judge misses concentrate in long-gap and temporal-reference failures, so Track-B recall for these families is conservative for retrieval memories.Twenty-eight of 32 judge-confirmed controls were confirmed, while eight of 224 audit units were unresolved.Source: §G.3 · p. 25

H.1. What the Closed-Loop Comparison Identifies

  • Because each policy writes its own history, Track-B compares complete policies rather than isolating the causal effect of swapping one reference at a fixed generation state.The proposed test branches from a shared prefix and matches prompt, seed, generator state, and reference count for reference-level interventions.Source: §H.1 · p. 26

H.2. Design Hypotheses and Falsification

  • The audit’s inclusion-weighted failure shares are conditional on sampled judge-flagged misses, not population failure frequencies.The proposed design hypotheses remain untested interventions, not established remedies.Source: §H.2 · p. 26

H.3. Limitations and Scope

The study’s conclusions are bounded by its rollout design, judges, and testbed scope. Its results do not establish how memory utility scales with storage capacity or generalizes across all models and story settings.

Internal validity

  • Track-B uses one rollout per story and condition, so story-clustered intervals do not measure variation across generation seeds.Conditions also generate distinct intermediate histories, preventing attribution to an isolated frame swap.Source: §H.3 · p. 26

Measurement validity

  • The pinned judge can miss entities or conflate appearance with identity, while identity similarities condition on visible or detected returns and can reward obsolete appearances.A second judge preserves aggregate directions but changes absolute rates and some fine-grained significance results; failure attribution does not provide a population-level causal partition.Source: §H.3 · pp. 26–27

External validity

  • Track-B covers 30 authored stories and one shared keyframe-plus-video pipeline, while the alternative-backend check covers eight stories; Track-A does not test downstream generation on its exact segments.The results do not establish monotonic effects of storage capacity or reference count, or generalize to all genres, interactive policies, or models.Source: §H.3 · p. 27

Related papers

Loading 2610.35001v1…