PaperlayerView exact revision on arXiv

Source-linked AI summary

The Past Frames the Future: Memory for Autoregressive Video Generation

Harold Haodong Chen, Rongjin Guo, Disen Lan, Wen-Jie Shu, Hongfei Zhang, Hanzhe Hu, Shengtao Yao, Zixin Zhang, Guibin Zhang, Zhefan Rao, Jinxiu Liu, Yexin Liu, Rui Peng, Yuhao Liu, Bin Ren, Shuai Yang, Yukang Chen, Salman Khan, Ying-Cong Chen, Ser-Nam Lim, Rynson W. H. Lau, Nicu Sebe, Yu Cheng, Ming-Hsuan Yang, Qifeng Chen

arXiv:2609.28466v1cs.CV

TL;DR

Autoregressive video rollouts can outlast bounded access to history, while the field lacks a shared taxonomy for memory mechanisms. This survey operationalizes memory and organizes prior work across its forms, functions, operations, learning, and evaluation, then distinguishes the bottlenecks and validation needs that shape reliable memory.

  • Problem

    Autoregressive generation must preserve relevant history beyond bounded local context, but memory mechanisms lack a shared taxonomy and consistent conceptual standards.

    Source: §1 · p. 4

  • Method

    The survey defines memory as persistent historical information that can condition later generation and organizes mechanisms by forms, functions, operations, learning, and evaluation.

    Source: §1 · pp. 5–6

  • Results

    The synthesis distinguishes discarded observations, stale states, inaccessible correct states, and retrieved content that does not affect generation, despite similar visual symptoms.

    Source: §9 · p. 78

  • Takeaways & Limitations

    Evaluation should create a history-query gap and test whether changing relevant past content changes later output, with read-time interventions testing retrieved content's contribution.

    Source: §9 · p. 78

    The survey excludes short-video generation without persistent cross-step memory and covers world-action models selectively when their memory mechanisms inform AR visual generation.

    Source: §1 · p. 6

Abstract

from arXiv · show

Advances in generative models have improved video fidelity, enabling long-horizon generation, interactive world modeling, and evolving visual environments. Autoregressive (AR) video generation extends visual sequences through causal rollouts. However, a fundamental bottleneck emerges: as the generated sequence expands, practical models must operate under strictly bounded context windows, storage, and computational limits. Consequently, critical historical information, e.g., entity identities, dynamic states, and intervention-induced causal changes, often leaves the active context long before its relevance diminishes. Overcoming this limitation and maintaining temporal persistence constitutes a fundamental memory problem. We present a systematic and comprehensive review of memory mechanisms in AR video generation. We formulate memory operationally as persistent historical information maintained across outer AR steps, capable of influencing future generation even after the originating evidence is no longer locally accessible. Building upon this unified framework, we organize the literature through five complementary perspectives: (I) Forms, the representational carriers of history; (II) Functions, the specific semantic and physical information requiring preservation; (III) Operations, the lifecycle of writing, reading, updating, managing, and integrating memory; (IV) Learning, the optimization of memory behaviors under closed-loop rollouts; and (V) Evaluation, the paradigms for diagnosing genuine memory capabilities. We conclude by synthesizing open challenges, including composable and resource-aware memory architectures, trustworthy state updating, self-rollout learning, and standardized evaluation. By bridging representations, mechanisms, and learning paradigms, this paper establishes a structured foundation for developing reliable, memory-conditioned video generation systems.

1. Introduction

Bounded causal rollouts make retention a central challenge: relevant history can leave local context before it stops mattering. This survey defines memory operationally and organizes mechanisms, learning, and evaluation into a shared framework.

1. Introduction

  • Under bounded history, information needed later may leave the immediate context, making selective retention central to temporal persistence.Without effective retention, recursive generation may forget or distort entities, appearances, spatial layouts, motion states, or events.Source: §1 · p. 4
  • Memory is persistent historical information that conditions later generation after its originating evidence is no longer locally accessible.The survey operationalizes memory by whether removing or modifying the state can affect later outputs.Source: §1 · p. 5
  • The field lacks a paradigm-specific taxonomy, while inconsistent terminology makes systematic comparison of memory mechanisms difficult.Functionally related mechanisms may be described as context, cache, history, or state.Source: §1 · p. 5
  • The survey organizes memory by representational forms, preservation functions, and the operations that write, read, update, manage, and integrate it.These categories cover carriers, historical properties to preserve, and the operational lifecycle during sequential rollout.Source: §1 · p. 5
  • It also reviews learning objectives, memory-state distributions, and optimization, distinguishes memory-revealing evaluation from general video measures, and synthesizes open challenges.The survey frames Forms, Functions, and Operations as system characterization, Learning as construction, and Evaluation as evidential validation.Source: §1 · pp. 5–6
  • The survey covers discrete-token and continuous causal AR generation when historical state is explicitly preserved, managed, or transformed, but excludes short-video generation without persistent cross-step memory.World-action models are covered selectively when their memory mechanisms directly inform AR visual generation.Source: §1 · p. 6

2. Background: Why is Memory Needed?

The section frames autoregressive video generation as a causal rollout over visual units and introduces persistent memory as a way to retain historical information beyond bounded local context. It connects this setting to generation paradigms, memory operations, and recurring long-horizon failure modes.

2.1. Autoregressive Video Generation

  • Autoregression is the outer causal rollout over visual units, while the inner process that synthesizes each unit can vary.Units may be tokens, frames, latents, clips, or chunks.Source: §2.1 · pp. 6–7, Figure 2 · p. 7

2.2. Paradigms of Step-wise Generation

  • Discrete token models generate units through next-token prediction, whereas continuous frame- or chunk-wise models synthesize a whole unit; both face long-horizon history constraints.Chunk-wise generation can shorten the outer rollout, but each unit still conditions later steps.Source: §2.2 · pp. 7–8

2.3. Bounded Historical Access

  • Bounded local context limits access to earlier evidence, so open-ended generation relies on managed historical state to approximate full-history conditioning.The constraint also affects interactive generation, where distant interventions may need to continue shaping later outputs.Source: §2.3 · pp. 8–9
  • Earlier state-changing interventions can leave the active window yet still need to constrain later outputs, making causal persistence a memory requirement.Observation-changing controls primarily stress spatial preservation, while state-changing interventions raise causal preservation needs.Source: §2.3 · p. 9

2.4. Memory-Conditioned Generation

  • Memory-conditioned generation adds persistent state to bounded context, reading and integrating history before generation and writing, updating, and managing state afterward.The state functions as memory when it preserves information unavailable in the active context and influences subsequent generation.Source: §2.4 · pp. 9–11

2.5. Failure Modes of Memoryless Rollouts

  • Memoryless rollouts may lose entity identity, appearance, spatial structure, or evolving motion across temporally separated observations.Reported symptoms include entity omission or duplication, appearance drift, incompatible layouts, and degraded dynamics.Source: §2.5 · p. 11
  • Semantic commitments may drift and intervention consequences may reset; these symptoms are not uniquely attributable to missing memory.The survey notes that dynamics modeling, control execution, or rendering can also produce similar failures.Source: §2.5 · pp. 11–13

3. Forms: What Carries Memory?

The section organizes memory by the computational object carrying historical information, distinguishing four carrier families and noting that systems may combine them.

3. Forms: What Carries Memory?

  • The taxonomy distinguishes visual, implicit-state, explicit-state, and adaptive-parametric memory by the object that carries history.The families respectively retain visual evidence, model-native latent states, structured world variables, or sequence-specific adapted parameters.Source: §3 · p. 13
  • Visual, implicit, and explicit carriers are runtime states or stored objects, while adaptive parametric memory places persistence in adapted parameters.A system may combine several carrier types within one generation pipeline.Source: §3 · p. 13

3.1. Visual Memory

Visual memory preserves historical information as visual evidence traceable to past observations, from renderable frames and clips to VAE latents. These carriers trade fidelity and temporal detail against storage, conditioning cost, and the risk of reusing outdated or erroneous evidence.

3.1. Visual Memory

  • Visual memory retains observation-aligned evidence, including RGB frames, clips, or VAE latents, when it remains traceable to specific historical observations.Its defining criterion is traceable correspondence, not human readability.Source: §3.1 · pp. 13–14

3.1.1. Pixel-Space Visual Memory

  • Pixel-space memory offers direct access to high-fidelity evidence; clips preserve short-range motion and temporal order beyond frames, but cost more to store and condition on.Frames support visual anchoring and state recall, while reliable reuse requires avoiding stale or erroneous observations.Source: §3.1.1 · p. 15

3.1.2. VAE-Space Visual Memory

  • VAE-space memory stores observation-aligned latents that can be approximately decoded, reducing storage and conditioning overhead relative to decoded RGB history.Compression can lose details, couple memory to particular representations, and allow retained generated-latent errors to accumulate; systems use carry-over, windowed, and retrieval-based access.Source: §3.1.2 · pp. 15–16

3.2. Implicit State Memory

Implicit state memory carries history in persistent learned neural states without requiring direct observation-level correspondence or a predefined world-state schema. Its forms range from itemized attention caches to consolidated recurrent states and separately encoded history, each balancing access, compactness, and information retention.

3.2. Implicit State Memory

  • Implicit state memory retains rollout history in persistent model-native neural states whose learned semantics condition subsequent generation.It includes attention-cache, recurrent or state-space, and encoded-history carriers; transient activations that disappear within one generation step do not qualify.Source: §3.2 · pp. 16, 18, 19

3.2.1. Attention-Cache States

  • Attention-cache memory preserves fine-grained historical states for later access, while bounded capacity trades historical coverage against cost and risks eviction or reuse of stale evidence.Retrieval, compression, and specialized cache organization can manage capacity, but retained states may remain difficult to access reliably.Source: §3.2.1 · pp. 19–20

3.2.2. Recurrent and State-Space States

  • Recurrent and state-space memory consolidates history into an evolving state that can remain compact as rollouts grow, but rare evidence may be lost or errors propagated.Local-global hybrids retain exact recent evidence alongside compact recurrent memory for more distant history.Source: §3.2.2 · pp. 20–21

3.2.3. Encoded History States

  • Encoded history memory re-encodes historical information into learned representations, ranging from source-aligned features to compressed sequences or fixed memory slots.Compression reduces retention cost but can discard evidence needed later, and usefulness depends on compatibility with the downstream generator.Source: §3.2.3 · p. 21

3.3. Explicit State Memory

Explicit state memory preserves interpretable world-state variables across autoregressive steps, updating them as the world evolves. It covers entity and spatial states, with usefulness bounded by the adequacy and accuracy of the maintained representation.

3.3. Explicit State Memory

  • Explicit state memory maintains persistent variables with declared world-level meanings, which can be updated as the generated world evolves.The carrier is defined by the meaning of its state, not by a particular update mechanism.Source: §3.3 · p. 22

3.3.1. Entity-Centric States

  • Entity-centric memory binds persistent identities to interpretable attributes and time-varying states, preserving object-specific information through occlusion, viewpoint changes, and interactions.Examples include identity, appearance attributes, pose, motion, visibility, contact, and event status [187] [365] [405] [352] [145] [395] [45].Source: §3.3.1 · p. 23
  • Entity-state reliability depends on the declared schema and update correctness, while fine-grained appearance often remains in visual or implicit memory.Restricted schemas may omit relevant properties, and estimation, action-binding, or synchronization errors can produce internally consistent but physically or causally incorrect trajectories.Source: §3.3.1 · p. 23

3.3.2. Spatial and Geometric States

  • Spatial and geometric memory anchors interpretable state to viewpoints, image planes, metric coordinates, or topological structures.Its typed variables may represent pose, depth, occupancy, visibility, or connectivity; opaque learned features remain visual or implicit components.Source: §3.3.2 · p. 23
  • View-aligned 2D fields remain viewpoint-dependent, whereas world-aligned 3D structures consolidate evidence in a shared coordinate frame.2D fields require correspondence or reprojection across views [38] [385]; 3D maps face fusion and dynamic-content challenges, and storage may grow with explored extent [44] [31] [397] [366].Source: §3.3.2 · p. 24
  • Topological layouts compactly represent connectivity and other non-metric spatial relations but cannot directly support viewpoint projection or rendering.Spatial edges belong to this category, while non-spatial event or causal relations are better represented as state-transition records.Source: §3.3.2 · p. 24

3.4. Adaptive Parametric Memory

Adaptive parametric memory preserves experience-dependent information in parameters that persist and influence later generation. Internal memory shares the generator's weights, while modular memory separates an adaptive component, trading closer integration for greater lifecycle control.

3.4. Adaptive Parametric Memory

  • Adaptive parametric memory stores sequence-, instance-, or experience-specific information in adapted parameters that persist and affect subsequent generation.The state need not change at every autoregressive step and may persist between adaptation events.Source: §3.4 · p. 25

3.4.1. Internal Parametric Memory

  • Internal parametric memory places rollout-dependent information in generator weights, using fast-weight recurrence or absorption of historical context into existing weights.LaCT [383] and RAD-TTT [42] propagate adapted weights across chunks, while ISPA [83] modulates attention projection weights to compensate for removed historical KV states.Source: §3.4.1 · pp. 25–26
  • Consolidating history into shared internal weights bounds persistence but makes individual contributions difficult to recover, while stale or erroneous adaptations may continue affecting generation.Their effects persist until the parameter state is revised or reset.Source: §3.4.1 · p. 26

3.4.2. Modular Parametric Memory

  • Modular parametric memory separates experience-dependent parameters from the base generator, making them more independently resettable, storable, replaceable, or reusable.Examples include temporary LoRA modules and dedicated fast-weight memory [117].Source: §3.4.2 · p. 26
  • Modular weights remain difficult to interpret, and multiple parameter memories raise retrieval, composition, interference, and stale-state management questions.Rollout-adaptive modules may also reinforce errors when generated experience is repeatedly absorbed into their parameters.Source: §3.4.2 · p. 26

4. Functions: What Does Memory Preserve?

This section examines what memory must preserve beyond bounded context, distinguishing information that should persist from legitimate changes in the generated world. It covers causal preservation across interventions and coordination among identity, spatial, dynamic, semantic, and causal constraints.

4. Functions: What Does Memory Preserve?

  • Five preservation functions address entity forgetting, layout inconsistency, lost motion, broken semantic commitments, and forgotten consequences of interventions.A carrier can support multiple functions, and a function can be realized jointly by several carriers.Source: §4 · p. 27, Figure 4 · p. 28

4.1. Identity Preservation

  • Identity preservation requires retaining discriminative evidence, associating it with the correct entity, and maintaining that association through absence.Evaluation should test reappearance after a verified information gap, ideally with increasing temporal, viewpoint, or shot separation.Source: §4.1 · p. 29

4.2. Spatial Preservation

  • Spatial preservation makes historical evidence actionable from later viewpoints through geometry-addressed retrieval, persistent scene states, or implicit alignment.Failures can arise from missing coverage, incorrect spatial addressing, corrupted world-state updates, or weak geometric utilization; scene revisitation tests whether spatial information survives an observation gap.Source: §4.2 · p. 33

4.3. Dynamic Preservation

  • Dynamic preservation retains enough motion or latent-process information to continue evolution across gaps, update estimates during absence, and limit recursive prediction errors.Replaying the last visible state is insufficient, and self-rollout errors can reinforce incorrect trajectories; longer or larger memory helps only when retained dynamics remain correct and usable.Source: §4.3 · p. 34

4.4. Semantic Preservation

  • Semantic preservation maintains commitments such as roles, relations, goals, and event states, with correct grounding and selective revision as conditions change.Evaluation should test delayed consistency after supporting evidence leaves local context, rather than relying only on prompt alignment or narrative fluency.Source: §4.4 · p. 35

4.5. Causal Preservation

  • Causal preservation retains intervention-induced state changes after the action and its immediate visual evidence leave bounded context.Later generation must respect what has been changed, avoiding state resets or restoration of obsolete configurations; this matters especially in interactive generation.Source: §4.5 · pp. 35–36
  • Memory must bind each intervention to the affected entity and state, then preserve the resulting post-intervention state and constraints after local evidence disappears.Recording that an action occurred is insufficient: the changed state must remain operative rather than reverting to a plausible canonical configuration.Source: §4.5 · p. 36
  • Causal memory must revise, overwrite, or invalidate earlier states in intervention order, while evaluation tests delayed outputs after removing local evidence of the earlier action.Otherwise, obsolete configurations can compete with newer state, producing oscillation, state averaging, or prior reversion; tests should distinguish state-changing interventions from observation-changing navigation.Source: §4.5 · pp. 36–37

4.6. Coupled Functions

  • Identity, spatial, dynamic, semantic, and causal constraints must remain mutually compatible, though different memory carriers may preserve complementary aspects of history.This includes grounding roles and relations to the intended entity, motion to scene geometry, and interventions to affected entities, regions, and downstream state changes.Source: §4.6 · p. 37
  • Preserving function depends on memory being written, read, updated, managed, and integrated so it is neither stale nor retrieved for the wrong query.Functional claims should specify the temporal or intervention gap, memory carrier and access mechanism, and computational regime demonstrating persistence.Source: §4.6 · pp. 37–38
  • Self Forcing [130], Rolling Forcing [189], Context Forcing [41], and PackForcing [216] support preservation through recursive self-generation but are not additional content functions.They respectively expose generation to self-produced histories, extend supervision across causal rollouts, target long-context consistency, and control historical-context representation.Source: §4.6 · p. 38

5. Operations: How Does Memory Work?

This section examines memory as an active interface across autoregressive rollout, covering how history is written, read, updated, managed, and integrated into the next visual unit. It focuses on how management selects the evidence and fidelity that persist, while integration governs its influence, within resource constraints and without stale or incompatible constraints.

5.1. Memory Writing

  • Writing decides what evidence persists and where it is bound, trading exhaustive coverage against redundancy and contamination, selective efficiency against irreversible omission, and structured access against misbinding.Transition-triggered writing can preserve sparse changes with long-lived consequences, but depends on reliable trigger detection and association.Source: §5.1 · pp. 39–40

5.2. Memory Reading

  • Reading selects and exposes retained evidence for the current prediction through temporal, content-based, semantic, spatial, or state-conditioned access, each balancing relevance and cost against distinct retrieval errors.Temporal access is efficient but recency can miss distant evidence; content, semantic, spatial, and state-based access can fail through ambiguity, binding or correspondence errors, or stale state estimates.Source: §5.2 · pp. 43–44

5.3. Memory Updating

  • Updating transforms admitted evidence into an evolving state through append-only, recurrent, cache, structured, world-state, or corrective mechanisms, balancing stability with responsiveness while risking stale, erased, or consolidated errors.Recurrent consolidation and cache refresh can propagate compressed or incompatible states; structured and world-state updates rely on accurate correspondence and transitions, while corrective revision depends on trustworthy inconsistency detection.Source: §5.3 · pp. 45–46

5.4. Memory Management

  • Management determines what updated information survives and at what fidelity under resource budgets, but future relevance and reliability are uncertain; rolling truncation controls cost while risking loss of needed remote evidence.The supplied part introduces anchor and multi-tier retention as an alternative that reserves capacity for long-range references alongside recent context.Source: §5.4 · p. 47
  • Utility-aware policies [205] [333] [32] [396] retain entries by estimated future value rather than age alone, but locally dispensable evidence may matter after revisitation, delayed interaction, or reappearance.Utility estimates may use attention, novelty, retrieval statistics, uncertainty, diversity, or predicted relevance, and remain inherently uncertain.Source: §5.4 · p. 47
  • Compression and consolidation [211] pool, subsample, quantize, or project older states to extend memory horizon, while risking loss of fine cues, small entities, relations, and rare events.Unlike recurrent updating, compression changes the representation of retained state under a resource budget.Source: §5.4 · pp. 47–48
  • Validity-aware forgetting removes obsolete or unreliable entries; StableWorld [351] does so for geometrically degraded frames, but premature invalidation erases long-range evidence while insufficient forgetting preserves errors.Forgetting controls whether an entry survives, whereas corrective revision changes what the memory state represents.Source: §5.4 · p. 48
  • Memory management balances coverage, fidelity, cost, and validity: truncation loses remote evidence, retention risks stale constraints, compression loses information, and forgetting requires reliable confidence signals.Persistent anchors can protect long-range stability but may suppress legitimate appearance, viewpoint, or world-state changes.Source: §5.4 · pp. 47–48

5.5. Memory Integration

  • Integration converts retrieved memory into an actionable constraint: accessible evidence can fail when weakly injected, incompatible with current representations, or overwhelmed by local context.It must make memory sufficiently influential and reliably accessible while preserving flexibility for legitimate changes in the generated world.Source: §5.5 · pp. 48, 50
  • Concatenation appends retrieved memory to current input [370] [380] [379], while dedicated attention [371] [372] [193] and direct insertion of retrieved transformer states into self-attention [347] [116] [333] provide distinct pathways.Concatenation offers limited utilization control and consumes the context budget; cross-attention effectiveness depends on compatibility and calibration between current queries and historical keys.Source: §5.5 · p. 49
  • Adaptive gating [64] [23], geometry-aligned conditioning [89] [374] [385], and structured semantic controls [232] [199] [405] regulate influence or encode spatial and explicit state.Gating requires calibration; geometry can transfer pose or depth errors into conditions, while semantic state alone may not specify visual appearance.Source: §5.5 · pp. 49–50

6. Learning: How Is Memory Learned?

Memory learning in autoregressive video generation must address a closed loop: generated outputs update memory, and prediction errors can alter later states. The survey organizes learning methods by what is supervised, which memory-state distributions are encountered, and whether training reflects deployment memory mechanisms.

Learning question

  • The learning question is how models acquire reliable memory behavior from data, supervision, and closed-loop interaction.Figure 8 depicts generated outputs being written back into memory and reused as conditions for future generation.Source: §6 · p. 50, Figure 8 · p. 52

Closed-loop learning

  • Prediction errors written into evolving memory can shift later conditioning and compound over extended autoregressive rollouts.Reliable learning must support accurate synthesis and persistent, informative, coherent memory states despite finite context and memory budgets.Source: §6 · pp. 51–52

Rollout formulation

  • A rollout step retrieves relevant memory, integrates it with generation context, generates a visual unit, then writes or updates memory for the next step.The transition may also include memory management; memory can take forms such as visual observations, latent or KV states, retrieval stores, structured states, or adaptive parameters.Source: §6 · p. 51

Learning taxonomy

  • Figure 7 organizes memory learning by supervision objectives, training-time memory-state distributions, and memory-aware learning under deployment mechanisms.These dimensions distinguish what is optimized, which histories are covered, and whether retention, compression, retrieval, or adaptation is reflected in training.Source: Figure 7 · p. 51, §6 · p. 52

6.1. Memory Learning Objectives

Memory learning separates prediction of future visual units from explicit supervision of persistent state content and its evolution. Output losses train synthesis but leave memory-specific requirements underconstrained, motivating state and dynamics objectives.

6.1.1. Output-Level Prediction Supervision

  • Output-level objectives predict future tokens, frames, latents, or chunks, but shape learnable memory only indirectly through its contribution to generated outputs.Discrete models typically use next-token cross-entropy, while continuous generators commonly use denoising or flow-matching objectives.Source: §6.1.1 · p. 53
  • Output losses can dilute distant-memory signals, delay credit assignment, and leave internal memory states underconstrained when immediate predictions are similar.These issues arise because distant information may matter only at sparse events and its consequences may appear many rollout steps later.Source: §6.1.1 · p. 54

6.1.2. Memory State Supervision

  • Memory-state supervision either directly aligns persistent state with step-specific targets or trains generation conditioned on constructed memory states.Targets may come from annotations, simulators, multi-view reconstruction, or pretrained perception models.Source: §6.1.2 · pp. 54–55
  • Explicit world-state targets—including poses, depth, occupancy, semantic maps, object attributes, trajectories, and 3D scene states—ground memory in shared spatial or semantic variables.Structured supervision can be costly or unavailable, and predefined variables may cover only part of what future generation requires.Source: §6.1.2 · p. 55
  • Learned perceptual or geometric features offer an alternative to specified state variables, but their usefulness depends on whether they capture information relevant to future generation.GIM-World aligns memory with frozen 3D foundation-model features across views to encourage view-consistent geometry.Source: §6.1.2 · p. 55
  • State-conditioned methods make memory part of the generation pathway, while direct content targets constrain what the state encodes; neither alone guarantees reliable long-horizon memory.Targets can omit later-needed information, and generators may bypass memory when local context suffices.Source: §6.1.2 · p. 55

6.1.3. Memory Dynamics Supervision

  • Memory-dynamics supervision constrains state evolution across multiple transitions by comparing future states with targets over a specified horizon.The objective can assess discrepancy after projection or decoding into image, latent, semantic, geometric, or feature space.Source: §6.1.3 · p. 56
  • Long-horizon outcome constraints test downstream coherence, whereas transition-path constraints guide evolution more locally, with each approach carrying distinct supervision costs and attribution limits.Outcome signals can be delayed; path-based methods may require extra rollouts, reverse processes, teacher trajectories, or structural assumptions.Source: §6.1.3 · p. 56

6.2. Memory State Distribution

Training memory states can differ from deployment states because training often uses ground-truth histories while inference recursively feeds generated outputs back into memory. The section contrasts teacher forcing, history augmentation, and self-rollout as increasingly direct ways to expose training to deployment-like memory states, with trade-offs in coverage and cost.

6.2. Memory State Distribution

  • Training on ground-truth histories induces memory states different from deployment rollouts, where generated outputs are recursively written into memory and can shift future conditioning.Errors in generated outputs may accumulate through memory updates and move rollouts into states poorly represented in teacher-forced training.Source: §6.2 · p. 57

6.2.1. Teacher Forcing

  • Teacher forcing constructs memory from ground-truth history and enables parallel training, but does not expose the model to errors fed back through persistent memory at inference.This limits exposure to the model-induced state distribution and cannot by itself eliminate memory-state exposure bias.Source: §6.2.1 · pp. 57–58

6.2.2. History Augmentation

  • History augmentation broadens training states through corrupted, masked, selected, temporally perturbed, or model-derived histories, without requiring complete deployment rollouts.Its induced distribution remains an approximation: prescribed perturbations may miss coupled rollout failures, while model-derived histories retain ground-truth initialization or recombination.Source: §6.2.2 · pp. 58–59

6.2.3. Self-Rollout

  • Self-rollout writes the model’s generated outputs into memory during training, directly exposing subsequent predictions to states induced by its own errors.It matches deployment states only when generation, memory updates, and memory access follow deployment procedures; truncated or mismatched rollouts approximate them.Source: §6.2.3 · p. 59
  • Self-rollout entails sequential sampling and activation-memory costs, while truncation or stochastic gradient methods reduce cost at the expense of credit assignment across distant updates.For N chunks with S denoising steps per chunk, the described rollout requires O(NS) sequential denoising-network evaluations.Source: §6.2.3 · pp. 59–60
  • Self-rollout approaches vary in how they package trajectories and propagate supervision through visited states, including rolling windows, corrective trajectories, context gradients, and teacher-derived signals.These methods supervise states induced by recursive student generation rather than ground-truth histories.Source: §6.2.3 · p. 60
  • Self-rollout most directly narrows the training-deployment memory-state gap, but deeper rollouts increase sampling cost, activation memory, and optimization difficulty.Finite horizons, mismatched procedures, and gradient truncation leave residual gaps or limit supervision across distant updates.Source: §6.2.3 · p. 60

6.3. Memory-Aware Learning

Memory-aware learning trains generation under deployment-like memory constraints, addressing the mismatch between rich training histories and limited inference interfaces. The survey organizes this learning around retaining, compressing, retrieving, and adapting historical information, each with distinct resource and reliability trade-offs.

6.3. Memory-Aware Learning

  • Training under deployment-time capacity, representation, and access constraints can align the generator—and, when applicable, its memory policy—with the interface used at inference.The interface changes both the amount and structure of available conditioning information.Source: §6.3 · pp. 60–61

6.3.1. Learning to Retain

  • Retention methods either train with predefined bounded-history policies or learn selection policies, but short-term salience may miss future utility and self-generated states may be unreliable.Examples include sink-based layouts and learned KV selection; policies must balance future utility, reliability, and temporal coverage.Source: §6.3.1 · p. 62

6.3.2. Learning to Compress

  • Compression either reallocates explicit history resolution or consolidates history into a bounded state, trading lower storage or attention costs against loss of future-relevant evidence.Training through the deployment compression pathway helps address the difference between compressed states and original history representations.Source: §6.3.2 · pp. 62–63

6.3.3. Learning to Retrieve

  • Retrieval can use predefined spatial cues or learned query and selection mechanisms, but relevance must complement active memory and remain compatible with the current world state.Geometric relevance alone may miss changes during occlusion or interaction, and learned retrieval depends on training coverage of revisitation, occlusion, and reappearance.Source: §6.3.3 · pp. 63–64

6.3.4. Learning to Adapt

  • Adaptation stores rollout experience in persistent parameter states that affect later generation, rather than supplying memory as an addressable context input.Methods include recurrent fast weights and localized parameter writing; repeated updates can incur compute overhead, interference, forgetting, and error accumulation.Source: §6.3.4 · pp. 64–65

7. Evaluation: How to Measure Memory?

The survey argues that video quality or long rollout duration alone cannot establish memory: a valid test must require information from earlier history that is unavailable in the current condition. It organizes benchmarks, information-gap protocols, target-specific measurements, and attribution controls around that criterion, while identifying gaps in standardization.

7. Evaluation: How to Measure Memory?

  • A memory-revealing evaluation makes the correct output depend on earlier rollout information unavailable from the current condition, verifies the behavior, and rules out non-memory explanations.Long duration alone is insufficient if the current condition still reveals the needed information.Source: §7 · pp. 65–66

7.1. Benchmark Landscape: Levels of Memory Evidence

  • Benchmarks differ in memory evidence: information-gap generation protocols can directly test history dependence, while adjacent understanding tasks and sequence stress tests do not necessarily require history for video generation.Even memory-oriented benchmarks provide direct evidence only when the queried state is unavailable from the current prompt and observation; component ablations add architecture-specific evidence but not model-independent protocols.Source: §7.1 · pp. 66–67

7.2. Memory-Revealing Evaluation Setups

  • The protocol landscape includes a visible-consistency baseline and established entity, scene, and hidden-state gaps, while goal continuation and delayed action effects remain less standardized or proxy directions.Figure 9 organizes protocols by whether the query requires rollout history; listed benchmarks may provide representative proxies or components rather than standardized memory-revealing evaluations.Source: §7.2 · pp. 68–69, Figure 9 · p. 68
  • History-query separation can be varied by recurrence gap, viewpoint displacement, hidden duration, intervening events, or interaction depth, but interpretation requires matched conditions and controlled resource budgets.Separation curves should not automatically be read as memory decay; some reported curves primarily measure rollout stability.Source: §7.2 · p. 69

7.3. Evaluation Targets and Metrics

  • A protocol makes prior history necessary, while the queried state determines the target-specific metric; visual quality gates and entity, spatial, state, and narrative measures address different evaluation targets.No score is memory-specific in isolation, and target-specific measurements do not by themselves attribute an outcome to memory use.Source: §7.3 · p. 70
  • Delayed-action evaluation should verify successful intervention before testing whether its state change persists, but the surveyed components do not yet form a standardized delayed-action memory metric.The action specification must also be insufficient to answer the later query.Source: §7.3 · p. 71

7.4. Validity Controls and Attribution

  • Valid attribution combines paired histories that isolate relevant prior information, prerequisite checks that verify task execution, and component interventions that localize dependence within a particular architecture.Paired histories hold the current observation and query fixed while changing relevant earlier history; component interventions do not substitute for model-independent protocols.Source: §7.4 · pp. 71–73
  • Evaluator reliability is part of validity: temporal sampling and calibration against human judgments or controlled corruptions help detect failures automatic measures may miss.The controls are complementary, and no single one is sufficient on its own.Source: §7.4 · pp. 72–73

8. Future: What Remains Open?

Memory for autoregressive video generation remains fragmented across carriers, operations, learning, and evaluation. The open challenge is to maintain useful, reliable, accessible state under bounded resources and recursive generation.

8. Future: What Remains Open?

  • The research agenda links composable, resource-aware memory and trustworthy updating with self-rollout learning, interactive causal state, and standardized evaluation.Figure 10 presents these as connected directions, alongside benchmarks for retention, attribution, and efficiency.Source: §8 · p. 73, Figure 10 · p. 74

8.1. Unified and Composable Memory

  • Shared identifiers, timestamps, coordinate frames, provenance, confidence, and validity intervals could align memory carriers without forcing one representation.Selective access, cross-carrier checks, revision, replacement, invalidation, and reset should preserve unrelated content; hierarchical composition can retain detailed evidence while consolidating stable history.Source: §8.1 · pp. 73–74

8.2. Resource-Aware Memory

  • Resource-aware memory must estimate downstream utility, since current salience or similarity can discard consequential events while redundant evidence consumes resources.Allocation should coordinate storage, access, retrieval latency, and computation rather than increase memory uniformly with rollout length.Source: §8.2 · pp. 74–75

8.3. Trustworthy Memory Updating

  • Trustworthy updating must judge whether new evidence warrants revising persistent state, since generated errors and overly persistent anchors can both mislead later rollouts.Provenance, confidence, and validity information could support selective, localized, reversible revisions; closed-loop training can test correction without disturbing unrelated commitments.Source: §8.3 · pp. 75–76

8.4. Self-Rollout Learning

  • Self-rollout learning faces a gap between idealized training histories and deployment-induced states, while long-horizon rollouts are costly and hard to stabilize.Curricula, mixed histories, trajectory replay, delayed-state supervision, and interventions on memory operations are proposed to improve coverage and localize failures.Source: §8.4 · p. 76

8.5. Interactive and Causal Memory

  • Action conditioning alone does not ensure that intervention effects remain persistent state after their action and visual evidence leave local context.Future systems should represent intervention-linked transitions explicitly, support controlled revision and branch separation, and test delayed effects after scene revisitation.Source: §8.5 · p. 77

8.6. Comparable Memory Evaluation Protocols

  • Memory evaluation remains strongest for identity recurrence and spatial revisitation, while correction, forgetting, rollback, and delayed intervention effects are underexplored.Comparable protocols should separate retention, state correctness, access, and use, while reporting history-query separation and resource costs.Source: §8.6 · pp. 77–78

9. Conclusion

The conclusion frames memory as history that remains consequential after its evidence leaves active context. It argues that reliable systems require verifiable state maintenance and evaluation that tests whether memory affects later generation.

9. Conclusion

  • Memory failures can arise from discarded information, incorrect or inaccessible state, or retrieved content that does not influence generation.Longer context, larger stores, and aggregate video-quality scores alone do not establish durable memory.Source: §9 · p. 78
  • The immediate priority is verifiable state maintenance: keep historical evidence distinct from revisable world estimates and localize justified updates.Updates should be robust to errors in self-generated histories.Source: §9 · p. 78
  • Evaluation should create a genuine history-query gap and test whether changing relevant past content or retrieved content changes later output.The longer-term objective is state that remains correct, accessible, and consequential through extended rollout, interaction, and revision.Source: §9 · p. 78
Loading 2609.28466v1…