PaperlayerView exact revision on arXiv

Source-linked AI summary

CineMem: Attention-Aligned Fine-grained Memory Retrieval for Multi-Shot Long Video Generation

Xiaoxue Wu, Xinyu Gao, Fang Yu, Yu Qiao, Xinyuan Chen

arXiv:2610.17926v1cs.CVcs.AI

TL;DR

Autoregressive multi-shot video generation needs long-range memory to maintain continuity across shot transitions, but existing retrieval is often frame-level and shared across query tokens. CineMem uses attention-informed online clustering and query-adaptive retrieval; experiments report stronger consistency than autoregressive baselines and competitiveness with large-scale bidirectional models. Its reported gains concern long-range consistency, particularly in revisitation scenarios.

  • Problem

    Existing frame-level retrieval shares historical context across query tokens, despite heterogeneous memory needs, while multi-shot generation requires continuity across shot transitions.

    Source: §1 · p. 2

  • Method

    CineMem analyzes attention patterns and combines semantic online clustering of historical KV-cache tokens with query- and head-wise cluster retrieval.

    Source: §1 · pp. 2–3

  • Results

    CineMem achieves superior consistency compared with existing autoregressive baselines, remains competitive with large-scale bidirectional models, and improves long-range consistency in revisitation scenarios.

    Source: §1 · p. 3, §5 · p. 11

  • Takeaways & Limitations

    The results highlight fine-grained memory modeling as important for autoregressive video generation.

    Source: §5 · p. 11

    Experiments use a relatively small autoregressive backbone and limited adaptation data, and generated-video quality may lag behind larger state-of-the-art diffusion models.

    Source: F LIMITATION AND FAILURE CASE · p. 33

Abstract

from arXiv · show

Autoregressive video generation relies on memory retrieval to model long-range dependencies. This capability becomes particularly critical for multi-shot generation, where models must reconcile substantial visual changes across shot transitions with persistent semantic continuity. Existing methods tend to perform memory retrieval at the frame level and share the same historical context across all query tokens, overlooking the heterogeneous information needs and leading to redundant retrieval and inefficient use of historical memory. To bridge this gap, we introduce CineMem, an attention-aligned fine-grained memory retrieval framework that combines online memory organization with query-adaptive retrieval for autoregressive multi-shot video generation. Specifically, guided by intrinsic memory-access patterns revealed through attention analysis, we propose an online-clustered KV cache as a memory organization strategy over all historical tokens, enabling semantic-aware memory grouping and more precise retrieval. In addition, we introduce a query-level retrieval mechanism that provides broader access to historical information under a fixed retrieval budget, thereby improving long-term memory utilization. Together, these designs enable the multi-shot video generator to preserve long-range consistency across shot transitions, particularly in revisitation scenarios, while approximating full-memory access under a constrained memory budget. To facilitate evaluation, we further introduce a suite of task-specific, VLM-assisted metrics for faithful inter-shot consistency assessment. Extensive experiments demonstrate the effectiveness of our framework.

1 INTRODUCTION

CineMem targets the coarse, shared frame-level memory routing used in autoregressive video generation, where query tokens can have heterogeneous historical-memory needs. It combines attention-informed online clustering with query-adaptive retrieval to improve long-range consistency across shot transitions, particularly when earlier content reappears.

1 INTRODUCTION

  • Multi-shot generation requires narrative continuity across substantial viewpoint changes, including recovery of previously seen subjects, scenes, or viewpoints after intervening shots.The paper calls recovery after multiple shot transitions “revisitation.”Source: §1 · p. 2
  • Existing memory routing is often frame-level or temporally fixed, sharing retrieval decisions across query tokens and underusing historical information with heterogeneous needs.Sliding windows discard distant information, while fixed anchors and frame-level retrieval do not fully exploit historical memory.Source: §1 · p. 2
  • Attention analysis finds semantic clustering among historical tokens and substantial variation in retrieval preferences across query tokens and attention heads.The analysis uses an autoregressive adaptation framework that lets each current chunk attend to the full generation history.Source: §1 · p. 2
  • Online clustering groups historical KV-cache tokens by semantic similarity, making content-aware clusters retrieval units rather than fixed frame groups or individual tokens.This design reduces redundancy from retrieving entire frames while avoiding the computational cost of treating every token as a separate memory unit.Source: §1 · pp. 2–3
  • Query- and head-wise retrieval selects relevant clusters under a fixed per-query budget, broadening historical access without increasing retrieval cost.Together, the two designs balance retrieval granularity, historical coverage, and computational efficiency.Source: §1 · p. 3
  • CineMem achieves superior consistency to existing autoregressive baselines, remains competitive with large-scale bidirectional models, and is effective as a plug-in without additional fine-tuning.The evaluation protocol includes video-level quality, intra-shot consistency, and VLM-assisted inter-shot consistency.Source: §1 · p. 3

2 RELATED WORK

Prior work spans multi-shot video generation and autoregressive memory routing, but the cited approaches leave fine-grained visual-memory retrieval for multi-shot generation underexplored. CineMem studies that setting, distinct from fine-grained retrieval aimed at single-shot video or text-domain inference acceleration.

Autoregressive Video Generation and Memory

  • Autoregressive video methods use fixed anchors, sliding windows, or coarse retrieval [Ji et al., 2025; Hu et al., 2026], while OminiMem's fine-grained head-adaptive retrieval primarily targets single-shot generation [Zhao et al., 2026].Fine-grained KV-cache organization in language models has mainly targeted text-domain inference acceleration [Liu et al., 2025; 2026c; Zhu et al., 2025].Source: §2 · pp. 3–4
  • CineMem instead investigates fine-grained retrieval over visual memory to preserve long-range consistency in autoregressive multi-shot video generation.This contrasts with text-domain inference acceleration and single-shot-focused retrieval.Source: §2 · pp. 3–4

Multi-Shot Video Generation

3 METHOD

CineMem aligns memory organization and retrieval with observed attention patterns for long-horizon autoregressive multi-shot generation. Its method adapts a bidirectional model, analyzes full-history memory selection, then combines online KV clustering with query- and head-adaptive retrieval.

3.1 PRELIMINARY

  • Flow matching supplies velocity-field generation, while autoregressive diffusion conditions each chunk on prior history and permits prompt changes across multi-shot transitions.Shot-level masking restricts each shot to the global description and its corresponding caption.Source: §3.1 · p. 4, §3.2 · p. 5

3.2 LONG-HORIZON AUTOREGRESSIVE ADAPTATION

  • Chunk-rollout adaptation converts a bidirectional model to long-horizon autoregressive generation; noisy resampled context simulates generated history to reduce train-inference discrepancy and stabilize rollout.The model is fine-tuned on 15-second multi-shot videos.Source: §3.2 · pp. 4–5

3.3 MEMORY SELECTION IN ATTENTION MODULE

  • Full-history analysis finds semantically coherent cross-frame token clusters and heterogeneous query-head preferences; selecting the top 30% per pair collectively covers nearly the full memory.The figure describes semantically meaningful clustering and broad coverage from the union of query-head selections.Source: §3.3 · pp. 5–6, Figure 4 · p. 6

3.4 ONLINE CLUSTERED KV CACHE

  • Online clustering groups historical tokens by cosine similarity into per-head retrieval units, retains first and latest chunks as anchors, and balances clusters through splitting, merging, load penalties, and assignment consistency.The resulting context-aware organization permits cross-frame token associations.Source: §3.4 · p. 6

3.5 QUERY-LEVEL RETRIEVAL

  • Each query-head pair selects relevant clusters from full history; combined with anchors, this expands effective access under the same budget and approximates full-memory attention.Training constructs KV caches from resampled ground-truth frames under low-level noise, matching the autoregressive adaptation setting.Source: §3.5 · pp. 6–7

4 EXPERIMENT

The experiments evaluate CineMem’s efficiency, streaming multi-shot video quality, and retrieval design. Results report strong inter-shot consistency, and ablations support the contributions of resampled-context training, clustered memory, and fine-grained retrieval.

4.1 IMPLEMENTATION DETAILS

  • Implementation uses a 45K-video multi-shot dataset and retrieves 28 groups per query, with target and maximum cluster sizes of 1,024 and 1,200 tokens.The model is based on Wan2.1-T2V-1.3B; the dataset is constructed from raw movie footage.Source: §4.1 · p. 7
  • CineMem reduces computation versus full-history attention and has lower self-attention cost than frame-level retrieval despite online-clustering overhead.The reported FLOPs include self-attention and retrieval-related computation.Source: §4.1 · p. 7, Table 1 · p. 9

4.2 EVALUATION AND RESULTS

  • Qualitative comparisons show CineMem following prescribed transitions and maintaining cross-shot consistency, including revisitation scenarios.The comparison includes representative causal and bidirectional baselines on 15-second multi-shot videos.Source: Figure 5 · p. 8, §4.2 · p. 9
  • CineMem achieves the best PTCS performance and surpasses larger bidirectional models with substantially more parameters.PTCS jointly considers unchanged elements and intended changes during shot transitions.Source: §4.2 · pp. 9–10, Table 2 · p. 10

4.3 ABLATION STUDY

  • Resampled-history training outperforms standard teacher forcing across metrics, especially in long-horizon consistency.The strategy makes training conditions more consistent with autoregressive inference.Source: §4.3 · p. 10
  • Query-wise retrieval improves performance with both frame-level and clustered memory, while clustering further improves transition quality and inter-shot consistency over frame grouping.Without additional training, the retrieval strategy performs comparably to full-memory attention and achieves higher inter-shot consistency under a constrained memory budget; query grouping remains slightly inferior to fully query-wise retrieval.Source: §4.3 · pp. 10–11

5 CONCLUSION

CineMem combines clustered KV-cache organization with query-adaptive retrieval for autoregressive multi-shot video generation. The paper reports improved long-range consistency in revisitation scenarios and analyzes memory-access patterns to motivate fine-grained retrieval.

5 CONCLUSION

  • CineMem combines an online clustered KV cache with query-adaptive retrieval to improve long-range consistency in revisitation scenarios.The conclusion frames fine-grained memory modeling as important for autoregressive video generation.Source: §5 · p. 11

A DETAILS OF MEMORY-ACCESS PATTERN

  • The appendix analyzes heterogeneous query-head memory preferences and retrieval overlap with full-memory attention during autoregressive rollout.These analyses provide further evidence for the memory-access patterns motivating the retrieval design.Source: A DETAILS OF MEMORY-ACCESS PATTERN · p. 16

A.1 HETEROGENEOUS QUERY AND HEAD MEMORY PREFERENCES

Memory-access analyses find heterogeneous selection across queries and attention heads, motivating independent fine-grained retrieval. Pre-RoPE clustering preserves semantic grouping more clearly, while online-clustered retrieval selectively approximates full-memory access.

A.1 HETEROGENEOUS QUERY AND HEAD MEMORY PREFERENCES

  • Selecting the top 30% of memory tokens per query-head pair covers nearly all historical memory frames.The visualization divides each frame into four spatial memory units and shows selection preferences across query-head pairs.Source: §A.1 · p. 16
  • Different queries select distinct local regions within the same historical frame, while different attention heads show substantially varied memory preferences.The query visualization groups neighboring 3 × 3 spatial tokens into memory units.Source: §A.1 · p. 16, Figure 7 · p. 17, Figure 8 · p. 18
  • Query-head-wise retrieval achieves the largest memory coverage, while head-wise selection shows faster coverage growth and lower overlap than query-wise selection.The comparison contrasts query-head-wise, query-wise, and head-wise selection granularities.Source: §A.1 · pp. 16, 19, Figure 9 · p. 19

A.2 PRE-ROPE CLUSTERING VS. POST-ROPE CLUSTERING

  • RoPE strengthens positional similarity and weakens semantic clustering, supporting clustering and retrieval in the Pre-RoPE space.Post-RoPE organization may become biased toward positional grouping and resemble frame-level partitioning.Source: Figure 10 · p. 19, §A.2 · p. 20

A.3 RETRIEVAL ALIGNMENT WITH FULL-MEMORY ATTENTION

  • Online-clustered retrieval has higher precision than frame-level grouping and sliding-window retrieval across almost the entire threshold range, and higher recall at low-to-moderate thresholds.At very high thresholds, recall is slightly lower than fixed-budget baselines because the clustered cache retrieves a compact set of relevant clusters.Source: Figure 11 · p. 20, §A.3 · p. 21
  • Overall, CineMem provides a more selective approximation to full-memory attention while preserving access to highly relevant historical information.The reported comparison characterizes this as a better precision-recall trade-off.Source: Figure 11 · p. 20, §A.3 · p. 21

B DETAILS OF METHODOLOGY

The methodology details an online-clustered KV cache and query-adaptive retrieval, organizing historical memory into semantic groups and routing memory access for each query-head pair.

B.1 ONLINE CLUSTERED KV CACHE

  • Historical memory is clustered independently for each transformer block and attention head, with clusters representing key-value tokens through assignments, counts, accumulated keys, and centroids.The cache is also maintained separately for each classifier-free guidance branch.Source: §B.1 · p. 21
  • As rollout grows, the cache adjusts its cluster count, splits oversized clusters, merges compatible clusters, and reorders tokens so each cluster is contiguous for retrieval.The target cluster size guides dynamic maintenance; split clusters exceed Smax = 1200, and merges must remain within that maximum.Source: §B.1 · p. 23

B.2 QUERY-ADAPTIVE RETRIEVAL

  • For each query position and head, normalized queries score cluster centroids and select the highest-scoring clusters independently.Routing uses pre-RoPE queries and centroids; positional encoding is applied to retrieved keys for final attention.Source: §B.2 · pp. 23–24
  • When history fits within the retrieval budget, the model uses full historical attention; otherwise, query-specific clusters supplement retained boundary context and are integrated into attention normalization.Attention runs over the current chunk and recent memory in one branch, and gathered historical tokens in another.Source: §B.2 · p. 24

B.3 TRAINING WITH MEMORY RETRIEVAL MECHANISM

Training uses sequential chunk-level rollout to preserve causal cache evolution, while resampled low-noise predictions expose later chunks to accumulated modeling errors.

B.3 TRAINING WITH MEMORY RETRIEVAL MECHANISM

  • Chunks are processed sequentially, and the cache is updated only after each chunk so retrieval uses progressively accumulated history as it does during inference.The current prediction therefore cannot retrieve its own future key-value representations.Source: §B.3 · p. 24
  • After the supervised loss, a one-step low-noise prediction estimates the clean chunk, whose detached key-value representations are added to the cache for subsequent chunks.The estimate is forwarded at t = 0, then the cache is updated using the inference-time clustering and layout procedures.Source: §B.3 · p. 25
  • Resampled context introduces accumulated modeling errors into later memory states, while gradients do not pass through previous chunks, cluster assignments, or the rollout.The current loss forward pass remains differentiable, including selected-cluster attention.Source: §B.3 · pp. 24–25

C ANALYSIS OF EFFICIENCY

The efficiency analysis compares full-history attention with frame-group and online-clustered retrieval in theoretical cost and practical rollout measurements.

C ANALYSIS OF EFFICIENCY

  • The theoretical comparison uses a 15-second Wan2.1-T2V-1.3B setting with 21 autoregressive rollout steps and three latent frames per chunk; frame-group retrieval selects 20 latent frames.Full-history attention exposes all historical memory, whereas frame-group retrieval shares a fixed set of retrieved frames across queries.Source: C ANALYSIS OF EFFICIENCY · p. 26
  • Both retrieval strategies substantially reduce theoretical self-attention cost versus full-history attention, and online-clustered retrieval has slightly lower attention cost than frame-group retrieval.Clustering adds refinement and maintenance overhead, but retrieval overhead remains small relative to self-attention computation.Source: C ANALYSIS OF EFFICIENCY · p. 26
  • Retrieval-based strategies maintain more stable denoising time than full-history attention, while CineMem’s latency overhead over frame-group retrieval remains limited and their peak memory growth is nearly identical.The practical measurements cover a 42-chunk rollout on four H200 GPUs; the figure reports per-chunk denoising time and peak allocated GPU memory.Source: C ANALYSIS OF EFFICIENCY · p. 26, Figure 12 · p. 27

D DETAILS OF EVALUATION

The evaluation covers video-level, intra-shot, and inter-shot properties, with PTCS assessing both semantic preservation and prompt-specified transition changes. The section also discusses PTCS validity and human-study results.

D.1 IMPLEMENTATION OF METRICS

  • Video-level evaluation measures visual quality, text-video alignment, and transition controllability, while intra-shot evaluation measures feature consistency between frames within each shot.The stated tools include LAION aesthetic score, ViCLIP-based similarity, TransNetV2-based transition evaluation, and DINO frame features.Source: §D.1 · p. 27
  • PTCS combines continuity preservation across applicable semantic categories with alignment to intended changes, penalizing missing transitions and unrequested semantic changes.Its components are PCP and PTA, combined as a geometric mean.Source: §D.1 · pp. 27–28

D.2 THE EFFECTIVENESS OF PROMPT-CONDITIONED TRANSITION-CONSISTENCY SCORE

  • Similarity-based inter-shot metrics can score visually similar shots highly despite failed transitions, whereas examples show PTCS reflecting preservation violations, missing transitions, and unexpected changes.The authors present PTCS as addressing this limitation by considering both preservation and prompt-specified changes.Source: §D.2 · p. 28
  • In a 30-prompt user study, PTCS trends aligned with human ratings, and CineMem had the highest inter-shot consistency MOS and competitive overall video quality.Participants rated videos from 1 to 5 on overall quality, intra-shot consistency, and inter-shot consistency.Source: §D.2 · p. 28

E ADDITIONAL RESULTS

E.1 QUALITATIVE RESULTS OF ABLATION STUDY

The ablation results favor fine-grained retrieval over frame-level grouping or shared retrieval, with training further improving visual consistency and detail preservation. The section also presents query-group retrieval as an alternative and illustrates how PTCS assesses transition correctness.

E.1 QUALITATIVE RESULTS OF ABLATION STUDY

  • Training-free CineMem outperforms frame-level grouping and shared retrieval, while training further improves visual consistency and detail preservation.The qualitative ablation shows consistency after multiple shot transitions.Source: §E.1 · pp. 28–29
  • PTCS better reflects transition correctness than similarity-based metrics by considering semantic preservation alongside prompt-specified changes.Figure 13 provides qualitative examples comparing the metrics.Source: Figure 13 · p. 29
  • Table 4 reports user-study ratings from three perspectives on a 1–5 scale.The supplied caption does not give the rating values.Source: Table 4 · p. 30
  • Query-wise retrieval selects memory independently for each token, whereas query-group retrieval recursively groups queries by content diversity and shares memory among similar preferences.Figure 14 presents this alternative retrieval design.Source: Figure 14 · p. 30

E.2 FINE-GRAINED ANALYSIS OF GENERATION QUALITY

CineMem performs strongly across content categories, shot counts, and revisit distances, with the highest PTCS across shot-count groups and at the longest revisit range. Additional qualitative results show coherent generation on 30-second videos.

E.2 FINE-GRAINED ANALYSIS OF GENERATION QUALITY

  • CineMem achieves the best or second-best PTCS in five of six content categories, with the highest scores on Objects and Still Life and Solo Activities.It remains competitive in conversations, multiple-entity scenarios, and animals.Source: §E.2 · p. 31, Table 5 · p. 32
  • CineMem achieves the highest PTCS across all shot-count groups, including prompts with six to seven shots.It also has the strongest transition control and intra-shot consistency across these groups.Source: §E.2 · p. 31, Table 6 · p. 32

E.3 LONGER VIDEO GENERATION

  • Qualitative results on 30-second videos show continued autoregressive generation with coherent visual content and consistency across multiple shot transitions.The authors describe these results as indicating potential for generation over extended durations.Source: §E.3 · pp. 32–33

E.2 FINE-GRAINED ANALYSIS OF GENERATION QUALITY

  • CineMem achieves the highest PTCS at the longest revisit range, 13–18 chunks, and maintains strong performance across revisit distances.Revisit distance is the maximum number of chunks between two revisited appearances in a prompt.Source: §E.2 · p. 31, Table 7 · p. 33

F LIMITATION AND FAILURE CASE

The authors identify limited model scale and training data as constraints, and complex multi-entity transitions as a remaining challenge. Future directions include improved memory organization, entity-aware retrieval, and longer generation horizons.

F LIMITATION AND FAILURE CASE

  • Limited computational resources constrain experiments to a relatively small autoregressive backbone and limited adaptation data, and generated-video quality may lag larger diffusion models.The authors expect the approach could benefit from larger-scale models and datasets.Source: F LIMITATION AND FAILURE CASE · p. 33
  • Complex multi-entity transitions remain challenging even for full-memory autoregressive generation or bidirectional models because they require reasoning beyond historical retrieval.The described failure includes missing characters and inconsistent details when a later shot requires a different character count and coordinated viewpoint changes.Source: F LIMITATION AND FAILURE CASE · p. 33, Figure 18 · p. 34

G FUTURE WORK

  • Future work could explore improved clustering and shared query retrieval to improve retrieval quality or efficiency.The query-group variant is cited as an example of an alternative to fully query-wise and head-wise retrieval.Source: G FUTURE WORK · pp. 33, 35
  • Entity-aware or structure-aware memory retrieval is proposed as a direction for addressing challenging multi-entity scenarios, alongside extending generation to longer horizons.These are identified as future directions for autoregressive video generation.Source: G FUTURE WORK · p. 35

Related papers

Loading 2610.17926v1…