PaperlayerView exact revision on arXiv

Source-linked AI summary

MosaiChunk: Compositing Spatio-Temporal Memory for Autoregressive Video Generation

Yiwen Zhang, Haocheng Xi, Michael Tian-Yue Liu, Alexei A. Efros, Hadar Averbuch-Elor, Qianqian Wang, Haiwen Feng

arXiv:2610.02153v1cs.CVcs.GR

TL;DR

Sliding-window generation evicts older KV, while whole-chunk retrieval can include many entries when reusable visual content occupies only a small subset. MosaiChunk composes selected historical KV sections across space and time using a lightweight router while keeping the video generator frozen. MosaiChunk raises median CLIP similarity from 0.841 to 0.936 on T2V and 0.807 to 0.863 on I2V versus the whole-chunk retrieval and conditioning baseline after 180° turns away and back.

  • Problem

    Sliding-window generation evicts older KV, while whole-chunk retrieval can include many entries when reusable visual content occupies only a small subset.

    Source: §1 · p. 2

  • Method

    MosaiChunk composes selected historical KV sections across space and time using a lightweight router while keeping the video generator frozen.

    Source: §1 · p. 2

  • Results

    MosaiChunk raises median CLIP similarity from 0.841 to 0.936 on T2V and 0.807 to 0.863 on I2V versus the whole-chunk retrieval and conditioning baseline after 180° turns away and back.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results motivate future video models that build memory into the generator so it can reuse visual content after it leaves active context.

    Source: §6 · p. 10

    Because the pretrained backbone remains fixed, the method inherits limitations of the backbone and source data, including potential biases and harmful behaviors.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Long-horizon autoregressive video generation is limited by a finite context window. When an object or scene falls out of context, its fine-grained visual details may be lost and difficult to recover upon reappearance. To retain access to such visual details, we introduce MosaiChunk, a spatio-temporal memory mechanism that composes a mosaic of selected historical key-value (KV) entries across space and time. Our approach is motivated by the observation that a frozen video generator can directly consume such non-contiguous historical KV and recover the corresponding visual content. We therefore keep the generator fixed and learn only a lightweight router that determines which historical sections to include in the mosaic under a fixed active-memory budget. We further introduce RememBench, a benchmark of long-horizon revisits with prompt-driven text-to-video (T2V) and camera-driven image-to-video (I2V) splits. Our experiments show that MosaiChunk consistently improves revisit consistency over both sliding-window inference and whole-chunk retrieval under matched memory budgets, across both T2V and I2V settings.

1 INTRODUCTION

Long-horizon generation can lose access to visual details after sliding-window eviction, while whole-chunk retrieval may include many irrelevant KV entries. MosaiChunk composes selected historical KV sections under a fixed memory budget, and its reported revisit results improve over whole-chunk retrieval in both T2V and I2V.

Motivation

  • Sliding-window inference evicts older KV, so a returning object may be synthesized with a different appearance after leaving the active context.Keeping all history active would grow both the KV cache and attention cost, while retrieved chunks may contain many entries unrelated to the reusable visual content.Source: §1 · p. 2

Contribution

  • MosaiChunk assembles historical KV from different spatial regions and times, keeping the generator fixed while training a lightweight router to select relevant sections.The mechanism is motivated by the observation that a frozen generator can consume non-contiguous historical KV and recover corresponding visual content.Source: §1 · p. 2
  • Self-distillation trains the router using richer oracle memory for the teacher and a smaller routed memory for the student, without rendered or recorded target videos.Both use the same frozen backbone and noisy latent, differing only in far memory.Source: §1 · p. 2

Evaluation

  • MosaiChunk improves median revisit CLIP similarity over whole-chunk retrieval from 0.841 to 0.936 on T2V and from 0.807 to 0.863 on I2V.The comparison is reported after the camera turns 180° away and back.Source: §1 · p. 2

Contribution

  • RememBench evaluates long-horizon revisits through prompt-driven T2V and camera-driven I2V splits, alongside contributions in compositional memory and self-distillation.The benchmark tests whether returning content preserves details established earlier in the rollout.Source: §1 · p. 2

2 RELATED WORK

Prior work spans autoregressive video generation, visual memory and KV-cache management, retrieval-based conditioning, and reuse of visual fragments. MosaiChunk positions itself as retrieving and composing fine-grained sections of original KV under a fixed active-cache budget.

Autoregressive video generation

  • Autoregressive video methods develop streaming and long-horizon generation, but sliding-window inference still evicts historical KV needed for revisits.The cited approaches include causal generation, self-generated histories, temporal and view-wise autoregression, historical-representation supervision, and longer rollouts with prompt transitions.Source: §2 · p. 3

Long-term visual memory

  • Visual-memory work retrieves frames or organizes past observations, while native-KV cache methods reduce storage and attention costs through compression and selection techniques.The passage groups prior memory representations as geometric, compressed, or learned, and KV methods as quantization, token selection, merging, and compaction.Source: §2 · p. 3

Retrieval-based methods

  • Retrieval-based methods select historical frames, chunks, or blocks, sometimes jointly optimizing selection and the generator; MosaiChunk instead retrieves content-based sections of original KV with a frozen backbone.Its selector is trained using denoising predictions under oracle context, without rendered ground-truth target videos.Source: §2 · p. 3

Representing visual content with reusable components

  • Earlier image and video methods reuse patches, frames, regions, epitomes, or local-feature statistics; MosaiChunk organizes historical KV into reusable content-based sections.The passage situates this design within a broader perspective of decomposing visual content into reusable components.Source: §2 · p. 3

3 METHOD

The method first demonstrates that a frozen generator can reuse non-contiguous historical KV, then builds a router to retrieve and compose such entries under a fixed active-cache budget. The router is trained by matching the predictions of a student with composed memory to those of the same frozen backbone with richer historical context.

3.1 REUSING NON-CONTIGUOUS HISTORICAL KV FOR VISUAL MEMORY

  • Supplying selected non-contiguous KV for the cookie region alongside the sliding window preserves its pink, star-shaped appearance when the tin reopens.This intervention shows that a frozen autoregressive generator can reuse its own cached KV after those entries leave the active context.Source: §3.1 · p. 4

3.2 COMPOSING HISTORICAL KV WITH MosaiChunk

  • The router partitions chunks into sections, encodes them as descriptors, queries history outside the sliding window, then selects and composes the highest-scoring sections into a MosaiChunk.Sections are groups of unrotated KV entries; the composed memory is supplied alongside the sliding window to the frozen DiT.Source: §3.2 · p. 5
  • Global top-N retrieval ranks candidate sections together, with the far-memory budget determining N; selected sections’ values are scaled by normalized softmax weights before their KV is concatenated.A larger N corresponds to a larger MosaiChunk budget and broader coverage of history.Source: §3.2 · p. 5
  • Self-distillation matches the student’s denoising velocity to the teacher’s prediction, using the same frozen DiT, noisy latent, and sliding window but different far memory.The teacher receives whole historical chunks covering the content to revisit, while the student receives the router’s smaller MosaiChunk; gradients update the descriptor encoder, not the backbone or stored KV.Source: §3.2 · pp. 5–6, Figure 4 · p. 5
  • Because teacher and student differ only in far memory, matching predictions supervises memory selection and weighting and encourages descriptors to favor relevant content over recency.The teacher’s whole chunks come from a much earlier visit to the same content.Source: §3.2 · p. 6

4 BENCHMARKS

RememBench targets long-horizon revisits where the content’s establishing frames have left the sliding window, addressing a gap in existing video benchmarks. It includes prompt-driven T2V and camera-driven I2V splits.

Benchmark design

  • RememBench evaluates consistency across revisits after the frames establishing target appearance have been evicted from the sliding window.The benchmark has prompt-driven T2V and camera-motion-driven I2V splits.Source: §4 · p. 6

The T2V split

  • The T2V split contains 100 samples with scenarios disjoint from router training and extends an appear–disappear–reappear prompt design to four segments.A separate segment keeps the object out of sight.Source: §4 · p. 6

The I2V split

  • The I2V split contains 150 DL3DV scenes, with 50 indoor and 100 outdoor, and uses camera trajectories not seen during training.All scenes include in-place rotations at 90°, 180°, and 360°; outdoor scenes also include rotation with translation.Source: §4 · p. 6

5 RESULTS

On RememBench, MosaiChunk improves revisit consistency over sliding-window Base and whole-chunk MoC; ablations favor learned descriptors and global selection.

5.1 EVALUATION PROTOCOL

  • Evaluation compares MosaiChunk with sliding-window Base and whole-chunk MoC on frozen H3-AR T2V and LingBot-World-Infinity I2V, at one- and two-chunk budgets.CLIP and LPIPS assess revisit consistency, while TempSSIM and Drift measure rollout quality; results are medians over scenes.Source: §5 · p. 6, §5.1 · pp. 6–7

5.3 IMAGE-TO-VIDEO RESULTS

  • At one chunk, median CLIP gains over Base are 0.072 under rotation and 0.060 with translation; LPIPS improves in all four settings.Figure 6 illustrates recovery of the garden entrance’s archway and the playroom’s layout and wall decorations.Source: Table 2 · p. 7, §5.3 · p. 8, Figure 6 · p. 9
  • On WBench and WorldMark, MosaiChunk improves every consistency score over sliding-window Base at both budgets while maintaining video quality.Table 3 covers the WBench navigation split and WorldMark first-person splits.Source: Table 3 · p. 8, §5.3 · p. 8

5.2 TEXT-TO-VIDEO RESULTS

  • 0.144 and 0.185 median CLIP gains over Base, and 0.067 and 0.095 over MoC, at one- and two-chunk budgets show MosaiChunk’s T2V advantage.Figure 5 shows MosaiChunk preserving the vegetables and their arrangement, and the cupboard’s contents, at a two-chunk budget.Source: Figure 5 · p. 7, §5.2 · p. 8

5.4 ABLATION STUDIES

  • Learned descriptors outperform mean-pooled keys, and global top-N improves both metrics at both budgets, making the full router the best-performing variant.The ablation compares descriptor learning and shared-budget global selection on 150 I2V rotation scenes.Source: §5.4 · pp. 8–9, Table 4 · p. 9

6 CONCLUSION

The conclusion presents MosaiChunk and RememBench, notes limits inherited from the fixed pretrained backbone, and points to native generator memory as future work.

6 CONCLUSION

  • MosaiChunk composes historical visual memory, while RememBench evaluates long-horizon revisits; self-distillation trains its router with richer historical context as teacher input.The conclusion suggests future video models could learn to reuse earlier generated content directly in the generator.Source: §6 · p. 10
  • Because the backbone remains fixed, MosaiChunk inherits its limitations in instruction following and learned generative distribution.The authors also note potential biases and harmful behaviors inherited from the pretrained backbone and source data.Source: §6 · p. 10, ETHICS STATEMENT · p. 10
  • Generative AI tools assisted with data construction, code implementation and debugging, results interpretation, and manuscript preparation; an author manually reviewed the outputs.The authors retain responsibility for the reported methods, results, and conclusions.Source: AI USE STATEMENT · p. 10
  • The authors say code, data, and trained router checkpoints will be publicly released to support reproduction and further research.Supplementary materials describe implementation, training, benchmark construction, and generation settings.Source: REPRODUCIBILITY STATEMENT · p. 10

Supplementary Material

The supplementary material points readers to interactive T2V and I2V visualizations and groups supporting material into implementation, benchmark, and results sections.

Supplementary Material

  • The Video Viewer provides interactive visualizations of text-to-video and image-to-video results.The document also provides implementation details, benchmark materials, and additional results.Source: Supplementary Material · p. 14
  • Section A contains implementation details.The passage identifies this as one part of the document’s supplementary material.Source: Supplementary Material · p. 14
  • Section B describes benchmark construction, scene visualizations, and evaluation protocols; Section C presents additional results.These sections complement the interactive visualizations in the Video Viewer.Source: Supplementary Material · p. 14

A IMPLEMENTATION DETAILS

MosaiChunk keeps the video generator fixed and adds a trainable router that composes selected historical KV sections as far memory. The section also details how the router is trained and how RememBench is configured.

A IMPLEMENTATION DETAILS

  • MosaiChunk adds a trainable router to a frozen generator, which reads the composed memory alongside a sliding window and attention sink.H3-AR uses text tokens as its sink; LingBot-World-Infinity uses the first video chunk’s tokens.Source: A IMPLEMENTATION DETAILS · p. 14, §A.1 · p. 15

A.1 MEMORY ROUTER

  • Balanced k-means partitions chunks into equal-size KV sections; pooled keys form descriptors, while verbatim history is stored on CPU.The same partition applies across layers, and sections may span non-contiguous positions and multiple frames.Source: §A.1 · pp. 14–15, Table 5 · p. 14
  • The latest chunk’s sections query historical sections outside the sliding window and sink, with centered, normalized descriptors scored by cosine similarity.The scoring favors sections matching the current query while complementing content missing from the sliding window.Source: §A.1 · p. 15
  • The router selects the highest-scoring sections within the far-memory budget, weights their values by normalized softmax scores, and concatenates their KV.Selected tokens retain spatial coordinates and receive adjusted temporal RoPE coordinates; the frozen DiT reads the result as far memory.Source: §A.1 · p. 15

A.2 SELF-DISTILLATION

  • Self-distillation trains the descriptor encoder to match a teacher’s predictions from richer far memory while teacher and student share the frozen DiT and other context.For T2V, the teacher retains the initial reveal chunks; for I2V, it retains up to three chunks nearest the current camera pose.Source: §A.2 · pp. 15–16
  • Only the router is optimized: gradients do not pass through discrete section selection, stored KV, or backbone parameters, and evaluation disables training-time candidate exploration.Training uses generated rollouts rather than ground-truth target videos; RememBench uses T2V and I2V settings distinct from the router’s training setup.Source: §A.2 · p. 16, Table 6 · p. 16, B BENCHMARK DETAILS · p. 17

B.1 DATASET CONSTRUCTION

RememBench evaluates prompt-driven T2V and camera-driven I2V revisits using screened scenes and controlled inputs. Its protocol selects departure–revisit pairs and applies consistent consistency and rollout-quality metrics.

B.1 DATASET CONSTRUCTION

  • T2V prompts stage an object’s appearance, disappearance, continued absence, and return, with the out-of-sight interval exceeding the largest sliding-window baseline’s recent context.One rollout per prompt is screened for compliance; 100 of 466 scenes are retained, with fixed scenes and noise seeds across methods and budgets.Source: §B.1 · p. 17
  • I2V uses 150 DL3DV scenes—50 indoor and 100 outdoor—with scene prompts and camera motion supplied separately; outdoor scenes also support translation trajectories.All scenes have 90°, 180°, and 360° rotation settings; outdoor scenes additionally combine rotation with translation.Source: §B.1 · pp. 17–18

B.2 VISUALIZATION OF FIRST FRAMES

  • The two splits illustrate diverse content through randomly sampled T2V reveal frames and I2V conditioning frames.Sixteen scenes are sampled at random from each split.Source: §B.2 · p. 18, Figure 8 · p. 19

B.3 DETAILED EVALUATION PROTOCOL

  • Comparisons match prompts and noise seeds for T2V, and conditioning frames and camera trajectories for I2V, while matching active KV budgets.T2V departure frames are shared across methods but revisit frames are marked per rollout; I2V revisit frames are selected from reconstructed camera poses, and CLIP, LPIPS, TempSSIM, and Drift assess consistency and rollout quality.Source: §B.3 · pp. 18–19, Table 9 · p. 20

C.1 EXTRA QUALITATIVE RESULTS

Additional qualitative examples show T2V and I2V revisits with intermediate rollout frames. Methods share scene inputs, so the displayed rows differ in the memory supplied to the backbone.

C.1 EXTRA QUALITATIVE RESULTS

  • The T2V examples show departure frames, sampled intermediate rollout frames, and marked revisit frames at a two-chunk far-memory budget.The rows compare Base, MoC, and MosaiChunk under shared prompts and noise seeds.Source: Figure 10 · p. 21, §C.1 · pp. 21–22
  • Within each scene, rows differ only in the memory supplied to the backbone.The displayed order is Base, MoC, then MosaiChunk.Source: §C.1 · pp. 21–22
  • The I2V examples show the conditioning frame, sampled camera-turn frames, and selected revisit frames at a one-chunk far-memory budget.Methods share the conditioning frame and input camera trajectory; insets schematically show the commanded rotation.Source: §C.1 · pp. 21–22, Figure 11 · p. 22

C.2 EXTRA QUANTITATIVE RESULTS

Across the reported evaluations, MosaiChunk improves several consistency measures, but gains are not uniform across metrics or camera trajectories. Full rotations remain challenging, and relative method ordering varies across settings.

C.2 EXTRA QUANTITATIVE RESULTS

  • MosaiChunk improves every consistency metric reported in Table 3 at both budgets and also improves video-quality scores.The evaluations include relatively small camera rotations that can preserve substantial overlap with earlier views; Base already achieves strong consistency scores.Source: §C.2 · p. 23
  • MosaiChunk has higher Background, Spatial, Gated Spatial, Perspective, Subject, and Geometric scores on WBench at both budgets.Segment is lower at two chunks, and Photometric is slightly lower at that budget.Source: §C.2 · p. 23, Table 10 · p. 24
  • WorldMark results show improved Global and Revisit Memory at both budgets, alongside higher Aesthetic and Perceptual scores.Local Memory is slightly lower, distinguishing improved recall from local temporal stability.Source: §C.2 · p. 23, Table 11 · p. 24
  • The 90° and 360° camera-trajectory results qualify the 180° findings: scores are similar at 90°, while 360° revisit consistency is lower and method ordering varies.The reported comparison covers rotation scenes and scenes combining rotation with translation.Source: §C.2 · p. 24, Table 12 · p. 25

C.3 EXTRA ABLATIONS

The extra ablations test pose-guided retrieval and varying far-memory budgets, while KV transplantation tests whether selected historical content transfers between frozen rollouts.

C.3 EXTRA ABLATIONS

  • For per-query selection, the descriptor encoder is retrained to choose historical sections independently for each latest-chunk query section; one or two sections per query define one- or two-chunk budgets.Source: §C.3 · p. 25
  • Pose guidance improves both CLIP and LPIPS over MosaiChunk in every setting; MosaiChunk-Pose has similar CLIP to WorldKV, with slightly higher scores at both two-chunk budgets.Figure 12 illustrates preservation of a background building’s staircase, though it occupies only a small part of the frame.Source: §C.3 · p. 25, Table 13 · p. 26, Figure 12 · p. 26
  • 0.899 vs. 0.841 CLIP: one-chunk T2V MosaiChunk exceeds two-chunk MoC; 0.815 vs. 0.807 CLIP: half-chunk I2V does likewise.Both MosaiChunk and MosaiChunk-Pose improve as the far-memory budget grows.Source: §C.3 · p. 26, Figure 13 · p. 27

C.4 KV TRANSPLANTATION

  • Swapping manually selected cookie KV between frozen rollouts transfers each cookie to the other tin while retaining each original tin.Figure 14 arranges tins by row and cookies by column; diagonal images are source frames and off-diagonal images show swapped results.Source: Figure 14 · p. 27, §C.4 · p. 27
Loading 2610.02153v1…