PaperlayerView exact revision on arXiv

Source-linked AI summary

Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrieval

Jiwen Yu, Jianhong Bai, Yiran Qin, Quande Liu, Xintao Wang, Pengfei Wan, Di Zhang, Xihui Liu

arXiv:2506.03141v2cs.CV

TL;DR

Existing interactive long-video methods struggle to maintain scene-consistent memory because they use limited historical context. Context-as-Memory stores and directly conditions on historical frames, while Memory Retrieval selects relevant frames through camera-pose FOV overlap. The method reports superior memory capabilities against SOTAs and effective generalization to unseen open-domain scenarios.

  • Problem

    Existing approaches struggle to maintain scene-consistent memory in long video generation because they use limited historical context.

  • Method

    Context-as-Memory stores generated frames as memory, concatenates historical context with predicted frames, and retrieves relevant frames using camera-pose FOV overlap.

  • Results

    Context-as-Memory achieves superior memory capabilities compared with SOTAs and generalizes effectively to unseen open-domain scenarios.

  • Takeaways & Limitations

    Historical generated frames can serve as memory for scene-consistent interactive long video generation within the paper’s evaluated scope.

  • Takeaways & Limitations

    The method is limited to static scenes, while FOV overlap can struggle in complex scenarios with multiple occlusions.

Abstract

from arXiv · show

Recent advances in interactive video generation have shown promising results, yet existing approaches struggle with scene-consistent memory capabilities in long video generation due to limited use of historical context. In this work, we propose Context-as-Memory, which utilizes historical context as memory for video generation. It includes two simple yet effective designs: (1) storing context in frame format without additional post-processing; (2) conditioning by concatenating context and frames to be predicted along the frame dimension at the input, requiring no external control modules. Furthermore, considering the enormous computational overhead of incorporating all historical context, we propose the Memory Retrieval module to select truly relevant context frames by determining FOV (Field of View) overlap between camera poses, which significantly reduces the number of candidate frames without substantial information loss. Experiments demonstrate that Context-as-Memory achieves superior memory capabilities in interactive long video generation compared to SOTAs, even generalizing effectively to open-domain scenarios not seen during training. The link of our project page is https://context-as-memory.github.io/.

1 INTRODUCTION

Interactive long video generation needs memory that preserves scene content and spatial relationships across time. Context-as-Memory addresses this by storing historical frames, conditioning on them directly, and retrieving relevant context using camera visibility.

  • Interactive long video generation supports applications such as gaming and simulation through streaming videos controlled by user interactions.
  • Current methods often maintain frame-to-frame continuity but fail to preserve scene consistency when the camera revisits previously viewed locations.The paper illustrates this failure with simple left-then-right camera movements producing different scenes.
  • Limited context windows prevent models from using distant historical frames that could preserve long-term scene consistency.The paper proposes treating previously generated context frames as memory for selecting and replicating relevant historical content.
  • Processing all historical frames is resource-intensive, wasteful when most are irrelevant, and potentially noisy, motivating retrieval of a small relevant subset.This subset is termed Memory Retrieval.
  • Context-as-Memory directly stores generated frames and concatenates context with predicted frames without post-processing or external control modules.The design avoids feature embedding extraction, 3D reconstruction, external adapters, and cross attention.
  • Memory Retrieval uses camera trajectories and FOV overlap to identify co-visible frames, supported by a scene-consistent dataset with precise camera annotations.The dataset includes diverse scenes and trajectories where the same regions appear across viewpoints and times.
  • Experiments report superior long-video memory, including effective performance in unseen open-domain scenarios, compared with SOTAs.

2 RELATED WORK

Related work covers video generation, controllable and streaming generation, memory capability, and emerging long-context methods. It situates this paper among diffusion-based systems and approaches addressing scene consistency through context or explicit representations.

  • Video Generation Model: Video generation models produce frame sequences and commonly use diffusion architectures for high-quality video synthesis.
  • Controllable Video Generation: Controllable video generation models video distributions conditioned on signals such as camera motion or agent actions.These controls support interactive exploration in games and simulators.
  • Streaming Video Generation: Streaming video generation continuously predicts new frames conditioned on previously generated frames, using diffusion-based or GPT-like approaches.
  • Streaming Video Generation: Diffusion-based methods are emphasized here because they generally provide higher visual quality and faster sampling speed than GPT-like alternatives.
  • Memory Capability for Video Generation: Existing long-video methods generally lack memory capability, maintaining local continuity while scenes change over time.One related direction constructs explicit 3D representations from generated videos.
  • Memory Capability for Video Generation: Recent methods explore long-context conditioning through multi-shot consistency, separate long- and short-term windows, and hierarchical context.

3 METHOD

The method injects historical video context directly into a full-sequence video diffusion model and retrieves relevant frames using camera-trajectory-based FOV overlap. This supports variable-length context while reducing redundant or irrelevant historical frames.

  • Context Conditioning: Context frames and predicted frames are concatenated along the frame dimension, allowing variable-length context conditioning without additional modules.The approach adapts a pre-trained full-sequence text-to-video model to context-conditioned generation.
  • Context Conditioning: The model preserves the pre-trained positional encoding for predicted latents while assigning new positional encodings to newly added context frames.This design aims to preserve the original model’s generation capability during adaptation.
  • Memory Retrieval: Random selection, recent-neighbor windows, and hierarchical compression either retain redundant frames or lose temporally distant historical information.The proposed search targets valuable old frames by visible-region overlap rather than temporal proximity or aggressive compression.
  • Memory Retrieval: Memory Retrieval searches historical frames using camera trajectories and selects frames whose visible regions overlap those of the frames to be generated.Camera poses are available from user-provided camera control, so an additional camera pose estimator is unnecessary.
  • Memory Retrieval: FOV overlap is estimated by testing intersections among the left and right rays from two camera origins and filtering intersections that are too far from the predicted camera.The XY-plane camera-motion restriction makes this four-ray test computationally simple, though occlusions can cause failures.
  • Memory Retrieval: When retrieved frames exceed the context limit, adjacent-frame redundancy can be reduced by randomly selecting one frame from each consecutive group.The paper states that this additional selection is unnecessary in most cases.

4 EXPERIMENTS

Experiments show that Context-as-Memory improves memory capability and video quality by using relevant historical context, while retrieval and context-size choices balance information, redundancy, computation, and speed. It also maintains memory capability in open-domain scenarios.

  • 4.2 Comparison Results: Context-as-Memory achieves the best memory capabilities and highest generated-video quality among the compared methods.The comparison includes single-frame, multi-frame, DFoT, and FramePack baselines.
  • 4.2 Comparison Results: PSNR and LPIPS favor Context-as-Memory because it retrieves useful context, whereas recent-frame methods suffer from adjacent-frame redundancy and information decay.Random context selection also outperforms DFoT and FramePack, possibly because it accesses more varied information on average.
  • 4.2 Comparison Results: FID and FVD show that Context-as-Memory achieves the best generation quality among all methods.The paper attributes this to stronger conditional guidance and reduced error propagation from earlier, less-accumulated-error context frames.
  • 4.2 Comparison Results: History Context Comparison is harder than Ground Truth Comparison, and global-context retrieval is needed for memory-aware generation under forward-and-backward camera trajectories.DFoT and FramePack access only the most recent context and therefore continuously generate new content.
  • 4.3 Ablation Study: A context size of 20 offers a performance-speed trade-off, while larger contexts improve memory capability but increase computational cost and slow generation.The paper reports a notable speed drop when context size reaches 30 compared with size 1.
  • 4.3 Ablation Study: FOV and Non-adj filtering improve memory capability by removing useless or redundant candidate frames, while Far-space-time has relatively minor impact.FOV filtering uses camera-pose field-of-view overlap, and Non-adj keeps only one frame from continuous sequences as a candidate.
  • 4.4 Open-Domain Results: Context-as-Memory retains good memory capability when internet images seed long videos in open-domain scenarios absent from training.The evaluation uses a rotate-away-and-rotate-back trajectory to test consistency after generating new content.

5 CONCLUSION

Context-as-Memory uses historical generated frames as memory and dynamically retrieves valuable context to support scene-consistent long video generation. Its limitations include restricted scene complexity and persistent error accumulation.

  • Context-as-Memory directly stores historical generated frames and conditions predicted frames on retrieved context to achieve scene-consistent long video generation.Memory Retrieval dynamically selects valuable context to avoid the computational overhead of lengthy history.
  • Memory Retrieval selects context dynamically to reduce the overhead caused by lengthy historical context.
  • Limitations and Future Work: The method is limited to static scenes, while dynamic-scene memory retrieval remains more challenging.FOV overlap may struggle in complex scenarios with multiple occlusions, such as interconnected indoor rooms.
  • Limitations and Future Work: Error accumulation persists in long video generation and currently requires larger datasets, more extensive training, and more powerful base models to address.

A INTRODUCTION OF THE BASE TEXT-TO-VIDEO GENERATION MODEL

The base text-to-video model is a transformer-based latent diffusion model that operates in a video latent space produced by a 3D-VAE.

  • The base model uses a transformer-based latent diffusion architecture for text-to-video generation.A 3D-VAE transforms videos from pixel space into latent space before video diffusion modeling.

B DETAILS OF COLLECTED DATASET

The collected dataset consists of diverse rendered 3D environments and long camera-trajectory videos, with camera parameters recorded for each frame.

  • 3D Environments: The dataset contains 12 visually realistic, stylized, or surreal 3D environments spanning indoor and outdoor settings.Examples include city streets, shopping malls, and countryside scenes.
  • Camera Trajectories: Smoothed polyline camera trajectories are generated from randomly sampled scene coordinates using B-spline curves.The trajectories are designed for smooth camera movement without abrupt speed or rotation changes.
  • 100 long videos were batch-rendered in Unreal Engine 5, each containing 7,601 frames at 30 fps.
  • Camera extrinsic and intrinsic parameters are recorded for every frame.The camera uses a 24mm focal length, aperture 10, and 52.67-degree field of view.

C ADDITIONAL OPEN-DOMAIN RESULTS

Additional experiments use diverse internet images as initial frames for open-domain long-video generation with rotate-away-and-back trajectories. The method generalizes through diverse training scenes and the pretrained base model, but remains limited on complex trajectories and dynamic exploration.

  • Open-Domain Results: Open-domain experiments initialize long-video generation from diverse internet images and use rotate-away-and-back camera trajectories.The source images cover varied styles and scenes.
  • Open-Domain Results: Open-domain generalization is attributed to diverse-scene training and the pretrained base model’s generative priors.
  • Open-Domain Results: The 1B-parameter base model produces good results only on simple trajectories, while complex trajectories lead to unacceptable error accumulation.Validation with larger-scale base models remains future work.
  • Open-Domain Results: The method does not yet support complex, diverse, and dynamic long-term scene exploration in open-domain settings.The stated goal is extended navigation from arbitrary images while maintaining memory consistency.
Loading 2506.03141v2…