Source-linked AI summary
VideoSSM: Autoregressive Long Video Generation with Hybrid State-Space Memory
Yifei Yu, Xiaoshan Wu, Xinting Hu, Tao Hu, Yangtian Sun, Xiaoyang Lyu, Bo Wang, Lin Ma, Yuewen Ma, Zhongrui Wang, Xiaojuan Qi
TL;DR
Long-horizon autoregressive video generation struggles with accumulated errors, motion drift, and content repetition. VideoSSM combines an evolving SSM global memory with a local context window, and experiments report strong temporal consistency, motion stability, minute-scale coherence, and prompt-adaptive interaction.
Problem
Autoregressive diffusion supports streaming and interactive synthesis, but maintaining coherence over minute-scale horizons remains difficult because of error accumulation, motion drift, and content repetition.
Method
VideoSSM augments autoregressive DiT with an SSM-compressed global memory and a causal sliding-window local cache that jointly track scene evolution and local motion details.
Results
VideoSSM achieves state-of-the-art temporal consistency and motion stability among popular autoregressive video generators, with minute-scale reductions in cumulative error, drift, and repetition.
Takeaways & Limitations
The hybrid memory design supports long-horizon coherence, content diversity, linear-time scalability, and prompt-adaptive interactive video generation.
Abstract
from arXiv · showhide
Autoregressive (AR) diffusion enables streaming, interactive long-video generation by producing frames causally, yet maintaining coherence over minute-scale horizons remains challenging due to accumulated errors, motion drift, and content repetition. We approach this problem from a memory perspective, treating video synthesis as a recurrent dynamical process that requires coordinated short- and long-term context. We propose VideoSSM, a Long Video Model that unifies AR diffusion with a hybrid state-space memory. The state-space model (SSM) serves as an evolving global memory of scene dynamics across the entire sequence, while a context window provides local memory for motion cues and fine details. This hybrid design preserves global consistency without frozen, repetitive patterns, supports prompt-adaptive interaction, and scales in linear time with sequence length. Experiments on short- and long-range benchmarks demonstrate state-of-the-art temporal consistency and motion stability among autoregressive video generator especially at minute-scale horizons, enabling content diversity and interactive prompt-based control, thereby establishing a scalable, memory-aware framework for long video generation.
1. Introduction
Long-video generation requires persistent identity and temporal coherence, but AR diffusion faces accumulated errors, motion drift, and repetition over extended horizons. VideoSSM addresses this with evolving global memory plus local context, achieving coherent, diverse, interactive generation at linear cost.
- 1. Introduction: Long-video generation aims to simulate evolving visual worlds with persistent identity and temporal coherence, but current systems are constrained by short context and quadratic full-attention cost.These constraints also impede real-time prompt-adaptive updates.
- 1. Introduction: VideoSSM combines an SSM-compressed global memory with a causal sliding-window local cache to model evolving scene state and fine motion details.The SSM continuously updates holistic scene context, while the local window preserves lossless short-term cues.
- 1. Introduction: Unlike fixed history anchors, the hybrid memory updates global context continuously, avoiding frozen patterns and supporting content diversity and interactive adaptability.The design is presented as a scalable alternative to static memory behavior.
- 1. Introduction: At minute-scale horizons, VideoSSM substantially reduces cumulative error, motion drift, and content repetition while supporting prompt switching.Interactive evaluations report smoother transitions, fewer residual semantics, and higher user preference.
- 1. Introduction: VideoSSM achieves state-of-the-art temporal consistency and motion stability among popular autoregressive video generators on short- and long-range VBench evaluations.The comparison covers both short- and long-range benchmarks.
2. Related Work
Related work contrasts bidirectional and autoregressive generation, along with local and global memory strategies. Existing approaches support either streaming or long-context retention imperfectly, motivating a hybrid design.
- 2. Related Work: Bidirectional video models generate frames jointly, whereas autoregressive methods synthesize frames sequentially to support streaming and interactive prompt control.AR methods follow a next-token or next-block prediction paradigm for spatiotemporal video tokens.
- 2. Related Work: Longer-video pipelines based on overlapping clips, hierarchical interpolation, or training-free extrapolation remain computation-heavy or unsuitable for real-time generation.AR models offer stronger scalability for variable-length and real-time generation.
- 2. Related Work: Standard DiT uses full self-attention for long-context modeling but lacks causality and streaming capability, while causal AR DiT enables streaming at the cost of weakened long-context consistency.The architectural comparison motivates adding memory to the causal block.
- 2. Related Work: Local sliding-window memory loses information outside its window, while fixed early-frame sinks can freeze global state and cause content repetition.These complementary weaknesses define the need for a continuously updated global memory.
3. Preliminary: From DiT to AR DiT
The preliminary section moves DiT from bidirectional full attention to causal autoregressive generation. Each new frame is conditioned on previously generated content, improving streaming capability but leaving long-horizon drift and repetition unresolved.
- 3. Preliminary: From DiT to AR DiT: Standard DiT applies full self-attention over spatiotemporal tokens within a bidirectional vision Transformer.The formulation operates on tokens obtained from video frames.
- 3. Preliminary: From DiT to AR DiT: Causal AR DiT restricts temporal self-attention so each new frame attends only to previous frame tokens.At inference, each synthesized frame updates the conditioning signal for subsequent generation.
- 3. Preliminary: From DiT to AR DiT: Autoregressive inference feeds newly synthesized frames back into the conditioning signal, replacing ground-truth history with generated content.This produces frame-wise causal synthesis suitable for long-horizon generation.
- 3. Preliminary: From DiT to AR DiT: Pushing AR diffusion to minute- or hour-long horizons leads to error accumulation, motion drift, and content repetition.The preliminary discussion motivates augmenting AR DiT with hybrid memory.
4. VideoSSM
VideoSSM augments causal autoregressive DiT with hybrid state-space memory that combines precise local context with evolving global memory. The design preserves streaming efficiency while addressing information drift, repetitive patterns, and long-range coherence.
- 4. VideoSSM: VideoSSM augments autoregressive DiT with a hybrid memory architecture that explicitly manages short-term and long-term information.The local component preserves fine-grained motion and appearance details, while the global component compresses longer history.
- 4. VideoSSM: Causal attention has O(T²) complexity, whereas sliding-window attention provides O(TL) streaming efficiency but loses information when early tokens are evicted.Attention-sink methods retain O(TL) complexity but can freeze memory and produce repetitive content.
- 4.2.1. Local Memory: Sliding Window Self-Attention: The local memory uses a causal sliding window and rolling KV cache to retain recent token representations for motion cues and visual details.Only sink and the L most recent tokens remain in the local cache.
- 4.2.2. Global Memory: Dynamic State Computation: The global memory compresses evicted tokens into a fixed-size state that evolves through gate caching, state updates, memory retrieval, and output gating.Selective updates retain unpredictable information while gated decay supports controlled forgetting and long-term stability.
- 4.2.3. Position-Aware Gated Fusion: A position-aware router fuses local and global streams, suppressing global memory early and increasing its contribution as more context accumulates.The fused representation is passed to subsequent cross-attention and feed-forward layers while retaining sliding-window efficiency and dynamic global context.
5. Experiments
VideoSSM is evaluated on short and minute-long videos using automated benchmarks, qualitative comparisons, interactive prompt switching, and a user study. Across these evaluations, it improves consistency, motion stability, dynamic evolution, and interactive transitions over autoregressive baselines.
- 5.2. Video Quality Evaluation: 83.95 Total and 84.88 Quality are the highest scores among evaluated autoregressive models on 5-second VBench videos.The comparison includes few-step distilled generators, long-range variants, other baselines, and larger models such as MAGI-1.
- 5.3. Long Video Generation: VideoSSM achieves the highest Subject and Background Consistency among autoregressive models on minute-long VBench videos.The evaluation uses train-short test-long generalization from 5-second training clips to 60-second generation.
- 5.3. Long Video Generation: 50.50 Dynamic Degree lets VideoSSM preserve long-term coherence while maintaining natural temporal evolution instead of static or frozen outputs.This value is markedly higher than those reported for LongLive and Self Forcing.
- 5.3. Long Video Generation: Over 60 seconds, VideoSSM maintains subject identity and dynamic motion, while baselines exhibit content collapse, severe drifting, motion stagnation, or hallucinated duplicate subjects.The qualitative comparison covers burger and underwater scenes and contrasts VideoSSM with SkyReels-V2, Self Forcing, CausVid, and LongLive.
- 5.4. Interactive Generation: KV recaching enables VideoSSM to refresh local memory after prompt changes, producing smooth transitions while preserving scene coherence.The interactive demonstration evaluates adaptation to updated narrative directives during long-video generation.
- 5.5. User Study: 41.07% of user-study votes favored VideoSSM, which achieved the best average rank of 1.85 across 40 participants.Participants ranked four one-minute videos generated from eight prompts; the study attributes preference to balancing dynamic realism with long-term consistency.
6. Conclusion
The conclusion presents VideoSSM as a hybrid-memory autoregressive diffusion model for coherent, adaptive minute-scale video generation. Its reported benefits include reduced long-horizon errors and linear-time scalability, with future work targeting richer conditioning and controllable editing.
- 6. Conclusion: VideoSSM combines an evolving SSM global memory with a local context window to support consistent, adaptive, linear-time long-video generation.The conclusion frames autoregressive diffusion as a recurrent dynamical process with complementary long- and short-term memory.
- 6. Conclusion: Experiments report reduced error accumulation, motion drift, and content repetition, enabling minute-scale coherence and prompt-adaptive interactive generation.The conclusion summarizes the empirical scope without adding results beyond the reported evaluations.
- 6. Conclusion: Future directions include multimodal conditioning, camera-aware and geometric priors, and controllable longform video editing.These directions are explicitly identified as extensions of the framework.