PaperlayerView exact revision on arXiv

Source-linked AI summary

LongLive: Real-time Interactive Long Video Generation

Shuai Yang, Wei Huang, Ruihang Chu, Yicheng Xiao, Yuyang Zhao, Xianbang Wang, Muyang Li, Enze Xie, Yingcong Chen, Yao Lu, Song Han, Yukang Chen

arXiv:2509.22622v2cs.CV

TL;DR

Long-video generation must balance efficiency, quality, and interactive prompt control, while existing approaches face attention, memory, and transition challenges. LongLive combines causal frame-level autoregression with KV-recache, streaming long tuning, and short-window attention plus a frame sink. It fine-tunes a 1.3B-parameter model in 32 GPU-days, sustains 20.7 FPS on one H100, and supports videos up to 240 seconds with strong VBench performance.

  • Problem

    Interactive long-video generation requires efficient long rollouts while maintaining visual consistency and semantic coherence across streaming prompt switches.

  • Method

    LongLive combines causal frame-level autoregression, KV-recache for prompt switches, streaming long tuning, and short-window attention with a frame sink.

  • Results

    32 GPU-days fine-tunes a 1.3B-parameter model, while inference reaches 20.7 FPS on one NVIDIA H100 and supports videos up to 240 seconds.

  • Takeaways & Limitations

    LongLive achieves real-time interactive long-video generation with strong VBench performance in both short- and long-video settings.

  • Takeaways & Limitations

    Performance is bounded by the pretrained base model, and self-supervised fine-tuning without curated real-video data limits correction of inherited errors and biases.

Abstract

from arXiv · show

We present LongLive, a frame-level autoregressive (AR) framework for real-time and interactive long video generation. Long video generation presents challenges in both efficiency and quality. Diffusion and Diffusion-Forcing models can produce high-quality videos but suffer from low efficiency due to bidirectional attention. Causal attention AR models support KV caching for faster inference, but often degrade in quality on long videos due to memory challenges during long-video training. In addition, beyond static prompt-based generation, interactive capabilities, such as streaming prompt inputs, are critical for dynamic content creation, enabling users to guide narratives in real time. This interactive requirement significantly increases complexity, especially in ensuring visual consistency and semantic coherence during prompt transitions. To address these challenges, LongLive adopts a causal, frame-level AR design that integrates a KV-recache mechanism that refreshes cached states with new prompts for smooth, adherent switches; streaming long tuning to enable long video training and to align training and inference (train-long-test-long); and short window attention paired with a frame-level attention sink, shorten as frame sink, preserving long-range consistency while enabling faster generation. With these key designs, LongLive fine-tunes a 1.3B-parameter short-clip model to minute-long generation in just 32 GPU-days. At inference, LongLive sustains 20.7 FPS on a single NVIDIA H100, achieves strong performance on VBench in both short and long videos. LongLive supports up to 240-second videos on a single H100 GPU. LongLive further supports INT8-quantized inference with only marginal quality loss.

1 INTRODUCTION

LongLive targets interactive long-video generation by combining causal frame-level autoregression with mechanisms for prompt switching, long-video training, and efficient inference. It reports real-time throughput, minute-long training, strong VBench performance, and generation up to 240 seconds.

  • Framework: LongLive uses causal, frame-level autoregressive generation to support efficient inference with KV caching.Its design targets interactive long-video generation rather than static prompt-only generation.
  • Framework: KV-recache updates cached states with new prompt embeddings to preserve smoothness and prompt adherence across switches.The mechanism addresses visual and semantic inconsistency during interactive prompt transitions.
  • Framework: Streaming long tuning aligns training and inference by training on long videos, addressing quality degradation in long-video autoregressive generation.The approach is presented as a response to the mismatch between short-clip training and long-video inference.
  • Framework: Short-window attention with a frame-level attention sink accelerates inference while preserving performance and long-range consistency.The frame sink is paired with short windows to reduce efficiency costs without discarding long-range behavior.
  • Results: 32 GPU-days fine-tunes a 1.3B-parameter model for minute-long videos, while 20.7 FPS on one H100 supports real-time interaction.LongLive also reports strong VBench scores, generation up to 240 seconds on one H100, and marginal quality loss with INT8 inference.

2 RELATED WORK

Related work spans diffusion–autoregressive hybrids that combine diffusion modeling with autoregressive prediction. Examples include systems using diffusion forcing, film-structure planning, and multimodal controls.

  • Diffusion–autoregressive methods: Recent video-generation methods integrate diffusion modeling with autoregressive prediction as an intermediate paradigm.This line of work lies between purely diffusion-based and purely autoregressive approaches.
  • Representative systems: SkyReels-V2 combines diffusion forcing with a film-structure planner and multimodal controls.The passage identifies it as an example of the hybrid approach.

3 METHOD

LongLive addresses interactive switching, long-video training, and efficient long inference with KV-recache, streaming long tuning, short windows, and a frame sink. These components preserve prompt alignment and temporal consistency while reducing memory and computation demands.

  • 3.1 KV RECACHE: KV-recache rebuilds the cache from generated frames and the new prompt, retaining visual continuity while updating prompt semantics.The cache is recomputed at each prompt boundary and then reused for subsequent causal generation.
  • 3.1 KV RECACHE: KV-recache supports smooth transitions with full new-prompt compliance, unlike clearing the cache or retaining it unchanged.Clearing causes abrupt changes, whereas retaining the full cache can delay or prevent adaptation to the new prompt.
  • 3.1 KV RECACHE: Training integrates recaching and supervises the post-switch rollout under the new prompt to reduce train–inference mismatch.The procedure performs recache once at a switch, continues with the updated cache, and conditions the teacher on the new prompt.
  • 3.2 STREAMING LONG TUNING: Streaming long tuning extends generation in successive 5s clips using historical KV caches and supervises only the newly generated clip.This keeps memory and teacher supervision local while training on long sequences.
  • 3.2 STREAMING LONG TUNING: Long-video tuning is a prerequisite for efficient long-inference strategies such as window attention and frame sink.The study links long-video tuning to both long-video performance and the effectiveness of these acceleration methods.
  • 3.3 EFFICIENT LONG INFERENCE: Short-window attention makes attention and KV-cache costs depend on window size rather than total video length.The fixed temporal window reduces computation and memory, but introduces a quality–efficiency trade-off.
  • 3.3 EFFICIENT LONG INFERENCE: Frame-sink tokens act as persistent global anchors that restore long-range consistency under short-window attention.The first frame chunk is retained in the KV cache and concatenated to every attention block’s keys and values.
  • 3.3 EFFICIENT LONG INFERENCE: Training maintains historical context without gradients while retaining the current supervised clip’s full KV cache with gradients.The local attention window is W frames and the teacher-supervised clip is T frames.

4 EXPERIMENT

Experiments evaluate LongLive on short, long, and interactive videos, showing strong quality, consistency, prompt adherence, and throughput. Ablations examine KV-recache, short-window attention, and the frame sink.

  • Long Video Generation: LongLive achieves state-of-the-art performance on 30-second VBench-Long videos while being the fastest among compared open-source models.The comparison includes SkyReels-V2, FramePack, and Self-Forcing.
  • Ablation Studies: KV-recache achieves the best consistency score and CLIP score among the evaluated cache strategies.The ablation compares clearing the cache, retaining it unchanged, and refreshing it using preceding frames and the new prompt.
  • Interactive Long Video Generation: Interactive evaluation uses 160 curated 60-second videos containing six successive 10-second prompts, with whole-sequence quality and segment-level CLIP scores.LongLive shows strong prompt compliance, smooth transitions, and long-range consistency; Self-Forcing degrades on longer horizons and SkyReels-V2 has weaker consistency.
  • Ablation Studies: Consistency improves as the short-window attention size grows and saturates around a 24-frame window.A frame sink mitigates the consistency loss from smaller windows.

5 CONCLUSION

LongLive is presented as a frame-level autoregressive framework for real-time interactive long-video generation. Its reported capabilities include efficient training and inference, long-duration generation, and INT8 quantization with minimal performance degradation.

  • Conclusion: LongLive introduces a frame-level AR framework for real-time and interactive long video generation.The framework combines KV-recache, streaming long tuning, short-window attention, and a frame sink.
  • Conclusion: 32 GPU-days are sufficient to fine-tune a model for long-video AR generation.The conclusion states that tuning on long videos also supports efficient inference strategies.
  • Conclusion: 20.7 FPS on a single NVIDIA H100 GPU supports video generation up to 240 seconds while maintaining high fidelity and temporal coherence.INT8 quantization compresses the model from 2.7 GB to 1.4 GB with minimal performance degradation.

D.1 DIFFUSION-BASED LONG VIDEO GENERATION

The supplied material surveys diffusion, hybrid, and interactive video-generation approaches, alongside prompt synthesis and streaming inference procedures. It also includes algorithm fragments for cache-based long tuning and interactive rollout.

  • Diffusion-Based Long Video Generation: Diffusion-based methods extend long-video generation through token compression, coarse-to-fine generation, frequency adjustment, noise rescheduling, and temporal-frequency blending.Examples include Phenaki, NUWA-XL, RIFLEx, FreeNoise, FreeLong, and FreeLong++.
  • Streaming and Interactive Algorithms: The streaming long-tuning algorithm initializes a KV cache, samples prompt pairs and a switch index, recaches at the switch, generates the next clip, computes DMD loss, and updates the generator.The interactive inference algorithm similarly updates the active prompt and recaches before generating subsequent frames.
  • Hybrid Diffusion-AR Methods: Diffusion-forcing combines diffusion quality with AR efficiency by injecting noise into future tokens and training the model to denoise them.StreamingT2V adds short- and long-term memory modules for coherent text-to-video generation.
  • Interactive Video Generation: Interactive video-generation systems support real-time user influence through text, keyboard prompts, navigation, reference frames, motion context, or actions.Several cited systems target explorable or game-like environments such as Minecraft and GTA.
  • Prompt Construction: The prompt-generation procedure asks an instruction-tuned language model to write the next moment of a continuous shot while preserving the original setting, subject, mood, style, and camera properties.It adds exactly one new action or object for the existing main subject.

F.1 IMPLEMENTATION

LongLive first converts Wan2.1-T2V-1.3B into a chunk-wise causal AR model, then applies streaming long tuning with short-window attention and frame-sink tokens.

  • Implementation: The initialization uses 3 latent frames per chunk, a 9-frame local attention window, and the first 3 latent frames as the sink.These settings are followed by streaming long tuning according to Algorithm 1.
  • Implementation: Streaming long tuning is applied after initialization to train the model on long-video generation.The passage states that this procedure follows the streaming long-tuning algorithm.

F.2 LORA TUNING

LongLive uses high-rank LoRA adapters for streaming long tuning, reducing trainable-state costs while retaining long-video quality. The experiments show rank 256 performs best among tested LoRA budgets, while quantization results are reported separately.

  • Training efficiency: 73% savings in parameter and optimizer-state footprint are achieved relative to full fine-tuning with LoRA.The resulting adapters require 256 ranks, making roughly 27% of model parameters trainable.

G QUANTIZATION

LongLive’s quantized and qualitative evaluations cover throughput, VBench quality, interactive prompt sequences, and ultra-long generation. The reported examples include sequential-prompt videos and smooth 240-second output.

  • Quantized inference: 1.9× smaller model size and 1.3× higher throughput are obtained with INT8 quantization while VBench quality degrades minimally.Table B reports FPS on a single NVIDIA 5090 GPU.
  • Interactive videos: Six sequential prompts are used to generate interactive 60-second videos for qualitative evaluation.The examples are presented in Figures A and B, with prompts covering evolving scenes and actions.
  • KV recaching: KV recache combines visual consistency with new-prompt adherence, unlike no-cache or retained-cache ablations.No KV cache causes abrupt transitions, while retaining the cache can delay or ignore new prompts.
  • Ultra-long generation: 240-second generation remains smooth and consistent in an ultra-long sequence experiment.The experiment demonstrates train-and-test capability on ultra-long sequences.
  • User study: The user study collected 1,248 judgments from 26 valid participants across 48 questions and four video-quality dimensions.Participants compared video pairs on overall preference, motion quality, instruction following, and visual quality.

M LIMITATION ANALYSIS

LongLive’s principal limitation is that fine-tuning remains bounded by the pretrained base model’s capacity and quality. The qualitative materials separately illustrate single-prompt, interactive, and KV-recache examples.

  • Limitation: Base-model capacity and quality bound LongLive’s ultimate performance.Because the method fine-tunes a pretrained model without additional curated real-video data, it primarily improves adaptation and stabilization rather than absolute quality ceilings.
  • Quality scope: Short video segments are unlikely to consistently exceed the base model’s quality, even when long-horizon consistency or instruction adherence improves.The limitation follows from the self-supervised fine-tuning setup and inherited systematic errors or biases.
  • Qualitative evidence: Interactive 60-second videos with sequential prompts are presented as qualitative examples.Figures A and B show prompt-driven video sequences, while Figure C shows single-prompt 60-second videos.
  • KV-recache ablation: KV recache is the only ablation shown to combine visual consistency with new-prompt adherence.The comparison contrasts no KV cache, full KV cache, and KV recache behavior.
Loading 2509.22622v2…