PaperlayerView exact revision on arXiv

Source-linked AI summary

LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation

Yukang Chen, Luozhou Wang, Wei Huang, Shuai Yang, Bohan Zhang, Yicheng Xiao, Ruihang Chu, Weian Mao, Qixin Hu, Shaoteng Liu, Yuyang Zhao, Huizi Mao, Ying-Cong Chen, Enze Xie, Xiaojuan Qi, Song Han

arXiv:2605.18739v2cs.CVcs.DC

TL;DR

Long video generation faces severe training and inference memory, efficiency, and latency challenges. LongLive-2.0 addresses them with NVFP4-based parallel training and inference, achieving up to 2.1× training speedup and 1.8× inference speedup while supporting 45.7 FPS.

  • Problem

    Long video generation faces excessive GPU memory use, low computational efficiency, and deployment latency challenges across training and inference.

  • Method

    LongLive-2.0 co-designs Balanced sequence-parallel autoregressive training with NVFP4 quantization, quantized inference, KV caching, and asynchronous streaming VAE decoding.

  • Results

    Up to 2.1× training speedup and 1.8× inference speedup are achieved, with LongLive-2.0-5B reaching 45.7 FPS.

  • Takeaways & Limitations

    The system directly fine-tunes diffusion models into long, multi-shot autoregressive models and supports real-time generation with lightweight LoRA weights.

  • Takeaways & Limitations

    NVFP4 inference acceleration is hardware-dependent and is available only on Blackwell GPUs with optimized kernel support.

Abstract

from arXiv · show

We present LongLive-2.0, an NVFP4-based parallel infrastructure throughout the full training and inference workflow of long video generation, addressing speed and memory bottlenecks. For training, we introduce sequence-parallel autoregressive (AR) training, instantiated as Balanced SP, which co-designs the efficient teacher-forcing layout with SP execution by pairing clean-history and noisy-target temporal chunks on each rank, enabling a natural teacher-forcing mask with SP-aware chunked VAE encoding. Combined with NVFP4 precision, it reduces GPU memory cost and accelerates GEMM computation during training, the proportion of which increases as video length grows. Moreover, we show that a high-quality infrastructure and dataset enable a remarkably clean training pipeline. Unlike existing Self-Forcing series methods that rely on ODE initialization and subsequent distribution matching distillation (DMD), LongLive-2.0 directly tunes a diffusion model into a long, multi-shot, interactive auto-regressive (AR) diffusion model. It can be further converted to real-time generation (4 to 2 denoising steps) with standalone LoRA weights. For inference on Blackwell GPUs, we enable W4A4 NVFP4 inference, quantize KV cache into NVFP4 for memory savings, and boost end-to-end throughput with asynchronous streaming VAE decoding. On non-Blackwell GPU architectures, we deploy SP inference to match the speed on Blackwell GPUs, while the quantized KV cache can lower inter-GPU communication of SP. Experiments show up to 2.15x speedup in training, and 1.84x in inference. LongLive-2.0-5B achieves 45.7 FPS inference while attaining strong performance on benchmarks. To our knowledge, LongLive-2.0 is the first NVFP4 training and inference system for long video generation.

NVIDIA

LongLive 2.0 supports NVFP4-based multi-shot long-video generation across training and inference. NVFP4 preserves BF16-quality scene and shot semantics while improving efficiency, and the system allows flexible shot durations.

  • LongLive 2.0 supports NVFP4-based multi-shot long-video generation for both training and inference.
  • NVFP4 preserves the BF16 baseline’s overall scene composition, subject structure, and shot-level semantics.
  • Shot durations can be customized flexibly in LongLive 2.0.
  • 2.15× faster training and 1.84× faster inference are achieved with NVFP4.

1. Introduction

LongLive-2.0 presents an NVFP4-based parallel infrastructure that jointly addresses the memory and efficiency bottlenecks of long-video training and inference. Its infrastructure co-design enables Balanced SP for scalable autoregressive training, full-NVFP4 inference with KV-cache compression, and a cleaner direct-training pipeline.

  • Motivation: Long video generation faces excessive GPU memory consumption and low computational efficiency in both training and inference.Training requires extensive computation over massive long-video datasets, while inference must meet strict latency requirements for interactive and real-time applications.
  • Overview: LongLive-2.0 introduces an NVFP4-based parallel infrastructure for long-video generation training and inference.The framework jointly co-designs infrastructure across both stages, addressing limitations in prior training and inference pipelines.
  • Training infrastructure: Balanced SP assigns clean-context and noisy-target latents from the same temporal chunk to each GPU, balancing loss-bearing tokens for sequence-parallel autoregressive training.This paired layout differs from treating the two latent streams as one concatenated sequence and enables a natural teacher-forcing mask.
  • Inference infrastructure: Full NVFP4 enables W4A4 inference on Blackwell GPUs, while NVFP4 KV-cache quantization saves memory and lowers inter-GPU communication for SP inference on other architectures.SP inference on non-Blackwell GPUs also supports real-time generation, and the system targets end-to-end generation speed.
  • Algorithm design: High-quality infrastructure enables direct and efficient training on long videos, yielding a cleaner pipeline than methods relying on ODE initialization, DMD, or additional long-tuning stages.Existing methods use complex multi-stage processes, while the original LongLive adds a long tuning stage for long and interactive generation.

2. Training Infrastructure

LongLive-2.0 makes long-video AR training practical through paired clean-history/noisy-target chunking, Balanced SP, and local VAE encoding. It uses NVFP4 throughout training and distillation to reduce memory cost and accelerate low-precision computation while maintaining a streamlined AR-to-real-time pipeline.

  • Clean training pipeline: Direct AR fine-tuning on long-video data, followed by standalone LoRA weights, enables few-step denoising and real-time inference without multi-stage ODE initialization or intermediate DMD.The resulting model supports long, interactive, multi-shot generation.
  • AR teacher forcing: One forward pass supervises all N noisy chunks by pairing clean and noisy VAE latent streams with a block-sparse AR attention mask.Each noisy chunk attends to preceding clean chunks and its own noisy tokens.
  • Balanced SP: Balanced SP assigns each rank matched clean-context and noisy-target chunks, making loss computation uniform across ranks.The layout avoids clean-heavy and noisy-heavy ranks caused by naively slicing the concatenated sequence.
  • Balanced SP: Balanced SP reduces per-rank VAE encoding cost from O(F) to O(F/P + h) by encoding only each local chunk plus a left receptive-field halo.The method preserves the DiT training objective while discarding halo latents after encoding.
  • NVFP4 training: NVFP4 reduces memory cost and accelerates low-precision GEMMs whose share grows with video length, supporting both AR training and DMD step distillation.The paper presents this as its first end-to-end NVFP4 recipe for long video generation.

3. Inference Infrastructure

LongLive-2.0 combines W4A4 NVFP4 inference, chunkwise NVFP4 KV-cache quantization, and asynchronous streaming VAE decoding to reduce memory use and improve long-video generation throughput. The infrastructure targets repeated GEMMs, history-growing KV caches, and VAE decoding latency.

  • W4A4 NVFP4 inference: W4A4 NVFP4 inference replaces BF16 GEMMs, reducing memory traffic with an ideal theoretical throughput speedup of up to 4×.Deployment supports either a quantized backbone with a separate LoRA branch or a merged W4A4+LoRA model with fused low-rank kernels.
  • Quantized KV cache: 3.6× KV-cache compression is achieved in practice by quantizing frame chunks with NVFP4 micro-block scaling.The storage cost changes from 4T_cHd bytes to 9/8T_cHd bytes, ignoring amortized scale and padding overhead, while preserving generation quality.
  • Asynchronous VAE decoding: Asynchronous streaming VAE decoding addresses the baseline’s long latency and O(C·T_c) VAE-side GPU memory cost.The redesigned 3D VAE supports chunk-by-chunk streaming decoding with immediate CPU offloading.

4. Algorithm-level Designs

LongLive-2.0 organizes generation around editable temporal chunks with per-chunk prompts and factorized cross-attention, while its streaming design combines sliding-window KV caching with multi-shot attention sinks to preserve global identity and shot-level coherence.

  • Chunk-wise Prompting: Each temporal latent chunk is treated as an editable generation unit bound to an individual text prompt, with cross-attention factorized per chunk.This replaces conditioning the whole video on a single global prompt and enables different shots to carry different prompts.
  • Few-step Distillation: Few-step distillation performs one-stage DMD on the AR-trained model, avoiding ODE initialization, short-video DMD, and streaming long-tuning DMD.The resulting formulation removes separate initialization and progressive long-tuning stages.
  • Multi-Shot Attention Sink: Sliding-window self-attention with KV caching caps per-step compute at O(W·L_c), but naive token eviction causes appearance drift and standard global attention sinks fail across multiple shots.W is the attention-window length in chunks, and L_c is the token length of each chunk.
  • Multi-Shot Attention Sink: The multi-shot attention sink uses a permanently fixed global sink for global identity and a shot-level sink rebound at each scene cut for local temporal coherence.The two anchor sets cooperate to preserve both video-wide identity and intra-shot consistency.
  • Multi-Shot Attention Sink: The effective key/value set combines global anchors, shot-level anchors, and the recent sliding window, while the shot-level sink adds zero memory overhead through two scalar pointers.The shot-level sink is virtually prepended only after the window passes it, avoiding data copying.
  • Interaction with Chunk-wise Prompting: A prompt switch defines a scene cut that rebinds the shot-level sink and reinitializes subsequent cross-attention cache while preserving the global sink and prior history.This decoupling supports minute-scale interactive generation without redundant recomputation.

5. Experimental Results

LongLive-2.0 improves long-video training and inference efficiency through Balanced SP, NVFP4, KV-cache quantization, and asynchronous decoding. Experiments also report strong short- and long-video generation performance, while noting that VBench scores depend on evaluation protocol.

  • AR Training Efficiency: 40.1s, 119.3s, and 639.5s are the NVFP4+Balanced SP training iteration times for 16s, 32s, and 64s videos, yielding 1.3×, 1.4×, and 2.1× speedups over BF16+SP.The gain is largest at 64s, where NVFP4+Balanced SP nearly halves iteration time versus BF16+Balanced SP and more than doubles throughput over BF16+SP.
  • NVFP4 DMD Training: Peak per-GPU memory for NVFP4 DMD training decreases from 70.5 GB to 49.0 GB, a 21.5 GB reduction or 0.69× of the BF16 baseline.The conversion path uses NVFP4 for the frozen real-score branch and NVFP4+LoRA for trainable branches.
  • Inference Efficiency: 45.7 FPS is achieved by the 2-step 5B model, while KV-cache quantization reduces peak memory from 29.7 GB to 19.4 GB with only a modest latency cost.The 4-step 5B model reaches 29.7 FPS and surpasses all listed baselines.
  • Inference Efficiency: 36.3s E2E latency is reached for 64s videos by the final 2-step system while maintaining a 19.4 GB memory footprint.Asynchronous decoding lowers end-to-end latency by overlapping denoising and VAE decoding.
  • Short-video generation: 45.7 FPS is reached at 720p with 2 denoising steps, demonstrating efficient real-time short-video generation under NVFP4 quantization.The 3-step configuration reaches 35.2 FPS, and LongLive-2.0 achieves the strongest performance at 1280×720 resolution.
  • Long Video Generation: LongLive-2.0 achieves the best average rank among compared methods on 60s generation, with the NVFP4 model obtaining a subject consistency score of 97.62.The evaluation uses MovieGenBench prompts and VBench-Long with 4 denoising steps; VBench scores depend on resizing and frame-sampling protocols.

6. Conclusion … B. Multi-shot Long-video Dataset

LongLive-2.0 co-designs NVFP4 quantization, sequence parallelism, and long-video data processing for efficient autoregressive generation. The paper situates these contributions within related work on long-video AR synthesis, FP4 systems, sequence parallelism, and multi-shot dataset construction.

  • 6. Conclusion: LongLive-2.0 combines Balanced SP and NVFP4 for training with W4A4 model and KV-cache quantization plus parallel dequantization for inference.The system is presented as an algorithm–infrastructure co-design for efficient long-video generation across the full training and inference workflow.
  • 6. Conclusion: NVFP4 inference acceleration is hardware-dependent, working on Blackwell GPUs while SP inference compensates on non-Blackwell architectures.The limitation passage specifically contrasts Blackwell GPUs such as GB200 with A100 and H100 systems lacking native optimized-kernel support.
  • A.1. Long Video Generation: Long-video research has shifted toward causal autoregressive synthesis, including frame- or chunk-level generation and few-step streaming models.Examples include CausVid, MAGI-1, and AAPT, which respectively address causal conversion, chunk-level scaling, and one-step interactive generation.
  • A.1. Long Video Generation: Related methods address autoregressive train–test mismatch, long-range memory, cache management, horizon extension, and interactive multi-shot storytelling.The surveyed techniques use mechanisms such as rollout-distribution training, causal attention, KV recaching, attention sinks, compressed memory, positional correction, and structured cache partitioning.
  • A.2. FP4 Quantization: FP4 research moves beyond FP8 to reduce memory traffic and matrix-multiplication cost, with block-scaled formats such as MXFP4 and NVFP4 addressing FP4’s coarse value set.NVFP4 uses finer 16-element blocks and FP8 E4M3 block scaling, while stable FP4 training also requires higher precision for numerically sensitive operations.
  • A.2. FP4 Quantization: Existing FP4 studies mainly target language-model training or general inference, whereas long-video AR generation adds long spatio-temporal sequences, repeated denoising GEMMs and attention, growing KV caches, and precision-sensitive deployment.LongLive-2.0 studies NVFP4 under these combined system pressures.
  • A.3. Sequence Parallelism: Sequence parallelism distributes long sequences across devices through ring-style chunk partitioning or attention-head partitioning with All-to-All communication.The paper also notes that Diffusion Transformer infrastructures customize these paradigms for multi-dimensional video data.
  • A.3. Sequence Parallelism: AR video teacher forcing requires specialized sequence parallelism because each temporal chunk appears as both clean context and noisy target under complex spatio-temporal masks.The passage states that simply combining this layout with an existing SP backend is insufficient because of load-balancing and execution challenges.

C. Balanced SP Details

Balanced SP partitions temporal tokens while preserving non-parallel behavior through global-coordinate computations, local SP-aware VAE encoding, and communication-native teacher forcing. Its partitioned error recycling and measured scaling make SP both faster overall and more memory-efficient at long contexts.

  • Hybrid parallelism: Balanced SP uses hybrid data/sequence parallelism, partitioning only temporal tokens while RoPE, attention, supervision, and loss masks use global coordinates.Ranks in each SP group share the same sample and prompt, and loss normalization uses the global sequence length.
  • Exact SP-aware VAE encoding: Each rank VAE-encodes its local raw-video chunk plus a left halo, producing exact local latents while reducing per-rank cost from O(F) to O(F/P + h).The halo covers the encoder’s left temporal dependency; discarded halo latents leave the local chunk identical to full-video encoding.
  • Natural teacher-forcing mask: A deterministic index mapping recovers clean/noisy identities and temporal positions directly in the interleaved Ulysses order, preserving teacher-forcing visibility without explicit permutations.The mapping is evaluated as a fused flex_attention predicate without materializing π on Q/K/V tensors.
  • SP-aware error recycling: SP-aware error recycling injects stored rollout errors into clean prefixes while sharding position- and timestep-indexed buffers by SP rank.Data-parallel ranks with the same SP rank exchange entries during warm-up, while timestep-sharded checkpoints prevent position misalignment on resume.
  • Performance: 1.12×–1.41× speedup over TP and 3.40×–3.86× over DP are achieved by SP, which becomes most memory-efficient at long contexts with 51.24/62.85 GB at sequence lengths 128/192.These measurements use 4 NVIDIA GB200 GPUs; TP is slightly more memory-efficient at short contexts, while SP is consistently fastest.

D. Sequence Parallelism Inference · E. Visual Ablation of Multi-Shot Attention Sink · F. Scale Search NVFP4 Quantization

LongLive-2.0 extends sequence parallelism to inference with NVFP4 communication and KV-cache compression, while multi-shot attention sinks stabilize long-video continuation and adaptive scale search improves NVFP4 quantization. Together, these methods reduce communication bottlenecks, preserve shot-level consistency, and support efficient W4A4 inference.

  • D. Sequence Parallelism Inference: SP reduces per-device memory to O(L/P), but its efficiency is bottlenecked by All-to-All communication for sequence-head transposition.The inference design extends DeepSpeed-Ulysses sequence parallelism from training to extreme long-video generation.
  • D. Sequence Parallelism Inference: NVFP4 communication casts runtime Q and retrieved K,V tensors into low precision before All-to-All, reducing effective payload from 16 bits to roughly 4.5 bits per element.After micro-block scaling overhead, empirical communication volume falls by roughly 3.6×, improving SP scalability for long-context AR inference.
  • D. Sequence Parallelism Inference: SP is compatible with other low-bit KV compression and attention-pruning methods, which can further reduce exchanged tensors and accelerate inference on non-Blackwell GPUs.The paper leaves systematic comparison of these alternatives to future work.
  • D. Sequence Parallelism Inference: On H100, SP=2 reduces BF16 end-to-end latency from 31.0s/50.2s/85.0s to 19.3s/38.1s/62.5s for 16s/32s/64s videos.KV-cache quantization cuts communication time from 1.8s to 1.1s for 16s videos at SP=2 and from 12.8s to 7.8s at SP=4.
  • E. Visual Ablation of Multi-Shot Attention Sink: Without the multi-shot attention sink, sliding-window generation can lose shot-local anchors, causing later-shot drift in subject appearance and scene layout.The qualitative ablation contrasts this failure with the stabilizing effect of the proposed sink.
  • E. Visual Ablation of Multi-Shot Attention Sink: The multi-shot attention sink combines a global sink preserving video-level identity with a shot-level sink anchoring the current shot.This produces more stable continuation during sliding-window generation.
  • F. Scale Search NVFP4 Quantization: Four-Over-Six adaptively chooses whether each NVFP4 block maps its maximum to 6 or 4 by comparing reconstruction error.The 6-based encoding preserves dynamic range, while 4-based encoding represents high-magnitude values more evenly; E4M3 scales support both with small kernel overhead.

G. Ablation of NVFP4 Quantization

The ablation shows that direct W4A4 NVFP4 post-training quantization degrades video quality, whereas pre-trained NVFP4 preserves sharper details and remains close to the BF16 baseline.

  • Qualitative comparison: Direct PTQ causes visible temporal-quality degradation, including blurred eye regions, while pre-trained NVFP4 preserves sharper facial details.Figure 11 compares PTQ and pre-trained NVFP4 qualitatively across the initial and subsequent frames.
  • Quantitative ablation: The BF16 reference and direct PTQ comparison shows that deployment-only W4A4 NVFP4 conversion introduces a clear quality drop.This indicates a non-negligible mismatch between BF16 training and low-precision W4A4 inference.
  • Precision alignment: Pre-trained W4A4 NVFP4 remains close to the BF16 baseline by aligning training precision with target deployment precision.The result supports using NVFP4 throughout training and inference rather than only as post-training compression.

H. DMD Training Strategies

The paper compares direct DMD fine-tuning of the AR model with standalone LoRA injection, finding that LoRA is more flexible operationally and produces more natural visual quality, while direct DMD yields higher-contrast, more synthetic videos.

  • Strategy comparison: Two DMD strategies are evaluated: direct fine-tuning of the AR model and standalone LoRA injection.In direct fine-tuning, the student, teacher, and critic are initialized from the multi-step AR DiT after AR training.
  • Strategy comparison: Standalone LoRA injection can transfer across AR checkpoints and run in parallel with AR training.Its independently trained LoRA module can be inserted into AR models trained on different video data, avoiding the need to wait for AR training to finish.
  • Visual characteristics: Direct DMD fine-tuning produces higher-contrast, more synthetic videos, whereas standalone LoRA injection yields more natural visual quality.The paper notes that visual preference is subjective, so higher-contrast direct-DMD results may appeal to some viewers.

I. Implementation Details

LongLive-2.0 is built on Wan2.2-TI2V-5B with frozen text encoder and VAE, BF16 mixed precision, gradient checkpointing, and AdamW. Its implementation uses sequence-parallel AR training and a separate DMD LoRA distillation stage with specified distributed-training configurations.

  • The system builds on Wan2.2-TI2V-5B while keeping the text encoder and VAE frozen throughout training.
  • Unless otherwise stated, training uses BF16 mixed precision, gradient checkpointing, and AdamW with weight decay 0.01.
  • NVFP4 training quantizes GEMM operands in forward, backward, and weight-gradient paths, while numerically sensitive operations and optimizer states remain higher precision.
  • AR training: AR training uses sequence parallelism on 32 NVIDIA GB200 GPUs, with SP size 4, hybrid-full FSDP, global batch size 16, and 600 iterations.The local batch size is 1 per SP group with gradient accumulation 2.
  • DMD LoRA distillation: DMD LoRA distillation freezes the pretrained backbone and trains LoRA adapters on 16 NVIDIA GB200 GPUs for 5000 iterations with global batch size 32.The generator learning rate is 1.0×10−5, the critic learning rate is 2.0×10−6, and generator updates occur every five steps while the critic updates every step.
Loading 2605.18739v2…