PaperlayerView exact revision on arXiv

Source-linked AI summary

SANA-Streaming: Real-time Streaming Video Editing with Hybrid Diffusion Transformer

Yuyang Zhao, Yicheng Pan, Qiyuan He, Jincheng Yu, Junsong Chen, Tian Ye, Haozhe Liu, Enze Xie, Song Han

arXiv:2605.30409v1cs.CVcs.AI

TL;DR

Real-time streaming video editing must preserve long-range temporal consistency while meeting consumer-GPU latency and memory limits. SANA-Streaming combines a hybrid diffusion transformer, Cycle-Reverse Regularization, and system co-design to enable minute-length editing at real-time speed, reaching 24 end-to-end FPS and 58 DiT FPS on an RTX 5090.

  • Problem

    Real-time streaming V2V editing must generate chronological frames, preserve appearance across hundreds of frames, and fit consumer-GPU latency and memory budgets.

  • Method

    SANA-Streaming combines GDN and softmax-attention blocks with Cycle-Reverse Regularization and fused-kernel, mixed-precision system co-design for streaming video editing.

  • Results

    24 end-to-end FPS and 58 DiT FPS are achieved on an RTX 5090, while minute-length editing maintains competitive editing quality.

  • Takeaways & Limitations

    SANA-Streaming provides a practical step toward interactive, high-resolution video editing by jointly addressing modeling, training, and system challenges.

  • Takeaways & Limitations

    High-quality long-video editing data remain scarce, and ambiguous instructions can produce inconsistent or incorrect edits.

Abstract

from arXiv · show

Real-time streaming video-to-video editing (V2V) is critical for interactive applications such as live broadcasting and gaming, yet it remains a formidable challenge due to the stringent requirements for temporal consistency and inference throughput. In this paper, we present SANA-Streaming, a system-algorithm co-designed framework for high-resolution, real-time streaming video editing on consumer GPUs, with the following three core designs: (1) Hybrid Diffusion Transformer architecture introduces softmax attention in part of the blocks to improve local modeling capabilities while preserving the efficiency of linear layers. (2) Cycle-Reverse Regularization is a novel training strategy that enforces semantic consistency by predicting source frames from generated content via flow matching, improving temporal consistency without requiring paired long edited videos. (3) Efficient System Co-design combines fused GDN kernels and Mixed-Precision Quantization (MPQ) optimized for the NVIDIA Blackwell (RTX 5090) architecture. By profiling real-world throughput, our MPQ maximizes Tensor Core utilization while maintaining generation quality. The resulting system achieves real-time 1280 x 704 resolution editing at 24 end-to-end FPS on a single RTX 5090 GPU, with the DiT core running at 58 FPS. Experimental results demonstrate that our co-design approach significantly outperforms existing SOTA methods in both temporal coherence and system throughput.

1. Introduction

SANA-Streaming addresses the difficulty of real-time streaming video-to-video editing by combining a hybrid diffusion transformer, Cycle-Reverse Regularization, and system-level efficiency co-design. The framework targets temporal consistency and throughput while preserving long-range information, locality, and constant-memory generation.

  • Motivation: Real-time streaming V2V editing must generate frames chronologically while preserving unchanged motion and content, making it harder than editing short clips.The introduction identifies temporal consistency and inference throughput as central challenges for streaming applications.
  • Hybrid Diffusion Transformer: The hybrid diffusion transformer combines GDN blocks for global information with evenly inserted softmax-attention blocks for locality and first-block consistency.It derives from SANA-Video’s all-linear backbone, replaces vanilla linear attention with GDN, and retains recurrent linear-state caching in most layers.
  • Cycle Consistent Streaming Training: Cycle-Reverse Regularization uses a reverse flow-matching objective to improve long-range consistency without paired long edited videos.It addresses drifting caused by short-teacher supervision during long rollout clips and leverages long source videos.
  • Efficient System Co-design: 1.5-2.2x speed up on different GPU architectures results from partitioning the frame-wise GDN kernel so state matrices fit in GPU SRAM.The system co-design also targets Blackwell Tensor Core efficiency by combining NVFP4 and FP8 with BF16 quality advantages.

2. Methodology

SANA-Streaming combines GDN linear attention with selective local softmax attention to support long, high-resolution streaming edits under fixed memory. Its methodology further uses cycle-reverse training, causal VAE distillation, and hardware-aware kernel and quantization designs for efficient streaming inference.

  • Hybrid Diffusion Transformer: Most blocks use GDN linear attention for compact global memory, while a few use sliding-window softmax attention with a persistent sink and recent window for local detail.The interleaved design preserves linear-attention streaming efficiency while adding fine-grained local context.
  • Global Accumulated Memory: GDN blocks update frame-wise recurrent states carried across chunks, keeping linear-attention memory independent of the number of streamed chunks.The state summarizes the generated prefix rather than caching raw tokens from all previous chunks.
  • Cycle-Reverse Regularization: Cycle-reverse regularization performs a forward edit followed by reverse reconstruction of the source chunk with flow matching, requiring only long source videos without paired edited targets.The generated edited chunk conditions the reverse pass, which targets the original source chunk.
  • Causal VAE Distillation: Causal VAE distillation replaces symmetric temporal padding with left-only padding so each decoded frame depends only on current and previous latent frames.Pretrained LTX2 VAE weights are reused and temporally remapped for stable initialization.
  • Efficient System Co-Design: 49% of DiT FLOPs come from FFN input/output point-wise convolutions, which cannot be demoted globally without collapsing quality; shallow blocks are most fragile to quantization.The policy keeps the sensitive pathway in FP8 and excludes CA-KV, which covers 1% of DiT FLOPs.

3. Data Pipeline

The data pipeline generates high-quality human-centric video-editing pairs and long-video editing prompts for live broadcasting, combining short-video pair construction, long-video instruction construction, and VLM-based filtering.

  • Data Pipeline: The pipeline targets live broadcasting by generating human-centric video-editing pairs and editing prompts for long videos.It comprises short-video pair construction, long-video instruction construction, and VLM-based filtering.

4. Experiments

Experiments evaluate SANA-Streaming on five OpenVE-Bench pixel-aligned edit categories and show state-of-the-art quality, while system co-design enables real-time consumer-GPU generation. Ablations further demonstrate improved temporal consistency from Cycle-Reverse Regularization and sharper causal-VAE decoding after distillation.

  • OpenVE-Bench Evaluation: SANA-Streaming achieves state-of-the-art performance on OpenVE-Bench, with nearly 2.5x smaller model size and 5x faster throughput than OpenVE.The evaluation covers global style, background change, local change, local remove, and local add.
  • Cycle-Reverse Regularization: Cycle-Reverse Regularization preserves non-edited regions and edited content consistently throughout long videos when paired long videos are unavailable.Without regularization, the edited crystal tree changes in later frames; with it, the edit remains consistent throughout the long video.
  • Causal VAE: Causal decoder distillation recovers sharper textures and object boundaries, achieving comparable performance with the bidirectional teacher.Direct causal conversion introduces visible blur and fine-detail loss because future latent frames are unavailable.

5. Related Work

Prior work addresses efficient diffusion architectures, instruction-guided video editing, and long-video generation through linear or sparse attention, supervised editing models, and causal generation paradigms. SANA-Streaming is positioned for real-time streaming V2V editing, contrasting with methods mainly targeting offline short-clip editing.

  • Hybrid Linear-Softmax Architectures: Full attention in Diffusion Transformers has quadratic complexity, motivating linear- and sparse-attention alternatives such as SANA-Video and SLA.These alternatives aim to reduce memory and latency costs for long sequences.
  • Video Editing: Instruction-guided video editing has shifted from tuning-free attention manipulation and mask-based control toward large supervised video editing models.VACE unifies video creation and editing through a video condition interface and context adapter, while InsViE and Ditto emphasize scalable synthetic paired-data construction.
  • Video Editing: OpenVE-3M and OpenVE-Bench provide a unified benchmark spanning spatially aligned and non-spatially aligned editing categories.Other systems improve instruction following and local edit fidelity through multimodal conditioning, in-context learning, or open-weight video editing backbones.
  • Video Editing: Most recent video editing systems target offline short-clip editing, whereas SANA-Streaming focuses on real-time streaming V2V editing.This distinction establishes the deployment setting addressed by SANA-Streaming.
  • Long Video Generation: Diffusion Forcing, Self-Forcing, and Causal-Forcing explore causal and autoregressive paradigms for generating long videos beyond pretrained short-video context lengths.LoL studies RoPE misbehavior in long-video generation and proposes scaling strategies for longer temporal contexts.

6. Conclusion

SANA-Streaming is a system–algorithm co-designed framework for real-time, high-resolution streaming video-to-video editing on consumer GPUs. It balances temporal consistency, visual quality, and inference efficiency, while remaining limited by scarce long-video data and ambiguous instructions.

  • Conclusion: SANA-Streaming combines a hybrid diffusion transformer, Cycle-Reverse Regularization, and efficient system design for streaming video editing on consumer GPUs.The framework targets real-time, high-resolution editing under streaming constraints.
  • Conclusion: Minute-length video editing is achieved at real-time speed while maintaining competitive editing quality.This result demonstrates the framework’s practical balance of temporal consistency, visual quality, and inference efficiency.
  • Limitations: High-quality long video editing samples remain scarce, hindering temporal consistency in complex scenarios.Cycle-Reverse Regularization mitigates paired-data scarcity but does not eliminate the shortage of suitable long-video samples.
  • Limitations: Ambiguous or underspecified instructions can produce inconsistent or incorrect edits because the model lacks explicit ambiguity-resolution and intent-faithfulness mechanisms.The limitation applies to SANA-Streaming as an instruction-guided generative model.

A. Mixed-Precision Quantization

Mixed-precision quantization searches for a policy that balances latent quality against efficiency relative to a BF16 reference. The evaluation uses full 30-second streaming generation on 45 calibration prompts and guides selection with quality metrics and Cost over Speedup.

  • Search setup and metrics: 45 calibration prompts are evaluated using full 30-second streaming generation under identical input noise, comparing output latents against the BF16 reference.Quality is reported with relative RMSE and LPIPS, averaged over the calibration set; lower values are better.
  • Quantization policy search: 9.84 T FLOPs are executed per DiT chunk, with FP4 and FP8 assignments evaluated by their parameter and FLOP shares.The hybrid DiT contains 2 B parameters, and per-chunk reporting preserves the relevant precision fractions and speedup factors.
  • Efficiency metric: 2x speedup is achieved with all-FP8 quantization, while all-FP4 quantization provides 4x over BF16 under the idealized roofline model.The remaining linear layers stay in FP8, and Speedup is defined relative to BF16.
  • Efficiency metric: 0.23 is the FP8 Cost over Speedup ratio, serving as the reference for identifying the best mixed-precision policy.Cost over Speedup is defined as RMSE divided by Speedup to combine quality and efficiency into one scalar.

B. Implementation Details

SANA-Streaming is implemented as a 2B hybrid diffusion transformer trained on millions of filtered video clips using staged training with cycle-reverse regularization.

  • Model Design: The 2B model uses the LTX2 VAE with 32×32×8 compression and 20 transformer blocks: 5 softmax-attention and 15 GDN blocks.Its hidden size is 2240 with 20 attention heads.
  • Dataset: About 10M short video clips are collected from Ditto, OpenVE, and human-centric editing data, then filtered to 7M pretraining clips and 1M SFT clips.The SFT stage applies stricter filtering criteria.
  • Model Training: The bidirectional model is trained for about 100K iterations on 32 NVIDIA H100 GPUs with batch size 2 per GPU, learning rate 5e-5, and AdamW.Long training follows ODE initialization, self-forcing training, and streaming long training.
  • Model Training: Cycle-reverse regularization is adopted during the long training stage.The long training procedure follows three stages: ODE initialization, self-forcing training, and streaming long training.

C. More Results

Additional results show that SANA-Streaming supports diverse video editing tasks while preserving temporal consistency, structural fidelity, original motion, and character identity. Figure 13 demonstrates style transfer, background replacement, and local object addition using original videos as source references.

  • Result organization: Each result group uses the original video as the source reference, followed by rows showing the edited outputs.
  • Editing capabilities: Figure 13 demonstrates three editing tasks: style transfer, background replacement, and local object addition.Examples include low-poly or mosaic styles, concrete-terrace or boxing-gym backgrounds, and added lanterns or dragonflies.
  • Editing capabilities: The method precisely applies these modifications while maintaining high temporal consistency and structural fidelity to the original motion and character identity.

D. GDN Kernel Details

The Triton GDN kernel is organized around padded recurrent states, specialized streaming phases, and precision-specific data paths. Its implementation is tuned separately across phases and evaluated under three RoPE/cache strategies with fixed chunking.

  • Tensor shapes and layout: Production inputs use D=112 padded to BLOCK_D=128, yielding 128 × 128 matrix and length-128 vector states per (B,H) stream.Frame summaries use (BH, F, 128, 128) for matrix terms and (BH, F, 128) for vector terms.
  • Implementation details: Phase A uses separate matrix and vector streams, Phase B performs an fp32 recurrent scan, and Phase C streams Q_f spatially in BLOCK_S tiles.The vector stream omits V and RoPE; bidirectional mode combines reverse contributions into the forward history before a single Phase C launch.
  • Precision modes: The kernel supports fp32, TF32, and bf16 dot-product modes, with bf16 summaries and Phase-C buffers reducing HBM traffic while Phase-B state remains fp32.TF32 uses tensor cores for matrix products, while bf16 uses bf16 tensor cores with fp32 accumulation.
  • Architecture-specific launch parameters: Phase A and C are tuned as spatial streaming kernels through BLOCK_S, whereas Phase B is tuned through warp count and optional D-tiling.fp32 and TF32 share Phase-A and Phase-C launch shapes, but Phase B dispatch depends on the exact dot-product mode.
  • Benchmark: The benchmark compares fixed-RoPE, rolling-RoPE with sink, and rolling-RoPE using an initial five-frame chunk followed by three-frame chunks.Table 6 reports total end-to-end sampling time alongside first-call and subsequent-call GDN timings.

E. Data Pipeline … E.3. Data Verification

The data pipeline builds high-quality human-centric video editing pairs through taxonomy-guided instruction and controllable generation, then verifies short and long samples for semantic fidelity, temporal coherence, and usable content.

  • E. Data Pipeline: The pipeline constructs high-quality human-centric video editing pairs for controllable semantic edits and temporally coherent motion.It covers short-video pair construction, long-video instruction construction, and VLM-based filtering.
  • E.1. Short Video: Short-video instructions sample local human, background, artistic style-transfer, or composite edits using taxonomy-guided prompting and Qwen3VL analysis.The resulting instruction describes concrete visual details for the requested transformation.
  • E.1. Short Video: Short-video generation combines an edited first frame, source video, Qwen3VL target caption, and source-derived pose video in Wan2.2-Fun-Control.The pose video serves as a motion anchor for preserving temporal dynamics.
  • E.2. Long Video: Long-video training adds paired forward and backward edit instructions generated from a representative anchor frame and the source video.Gemini-3-Flash generates the forward instruction p+, while the backward instruction p− infers the inverse operation recovering the source content.
  • E.3. Data Verification: VLM verification compares each short-video candidate with its source, instruction, and predicted caption to detect edit failures, non-edit changes, and temporal artifacts.Candidates are evaluated for instruction alignment, non-edit consistency, temporal stability, physical plausibility, and video quality.
  • E.3. Data Verification: Each verification dimension receives a 0–10 score, with comments recording strengths and failure cases for selecting final training data.Scores and textual comments are saved for every sample.
  • E.3. Data Verification: Long-video checks sample the first and last 10 seconds at approximately 2 FPS and discard videos with excessive near-black pixels or ineffective teacher edits.This removes title-card or black-screen segments and samples unsuitable for long-video self-forcing distillation.

F. Causal VAE Training … I. Broader Impacts

The paper trains a causal VAE decoder with reconstruction, perceptual, wavelet, and teacher-feature losses, and develops GDN blocks with finite recurrent memory for streaming generation. Deployment adds layered safeguards and acknowledges both beneficial applications and misuse risks of real-time video editing.

  • F. Causal VAE Training: The causal VAE keeps the pretrained encoder fixed, trains only the decoder, and distills intermediate features from a frozen bidirectional LTX2 teacher.Training combines Charbonnier reconstruction, frame-wise AlexNet-based LPIPS, Haar high-frequency supervision, and intermediate feature alignment.
  • G. Frame-wise Gated DeltaNet: GDN extends Gated DeltaNet to frame-wise streaming by carrying terminal recurrent states instead of caching raw key/value tokens from previous chunks.This provides compact finite recurrent memory for causal streaming generation.
  • G.1. Notation; G.2. Feature parameterization: Each head processes frame tokens with linear projections, RMSNorm, a ReLU feature map, and fixed 3D RoPE, rotating keys only for numerator memory updates and readout.Unrotated keys remain in the normalizer state to preserve mass conservation in the linear-attention denominator.
  • G.3. Gate parameterization: GDN uses frame-wise decay, token-wise write, and output gates; its Mamba-style decay parameterization is designed for stable long-horizon recurrence.The write gate controls memory updates, while the output gate selects attention channels through a SiLU-modulated projection.
  • G.4. Frame-wise delta-rule update: The recurrent update decays prior states, predicts current values, and writes only the remaining delta-rule error, reducing interference between stored associations.This correction-based memory is intended to preserve identity and scene information across thousands of frames.
  • G.6. Streaming cache: Streaming inference caches only terminal per-head recurrent states, yielding cache size independent of streamed chunks or total length while softmax blocks refine recent tokens locally.GDN therefore compresses streaming history into a fixed-size matrix state.
  • H. Safeguards; I. Broader Impacts: Deployment applies policy checks before, during, and after generation, while broader impacts include useful editing applications and risks from misleading real-time deepfakes.Safeguards include input screening, generation controls, output checks, provenance, audit logs, red-team evaluation, and acknowledgment that they reduce rather than eliminate risk.
Loading 2605.30409v1…