PaperlayerView exact revision on arXiv

Source-linked AI summary

LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editing

Xinyu Wang, Chongbo Zhao, Fangneng Zhan, Yue Ma

arXiv:2606.26740v2cs.CV

TL;DR

Real-time streaming video editing must preserve unedited content while processing video causally without future frames. LiveEdit combines progressive distillation with an AR-oriented mask cache, achieving state-of-the-art streaming quality and 12.66 FPS.

  • Problem

    Streaming video editing lacks practical low-latency causal processing that preserves backgrounds and unedited regions without future-frame access.

  • Method

    LiveEdit distills a bidirectional diffusion teacher into a unidirectional streaming editor and uses an AR-oriented mask cache to route computation to edited tokens.

  • Results

    12.66 FPS; LiveEdit achieves state-of-the-art visual quality, temporal consistency, and throughput among streaming video editing baselines.

  • Takeaways & Limitations

    LiveEdit supports pure streaming video editing with four-step inference, high-quality edits, and zero visual degradation in unedited regions.

Abstract

from arXiv · show

Streaming video editing has made rapid progress, yet practical deployment is still limited by two core issues: maintaining stable backgrounds and non-edited regions over time, and achieving the low latency required for real-time interactive scenarios. Meanwhile, recent streaming video generation methods are mostly developed for synthesis and cannot be directly applied to editing due to the strict preservation requirement and region-specific control. In this work, we present a novel streaming video editing framework that performs causal, frame-by-frame editing with strong content preservation and real-time responsiveness. Our key design is a three-stage distillation pipeline that progressively transfers editing capability from a powerful bidirectional foundation model to an efficient unidirectional streaming editor, enabling stable long-horizon edits without sacrificing visual fidelity. To further support real-time deployment, we introduce an AR-oriented mask cache that reuses region-related computation across frames, substantially reducing redundant processing and accelerating inference. Finally, we establish a dedicated benchmark for streaming video editing. Extensive evaluations demonstrate that our method achieves state-of-the-art visual quality among streaming baselines while drastically boosting inference speed to 12.66 FPS, making it suitable for interactive and augmented reality applications.

1 THU 2 HKUST · 1. Introduction

LiveEdit presents a causal, chunk-by-chunk streaming video editing framework designed to preserve unedited content while enabling ultra-low-latency inference. Its three-stage distillation pipeline and AR-oriented Mask Cache address the stability and computational challenges of real-time editing.

  • 1. Introduction: Real-time editing is increasingly important for augmented reality and live-streaming applications, but practical low-latency deployment remains challenging.
  • 1. Introduction: Bidirectional or global attention models are difficult to adapt causally because unavailable future frames can cause forgetting effects or severe flickering.
  • 1. Introduction: LiveEdit performs causal, chunk-by-chunk editing with strong content preservation and ultra-low latency through progressive distillation into a unidirectional streaming editor.
  • 1. Introduction: Stage 1 equips a Bidirectional Diffusion Transformer with editing abilities, while Stage 2 uses teacher forcing and chunk-wise causal attention to transition toward sequential inputs.
  • 1 THU 2 HKUST: LiveEdit combines a Causal DiT with mask-guided caching to target high-fidelity and efficient editing, unlike bidirectional models with inefficient inference and prior streaming models with inaccurate preservation.
  • 1. Introduction: The AR-oriented Mask Cache extracts editing masks from L2 distances between edited outputs and sources, reducing redundant full-DiT processing across frames.
  • 1. Introduction: 12.66 FPS is achieved by the proposed causal, chunk-by-chunk framework, while its distillation pipeline migrates editing knowledge to a 4-step Causal DiT student.

2. Related Work · 3. Method

The paper positions streaming video editing as distinct from offline generation because it requires causal, low-latency processing while preserving source content and unedited regions. It addresses this gap with three-stage distillation and an AR-oriented Mask Cache that transfers high-fidelity editing to a fast causal editor and reuses stable background computation.

  • 2. Related Work: Offline controllable-editing methods achieve versatile visual control but jointly process entire videos and dense conditions, making them computationally expensive and non-causal.Unified architectures and in-context approaches use massive joint representations or extended context sequences.
  • 2. Related Work: Generation-oriented streaming mechanisms do not directly fit editing because editing depends on continuous source frames, precise spatial restoration, and strict preservation rather than free-form motion synthesis.Reliance on historical generated outputs can conflict with the preservation requirements of general video editing.
  • 3.1. Motivation: Bidirectional editing models use future-aware dense temporal attention, but causal truncation spreads attention across historical frames and can disrupt the temporal consistency of unedited regions.The paper identifies attention distribution shift and globally computed scene synthesis as two inefficiencies in adapting these models to streaming editing.
  • 3.2. Three-Stage Distillation Pipeline: A three-stage distillation pipeline transfers editing capability from a bidirectional foundation model to an efficient unidirectional streaming editor.The stages acquire editing ability, establish chunk-wise causal behavior through Teacher Forcing, and distill the causal model for low-latency inference.
  • 3.3. AR-oriented Mask Cache: The AR-oriented Mask Cache derives a spatial editing mask from preceding source and edited latents, separating active editing regions from unedited areas.The threshold is dynamically determined from token redundancy, and the mask remains structurally stable across the sequence.
  • 3.3. AR-oriented Mask Cache: Active tokens receive full attention and feed-forward computation, while unedited tokens reuse cached features, reducing complexity and preserving absolute background consistency.The routing strategy dynamically decouples computation across spatial regions and streaming chunks.

4. Experiment

Experiments show that the three-stage framework enables real-time streaming editing while preserving unedited content and accurately following complex prompts. On a 120-pair benchmark, it outperforms strong baselines across key quality dimensions, with SA-feature caching providing the best tradeoff for temporal reuse.

  • Efficiency: 79ms per-frame latency enables 4-step streaming editing with real-time responsiveness and no background flickering.The 4-step generator bypasses expensive ODE initialization through autoregressive Stage 2 training.
  • Qualitative Comparison: The method applies textures and colors only to target regions while preserving backgrounds, lighting, shadows, and subject identities without visual degradation.Existing methods either fail to make localized edits or cause color bleeding by applying attributes across entire frames.
  • Quantitative Comparison: 120 benchmark pairs and six automated metrics evaluate generation quality and editing accuracy against bidirectional offline and streaming baselines.The metrics are Text Alignment, Background Consistency, Motion Smoothness, Dynamic Degree, Aesthetic Quality, and Imaging Quality.
  • Ablation Study: 4 NFEs and removal of CFG in Stage 3 eliminate the latency that prevents Stages 1 and 2 from supporting real-time deployment.Stages 1 and 2 each require 100 NFEs and retain CFG, whereas Stage 3 directly initializes from autoregressive Stage 2 weights.
  • Cache Analysis: SA-layer caching achieves superior performance across evaluated dimensions, whereas FFN caching degrades quality because FFN representations contain less temporally redundant, high-frequency spatial information.Visual comparisons show FFN caching produces severe blurring and structural instability, while SA caching preserves fine-grained edits and natural color saturation.

5. Conclusion

LiveEdit adapts bidirectional diffusion priors to efficient unidirectional autoregressive streaming video editing. Its progressive three-stage distillation pipeline addresses attention-distribution shift during causal execution and bridges offline global processing with online continuous video processing.

  • LiveEdit adapts powerful bidirectional diffusion priors to an efficient, unidirectional autoregressive paradigm.
  • A progressive three-stage distillation pipeline addresses the attention distribution shift inherent in causal execution.The stages are Foundation Tuning, Teacher Forcing, and DMD.
  • The architecture bridges offline global processing and online continuous video processing.

A. Discussion with Previous Methods S1 … A. Discussion with Previous Methods

The discussion contrasts LiveEdit with Self-Forcing and EgoEdit, emphasizing that LiveEdit avoids costly ODE initialization and uses autoregressive distillation plus an AR-oriented Mask Cache for source-fidelity preservation. Self-Forcing targets text-to-video generation, while EgoEdit focuses on egocentric editing within the existing Self-Forcing pipeline.

  • A. Discussion with Previous Methods: The comparison identifies Self-Forcing and EgoEdit as the two methods most similar to LiveEdit among existing video diffusion works.The discussion frames both methods as relevant reference points for positioning the proposed framework.
  • A. Discussion with Previous Methods: Self-Forcing targets Text-to-Video generation and uses self-generated conditioning to bridge exposure bias.Its design is oriented toward synthesis rather than source-preserving video editing.
  • A. Discussion with Previous Methods: ODE initialization creates a severe computational bottleneck for Self-Forcing by establishing its initial causal checkpoint.The trajectory is manageable when sampled from text in T2V, but remains costly for streaming editing.
  • A. Discussion with Previous Methods: LiveEdit bypasses costly ODE initialization by directly using Stage 2 autoregressive weights as a stable distillation starting point.This design removes the initialization route that limits Self-Forcing’s deployment efficiency.
  • A. Discussion with Previous Methods: LiveEdit introduces an AR-oriented Mask Cache to strictly preserve source fidelity through precise spatial restoration.The cache prioritizes region preservation over free-form video synthesis.
  • A. Discussion with Previous Methods: EgoEdit pioneers streaming models for egocentric video editing and validates task-specific data within an established Self-Forcing pipeline.Its contribution includes a continuous first-person visual dataset and benchmark.
  • A. Discussion with Previous Methods: EgoEdit structurally relies on the existing Self-Forcing route without further inference acceleration for streaming video editing.The passage contrasts this with LiveEdit’s explicit focus on reducing streaming inference costs.
  • A. Discussion with Previous Methods: EgoEdit’s research scope remains specialized and tailored to egocentric tasks, limiting its broader applicability.The passage states that this specialization makes it difficult to generalize beyond egocentric settings.

B. User Study

A 20-volunteer user study compared LiveEdit with six state-of-the-art baselines across instruction consistency, background preservation, and overall quality. LiveEdit dominated human preferences, supporting accurate editing and strict preservation of unedited content.

  • User Study Design: 20 volunteers ranked LiveEdit against six baselines on Instruction Consistency, Background Preservation, and Overall Quality.The baselines were InsV2V, LucyEdit, VideoCoF, StreamDiffusion, StreamDiffusionV2, and StreamV2V.
  • Background Preservation: 75.0% of “Best” votes and an 87.5% top-3 preference rate were achieved for Background Preservation.LucyEdit also reached an 87.5% top-3 rate but received only 12.5% of “Best” votes, reflecting structural shifts or temporal inconsistencies.
  • Overall Quality: 95.8% top-3 preference rate was achieved for Overall Quality, outperforming both offline and streaming baselines.The study confirmed a balance between high-fidelity semantic editing and strict spatial-temporal consistency.

C. More cases · D. More Comparisons

The paper presents additional cases demonstrating the method’s effectiveness and further comparisons between LiveEdit and baselines. These results are shown in Figures 2–6.

  • C. More cases: Additional cases are provided to demonstrate the effectiveness of the proposed method.The cases are shown in Figures 2, 3, and 4.
  • C. More cases: Figure 2 presents an additional case illustrating the method’s effectiveness.
  • C. More cases: Figure 3 presents another case illustrating the method’s effectiveness.
  • C. More cases: Figure 4 presents another case illustrating the method’s effectiveness.
  • D. More Comparisons: Additional comparisons are provided between the baseline and LiveEdit.The comparisons are shown in Figures 5 and 6.
  • D. More Comparisons: Figure 5 shows a further comparison between the baseline and LiveEdit.
  • D. More Comparisons: Figure 6 shows another comparison between the baseline and LiveEdit.
Loading 2606.26740v2…