PaperlayerView exact revision on arXiv

Source-linked AI summary

Streaming Video Editing with Easy Adaptation

Yujia Hu, Jiajun Li, Zihao He, Songhua Liu

arXiv:2609.24788v1cs.CV

TL;DR

Streaming video editing remains difficult because existing editing approaches rely on bidirectional full-sequence processing, while training a competitive streaming editor from scratch is costly. SVEET learns control on a pretrained bidirectional model using frame-independent conditioning and orthogonal decoupled training, then transfers it to streaming. The authors report strong editing quality with real-time performance, including 15 FPS on one H100 GPU without auxiliary acceleration.

  • Problem

    Existing editing methods rely on bidirectional full-sequence processing that conflicts with causal streaming, while training a competitive streaming editing model from scratch is prohibitively expensive.

    Source: §1 · p. 2

  • Method

    SVEET uses a temporally independent auxiliary control branch and orthogonal decoupled training to transfer editing capability learned on a pretrained bidirectional model to streaming.

    Source: §1 · p. 2

  • Results

    SVEET achieves strong editing quality while running at 15 FPS on a single H100 GPU without auxiliary acceleration.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    SVEET provides a paradigm for bridging offline and streaming video generation without costly retraining or distillation.

    Source: §6 · p. 9

Abstract

from arXiv · show

In this paper, we propose SVEET, a framework that requires merely training on a pretrained bidirectional video diffusion model but supports high-quality streaming video editing in an auto-regressive fashion. To tackle this problem, we first systematically revisit existing video-to-video diffusion approaches and identify two key principles for such streaming adaptation: backbone feature disentanglement and conditional frame independence. Building on these insights, we develop a novel paradigm for controllable video generation. At its core, an auxiliary model branch encodes source video inputs with temporally independent self-attention, and the intermediate features are injected into the corresponding backbone blocks for streaming-compatible control. Moreover, to bridge the discrepancy between the feature spaces of bidirectional and streaming models, we propose a decoupled training scheme that explicitly enforces the orthogonality between the optimization directions of video controllability and model causality. Such disentanglement ensures compatibility between the two objectives at inference and facilitates smooth zero-shot knowledge transfer across heterogeneous backbone architectures. Extensive experiments demonstrate that SVEET achieves superior editing quality while maintaining real-time performance, attaining 15 FPS on a single H100 GPU 17 without any auxiliary acceleration techniques. Codes are available at https://github.com/YujiaHu1109/SVEET.

1 INTRODUCTION

The paper asks whether editing learned on a pretrained bidirectional model can transfer to streaming without retraining. SVEET combines a temporally independent control branch with orthogonal decoupled training to support that transfer.

1 INTRODUCTION

  • Existing video editing methods use bidirectional full-sequence processing, while training a competitive streaming editor from scratch is prohibitively expensive.Prior streaming distillation can take 128 H100 GPU-days, and adding control signals further increases training cost.Source: §1 · p. 2
  • The paper identifies backbone feature disentanglement and conditional frame independence as principles for streaming-compatible control.The control mechanism should remain decoupled from the base model, and source-video encoding should be per-frame without temporal dependency.Source: §1 · p. 2
  • SVEET encodes source video with temporally independent self-attention in an auxiliary branch and injects its features into corresponding backbone blocks.Figure 2 depicts the VACE-based design, its temporally independent 2D control attention, and the orthogonal decoupled training scheme.Source: §1 · p. 2, Figure 2 · p. 3
  • Orthogonal Decoupled Training constrains controllability and causality optimization directions to address bidirectional-to-autoregressive feature mismatch and support zero-shot transfer.The paper describes the scheme as enabling transfer across heterogeneous backbones.Source: §1 · p. 2
  • In extensive experiments across multiple editing tasks, SVEET achieves strong editing quality while running at 15 FPS on a single H100 GPU, without auxiliary acceleration.Source: §1 · p. 2

2 RELATED WORK

Prior video editing methods mostly process full sequences bidirectionally, limiting real-time streaming use. Streaming generation methods address causal generation or acceleration, but generally target generation rather than editing and can require costly adaptation.

2.1 VIDEO EDITING WITH DIFFUSION MODELS

  • Most diffusion video editing approaches rely on bidirectional full-sequence processing, limiting their applicability to real-time streaming scenarios.The literature spans appearance editing, structure-guided generation, object manipulation, and unified conditional architectures.Source: §2.1 · p. 3

2.2 REAL-TIME STREAMING VIDEO GENERATION

  • Real-time streaming work focuses on causal autoregressive generation or inference acceleration, while many recent methods combine autoregression with diffusion distillation.These approaches primarily target video generation, often require costly adaptation or distillation, and the paper instead studies direct editing-capability transfer to a pretrained streaming backbone.Source: §2.2 · p. 3

3 METHODS

Transfer experiments motivate temporally independent conditioning, while SVEET pairs a framewise control branch with orthogonal decoupled training to address causal compatibility and backbone feature mismatch.

3.1 PRELIMINARIES

  • After bidirectional-to-stream transfer, VACE-Fun nearly fails, full spatiotemporal VACE retains only partial editing with unstable temporal behavior, and 2D-attention VACE transfers more reliably.These comparisons motivate keeping control learning decoupled from the backbone and avoiding cross-frame dependencies in source conditioning.Source: §3.1 · p. 4

3.2 SVEET

  • SVEET addresses two transfer challenges with a temporally independent control branch and Orthogonal Decoupled Training for feature discrepancy.Its target is to learn editing on a frozen bidirectional backbone and transfer it directly to a frozen causal streaming backbone without retraining or fine-tuning the latter.Source: §3.2 · p. 4
  • The control branch replaces full spatiotemporal self-attention with per-frame 2D spatial attention, so each frame's conditioning representation depends only on that source frame.The causal backbone models generated-video temporal coherence through its autoregressive history; the control branch needs no additional key/value cache.Source: §3.2 · pp. 4–5
  • ODT addresses degraded control transfer caused by the different parameters and feature distributions of bidirectional and causal backbones.The method explicitly decouples controllability learning from the bidirectional-to-causal feature discrepancy.Source: §3.2 · p. 5
  • SVEET estimates the feature gap from paired inputs to both backbones, using per-block hidden states and ridge regression to obtain a linear feature map.With bidirectional features as the reference, the residual transformation represents the update direction toward the causal counterpart.Source: §3.2 · p. 5
  • ODT constrains the trainable control update to the orthogonal complement of dominant bidirectional-to-causal discrepancy directions.The discrepancy subspace is derived from dominant singular directions, and the control update is parameterized in practice with LoRA adapters.Source: §3.2 · p. 5

4 THEORETICAL ANALYSIS

The analysis examines how orthogonalizing functionality-specific updates limits their interference and improves compositionality. Its bounds depend on residual singular-value energy and on how much of the second update overlaps the removed subspace.

4 THEORETICAL ANALYSIS

  • The interference from a second functionality on the first is upper-bounded by residual tail energy outside the first update’s dominant singular subspace.If the first functionality is approximately low-rank, the induced interference becomes negligible.Source: §4 · p. 6
  • The second functionality’s capacity reduction depends only on its overlap with the singular subspace removed from its update.Source: §4 · p. 6
  • Removing the dominant singular subspace of one update bounds higher-order interaction by its residual tail singular values.When the first update is approximately low-rank, the induced interference becomes negligible.Source: §4 · p. 6
  • Orthogonal subspace projection suppresses nonlinear coupling between functionalities and improves output compositionality.The analysis frames this as a theoretical property of the proposed orthogonal decoupled training strategy.Source: §4 · pp. 5–6
  • The analysis motivates orthogonality as a way to preserve feature components needed by controllability and causality while bounding output error.The formal statement and proof are in Appendix B.Source: §4 · p. 6

5 EXPERIMENTS

Experiments evaluate SVEET on stylization, inpainting, and depth-to-video generation. The reported comparisons favor SVEET overall, while ablations examine attention design, orthogonal decoupled training, transfer strategies, and subspace-rank selection.

5.1 EXPERIMENTAL SETTINGS

  • Experiments use Wan2.1-1.3B-VACE as the bidirectional backbone and chunk-wise Causal Forcing as the default streaming backbone, with three editing tasks and held-out test sets.Training uses 81-frame clips at 480 × 832 resolution; each task has approximately 6K–13K training samples.Source: §5.1 · p. 6

5.2 MAIN RESULTS

  • SVEET achieves the best average performance across most metrics and tasks, consistently leads baselines on VLM scores, and is preferred in the human evaluation.The quantitative comparison covers three editing tasks; the human evaluation is reported separately.Source: Table 1 · p. 7, §5.2 · p. 7, Table 3 · p. 9
  • Qualitative comparisons report stronger stylization structure preservation, style transfer, and temporal consistency, plus higher-quality, more temporally stable inpainting and depth-to-video outputs.The comparisons include SDEdit+CF, StreamDiffusionV2, Daydream+CF, LiveEdit, Channel Concat, and Full-Attn variants.Source: §5.2 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8

5.3 ABLATION STUDIES

  • 2D attention improves source preservation and streaming stability, while ODT reduces transfer-induced appearance drift; their combination gives the best overall trade-off.Post-hoc inference-stage projection causes substantial structural drift, while two-stage teacher forcing improves stability but degrades local content and requires additional streaming-backbone adaptation.Source: §5.3 · pp. 8–9
  • Adaptive rank selection outperforms fixed-rank settings in general, with the 80% spectral-energy variant performing comparatively better across most metrics.The study compares fixed ranks k = 32, 64, 128 with adaptive energy proportions r = 60%, 80%.Source: §5.3 · p. 9, Figure 7 · p. 9

6 CONCLUSIONS

The conclusion presents SVEET as a streaming-editing framework trained on a pretrained bidirectional diffusion model. It reports strong editing quality with real-time performance, while noting dependence on the underlying editor and limits to the tasks and backbones studied.

6 CONCLUSIONS

  • SVEET supports high-quality streaming video editing while training solely on a pretrained bidirectional video diffusion model.Source: §6 · p. 9
  • Its transfer approach combines a temporally independent control branch with orthogonal decoupled training to bridge bidirectional and autoregressive feature spaces.Source: §6 · p. 9
  • The conclusion reports strong editing quality and real-time performance without costly backbone retraining or distillation.Source: §6 · p. 9
  • The method still relies on the underlying bidirectional editing model’s capability; broader tasks and more heterogeneous backbones remain future work.Source: §6 · p. 9

A EMPIRICAL VERIFICATION OF THE ASSUMPTION

The empirical verification finds that bidirectional-to-causal discrepancy energy is concentrated in a relatively small set of singular directions. This observation supports a compact discrepancy subspace and motivates layer-adaptive rank selection.

A EMPIRICAL VERIFICATION OF THE ASSUMPTION

B THEORETICAL DERIVATION

The appendix formalizes interference, capacity, and output-composition trade-offs under orthogonal subspace projection, then details training and zero-shot streaming transfer.

B THEORETICAL DERIVATION

  • The removed energy is exactly characterized by the overlap between Delta W2 and the dominant singular subspace of Delta W1.Source: B THEORETICAL DERIVATION · p. 15
  • Orthogonal subspace projection bounds output-composition error by the residual tail singular value outside functionality 1's dominant subspace.The derivation also analyzes output compositionality under nonlinear transformations.Source: B THEORETICAL DERIVATION · p. 16

C.1 TRAINING DETAILS

  • SVEET uses frozen Wan2.1-1.3B-VACE and Causal Forcing backbones, optimizing only the VACE control branch with frame-wise spatial attention.The control branch uses rank-128 LoRA adapters in specified linear layers.Source: §C.1 · p. 17
  • ODT constructs a fixed orthogonal projector per DiT block from calibration features, selecting rank to capture 80% of cumulative discrepancy energy.The calibration set contains over 500 samples; the learnable LoRA update is projected onto the orthogonal complement during optimization.Source: §C.1 · p. 17
  • Each task-specific control branch is trained for 10 epochs on one A100 GPU using 81-frame videos at 480 × 832 resolution.Training uses AdamW, batch size 1, learning rate 5 × 10^-5, and weight decay 10^-2.Source: §C.1 · p. 18
  • After training, the control branch attaches to a frozen streaming backbone without fine-tuning or distillation, and transfers to alternative streaming backbones unchanged.Its temporally independent 2D attention allows frame-by-frame conditioning without an additional temporal KV cache.Source: §C.1 · p. 18

C.2 DATASET DETAILS

The experiments use separate, disjoint training and evaluation datasets for style transfer, video inpainting, and depth-to-video generation.

C.2 DATASET DETAILS

  • Style transfer uses about 13K training pairs across 26 styles and 120 held-out evaluation videos from Ditto.Each training sample pairs a source video with a text instruction specifying the target style.Source: §C.2 · p. 18
  • Video inpainting uses more than 6K VPData training samples and 80 held-out samples, with masks from the dataset's original annotations.Each sample includes a target video and its masked conditioning input.Source: §C.2 · p. 18
  • Depth-to-video uses about 7K VPData training samples and 80 held-out videos, conditioning on frame-wise depth maps extracted with Video-Depth-Anything.The same depth-extraction pipeline is applied to training and evaluation data.Source: §C.2 · p. 18
  • All clips are converted to 81 frames at 480 × 832 resolution, with task-specific conditioning signals aligned to target frames.The resulting evaluation sets contain 120 style-transfer samples and 80 samples each for inpainting and depth-to-video.Source: §C.2 · pp. 18–19

D.1 RESULTS ON OTHER STREAMING MODELS

The authors evaluate SVEET with both Causal Forcing and Self Forcing streaming backbones and report consistent qualitative editing effects on each.

D.1 RESULTS ON OTHER STREAMING MODELS

  • Figure 10 presents qualitative transfer results across the two streaming settings.CF denotes Causal Forcing and SF denotes Self Forcing.Source: Figure 10 · p. 19
  • SVEET produces consistent editing effects with both Causal Forcing and Self Forcing while preserving source-video spatial structure and temporal dynamics.The comparison examines transfer to Self Forcing in addition to the Causal Forcing backbone used in the main experiments.Source: §D.1 · p. 19, Figure 10 · p. 19
  • The reported results indicate that SVEET's bidirectional-to-streaming transfer strategy is not restricted to one streaming backbone.The trained control branch is evaluated with alternative streaming backbones.Source: §D.1 · p. 19

D.2 MORE VISUAL RESULTS

Additional visual examples depict SVEET streaming edits across varied videos, instructions, and control conditions.

D.2 MORE VISUAL RESULTS

  • Additional examples cover diverse source videos, editing instructions, and control conditions.These examples are presented as further visual results in Figure 11.Source: §D.2 · p. 19, Figure 11 · p. 20
  • The examples are reported to show stable and visually consistent streaming editing across a wide range of scenarios.Source: §D.2 · p. 19
  • Figure 11 contains the paper's additional visualization results.Source: Figure 11 · p. 20

Related papers

Loading 2609.24788v1…