PaperlayerView exact revision on arXiv

Source-linked AI summary

EditStream: A Unified Autoregressive Framework for Interactive Video Generation and Editing

Yuqian Zhou, Zhenghong Zhou, Zongze Wu, Cameron Smith, Richard Zhang, Jiebo Luo, Eli Shechtman, Zhe Lin

arXiv:2608.21424v1cs.CVcs.GRcs.HCcs.LGcs.MM

TL;DR

Interactive video workflows need one system that supports diverse generation and editing tasks while responding with low latency. EditStream unifies these tasks in a DiT and distills it into a few-step autoregressive model using VMM, unrolling, and energy annealing. Across multiple tasks, it reports high generation quality and inference efficiency while simplifying conversion, but its current design is limited to short-form horizons.

  • Problem

    Interactive creative workflows require diverse video generation and editing operations, while converting strong bidirectional diffusion teachers into reproducible few-step causal students remains challenging.

  • Method

    EditStream uses dual-type conditioning in one DiT and a bidirectional-to-autoregressive recipe combining VMM, warm-up, unrolling, and Energy Annealing.

  • Results

    Experiments across generation and editing tasks demonstrate high generation quality and inference efficiency, with VMM-forcing unrolling reaching 84.42 overall VBench on T2V.

  • Takeaways & Limitations

    EditStream provides a unified framework that supports interactive streaming inference across multiple video generation, transformation, and editing tasks.

  • Takeaways & Limitations

    The current model is primarily designed and evaluated for short-form video because it lacks specialized long-context or long-memory modeling and does not investigate long-horizon error accumulation.

Abstract

from arXiv · show

Interactive video generation and editing are becoming increasingly important for creative design. In this report, we introduce EditStream: a unified framework for interactive video generation and editing. EditStream unifies multiple video creation and manipulation tasks within a single DiT-based model through flexible task-specific conditioning, and further transforms it into a fast, few-step autoregressive model for efficient streaming. It supports Text-to-Video, Image-to-Video, Video-to-Video, Editing Propagation, Reference-guided Video Editing, and Camera Pose Change, enabling flexible control over video generation, transformation, and editing within one system. To make the unified model practical for interactive use, we develop a two-stage distillation approach that combines Velocity Moment Matching (VMM) with autoregressive unrolling. VMM matches conditional velocity moments at student-reached intermediate states to preserve generation quality and motion, while unrolling exposes the student to its own autoregressive predictions to improve temporal stability. Together, they alleviate common challenges in few-step autoregressive video generation, including over-saturation, degraded motion, temporal instability, and complex training. EditStream provides a practical and scalable solution that bridges high-quality diffusion-based video models with interactive creative workflows.

1 Introduction

EditStream addresses the need for interactive video creation and editing by unifying diverse tasks in one model and converting it into a few-step autoregressive system. Its VMM-based conversion recipe and energy annealing target quality, motion, stability, and training complexity challenges.

  • Interactive workflows require generation, animation, video manipulation, editing propagation, reference insertion, and camera redirection during streaming.
  • Existing unified models and autoregressive methods largely address task breadth and streaming conversion separately, while causal distillation remains difficult to reproduce.
  • EditStream formulates T2V, I2V, V2V, EditProp, refV2V, and ReShoot within one DiT using dual-type conditioning.
  • VMM converts a full-attention teacher into a causal student through warm-up and unrolling without offline ODE-pair preparation or teacher causalization.
  • Energy Annealing and the conversion recipe improve motion dynamics, temporal stability, and color quality during autoregressive generation.

2 Related Works

Related work has pursued unified video modeling, streaming autoregressive generation, and causal video editing through separate lines of research. EditStream combines these directions while emphasizing efficient conditioning and a simpler conversion-and-distillation recipe.

  • Unified Video Generation and Editing: Unified video models combine generation and editing through shared conditional interfaces, but prior approaches differ in conditioning design and data strategy.
  • Unified Video Generation and Editing: Channel concatenation reduces conditioning-token growth for pixel-aligned inputs, whereas token concatenation represents non-spatial reference conditions.
  • Streaming Autoregressive Video Generation and Editing: Full-sequence diffusion provides global spatiotemporal refinement but becomes costly as video duration and spatial resolution increase.
  • Streaming Autoregressive Video Generation and Editing: Diffusion Forcing, CausVid, and subsequent methods progressively develop causal streaming generation and reduce the training-inference exposure gap.
  • Streaming Autoregressive Video Generation and Editing: Long-form streaming methods address error accumulation, memory, and short-clip generalization, while recent systems increasingly use inherently causal video editors.
  • Streaming Autoregressive Video Generation and Editing: EditStream emphasizes broad task unification together with a simpler, more efficient, and more stable conversion-and-distillation recipe.

3 Unified Video Generation and Editing

EditStream represents generation and editing tasks in one latent video diffusion model, differing primarily through conditional inputs and task tokens. Its dual-type interface uses compact channel concatenation for aligned controls and token concatenation for reference signals.

  • 3.1 Problem formulation: The model predicts target video latents from task-dependent context and control signals, while sharing one diffusion formulation across tasks.
  • 3.1 Problem formulation: T2V, I2V, V2V, refV2V, EditProp, and ReShoot share a backbone but differ in conditional inputs and special task tokens.
  • 3.2 Dual-type Conditional Modeling: Pixel-aligned source videos, masks, and camera-warped inputs use channel concatenation to keep conditioning compact and avoid increasing attention length.
  • 3.2 Dual-type Conditional Modeling: Reference images and first-frame inputs use sequence token concatenation to leverage self-attention for in-context learning.

4 Velocity Moment Matching (VMM)

VMM distills a full-attention teacher into a few-step causal student by matching velocity moments at student-induced states. Energy Annealing complements this process by addressing guidance-related over-saturation and improving color stability.

  • 4.1 Distillation Scheme: VMM first performs a student coarse denoising transition, then matches teacher and auxiliary conditional moments at the resulting student-induced state.
  • 4.1 Distillation Scheme: VMM performs moment matching in velocity-output space, aligning the objective with flow-matching models and avoiding timestep-dependent gradient scaling from clean-sample matching.
  • 4.1 Distillation Scheme: The frozen teacher supplies target velocities, the auxiliary model estimates the student distribution, and the student becomes causal for streaming inference.
  • 4.1 Distillation Scheme: The student follows an ODE update from t to s, after which teacher and auxiliary predictions are evaluated at the same fake student state.
  • 4.1 Distillation Scheme: Sampling t from the discrete student schedule and s within the corresponding interval improves convergence efficiency for few-step distillation.
  • 4.2 Energy Annealing Strategies: Energy Annealing mitigates over-saturation propagated from large classifier-free guidance during teacher inference and distillation.
  • 4.2 Energy Annealing Strategies: The strategy converts velocity predictions to x0-space, uses a lower-guidance reference, and rescales channel statistics toward that reference.
  • 4.2 Energy Annealing Strategies: Moderate energy annealing improves saturation and color stability, but its factor requires manual tuning across models.

5 From Bidirectional Diffusion to Autoregressive Streaming

EditStream converts a full-context bidirectional diffusion teacher into a few-step causal student for streaming generation while preserving the teacher’s bidirectional attention. A two-stage process combines teacher-forced warm-up with VMM-forcing unrolling under the student’s own generated history.

  • Bidirectional video models provide strong full-sequence quality but are unsuitable for streaming generation and long-context extension.
  • Teacher preparation adds first-block cold-start training and prefix-suffix teacher forcing alongside full-sequence diffusion training.First-block training targets context-free initial generation, while prefix-suffix training denoises one noisy block using clean surrounding anchors.
  • The teacher remains bidirectional, while causality is introduced through noising and loss masks rather than modifying its attention mask.This avoids potentially reducing the pretrained teacher’s modeling capacity.
  • Warm-up distillation initializes the causal student on single blocks with clean prefix-suffix context, then unrolling distillation trains it on autoregressively generated history with KV caching.Training can transition from warm-up to unrolling after a small number of iterations.
  • During warm-up, active blocks contain noise while other blocks remain clean at timestep zero, and the student sees only the clean prefix plus active block.The teacher and auxiliary model can also use the clean suffix to construct the VMM correction.
  • VMM-forcing unrolling reproduces inference by generating each block from noise, committing it to the KV cache, and applying local VMM supervision at student-induced intermediate states.The committed zero-timestep block supplies later causal context, whereas the intermediate state provides trajectory-local supervision.

6 Data Construction

EditStream is trained on a heterogeneous dataset mixture spanning generation, editing, physical interactions, and camera control. The resulting balance is intended to support diverse editing behaviors while preserving temporally coherent streaming generation.

  • The unified model supports generation, instruction editing, propagation, reference guidance, physical object editing, egocentric editing, effect removal, and camera control.Training uses 81-frame clips resized into 720p aspect-ratio buckets.
  • Roughly half of the training mixture is allocated to general video generation and image-conditioned video generation.Sources include LTX2.3-video and OpenVid-1M, with LTX2.3 inference used on prompts from Self-forcing.
  • Paired editing data teaches instruction-following transformations, while propagation samples train edits from an edited first frame or weak propagation prompt across the video.
  • Reference-guided data contains roughly 476K source-target-reference triplets for transferring subjects or backgrounds while preserving source-video consistency.
  • Physically grounded data includes approximately 21K HUMOTO human-object interaction clips for object addition, removal, and edit propagation, supplemented by synthetic Kubric-style object editing.Rendered pairs are filtered for penetration, floating artifacts, unclear interactions, heavy occlusion, and strange viewpoints.
  • Multi-view camera-control data uses MultiCamVideo scenes with 10 cameras, 81 frames, and four focal-length/aperture groups, with geometric conditioning from reconstructed views.The target video supervises camera-pose transformation, while point-cloud renders and alpha masks provide conditioning.
  • The final dataset balances multiple task families to support strong generative quality and temporally coherent editing in one streaming model.

7 Experiments

EditStream delivers strong quality across text-to-video, video editing, reference-guided editing, and camera-pose control while retaining low-latency streaming operation. VMM-forcing generally improves motion diversity, temporal stability, and realism, though some comparisons have attribution and metric caveats.

  • Text-to-Video: 84.42 VBench overall score makes VMM-forcing Unrolling the strongest reported Text-to-Video variant.It exceeds the fine-tuned teacher at 84.05 and Wan2.2 at 83.60.
  • Text-to-Video: 79.17 dynamic degree, 87.82 color quality, 98.00 human action, and 85.36 perceptual quality drive VMM-forcing’s improvement.At 720p, EditStream reaches 10.56 FPS on one H100 GPU, or 16.10 FPS with parallelized VAE decoding on two GPUs.
  • Text-to-Video: VMM-forcing produces richer scene evolution and more dynamic camera and object motion than Self-forcing while avoiding its plastic-like, over-sharp appearance.The qualitative comparison links these differences to greater temporal diversity and preserved visual fidelity.
  • Video-to-Video Editing: On EditVerse video-to-video editing, VMM-forcing preserves most teacher performance and achieves the highest VLM-judged video quality, while qualitative results show stronger semantics, fidelity, and color quality.The comparison covers object addition, removal, replacement, style transfer, text rendering, and compositional edits.
  • Reference-Guided Editing: EditStream achieves the highest reported RefVIE-Bench overall score of 3.53 and best subject-reference consistency of 4.01 among open-source methods.It also preserves identity, object scale, scene composition, and pixel-level detail in low-latency streaming.
  • Camera Pose Control: For camera-pose control, VMM-forcing leads streaming variants in dis-occluded completion, temporal stability, and motion smoothness, while EditStream produces milder distortions than offline baselines.The offline baselines can hallucinate unsupported objects or environmental elements, whereas EditStream operates in a low-latency streaming setting.

8 Ablation Studies

The ablations show that HOI supervision improves human–object interaction, while VMM initialization, warm-up, unrolling, committed-context conditioning, and Energy Annealing shape generation quality, stability, and color balance.

  • 8.1 Data Ablation: HOI supervision makes edited virtual objects behave naturally under user manipulation, maintaining placement and avoiding disappearance, geometric morphing, and interpenetration artifacts.The model can grasp, lift, place, and release generated objects without task-specific inference procedures.
  • 8.2 Distillation Modules Ablation: Energy Annealing balances colorfulness and pixel stability: no-EA over-saturates, EA=0.2 appears flatter and duller, and EA=0.7 lies between them.No-EA has the highest colorfulness but also the highest clipping and color flickering, while EA=0.2 is most temporally stable and least clipped.
  • 8.2 Distillation Modules Ablation: VMM initialization converges within 200 iterations and improves T2V overall, V2V, and RefV2V scores over ODE initialization.The reported scores increase from 80.24 to 83.31 for T2V overall, 6.98 to 7.13 for V2V, and 2.68 to 3.55 for RefV2V.
  • 8.2 Distillation Modules Ablation: Replacing Self-forcing initialization with VMM initialization increases T2V overall from 83.67 to 84.13, V2V from 7.01 to 7.25, and RefV2V from 3.18 to 3.72.The comparison isolates initialization while keeping the Self-forcing unrolling procedure unchanged.
  • 8.2 Distillation Modules Ablation: Removing warm-up degrades all tasks and introduces discontinuities between autoregressive blocks, whereas a 1:4 warm-up/unrolling schedule outperforms or matches alternatives under a 1K-iteration budget.Increasing warm-up raises the T2V semantic score from 80.64 to 80.85, but does not improve overall performance.
  • 8.2 Distillation Modules Ablation: VMM-forcing improves the T2V overall score from 84.13 to 84.42, with gains mainly in Dynamic Degree, Color, Human Action, and perceptual metrics.The two compared methods perform largely comparably on conditional video editing tasks.
  • 8.2 Distillation Modules Ablation: Removing committed-context conditioning or alternating distillation modestly lowers T2V overall performance, while committed-context conditioning more clearly stabilizes unrolling losses.T2V overall decreases from 84.42 to 84.07 and 83.87 for the two ablated variants; editing performance remains largely unchanged.

9 Limitations and Future Work

The current release is primarily designed for short-form video and is not yet optimized for real-time 720p generation. Future work targets long-context modeling and more efficient high-resolution inference.

  • EditStream lacks specialized long-context or long-memory designs and does not investigate error accumulation over long generation horizons.
  • Consequently, the current model is primarily designed and evaluated for short-form video generation and editing.
  • At 720p, the release prioritizes visual quality over further latency optimization and is not yet optimized for real-time generation.
  • Integrating long-context modeling and efficient high-resolution inference is identified as a promising future direction.

10 Conclusion

EditStream unifies diverse video generation and editing tasks through dual-type conditioning and converts a bidirectional model into a causal streaming model with VMM distillation. Experiments report high quality and efficient inference across tasks, while VMM warm-up consistently improves the baseline pipeline and VMM-Forcing better preserves motion and temporal diversity in text-to-video.

  • EditStream unifies diverse video generation and editing tasks within one video DiT through dual-type conditioning.
  • The framework converts bidirectional video diffusion into causal streaming inference using Velocity Moment Matching distillation.
  • Experiments across generation, editing, propagation, reference-guided editing, and camera pose control demonstrate high quality and inference efficiency.
  • VMM warm-up consistently improves the original ODE-initialized Self-Forcing pipeline.
  • Under identical VMM initialization, VMM-Forcing and Self-Forcing perform comparably on conditional editing, while VMM-Forcing better preserves motion and temporal diversity in text-to-video.

A Implementation Details

The implementation trains a fully fine-tuned teacher and distills a strictly block-causal student through VMM warm-up and VMM-forcing unrolling, with a self-forcing baseline for comparison.

  • Teacher training: The teacher fully fine-tunes all 5B Wan2.2-TI2V-5B parameters on 8 nodes with 8 A100-80GB GPUs each.
  • Teacher training: Teacher training uses 704×1280 / 1280×704 resolution buckets, 81 frames, flow matching, AdamW, effective batch 192, and EMA weights.
  • Student distillation: The student is strictly block-causal and undergoes VMM warm-up followed by VMM-forcing unrolling.
  • Student distillation: Student, auxiliary, and teacher models share the teacher’s EMA checkpoint after teacher-forcing adaptation.
  • Self-forcing baseline: The self-forcing baseline uses an offline multi-task ODE-pair generation-and-regression checkpoint instead of direct initialization from bidirectional teacher weights.
  • Table 9 summarizes implementation across the three training stages.

B.1 Camera Pose Control Metrics

Camera pose changes create reprojection holes that require generation, so evaluation separates hole filling, temporal stability, valid-region preservation, and color quality using mask-based metrics.

  • Mask-based evaluation: Camera pose reprojection defines existing pixels as valid and previously occluded or absent pixels as black-pixel holes, represented by a binary mask.
  • Hole completion: Hole Complete measures whether masked hole pixels remain near-black or are filled with new content, reaching 100% only when every hole pixel is rendered.
  • Temporal stability: Hole Temp. Stab. compares optical-flow warp errors in hole and valid regions across consecutive frames to assess temporal consistency.
  • Valid-region preservation: Valid SSIM evaluates preservation of the original video appearance within valid regions against the point-cloud-rendered reference.
  • Hole-fill quality: Gated Hole Fill Qual. evaluates plausibility in the largest hole component while requiring valid-region preservation to prevent full-video regeneration from inflating scores.
  • Color quality: Color quality includes BRISQUE-based hole assessment, Hasler–Süsstrunk colorfulness, luminance dynamic range, clipping fractions, and temporal color flicker.
  • Color quality: Dynamic range is computed from percentile luma values, while shadow and highlight clipping use thresholds τdark = 0.02 and τbright = 0.98.
  • Color quality: Temporal color flicker uses optical-flow backward warping and mean CIEDE2000 difference, where lower values are better.

C.1 A Unified View of Related Distillation Objectives

The section contrasts distillation objectives by where supervision is applied and what is matched. VMM uses student-reached intermediate states for trajectory-conditioned velocity correction, combining local quality refinement with constraints on few-step dynamics.

  • Distribution Matching Distillation: DMD matches student and teacher endpoint distributions using a teacher–fake score residual at the same re-noised endpoint state.Its independently sampled noise scale examines endpoint distributions rather than the transition from source timestep t.
  • Trajectory-based objectives: Consistency Distillation, CTM, and teacher-distilled MeanFlow derive supervision from teacher or reference trajectories, matching endpoints, flow maps, or average velocities.These objectives use teacher trajectories to specify target states or finite-time transport.
  • Velocity Moment Matching: VMM first generates the destination state with the student transition, then evaluates teacher–auxiliary velocity residuals at that student-induced intermediate state.This makes the student destination a query state for velocity correction rather than a reference finite-time target.
  • Velocity Moment Matching: VMM is intended to balance DMD-like local detail recovery with teacher-consistent few-step trajectories by constraining local sampling dynamics.The method applies residual correction at student-generated states while retaining an association with a specific t→s inference transition.
  • Causal few-step initialization: VMM warm-up avoids precomputed teacher ODE pairs, a separate causal teacher, paired teacher destinations, and JVP-based consistency training.The bidirectional teacher can be retained, and the same VMM objective continues from warm-up into unrolling.
  • Causal few-step initialization: During unrolling, later-block history is produced by the complete sampler used at inference, while VMM remains applicable in both warm-up and unrolling.DMD may favor perceptual sharpness, whereas VMM is more closely aligned with a fixed deterministic ODE schedule.
Loading 2608.21424v1…