Source-linked AI summary
JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion
Yicheng Xiao, Wenxun Dai, Xinran Qin, Lin Song, Maoquan Zhang, Hang Xu, Yukang Chen, Yitong Li, Guohui Zhang, Yuan Zhang, Xuying Zhang, Tommy Zhang, Jianlong Yuan, Peihao Li, Shuai Lu, Siming Fu, Chuyang Zhao, Xin Han, Jie Huang, Wenbo Li, Guoqing Ma, Wei Huang, Xiaojuan Qi, Haoyang Huang, Nan Duan
TL;DR
Real-time open-ended video editing must emit causally with bounded computation while preserving source fidelity and long-term consistency. JoyAI-Video-Edit addresses this with chunk-wise autoregressive adaptation, source-anchored two-step distillation, and long-horizon training, substantially outperforming streaming editors while reaching approximately 30 FPS for 720p editing on one Nvidia B200 GPU.
Problem
Real-time video editing requires causal generation with bounded computation while preserving source fidelity and long-term temporal consistency.
Method
JoyAI-Video-Edit combines chunk-wise causal adaptation, Source-Anchored Distribution Matching Distillation, and Long-Horizon Autoregressive Distillation in a 16B-parameter autoregressive diffusion framework.
Results
JoyAI-Video-Edit substantially outperforms existing streaming editors, remains competitive with strong offline systems on short and long videos, and reaches approximately 30 FPS for 720p editing on one Nvidia B200 GPU.
Takeaways & Limitations
The system supports continuous editing while preserving subject identity, motion, spatial structure, and unrelated regions as new source frames arrive.
Abstract
from arXiv · showhide
Real-time video editing requires low-latency causal generation with bounded computational resources while preserving source fidelity and long-term temporal consistency. We present JoyAI-Video-Edit, a 16B-parameter autoregressive diffusion framework for real-time, open-ended video editing without access to future frames or a predefined video duration. Our method combines chunk-wise autoregressive adaptation, Source-Anchored Distribution Matching Distillation (SA-DMD), and Long-Horizon Autoregressive Distillation to reduce train--inference mismatch, preserve source fidelity during two-step generation, and mitigate accumulated temporal drift. Extensive automatic and human evaluations show that JoyAI-Video-Edit substantially outperforms existing streaming editors and remains competitive with strong offline systems on both short and long videos. The complete system achieves end-to-end 720p video editing at approximately 30 FPS on a single Nvidia B200 GPU. Code is available at https://github.com/jd-opensource/JoyAI-Video-Edit.
1 Introduction
JoyAI-Video-Edit targets open-ended causal video editing under strict latency and memory constraints, addressing train–inference mismatch and long-term source drift. It combines chunk-wise adaptation, source-anchored two-step distillation, and long-horizon training, achieving strong quality and real-time throughput.
- Motivation: Offline editors require future frames and growing temporal state, preventing causal output with bounded computation.Global temporal interactions and memory scale with clip length, while independently editing consecutive clips can disrupt continuity.
- Motivation: Autoregressive inference can accumulate reconstruction, color, and appearance errors because training uses clean history while inference uses imperfect predictions.Video editing additionally requires source alignment, untargeted-content preservation, and consistent transformation across chunks.
- Approach: JoyAI-Video-Edit converts a bidirectional editor into chunk-wise causal generation with bidirectional within-chunk attention and causal cross-chunk attention.A sliding temporal window retains recent chunks and the first chunk as a global sink, bounding temporal state and per-chunk computation.
- Approach: SA-DMD distills iterative diffusion into a two-step generator while using temporally aligned source chunks to counteract drift from imperfect autoregressive history.Long-Horizon Autoregressive Distillation supervises states affected by accumulated errors through segmented extended rollouts.
- Evaluation: The unified framework includes LongV2VBench, a long-video benchmark covering representative global and local editing tasks.The benchmark supports sustained-stream evaluation alongside the proposed training framework.
- Results: 720p editing runs at approximately 30 FPS on one Nvidia B200 GPU while preserving identity, motion, spatial structure, and unrelated regions.Evaluations report substantially better performance than streaming editors and competitiveness with strong offline systems on short and long videos.
2 Related Work
Related work spans offline editing, efficient video diffusion, streaming systems, autoregressive training, and few-step distillation. These directions address complementary quality, efficiency, causality, or stability challenges, but do not jointly provide JoyAI-Video-Edit’s open-ended streaming framework.
- Offline Editing: Offline video editors achieve diverse transformations using global or bidirectional temporal interactions, whereas streaming requires causal emission and reusable temporal state.Their fixed-clip formulation is effective offline but does not directly support continuous input streams.
- Efficiency: Efficiency-oriented methods reduce video diffusion cost through compressed latent spaces or sparse token processing, but do not provide general open-ended instruction-guided streaming editing.These approaches target computational bottlenecks complementary to streaming framework design.
- Streaming Editing: Streaming methods use causal or incremental computation for interactive video generation and editing, including long-video training, mask-cache reuse, and live-camera manipulation.The passage characterizes these systems as practical streaming approaches while noting their trade-offs.
- Autoregressive Generation: Diffusion-forcing and causal-generation systems model sequences with reusable or bounded histories, while Self Forcing explicitly unrolls generated autoregressive trajectories.Self Forcing uses post-training, few-step generation, and truncated gradients to control sequential rollout cost.
- Positioning: JoyAI-Video-Edit combines an MLLM, causal video VAE, and MM-DiT to jointly process conditions and latent visual tokens for edited-video generation.The architecture connects multimodal conditioning, causal latent representation, and diffusion transformation in one system.
- Distillation: Distribution Matching Distillation compresses iterative sampling, while Causal Forcing emphasizes architectural and state-distribution mismatch between bidirectional teachers and causal students.Long-duration generation methods further motivate accounting for long-horizon behavior during distillation.
3 Model
JoyAI-Video-Edit is a multimodal autoregressive diffusion model initialized through progressive text-to-video and bidirectional editing training. Its architecture combines semantic conditioning, causal latent video encoding, and diffusion-based editing for V2V and IV2V inputs.
- Architecture: JoyAI-Video-Edit is a unified autoregressive diffusion model comprising an MLLM, causal video VAE, and MM-DiT.It supports both video-to-video and image-and-video-to-video editing under different input conditions.
- Architecture: The MLLM extracts condition tokens from the source’s first frame and editing instruction to encode source content and intended transformation.These tokens provide semantic guidance to the diffusion transformer throughout denoising.
- Architecture: The causal video VAE encodes source videos and optional reference images into a shared latent space with temporal and spatial compression.Its compression ratio is 8 × 24 × 24, so each latent frame represents eight video frames.
- Training: Foundation training progresses from text-to-image alignment to text-to-video modeling, higher-quality supervised fine-tuning, and continual training.The curriculum gradually increases resolution and frame rate while retaining spatial-generation objectives.
- Training: Bidirectional V2V and IV2V supervision teaches instruction-conditioned transformation and content preservation before causal adaptation.Bidirectional target attention exposes the model to complete edited clips and initializes temporally coherent editing.
- Training: For noisy target training, the causal VAE produces source, reference, and target latents from the editing inputs.The reference latent is omitted for V2V samples without a reference image.
4 JoyAI-Video-Edit
The system adapts bidirectional editing to chunk-wise causal generation, then accelerates and stabilizes it with source-anchored distillation and segmented long-horizon optimization. Bounded temporal state supports sustained streaming without requiring full-video computation.
- Chunk-wise Adaptation: Causal adaptation generates aligned source and target chunks sequentially, using bidirectional attention within each chunk and causal attention across chunks.Each target chunk accesses only current and preceding chunks, enabling online decoding without future frames.
- Chunk-wise Adaptation: A fixed recent-history window plus the first chunk as a global sink keeps attention context constant as video length grows.The sink acts as a persistent anchor for long-horizon generation, while source, condition, reference, and history tokens are controlled by an attention mask.
- SA-DMD: DMD training uses a causal generator, trainable fake-score model, and frozen real-score model initialized under a shared-backbone LoRA configuration.The objective pulls the student toward the teacher through a mode-seeking reverse-KL formulation.
- SA-DMD: SA-DMD guides the teacher along text-conditioning and source-fidelity axes, producing a source-anchored target for the few-step generator.The source-aware guidance is absorbed into the generator so deployment can preserve fidelity with a single conditional branch.
- Long-Horizon Distillation: Long-horizon optimization divides extended rollouts into shorter clips, clears each graph after its backward pass, and accumulates gradients across clips.This exposes training to compounded autoregressive errors while keeping memory bounded; dynamic mirror looping extends conditioning when source video is shorter.
5 Real-Time Deployment
The deployment pipeline processes incoming video in consecutive chunks, using cached history and optimized execution to deliver real-time 720p editing.
- Chunk-wise processing: Incoming streams are divided into consecutive eight-frame chunks, which are encoded, edited with cached history, and decoded immediately without waiting for future frames.Clean key-value states are cached for subsequent chunks.
- Runtime: 30.1 FPS is achieved on a single Nvidia B200 GPU through a 266-ms complete cycle for each chunk.The cycle includes 22 ms for VAE encoding, 185 ms for DiT denoising, and 19 ms for VAE decoding, plus cache construction and pseudo encoding.
6 Data
The data pipeline combines filtered, diverse text-to-video and paired video-editing data, including global, local, and subject-level transformations.
- Text-to-video data: Text-to-video data span people, lifestyle, entertainment, nature, objects, and urban scenes, with quality, motion, stability, and deduplication filtering.Dominant categories are downsampled while long-tail concepts are retained as much as possible.
- Video-editing data: Paired video-editing data transfer image-editing supervision to video through image-to-image and reference-to-image sources.This addresses the difficulty of collecting high-quality paired video-editing data at scale.
- Pair construction: Paired videos are generated through keyframe-guided edit propagation or latent-shared image-to-video generation.The pipelines preserve original motion and unedited content, or share early denoising latents for consistency.
- Filtering and coverage: Generated pairs are filtered for visual quality, editing correctness, content preservation, and temporal consistency, then recaptioned by an MLLM.The resulting data cover local and global edits, including subject addition, replacement, and removal.
7 Experiments
Experiments evaluate short- and long-video editing quality, human preferences, and deployment throughput against streaming and offline systems. JoyAI-Video-Edit leads streaming baselines across reported quality and efficiency evaluations.
- Evaluation setup: The evaluation compares JoyAI-Video-Edit with streaming baselines including StreamDiffusionV2, SANA-Streaming, LiveEdit, and XMax-X2.0, plus numerous offline editors.Short-video evaluation uses the five-category OpenVE-Bench protocol.
- Short-video editing: 3.60 overall score on OpenVE-Bench exceeds SANA-Streaming, LiveEdit, XMax-X2.0, and StreamDiffusionV2 by 0.98, 1.60, 1.73, and 2.37 points, respectively.It ranks first among streaming methods in four of five categories and remains comparable to strong offline editors.
- Long-video editing: 3.30 overall score on LongV2VBench ranks first in all five editing categories and exceeds XMax-X2.0 by 1.59 points.LongV2VBench contains 229 one-minute editing tasks across five global and local categories.
- Throughput: 30.19 FPS at 720×1280 is 44.4% faster than XMax-X2.0 at a substantially lower resolution and more than twice as fast as SANA-Streaming at a comparable resolution.The bounded-history causal design reuses fixed temporal state and maintains stable per-chunk computation as streams grow.
- Human evaluation: 90%, 87%, 81%, and 87% preference votes were received against LiveEdit, SANA-Streaming, XMax-X2.0, and StreamDiffusionV2, respectively.Against Bernini-R, JoyAI-Video-Edit received 48% versus Bernini-R’s 44%.
- Ablation study: SA-DMD raises global style from 3.61 to 4.24 and local change from 3.43 to 4.00, while LHAD increases the overall score to 3.06 before both strategies reach 3.30.The reported results associate source anchoring with source-drift reduction and LHAD with stabilization of late, error-accumulated states.
- Visualization: Visualization results show instruction following while preserving subject identity, spatial layout, motion trajectories, and unrelated regions as new frames arrive.The causal pipeline provides immediate visual feedback for continuous instruction-level interaction.
8 Authors
The paper lists its authors and identifies contribution, project-leadership, and corresponding-author roles.
- Author list: The author list includes Yicheng Xiao, Wenxun Dai, Xinran Qin, Lin Song, and many additional contributors.The complete author list is presented across the supplied passages.
- Author roles: ∗ marks equal contribution, † marks the project leader, ‡ marks corresponding authors, and § indicates alphabetical contributor listing.These symbols qualify the author names in the paper.