Source-linked AI summary
Vidu S2: Real-Time Interactive, Editable, and Spatial Video Generation
Jintao Zhang, Kai Jiang, Jintao Chen, Xu Wang, Deyuan Liu, Jungang Li, Dechuang Chen, Ming Lin, Jingjiang Zhou, Haopeng Jin, Qi Jia, Xiaohang Wang, Yaole Wang, Zhanqiang Zhang, Ran Li, Zhengkun Huang, Shuyue Xiong, Yuji Wang, Zikun Dai, Hui He, Yang Luo, Mang Ning, Weiqi Feng, Chengyang Ye, Xinyue Lin, Min Zhao, Hongzhou Zhu, Hengkai Tan, Zeyuan Wang, Chendong Xiang, Kaiwen Zheng, Zhijie Deng, Fan Bao, Jianfei Chen, Jun Zhu
TL;DR
Vidu S2 addresses the need for video generation and editing that responds immediately during interactive visual experiences. It combines a real-time interactive avatar model with a real-time stream-editing model and explores spatial video, with experiments showing that Vidu S2 outperforms all baselines.
Problem
Offline, one-shot video generation requires users to wait for complete videos and does not support active interaction during generation.
Method
Vidu S2 combines Vidu S2-Avatar for interactive digital-character generation, Vidu S2-Editing for real-time stream editing, and spatial-video generation and editing.
Results
Vidu S2 outperforms all baselines while Vidu S2-Avatar supports real-time 720p generation, dynamic references, and stronger instruction following.
Takeaways & Limitations
The system supports continuously updated interactive visual experiences across avatar generation, video editing, and spatial video.
Takeaways & Limitations
Real-time spatial video still requires higher resolution and lower latency than conventional real-time monocular video generation, creating a key technical challenge.
Abstract
from arXiv · showhide
We present Vidu S2, which comprises Vidu S2-Avatar, a real-time interactive digital-character model, and Vidu S2-Editing, a real-time video editing model. Moreover, we explore the feasibility of real-time spatial video generation for both Vidu S2-Avatar and Vidu S2-Editing. Compared with Vidu S1, Vidu S2-Avatar supports real-time 720p video generation, generation with dynamic references that can be updated at any moment, and stronger instruction following, such as dancing. Vidu S2-Editing supports editing a video stream in real time, including style rendering, clothing replacement, character replacement, and background replacement. Experiments show that Vidu S2 outperforms all baselines. A playable online demo is available at https://vidu.com/vidu-stream.
1 Introduction
Vidu S2 targets the gap between offline, one-shot video generation and interactive visual experiences requiring immediate user response. It introduces real-time avatar generation, stream editing, spatial-video support, and an efficient serving stack, with experiments reporting superiority over all baselines while meeting real-time requirements.
- Unlike offline diffusion systems that generate complete videos after many denoising steps, Vidu S2 addresses interactive experiences requiring immediate responses.
- Vidu S2-Avatar supports real-time 720p generation, dynamically updated references, and stronger instruction following for motions such as dancing.
- Vidu S2-Editing edits incoming video streams on the fly through style transfer, virtual try-on, character replacement, and background replacement.
- Vidu S2 extends real-time avatar generation and video editing to synchronized spatial-video experiences for VR head-mounted displays.
- The system uses attention, quantization, kernel, and multi-GPU scheduling optimizations to make real-time models practical on low-cost GPUs.
- Vidu S2 outperforms all baselines while fully meeting real-time inference requirements.
2 Vidu S2-Avatar
Vidu S2-Avatar combines expanded and refined data preparation with an audio-visual diffusion model adapted for real-time streaming. Its training uses segment-wise conditioning, causal adaptation, and Self-Replay Forcing to improve instruction following, streaming robustness, and cross-segment optimization.
- 2.1 Data Preparation: Avatar training data expand toward dance and animation while refining clipping, filtering, captioning, and stabilization to improve motion diversity and training-data quality.The pipeline retains five stages and adds operators for subtle edit-point detection, high-clarity selection, and background stabilization; captions emphasize temporal structure.
- 2.2 Method: The audio-visual joint Diffusion Transformer predicts video-audio segments conditioned on a reference image and a sequence of segment-specific captions.The model jointly predicts clean video-audio latents for multiple segments, with the reference shared across them.
- 2.2 Method: Vidu S2-Avatar jointly trains image-to-video and reference-to-video generation with segment-wise conditions rather than a single prompt for the full sequence.The reference image is shared across segments to maintain appearance and identity, while segment-wise supervision improves instruction following.
- 2.2 Method: The model adapts a pretrained bidirectional video-audio model with block-wise causal attention and combined Teacher Forcing and Diffusion Forcing for streaming generation.Training alternates clean and noisy historical states, improving robustness to accumulated generated-history errors.
- 2.2 Method: Vidu S2-Avatar uses Self-Replay Forcing to replay self-generated, re-noised trajectories while preserving gradient flow across replayed segment boundaries.The detached rollout avoids retaining the full computation graph, while gradient-enabled replay applies DMD supervision across blocks.
3 Vidu S2-Editing
Vidu S2-Editing performs instruction- and reference-conditioned video edits in real time while preserving source motion through frame-aligned conditioning. Its training combines bidirectional editing pretraining with causal streaming methods and inference optimizations.
- 3.1 Overview: Vidu S2-Editing supports real-time style transfer, clothing replacement, character replacement, and background replacement from text instructions and optional reference images.Frame-aligned attention reads the source frame at the same time step for each target frame, preserving motion and temporal correspondence.
- 3.2 Method: The model encodes source-video and reference-image conditions alongside noisy target-video tokens, using condition-specific RoPE to represent their spatial and temporal positions.The source supplies spatial and motion context, while the reference image adds appearance information when available.
- 3.2 Method: Bidirectional training establishes visual and motion priors while restricting source-target interaction to temporally aligned frame pairs.This avoids unrestricted mixing between misaligned source and target contents.
- 3.2 Method: Causal streaming training replaces bidirectional temporal attention with block-wise causal attention and combines Teacher Forcing with Diffusion Forcing and Self-Replay Forcing.The procedure is designed to improve robustness to imperfect historical contexts during streaming inference.
- 3.3 Inference Optimization: Fine-grained inter-module scheduling shares GPU capacity among the VAE encoder, backbone, refiner, and decoder to reduce resource waste and end-to-end latency.Modules can use GPUs when other modules are idle instead of reserving fixed groups throughout inference.
4 Real-Time Spatial Video Generation and Editing
Vidu S2 extends real-time monocular generation and editing into synchronized spatial-video experiences for head-mounted displays. The pipelines support monocular-to-spatial conversion and direct stereoscopic editing, while high resolution and low latency remain deployment challenges.
- 4.1 Real-Time Spatial Video Generation: Spatial video increases perceived depth, scale, object positioning, and presence compared with flat video, supporting a more immersive entertainment experience.The motivation is to preserve continuous interaction while improving the spatial sense of generated characters.
- 4 Real-Time Spatial Video Generation and Editing: Vidu S2 converts avatar-generated or edited monocular streams into synchronized left- and right-eye views and can directly edit existing spatial video.These workflows target immersive viewing through VR head-mounted displays.
- 4.1 Real-Time Spatial Video Generation: The generation pipeline estimates per-frame depth, maps it to horizontal disparity, warps a center view in opposite directions, fills disocclusion holes, and temporally stabilizes depth.The resulting synchronized chunks preserve real-time character interaction while conveying object distance, scale, and scene depth.
- 4.2 Real-Time Spatial Video Editing: For monocular inputs, the editor edits first and then applies spatial conversion; for stereoscopic inputs, it edits horizontally joined views before splitting them back into eye views.Both workflows use the same visual references across the two views.
- 4.3 Discussion: Headset deployment requires high resolution and low latency because spatial video occupies much of the field of view and delay can desynchronize head motion from passthrough views.Maintaining visual quality while reducing end-to-end latency remains a key technical challenge.
- 4.3 Discussion: A proposed future direction is panoramic spatial video that lets users turn their heads and explore generated scenes freely.The paper connects this direction to interactive, real-time metaverse content.
5 Experiments
The experiments evaluate Vidu S2 on streaming digital-character generation and streaming video editing using public benchmarks and internal comparisons with commercial systems.
- 5 Experiments: Vidu S2 is evaluated on two complementary tasks: streaming digital-character generation and streaming video editing.The experiments include public-benchmark comparisons with open-source systems and randomized Good/Same/Bad pairwise comparisons with commercial systems.
5.1 Experimental Setup
The evaluation combines standardized public benchmarks with internal randomized pairwise comparisons across streaming character generation and video editing. Metrics cover visual, audio, alignment, consistency, instruction-following, and editing quality dimensions.
- 5.1.1 Tasks and Benchmarks: The setup evaluates digital-character generation with StreamAV-Bench and an internal benchmark covering streaming quality, long-horizon stability, updates, interruption recovery, and audio–video synchronization.StreamAV-Bench contains 160 scenarios in each of its Progressive and Interactive tracks; the Interactive Track receives five updates every 30 seconds and runs up to 180 seconds.
- 5.1.1 Tasks and Benchmarks: Video editing is evaluated with OpenVE-Bench, Sparkle-Bench, RefVIE-Bench, ViViD, and an internal benchmark spanning style transfer, virtual try-on, subject replacement, and background replacement.The public benchmarks cover instruction-guided, reference-conditioned, and unpaired virtual try-on settings.
- 5.1.3 Evaluation Protocols: Public comparisons use identical inputs, preprocessing, temporal sampling, evaluators, and scoring rubrics, while inherited benchmark metrics retain their original procedures and directionality.Published results are reused only when benchmark, split, and metric match.
- 5.1.3 Metrics: Character-generation metrics measure visual appeal, fidelity, audio quality, audio–visual alignment and synchronization, instruction fulfillment, and long-horizon subject and background consistency.AVSync estimates temporal synchronization error, so lower values are better.
- 5.1.3 Metrics: Editing metrics retain task-specific definitions for global style, background change, instruction compliance, motion preservation, reference fidelity, visual harmony, and temporal consistency.OpenVE Overall, RefVIE Overall, and Joint Overall are benchmark-defined aggregates.
- 5.1 Experimental Setup: Vidu S2-Avatar achieves the best value in every reported StreamAV-Bench metric, with gains distributed across visual, audio, alignment, and consistency dimensions.The reported table includes VA, VQ, PQ, AQ, AVAlign, AVSync, AIF, SC, and BC measurements.
- 5.1.3 Evaluation Protocols: Internal comparisons randomize pair presentation and use 20 calibrated professional evaluators who label each output Good, Same, or Bad.The protocol records input pairs, random seeds, model versions, service regions, and annotation records; product-page claims do not replace measured results.
5.2 Public-Benchmark Results
On public benchmarks, Vidu S2-Avatar leads across reported digital-character metrics, while Vidu S2-Editing achieves top results across multiple editing evaluations.
- Digital-Character Generation: Vidu S2-Avatar achieves the best value on every reported StreamAV-Bench metric, with gains distributed across visual, audio, synchronization, instruction, identity, and scene dimensions.The results indicate balanced performance rather than gains concentrated in one aspect.
- Video Editing: Vidu S2-Editing achieves the highest Sparkle-Bench Overall score of 3.74 and leads the reported global, foreground, and background criteria.The listed category scores include 4.00 for global instruction, 3.98 for foreground instruction, and 4.00 for foreground motion.
- Video Editing: Vidu S2-Editing achieves the highest reported scores across all OpenVE and RefVIE metrics, with a Joint Overall score of 4.26 that surpasses Bernini-R 14B by 0.34.It also reaches 4.42 Overall on OpenVE and 3.78 Overall on RefVIE, exceeding the streaming baseline Decart-Lucy2.5 by 0.07 on RefVIE.
- Video Editing: Vidu S2-Editing achieves a VFIDI of 9.9515 on the unpaired ViViD virtual try-on benchmark, outperforming CatV2TON and ViViD.The reported values are 19.5131 for CatV2TON and 21.8032 for ViViD, with lower values favored by the evaluation.
5.3 Complementary Human Preference Evaluation
Human evaluations assess quality over duration and through paired comparisons. Vidu S2-Avatar maintains strong ratings during longer streams and is consistently preferred over closed-source systems, while Vidu S2-Editing is evaluated on controlled paired cases.
- Digital-Character Generation: The duration-stratified ratings complement paired GSB judgments by measuring how streaming quality evolves as generation extends from 10 to 90 seconds.The evaluation separates sustained long-horizon behavior from clip-level preference judgments.
- Digital-Character Generation: Vidu S2-Avatar consistently achieves the highest ratings across five dimensions from 10 to 90 seconds, with consistency remaining high through 90 seconds.The evaluated dimensions are overall quality, consistency, video quality, motion quality, and emotional expression.
- Digital-Character Generation: Vidu S2-Avatar is preferred across overall quality, motion, expression, semantic adherence, and temporal consistency against three closed-source systems.Overall-quality preference reaches 85.7% against Runway Character GWM-1 and 100% against PixVerse Image Avatar and HeyGen.
- Video Editing: Vidu S2-Editing is evaluated on 150 paired cases using identical source videos, reference images, prompts, and temporal sampling across systems.The internal benchmark reports overall quality, video quality, temporal consistency, and semantic adherence; mean consistency scores are 3.56, 2.59, and 1.67 for Vidu S2-Editing, Decart-Lucy2.5, and XMax-X2.0, respectively.
5.4 Qualitative Case Studies
Qualitative cases show Vidu S2 preserving identity, motion, geometry, boundaries, and fine details during avatar generation and video editing, while spatial outputs provide separated foreground and background views.
- Digital-Character Generation: Vidu S2-Avatar preserves facial identity, hairstyle, facial proportions, product geometry, and hand structure across sampled frames under fixed identity, scene, and motion conditions.The comparison reports facial-identity and body-proportion drift for PixVerse Image Avatar and hair and finger distortions for Runway Character GWM-1.
- Video Editing: In virtual try-on, Vidu S2-Editing preserves body motion, garment boundaries, material texture, and hand–cloth occlusion relationships while transferring target garments.These properties are explicitly identified in the virtual try-on cases.
- Video Editing: Vidu S2-Editing transfers reference styles, garments, identities, and backgrounds while preserving source motion, pose, geometry, and temporal boundaries.The cases cover style transfer, virtual try-on, subject replacement, and background replacement.
- Spatial Video Generation: Spatial-video examples separate foreground subjects from backgrounds, producing coherent scene depth and a stronger sense of immersion.The examples show generated left- and right-eye views.
6 Conclusion
Vidu S2 combines real-time interactive avatar generation, real-time video editing, and an exploration of real-time spatial video generation. Its avatar model supports 720p output and dynamic references, while the editing model covers several stream-editing tasks and outperforms the evaluated baselines.
- Conclusion: Vidu S2 comprises Vidu S2-Avatar, Vidu S2-Editing, and an exploration of real-time spatial video generation.The two models target interactive digital-character generation and real-time video editing, respectively.
- Conclusion: Compared with Vidu S1, Vidu S2-Avatar supports real-time 720p generation, dynamically updated references, and stronger instruction following such as dancing.These capabilities are stated as improvements over Vidu S1.
- Conclusion: Vidu S2-Editing edits incoming video streams in real time through style transfer, virtual try-on, character replacement, and background replacement.The conclusion lists the four supported editing categories.
- Conclusion: Experiments show that Vidu S2 outperforms all baselines.This is the paper’s overall conclusion about the reported evaluations.