Source-linked AI summary
Vidu S1: A Real-Time Interactive Video Generation Model
Jintao Zhang, Kai Jiang, Jintao Chen, Xu Wang, Yang Luo, Yuji Wang, Dechuang Chen, Jungang Li, Chengyang Ye, Marco Chen, Hongzhou Zhu, Min Zhao, Yuxuan Jiang, Zhengkun Huang, Chendong Xiang, Kaiwen Zheng, Haoxu Wang, Xiaohang Wang, Qi Jia, Xin Chen, Yimin Chen, Youhe Jiang, Fangcheng Fu, Zhijie Deng, Fan Bao, Jianfei Chen, Jun Zhu
TL;DR
Existing video generators largely operate offline and lack direct speech control, real-time interaction, and stable long-horizon generation. Vidu S1 addresses these limitations with voice-guided interactive streaming and an efficient inference stack, achieving leading quality while generating 540p video at up to 42 FPS on consumer GPUs.
Problem
Existing video generation models mainly use offline, one-shot generation, while current autoregressive systems often lack real-time interaction, explicit speech control, or stable long-horizon generation.
Method
Vidu S1 combines continuous real-time interaction, speech-guided control of future video content, stable long-horizon streaming, and an efficient inference and serving stack.
Results
Vidu S1 achieves the best CSIM (0.9192), Sync-D (7.847), and DOVER (0.5660) while supporting speech-guided control and stable long-horizon streaming.
Takeaways & Limitations
Vidu S1 demonstrates practical, controllable, and personalized real-time video generation, producing 540p videos at up to 42 FPS on consumer GPUs.
Takeaways & Limitations
Existing long-horizon video generation remains challenging because accumulated errors can cause drift, instability, and eventual visual collapse.
Abstract
from arXiv · showhide
We introduce Vidu S1, a real-time interactive video generation model supporting voice control of digital characters. Users can control video generation content at any moment through voice instructions. Vidu S1 supports infinite-length real-time video generation without blurring, drift, or visual distortion. Built with TurboDiffusion and TurboServe, Vidu S1 outputs 540p real-time videos at up to 42 FPS on regular consumer GPUs. Users can upload custom images of real people, anime, and pets, and choose different voice tones for personalized experiences. Experiments show that Vidu S1 achieves the best performance across all test metrics while fully meeting real-time inference requirements. A playable online demo is available at https://vidu.com/vidu-stream.
1 Introduction
Vidu S1 addresses the gap between offline, one-shot video generation and interactive visual experiences by enabling real-time user interaction, speech control, and stable long-horizon streaming. Its inference and serving stack supports 540p generation at 42 FPS, while experiments report leading quality and real-time interactivity.
- Motivation: Existing video generators mainly operate offline, requiring users to wait for complete videos and preventing interaction during generation.This paradigm denoises the entire video synchronously and produces a clean result only at the end.
- Limitations: Real-time interactive generation remains limited by absent speech control, long-horizon drift, and demanding inference infrastructure.Autoregressive generation alone does not ensure interaction, open-ended stability, or practical deployment.
- Vidu S1: Vidu S1 enables continuous interaction and uses speech as an explicit signal to guide future video content during generation.Users can intervene at any moment rather than specifying all controls before generation begins.
- Vidu S1: Vidu S1 mitigates streaming error accumulation to support indefinitely long video generation without drift or collapse.The model is designed for stable long-horizon real-time generation.
- System performance: 540p at 42 FPS makes Vidu S1 a real-time video generator on low-cost or commodity GPUs through its efficient inference and serving stack.The stack includes TurboDiffusion and TurboServe.
- Evaluation: Vidu-StreamBench and HDTF experiments report best CSIM (0.9192), Sync-D (7.847), and DOVER (0.5660), alongside speech-guided control and stable long-horizon streaming.The reported results combine quality metrics with real-time interactivity and streaming stability.
2 Method
Vidu S1 is trained through progressive data preparation and a three-stage video-audio generation pipeline. Bidirectional pretraining is adapted to causal streaming generation, then optimized for stable autoregressive rollout and long-term consistency.
- Data Preparation: The data pipeline progressively converts raw videos into single-shot, filtered, diarized, and captioned training clips.It applies prefiltering, shot segmentation, subject, cleanliness, quality, and safety checks before speech diarization and multilevel captioning.
- Training Objective: The training framework targets temporally coherent video with synchronized audio under multimodal conditioning.The unified conditioning interface includes speech, text prompts, and reference images, while joint video-audio states are modeled together.
- Stage 1: Bidirectional Teacher Training: Stage 1 trains a bidirectional teacher on complete video-audio sequences conditioned on the full condition sequence.The teacher denoises joint latent states across a finite sequence of N frames.
- Stage 2: Causal Teacher Training: Stage 2 adapts the bidirectional teacher to causal autoregressive generation using a causal attention mask and historical video-audio prefixes.For each frame i > 1, only the current joint state is denoised using available conditions and the historical prefix.
- Stage 2: Causal Teacher Training: Hybrid Teacher Forcing and Diffusion Forcing training improves both synchronization supervision and robustness to imperfect historical prefixes.Teacher Forcing uses clean ground-truth prefixes, whereas Diffusion Forcing uses noisy historical states during training.
- Streaming Generation: The adapted causal teacher retains bidirectional generation quality while acquiring stable streaming capability, and TwinCache balances long-term consistency with visual fidelity.TwinCache separates temporal propagation from appearance refinement through stage-aware cache scheduling.
3 Evaluations
Vidu-StreamBench and HDTF evaluate whether Vidu S1 can combine instruction-following, avatar quality, and real-time interactive generation. Results report strong human preference, leading identity, synchronization, and perceptual metrics, plus 42 FPS 540p streaming on RTX 5090 GPUs.
- Benchmark design: Vidu-StreamBench contains 500 samples pairing action instructions, reference frames, and audio to test natural motion, stability, identity preservation, and instruction following.The benchmark also covers varied image styles, speaker attributes, emotions, and application scenarios.
- Human preference: Vidu S1 is consistently preferred over leading commercial avatar systems in pairwise comparisons on Vidu-StreamBench.The evaluation compares Vidu S1 with HeyGen, LemonSlice, and Kling-Avatar-2.0.
- Human preference: 100% preference against HeyGen and LemonSlice is achieved for subject controllability, where raters favored explicit action following.The reported advantage concerns visible avatar behavior, including gestures, head movement, and controllable action execution.
- Qualitative comparison: Vidu S1 demonstrates stronger visual stability and instruction following than Kling Avatar 2.0 given the same reference image, audio, and action instruction.The qualitative comparison highlights correct head-raising and thumbs-up gestures for Vidu S1 versus failures in the compared method.
- Public benchmark: Vidu S1 achieves leading HDTF performance across identity preservation, audio-visual synchronization, and perceptual quality while supporting real-time generation and instruction following.The reported metrics are CSIM, Sync-D, and DOVER.
- Real-time capability: 42 FPS average throughput is reached for 540p video with a 3-step configuration on RTX 5090 GPUs, exceeding the 30-FPS real-time playback threshold.The paper states that this efficiency preserves identity consistency, audio-visual synchronization, and perceptual quality.
4 Conclusion
Vidu S1 is presented as a real-time interactive video generation model for voice-controlled digital characters. It supports voice-guided control at any moment, infinite-length generation with stable visual quality and identity, and 540p output at up to 42 FPS on consumer GPUs.
- Conclusion: Vidu S1 enables users to guide video content at any moment through voice instructions and supports infinite-length generation with stable visual quality and subject identity.The conclusion frames these capabilities as supporting practical, controllable, and personalized video generation.
- Conclusion: 540p videos are produced at up to 42 FPS on consumer GPUs while experiments report best overall performance and real-time inference requirements are met.The conclusion presents these results as a step toward realistic, consistent, and easy-to-control generative video systems.