Source-linked AI summary
SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model
Guibin Chen, Dixuan Lin, Jiangping Yang, Youqiang Zhang, Zhengcong Fei, Debang Li, Sheng Chen, Chaofeng Ao, Nuo Pang, Yiming Wang, Yikun Dou, Zheng Chen, Mingyuan Fan, Tuanhui Li, Mingshan Chang, Hao Zhang, Xiaopeng Sun, Jingtao Xu, Yuqiang Xie, Jiahua Wang, Zhiheng Xu, Weiming Xiong, Yuzhe Jin, Baoxuan Gu, Binjie Mao, Yunjie Yu, Jujie He, Yuhao Feng, Shiwen Tu, Chaojie Wang, Rui Yan, Wei Shen, Jingchen Wu, Peng Zhao, Xuanyue Zhong, Zhuangzhuang Liu, Kaifei Wang, Fuxiang Zhang, Weikai Xu, Wenyan Liu, Binglu Zhang, Yu Shen, Tianhui Xiong, Bin Peng, Liang Zeng, Xuchen Song, Haoxiang Guo, Peiyu Wang, Max W. Y. Lam, Chien-Hung Liu, Yahui Zhou
TL;DR
Existing video systems remain fragmented across multimodal conditioning, synchronized audiovisual generation, and generation, inpainting, or editing. SkyReels-V4 unifies these capabilities with dual-stream MMDiT, a shared MLLM encoder, channel-concatenated inpainting, and an efficiency strategy for cinematic outputs. It reports top-tier Arena performance and the highest overall SkyReels-VABench average, with particular strengths in prompt following and motion quality.
Problem
Existing systems do not simultaneously support rich multimodal inputs, joint video-audio generation, and unified generation, inpainting, and editing.
Method
SkyReels-V4 combines dual-stream MMDiT video and audio branches, a shared MLLM encoder, channel-concatenated inpainting, and efficient low-resolution/keyframe generation.
Results
SkyReels-V4 achieves the highest overall average score on SkyReels-VABench and ranks among the top systems in Artificial Analysis Arena, with strong prompt following and motion quality.
Takeaways & Limitations
The model provides a unified foundation for multimodal video creation and editing with synchronized audio at cinematic quality and scale.
Abstract
from arXiv · showhide
SkyReels V4 is a unified multi modal video foundation model for joint video audio generation, inpainting, and editing. The model adopts a dual stream Multimodal Diffusion Transformer (MMDiT) architecture, where one branch synthesizes video and the other generates temporally aligned audio, while sharing a powerful text encoder based on the Multimodal Large Language Models (MLLM). SkyReels V4 accepts rich multi modal instructions, including text, images, video clips, masks, and audio references. By combining the MLLMs multi modal instruction following capability with in context learning in the video branch MMDiT, the model can inject fine grained visual guidance under complex conditioning, while the audio branch MMDiT simultaneously leverages audio references to guide sound generation. On the video side, we adopt a channel concatenation formulation that unifies a wide range of inpainting style tasks, such as image to video, video extension, and video editing under a single interface, and naturally extends to vision referenced inpainting and editing via multi modal prompts. SkyReels V4 supports up to 1080p resolution, 32 FPS, and 15 second duration, enabling high fidelity, multi shot, cinema level video generation with synchronized audio. To make such high resolution, long-duration generation computationally feasible, we introduce an efficiency strategy: Joint generation of low resolution full sequences and high-resolution keyframes, followed by dedicated super-resolution and frame interpolation models. To our knowledge, SkyReels V4 is the first video foundation model that simultaneously supports multi-modal input, joint video audio generation, and a unified treatment of generation, inpainting, and editing, while maintaining strong efficiency and quality at cinematic resolutions and durations.
1 Introduction
SkyReels-V4 addresses fragmented multimodal video systems with a unified model for synchronized video-audio generation, inpainting, and editing. Its dual-stream architecture, inpainting formulation, and efficiency strategy target cinematic, long-duration creation.
- Existing systems do not simultaneously unify multimodal inputs, joint video-audio generation, comprehensive inpainting, and editing in one framework.
- SkyReels-V4 uses dual-stream MMDiT branches for video and audio, sharing an MLLM-based text encoder for multimodal instruction following.
- Channel concatenation reformulates image-to-video, video extension, and editing as masked inpainting operations within one video architecture.
- 1080p, 32 FPS, and 15-second generation is made computationally feasible through joint low-resolution sequence and high-resolution keyframe generation.
- SkyReels-V4 is presented as the first system combining rich multimodal inputs, joint video-audio generation, unified generation/inpainting/editing, and high-resolution, long-duration outputs.
- Experiments report state-of-the-art Arena results and strong human-evaluation performance in instruction following, motion quality, and complex multi-shot narratives.
2 Related Work
Related work has progressed from diffusion-based video synthesis toward joint audiovisual generation, but approaches differ in transparency, architecture, and multimodal integration.
- Video generation evolved from early 2D+1D architectures to DiT-based frameworks, with commercial systems leading capabilities while open-source models provide alternatives.
- Joint text-to-audio-and-video methods span coupled U-Nets, adapter-based and expert-orchestration approaches, and dual-stream architectures.
3 Model Design
SkyReels-V4 uses parallel video and audio MMDiT branches with shared multimodal text conditioning to jointly generate synchronized audiovisual content. Its unified conditioning and in-context mechanisms support generation, inpainting, editing, and vision-referenced workflows.
- Dual-Stream Architecture: SkyReels-V4 uses parallel video and audio branches built on symmetric MMDiT backbones, with the video branch initialized from a pretrained text-to-video model and the audio branch trained from scratch.Both branches use matching architectural specifications.
- Dual-Stream Architecture: Hybrid transformer blocks use separate early-stream parameters with joint self-attention, then concatenate modality and text tokens in shared later layers for efficiency.The paper states that this hybrid strategy achieves faster convergence than either pure approach.
- Dual-Stream Architecture: Bidirectional audio-video cross-attention exchanges synchronization cues throughout the network, while scaled audio RoPE aligns 218 audio tokens with 21-frame video latents.Audio RoPE frequencies are scaled by 21/218 ≈0.09633.
- Dual-Stream Architecture: A shared frozen MLLM encoder produces multimodal embeddings consumed independently by both branches, simplifying conditioning while supporting cross-modal alignment.The combined prompt can include visual and acoustic descriptions.
- Unified Video Inpainting: Channel concatenation combines noisy video latents, conditional frames, and binary masks, expressing text-to-video, image-to-video, extension, interpolation, and editing through mask configurations.The formulation also supports fixed or dynamic spatial-temporal editing masks.
- Multimodal In-Context Learning: During video inpainting and editing, the audio branch generates sound from the modified visual content, while multimodal references and in-context learning provide fine-grained visual guidance.Reference images or clips can support identity-preserving and other vision-referenced generation tasks.
- Captioning: Structured captions encode text, sound effects, dialogue, singing, and background music with special tokens, and final training stages use this format exclusively.A prompt enhancer reformats free-form user inputs into the structured representation.
4 Training Strategy
SkyReels-V4 uses progressive multimodal training to develop video, audio, alignment, inpainting, and editing capabilities. A cascaded Refiner then combines super-resolution, keyframes, interpolation, and sparse attention for efficient cinematic outputs.
- Video Pretrain: Progressive training increases resolution, temporal length, and task complexity across video pretraining stages.The schedule begins with T2I, adds T2V, expands to inpainting and editing, and later introduces mixed and high-resolution training.
- Video Pretrain: Multimodal conditioning pretraining adds image and video references for generation and inpainting alongside text-to-video data.Reference-conditioned tasks each comprise 20% of the stage, with the remaining 60% dedicated to T2V.
- Video-Audio Training: Joint training simultaneously optimizes text-to-video, text-to-audio-video, and text-to-audio generation for synchronized audiovisual outputs.Half of the video-pretraining data is allocated to T2AV, while T2A data supports synchronized audio-visual generation.
- Video-Audio Supervised Fine-tuning: Supervised fine-tuning focuses on multimodal joint-generation data and curated high-quality videos to refine quality, motion coherence, and audio-visual alignment.The final fine-tuning step uses 1 million manually curated videos.
- Refiner: The Refiner jointly performs video super-resolution and frame interpolation using low-resolution outputs and high-resolution keyframes from the base model.It supports generation, inpainting, and editing, with masks preserving regions that do not require refinement.
- Computational Efficiency: Video Sparse Attention reduces computational overhead by selecting critical spatiotemporal regions before applying denser attention.This mechanism addresses long temporal contexts and high-resolution inputs in video diffusion transformers.
5 Model Performance
SkyReels V4 is evaluated through public arena rankings and structured human assessments covering audiovisual quality and multimodal generation. It achieves strong results across generation tracks and leads or outperforms baselines across broad evaluation dimensions.
- Evaluation Methodology: The evaluation combines public arena benchmarking with human assessments across five audiovisual dimensions.The dimensions are Instruction Following, Audio-Visual Synchronization, Visual Quality, Motion Quality, and Audio Quality.
- Artificial Analysis Arena: SkyReels V4 ranks 1st in text-to-video with audio, 2nd without audio, 4th in image-to-video with audio, and 7th without audio on the Artificial Analysis Arena.These rankings are based on public user preferences as of 2026-03-18.
- Evaluation Methodology: The human study uses 50 professional evaluators and both 5-point absolute scoring and pairwise Good-Same-Bad comparisons.Absolute scores standardize ratings, while GSB comparisons distinguish clearly better, comparable, and clearly worse outputs.
- Human Assessment Results: SkyReels V4 achieves the highest overall average absolute score, with particular strengths in Prompt Following and Motion Quality.It performs comparably to the strongest competitors in Visual Quality and maintains state-of-the-art performance in synchronization and audio quality.
- Human Assessment Results: In GSB comparisons, SkyReels V4 receives a higher proportion of “Good” ratings overall against every competing model.It outperforms Kling 2.6, Seedance 1.5 Pro, Veo 3.1, and Wan 2.6 across most evaluation dimensions.
6 Conclusion
SkyReels-V4 unifies multimodal conditioning, synchronized video–audio generation, and generation, inpainting, and editing in one architecture. Evaluations report strong performance and consistent improvement over competing baselines.
- SkyReels-V4 accepts text, images, video clips, masks, and audio references while jointly generating video and audio.It uses a dual-stream MMDiT design with a shared MLLM-based text encoder.
- Channel concatenation reformulates generation, inpainting, and editing as inpainting problems under specific mask configurations.Temporal concatenation incorporates multimodal references such as images, video clips, and audio.
- The joint low-resolution/high-resolution keyframe strategy enables efficient generation at scale.
- SkyReels-V4 achieves the highest overall average score on SkyReels-VABench and strong performance in Prompt Following and Motion Quality.It also ranks among the top systems in the text-to-video-with-audio track on the Artificial Analysis Arena.
- Pairwise comparisons consistently show SkyReels-V4 outperforming competing baseline systems.
- The authors describe SkyReels-V4 as the first model to combine multimodal inputs, joint video–audio generation, and generation, inpainting, and editing at cinematic quality and scale.
7 Contributors
The contributors are organized by project leadership and specialized roles spanning infrastructure, data and video understanding, model training, multimodal training, and evaluation.
- Project sponsorship and leadership are credited to Yahui Zhou and Guibin Chen.
- Infrastructure and data and video understanding contributors supported the project’s technical foundation and multimodal data work.
- Separate teams contributed to video model training, audio model training, and multimodal training.
- The model evaluation role is credited to Binglu Zhang, Yu Shen, Tianhui Xiong, and Bin Peng.
A Application Examples
The appendix presents application cases for video-audio generation, inpainting, and editing using flexible multimodal reference inputs. It organizes these examples into three categories: Generation, Inpainting, and Editing.
- The application examples cover video-audio generation, inpainting, and editing.
- The model processes multimodal references including images, audio, and motion information.
A.1.1 Multiple Image and Audio Reference Generation
The example demonstrates generation from multiple reference images and audio inputs. The resulting video is described as stylistically consistent with the images and matched to the audio across a multi-scene instruction.
- Multiple reference images and audio inputs guide the generated video.
- The generated result is stylistically consistent with the reference images and audio-matched.
- The instruction specifies multiple indoor scenes, character actions, dialogue, and changing background music across shots.
- Figure 6 presents an example using multiple images and audio references.
A.1.2 Image Reference and Motion Reference Generation
SkyReels-V4 combines image references for content or style with video references for motion, enabling controlled video generation. It can transfer motion patterns while preserving the referenced subject.
- Image references determine content and style, while motion references control the generated video's dynamics.
- A demonstrated instruction animates a person from an image using movements from a reference video.
- The model transfers motion from reference videos, including movement patterns, action rhythm, and trajectory.
A.2 Video Inpainting
SkyReels-V4 supports video inpainting and editing across local regions, global scene properties, and multimodal references. These capabilities include object-level manipulation, style and camera changes, and reference-guided generation.
- Video Inpainting: The model precisely inpaints video regions for subject replacement, attribute modification, and background replacement.
- Local Editing: It supports fine-grained local editing of subjects, attributes, and elements, including color, texture, and shape changes.
- Element Removal: The model removes watermarks, subtitles, logos, and other video elements while maintaining content coherence and naturalness.
- Subject Manipulation: Subject manipulation supports adding, deleting, and modifying subjects while maintaining temporal consistency.
- Global Editing: Global modifications cover style transfer, camera properties, and scene attributes such as weather, lighting, color tone, and time of day.
- Reference-Guided Editing: Image references can guide subjects, backgrounds, and first frames, while reference videos provide motion, expression, or visual-effects guidance.