PaperlayerView exact revision on arXiv

Source-linked AI summary

SkyReels-V3 Technique Report

Debang Li, Zhengcong Fei, Tuanhui Li, Yikun Dou, Zheng Chen, Jiangping Yang, Mingyuan Fan, Jingtao Xu, Jiahua Wang, Baoxuan Gu, Mingshan Chang, Wenjing Cai, Yuqiang Xie, Binjie Mao, Youqiang Zhang, Nuo Pang, Hao Zhang, Yuzhe Jin, Zhiheng Xu, Dixuan Lin, Guibin Chen, Yahui Zhou

arXiv:2601.17323v2cs.CV

TL;DR

Modeling real-world dynamics under multimodal conditions is central to world models, and video generation encodes geometric, semantic, and physical knowledge for this purpose. SkyReels-V3 unifies three video-generation capabilities in a multimodal in-context framework, with evaluations reporting industry-leading or better performance across key metrics.

  • Problem

    World models aim to capture and forecast complex real-world dynamics, while video generation encodes geometric, semantic, and physical knowledge for this purpose, especially under multimodal conditions.

    Source: §1 · p. 1

  • Method

    SkyReels-V3 uses a unified multimodal in-context video-generation framework to support reference image-to-video synthesis, video extension, and audio-guided generation.

    Source: §1 · p. 1

  • Results

    Evaluations report that SkyReels-V3 reaches or surpasses industry-leading performance across key metrics.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SkyReels-V3 advances video generation from short, frame-level synthesis toward coherent, narrative-level content creation.

    Source: §3 · p. 7

Abstract

from arXiv · show

Video generation serves as a cornerstone for building world models, where multimodal contextual inference stands as the defining test of capability. In this end, we present SkyReels-V3, a conditional video generation model, built upon a unified multimodal in-context learning framework with diffusion Transformers. SkyReels-V3 model supports three core generative paradigms within a single architecture: reference images-to-video synthesis, video-to-video extension and audio-guided video generation. (i) reference images-to-video model is designed to produce high-fidelity videos with strong subject identity preservation, temporal coherence, and narrative consistency. To enhance reference adherence and compositional stability, we design a comprehensive data processing pipeline that leverages cross frame pairing, image editing, and semantic rewriting, effectively mitigating copy paste artifacts. During training, an image video hybrid strategy combined with multi-resolution joint optimization is employed to improve generalization and robustness across diverse scenarios. (ii) video extension model integrates spatio-temporal consistency modeling with large-scale video understanding, enabling both seamless single-shot continuation and intelligent multi-shot switching with professional cinematographic patterns. (iii) Talking avatar model supports minute-level audio-conditioned video generation by training first-and-last frame insertion patterns and reconstructing key-frame inference paradigms. On the basis of ensuring visual quality, synchronization of audio and videos has been optimized. Extensive evaluations demonstrate that SkyReels-V3 achieves state-of-the-art or near state-of-the-art performance on key metrics including visual quality, instruction following, and specific aspect metrics, approaching leading closed-source systems. Github: https://github.com/SkyworkAI/SkyReels-V3.

1 Introduction

The introduction positions multimodal in-context learning for video generation as under-explored and presents SkyReels-V3 as a unified framework for conditional video synthesis. It supports reference-image-to-video, video extension, and audio-guided video generation, with design choices aimed at high-fidelity motion, identity preservation, synchronization, and coherent narratives.

  • The introduction motivates video generation as a way to encode geometric, semantic, and physical knowledge for modeling and predicting the physical world, while multimodal in-context video generation remains under-explored.It notes advances from diffusion-based Transformers and strong performance from commercial and open-source systems.Source: §1 · p. 1
  • The framework unifies visual reference, video, audio, and textual inputs for three capabilities: reference-image-to-video, video-to-video extension, and audio-guided video generation.The audio-guided capability is also called talking avatar.Source: §1 · p. 1
  • Large-scale diffusion Transformers, multimodal alignment, spatiotemporal consistency modeling, hybrid image–video training, and multi-resolution joint optimization support instruction following, motion generation, identity preservation, and audio-visual synchronization.The report says these design choices enable coherent narrative progression and cinematic-quality visual composition beyond frame-level synthesis.Source: §1 · p. 2
  • SkyReels-V3 reaches or surpasses industry-leading performance across key metrics, according to the introduction’s evaluation summary.The report presents it as an open-source foundation for video-generation research and applications.Source: §1 · p. 2

2 Methods and Evaluation

SkyReels-V3 brings reference image-to-video synthesis, video extension, and talking-avatar generation into one model family. The section describes their conditioning and training designs, supported capabilities, and evaluations.

2.1 Reference Images to Video

  • The reference-to-video model accepts one to four images with a text prompt, preserving identity, composition, and narrative continuity while following high-level instructions.It jointly encodes each reference image through the video VAE and concatenates its latent representation with video latents, enabling fine-grained subject and background control without manual composition.Source: §2.1 · pp. 2, 4
  • Cross-frame pairing, editing, semantic rewriting, and filtering construct diverse reference-video training pairs, while hybrid image-video and multi-resolution training target generalization and robustness.The pipeline selects temporally diverse but semantically consistent frames, extracts subjects and completes backgrounds, and filters distorted or inconsistent edited references to reduce copy-and-paste artifacts.Source: §2.1 · pp. 3–4
  • SkyReels-V3 achieves state-of-the-art performance within the industry on reference-image video generation, evaluated for reference consistency, instruction following, and visual quality.The benchmark uses 200 data pairs spanning film and television, e-commerce, and advertising scenarios.Source: Table 1 · p. 4, §2.1 · p. 4

2.2 Video Extension

  • Video extension supports single-shot continuation and five shot-transition types, with 720p output, 5–30-second single-shot durations, and five aspect ratios.A transition detector classifies cuts; unified positional encoding and hierarchical training support smooth multi-segment transitions, while spatiotemporal modeling targets rapid motion, interactions, and abrupt changes. The report describes sharp, natural-motion results across cinematic, series, game, and enhancement scenarios.Source: §2.2 · p. 5

2.3 Talking Avatar

  • Talking-avatar generation creates minute-level, 720p videos at 24 fps from a portrait and audio, with phoneme-aligned lip motion, multi-style characters, and masked multi-character role assignment.Audio-visual alignment with region masking and key-frame-constrained generation support expressive synchronization and temporal stability; internal evaluations report advantages over representative mainstream models in visual quality, lip synchronization, and expressive realism.Source: §2.3 · p. 6

3 Conclusion

SkyReels-V3 unifies reference-based video synthesis, video extension, and audio-driven talking-avatar generation in one multimodal in-context learning framework. The report presents it as achieving competitive benchmark performance while supporting coherent, narrative-level video creation.

3 Conclusion

  • SkyReels-V3 achieves competitive performance across multiple benchmarks, rivaling leading closed-source models in various domains.The report highlights strong subject consistency, high-fidelity motion generation, and robust instruction following across diverse tasks and aspect ratios.Source: §3 · p. 7
  • The framework integrates reference-based video synthesis, video extension, and audio-driven talking-avatar generation within a unified multimodal in-context learning paradigm.It jointly models visual, temporal, and auditory signals to move beyond short, frame-level synthesis toward coherent, narrative-level content creation.Source: §3 · p. 7
  • SkyReels-V3 combines multimodal conditioning, hybrid image–video training, hierarchical spatiotemporal modeling, and efficient token fusion strategies.The report positions the system as a foundation for future scalable, controllable, general-purpose video generation and cinematic-level synthesis research.Source: §3 · p. 7

Related papers

Loading 2601.17323v2…