Source-linked AI summary
SkyReels-A2: Compose Anything in Video Diffusion Transformers
Zhengcong Fei, Debang Li, Di Qiu, Jiahua Wang, Yikun Dou, Rui Wang, Jingtao Xu, Mingyuan Fan, Guibin Chen, Yang Li, Yahui Zhou
TL;DR
Elements-to-video generation must preserve multiple reference elements while composing natural, text-guided videos, a gap left by conventional text-to-video and image-to-video settings. SkyReels-A2 addresses this with curated triplet data, joint semantic-spatial image-text injection, inference optimization, and A2-Bench. The authors report high-quality, diverse, editable, temporally consistent outputs with strong identity fidelity and favorable comparisons to commercial models.
Problem
Existing text-to-video and image-to-video models do not directly address multi-element composition with predictable reference fidelity and flexible textual control.
Method
SkyReels-A2 combines text-reference-video triplet data, joint semantic-spatial image-text injection, optimized inference, and the A2-Bench evaluation framework.
Results
The authors report high-quality, diverse, editable, temporally consistent videos with strong identity fidelity and favorable performance against advanced commercial models.
Takeaways & Limitations
SkyReels-A2 provides an open framework for composing multiple visual elements in controllable videos across diverse application scenarios.
Abstract
from arXiv · showhide
This paper presents SkyReels-A2, a controllable video generation framework capable of assembling arbitrary visual elements (e.g., characters, objects, backgrounds) into synthesized videos based on textual prompts while maintaining strict consistency with reference images for each element. We term this task elements-to-video (E2V), whose primary challenges lie in preserving the fidelity of each reference element, ensuring coherent composition of the scene, and achieving natural outputs. To address these, we first design a comprehensive data pipeline to construct prompt-reference-video triplets for model training. Next, we propose a novel image-text joint embedding model to inject multi-element representations into the generative process, balancing element-specific consistency with global coherence and text alignment. We also optimize the inference pipeline for both speed and output stability. Moreover, we introduce a carefully curated benchmark for systematic evaluation, i.e, A2 Bench. Experiments demonstrate that our framework can generate diverse, high-quality videos with precise element control. SkyReels-A2 is the first open-source commercial grade model for the generation of E2V, performing favorably against advanced closed-source commercial models. We anticipate SkyReels-A2 will advance creative applications such as drama and virtual e-commerce, pushing the boundaries of controllable video generation.
1 Introduction
SkyReels-A2 addresses elements-to-video generation by combining multiple reference images with textual instructions while targeting fidelity, controllability, and coherent video composition. It introduces a dedicated data structure and joint image-text injection framework, with experiments reporting favorable multi-element video quality against commercial models.
- Research Motivation: Elements-to-video generation must preserve multiple reference elements while producing coherent, controllable videos from textual instructions.The framework targets characters, objects, and backgrounds rather than relying on a single initial frame.
- Contributions: SkyReels-A2 constructs text-reference-video triplets with multiple element images sampled across videos to avoid simple image copying.Captions are re-annotated to describe element appearance and actions.
- Contributions: The framework injects image information through complementary semantic and spatial pathways to support element consistency and text-guided composition.The contribution is described as a unified image-text injection design based on existing video foundation models.
- Contributions: SkyReels-A2 introduces a curated dataset and A2-Bench for comprehensive and automated evaluation of elements-to-video generation.The benchmark supports systematic assessment alongside the training triplets.
- Contributions: Experiments report high-quality, editable, and temporally consistent multi-element videos that perform favorably against advanced closed-source commercial models.The paper states that this comparison is supported by both qualitative and quantitative analyses, and that code, models, and the benchmark are released.
2 Methodology
SkyReels-A2 combines multi-reference video conditioning, a triplet-based data pipeline, and inference optimization for controllable elements-to-video generation. Its training data and benchmark are designed around diverse visual elements, while the model uses spatial and semantic reference features within a diffusion-transformer pipeline.
- Task Formulation: The task maps N reference images describing subjects and backgrounds to natural videos that follow a textual prompt.The formulation treats the references as composition elements and targets high-quality video generation.
- Video Diffusion Preliminaries: Latent video diffusion transforms Gaussian noise into video representations conditioned on textual prompts and a text encoder.The paper notes that SkyReels-A2 follows flow matching to optimize velocity with mean-squared error loss.
- Reference Conditioning: Image conditioning combines CLIP-derived semantic features with VAE-derived spatial latent features integrated into the video diffusion model.Semantic features enter through cross-attention, while spatial features are concatenated with input noise along the channel dimension.
- Model Architecture: SkyReels-A2 encodes references through spatial and semantic branches, injecting spatial features before diffusion-transformer blocks and semantic features through supplementary cross-attention.The spatial branch uses a fine-grained 3D VAE, while the semantic branch uses a CLIP vision encoder and MLP projection.
- Model Architecture: The architecture preserves reference identities and backgrounds while following textual prompts with flexible layouts, using minimal structural modifications to a video diffusion transformer.Subjects are segmented with white grounding, excluding background references, to avoid additional noise.
- Training: Training reconstructs target videos in latent space with diffusion MSE loss while optimizing cross-attention, patch embedding, and image-condition embedders.The remaining model components are kept frozen during training.
- Dataset Construction: The data pipeline filters and segments videos, generates holistic and structured captions, extracts visual elements, and matches them into video-caption-multi-reference triplets.Detection, face detection, human parsing, CLIP matching, and cross-clip reference retrieval provide element-level annotations and references.
- Dataset Construction: Approximately 2 million high-quality video-reference-prompt triplets are compiled for training.The triplets are the final product of the dataset construction pipeline.
3 Experiments
Experiments evaluate SkyReels-A2 against closed-source commercial baselines using automatic metrics, user studies, qualitative comparisons, and ablations. Results show competitive composition and video quality, with strengths in subject consistency, motion naturalness, and selected design choices.
- Experimental Settings: SkyReels-A2 is evaluated against Pika, Vidu, and Keling, the available state-of-the-art closed-source commercial baselines for elements-to-video.
- Quantitative Results: SkyReels-A2 achieves comparable composition performance, particularly in objective consistency, according to A2-Bench automatic metrics.A2-Bench covers composition consistency, visual quality, and prompt following.
- Quantitative Results: User studies find multi-reference performance comparable to commercial solutions, with advantages in dress consistency and human motion smoothness.Users rated methods from 1 to 5, with higher scores indicating better human-perceived results.
- Qualitative Results: Qualitative comparisons report balanced performance for SkyReels-A2 and Vidu across subject consistency, visual effect, and text response, while Pika shows weaker consistency and less motion.
- Ablation Study: Ablations select VAE frame repetition, cross-1 training, and a 1:0 single-to-multiple-element data ratio for the final model.Removing frame repetition reduces image information and composition consistency; latent-space concatenation can cause interaction loss between references.
- Qualitative Results: Overall results show particular advantages in subject character consistency and motion naturalness, while Keling exhibits a mirror movement effect.The mirror movement effect is described as motion from far to near or vice versa.
- Ablation Study: Training every cross-attention layer is selected as optimal under the balance between performance and data scale, although full fine-tuning improves image quality and naturalness.
- Applications: Applications include music-video creation with instruments and scenes, and virtual e-commerce promotions using product imagery, targeted scenarios, celebrities, and voice-overs.
4 Related Works
Video generation has progressed from GANs and VAEs to diffusion-based and transformer architectures, while personalization methods remain constrained by continual adaptation and closed-source systems. SkyReels-A2 addresses scalable, controllable open video composition with coherent, high-fidelity, editable outputs and subject consistency.
- Diffusion-based video models emerged to address temporal discontinuities found in early GAN-driven approaches.
- Large-scale text-to-video models improved fine-grained control through high-capacity textual representations and attention-based architectures.
- Recent personalization methods improve image and video generation but rely on continual model adaptation, hindering scalability and broader deployment.
- Lightweight tuning-free solutions reduce resource requirements, but leading systems such as Keling, Vidu, Pika, and MovieGen are closed-source.
- SkyReels-A2 publicly releases technique details and targets coherent, high-fidelity, editable videos with subject-specific consistency.
5 Conclusion
SkyReels-A2 is an elements-to-video framework that integrates multiple visual reference images through joint text-image injection. The reported results show high-quality, diverse, editable compositions with strong identity fidelity, positioning the framework for future controllable-video research.
- SkyReels-A2 generates elements-to-video compositions by integrating multiple visual reference images.
- Its joint text-image model injection uses semantic and spatial perspectives to integrate cross-modal data in existing video foundation models.
- Empirical results demonstrate high-quality, diverse composition synthesis with robust editability and strong identity fidelity.
- The framework is presented as a replicable basis for future research on controllable video generation.