Source-linked AI summary
VACE: All-in-One Video Creation and Editing
Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, Yu Liu
TL;DR
Unified video synthesis lacks a common way to handle the temporal, spatial, and multimodal requirements of creation and editing tasks. VACE addresses this gap with a Video Condition Unit and context-adapter architecture, achieving competitive performance across tasks while supporting compositional applications.
Problem
Unified video synthesis remains challenging because creation and editing tasks require consistency across temporal and spatial dynamics, while comprehensive benchmarks for derivative video tasks are lacking.
Method
VACE organizes text, image, video, and mask conditions in a Video Condition Unit and injects them into a Diffusion Transformer through context-token processing and Context Adapter Tuning.
Results
VACE performs competitively with specialized models across multiple video tasks, with user studies consistently favoring its results across multiple tasks.
Takeaways & Limitations
A single model can complete most video creation tasks and flexibly combine task capabilities for applications such as long video re-rendering.
Takeaways & Limitations
Unified models have less large-scale training than text-to-video foundation models, limiting identity preservation in reference generation and complete input control in compositional tasks.
Abstract
from arXiv · showhide
Diffusion Transformer has demonstrated powerful capability and scalability in generating high-quality images and videos. Further pursuing the unification of generation and editing tasks has yielded significant progress in the domain of image content creation. However, due to the intrinsic demands for consistency across both temporal and spatial dynamics, achieving a unified approach for video synthesis remains challenging. We introduce VACE, which enables users to perform Video tasks within an All-in-one framework for Creation and Editing. These tasks include reference-to-video generation, video-to-video editing, and masked video-to-video editing. Specifically, we effectively integrate the requirements of various tasks by organizing video task inputs, such as editing, reference, and masking, into a unified interface referred to as the Video Condition Unit (VCU). Furthermore, by utilizing a Context Adapter structure, we inject different task concepts into the model using formalized representations of temporal and spatial dimensions, allowing it to handle arbitrary video synthesis tasks flexibly. Extensive experiments demonstrate that the unified model of VACE achieves performance on par with task-specific models across various subtasks. Simultaneously, it enables diverse applications through versatile task combinations. Project page: https://ali-vilab.github.io/VACE-Page/.
1. Introduction
VACE presents an all-in-one video creation and editing model that unifies diverse task inputs and supports flexible task combinations. It is evaluated against specialized models across multiple tasks, with competitive quantitative and qualitative performance.
- VACE unifies reference-to-video generation, video-to-video editing, masked video-to-video editing, and free composition in one model.
- The framework uses a Video Condition Unit to integrate multimodal inputs such as images and videos for different video tasks.
- Collaborative spatiotemporal representation enables adaptive processing of unified video tasks.
- 480 evaluation samples covering 12 tasks support comparisons between VACE and existing specialized models.
- VACE reports competitiveness in quantitative and qualitative analyses and supports compositional expansion such as long video re-rendering.
2. Related Work
Visual generation and editing research has expanded from foundational image and video generation toward multimodal control and flexible editing. Figure 2 organizes VACE’s covered video tasks into categories whose combinations create many possibilities.
- Figure 2 depicts task categories covered by VACE, with four basic tasks combinable into many possibilities.
- Unified visual models such as ACE, OmniGen, OmniControl, and UniReal broaden controllable generation, local editing, and reference-guided generation.
3. Method
VACE represents diverse video tasks through four multimodal input types and a unified Video Condition Unit, then restructures a Diffusion Transformer to process those conditions. Its design supports task composition and context-adapter-based conditioning.
- 3.1. Multimodal Inputs and Video Tasks: VACE groups video tasks according to requirements for text, image, video, and mask inputs, using a Video Condition Unit for unified conditioning.
- 3.1. Multimodal Inputs and Video Tasks: The task categories are text-to-video, reference-to-video, video-to-video editing, masked video-to-video editing, and task composition.
- 3.2. Video Condition Unit: A VCU organizes each task as textual input, context-frame sequence, and mask sequence, with frames and masks aligned spatially and temporally.
- 3.2. Video Condition Unit: Reference frames use all-zero masks to preserve them, video-to-video editing uses input frames with all-one masks, and masked editing uses both context videos and masks.
- 3.2. Video Condition Unit: VCU composition combines reference frames, masks, and control inputs to support reference-inpainting and scribble-based video extension.
- 3.3. Arichitecture: The restructured DiT tokenizes context frames and masks, combines context tokens with noisy video tokens, and can inject context through Context Adapter Tuning.
- 3.3.1. Context Tokenization: Concept Decoupling separates pixels to be changed from pixels to be preserved using mask-based reactive and inactive frame sequences.
- 3.3. Arichitecture: Fully fine-tuning updates the DiT and Context Embedder, while Res-Tuning processes context tokens in a bypass branch for faster convergence and pluggability.
4. Datasets
VACE expands data construction and introduces VACE-Benchmark to evaluate diverse video generation and editing tasks. The benchmark covers multiple modalities, task types, automatic quality and consistency metrics, and human assessment.
- VACE data construction expands beyond text-video pairs to include target and source videos, local masks, references, and other modalities.
- The pipeline filters videos by resolution, aesthetic score, motion amplitude, and target-area size before obtaining instance-level temporal segmentation.
- Task-specific data preparation extracts depth, scribble, pose, and optical flow controls and constructs repainting data with masks.
- 4.2. VACE-Benchmark: VACE-Benchmark addresses the lack of comprehensive evaluation for derivative video tasks such as reference generation and editing.
- 4.2. VACE-Benchmark: The benchmark contains 240 high-quality videos across 12 task types, averaging 20 samples per task.
- 4.2. VACE-Benchmark: Table 2 compares unified VACE based on LTX-Video with proprietary models using automated metrics and human-study results.
5. Experiments
VACE is evaluated across diverse video creation and editing tasks using automatic metrics, user studies, qualitative results, and ablations. The experiments compare unified modeling choices and show strong performance across many tasks while identifying trade-offs in reference generation and model configuration.
- Qualitative Results: VACE supports compositional applications such as reference-, move-, animate-, swap-, and expand-anything.The “Move Anything” example combines an input image with a movement trajectory while maintaining character coherence and narrative consistency.
- Evaluation: VACE-Benchmark evaluation combines automatic scoring and user studies across video quality and consistency indicators.The automatic evaluation uses eight selected VBench indicators, while manual assessment captures user preferences.
- Quantitative Evaluation: VACE outperforms other open-source methods across eight indicators for I2V, inpainting, outpainting, depth, pose, and optical-flow tasks.The comparison uses normalized average metrics on the VACE-Benchmark; some competing methods have lower resolution, shorter durations, or unstable temporal coherence.
- Quantitative Evaluation: Reference-to-video performance remains below commercial models for the small fast-generation model but is comparable to Vidu 2.0.Human studies nevertheless report consistently better evaluation metrics across multiple tasks, aligning with user preferences.
- Ablation Studies: Additional-parameter fine-tuning converges faster than full fine-tuning while producing similar effects for the unified VCU input paradigm.The ablations also examine weighting schemes, timestamp shifting, p-zero, and the number and distribution of Context Adapter blocks.
6. Conclusion
VACE unifies diverse video creation and editing inputs in one framework, supporting flexible combinations of video tasks. The paper presents this as a step toward unified visual generative models with multimodal inputs.
- Conclusion: VACE unifies the diverse multimodal inputs required for multiple video creation and editing tasks.
- Conclusion: Most video AI creation tasks can be completed with a single inference of a single model.
- Conclusion: Flexible combinations of video tasks expand application scenarios and accommodate a wide range of user creative needs.
A. Implementation Details
Training uses the foundational LTX-Video and Wan-T2V text-to-video models, with LTX-Video supporting fast inference under limited resources.
- Training is based on the foundational text-to-video generation models LTX-Video and Wan-T2V.The hyperparameter settings are summarized in Tab. 3.
- LTX-Video enables quick inference with limited resources.
- 24 seconds are required to sample 40 steps for an approximately 5-second video on a single A100 card without dedicated acceleration.The passage states that this meets general users’ video-processing needs.
B. Additional Results
Additional Wan-T2V results cover a broad set of video generation and editing tasks. These examples include spatial control, appearance changes, temporal extension, and reference-driven generation.
- Additional Results: The additional qualitative results are based on Wan-T2V.
- Additional Results: The showcased tasks include outpainting, inpainting, and extension.
- Additional Results: The results also include grayscale, depth, scribble, pose, layout, face-reference, and object-reference tasks.
B.2. Visualization Comparison
Figure 8 compares LTX-Video-based VACE with specialized models across extension, inpainting, outpainting, depth-controlled, and pose-controlled generation tasks.
- VACE is compared with specialized models across extension, unconditional inpainting, outpainting, depth-controlled, and pose-controlled generation.The cited passage names I2VGenXL, CogVideoX, LTX-Video-I2V, ProPainter, Follow-Your-Canvas, M3DDM, Control-A-Video, VideoComposer, ControlVideo, and Text2Video-Zero as comparison models.
C.1. Limitations
The paper identifies limitations involving foundation-model trade-offs, insufficient scaling of unified models, and the operational complexity of multimodal video inputs.
- Foundation models influence generated-content quality and overall style.The paper states that smaller models favor rapid generation, whereas larger models improve creative-output success but increase inference time and resource consumption.
- Unified models have not yet received the large-scale data and computational resources used by text-to-video foundation models.The paper links this boundary to incomplete identity preservation during reference generation and imperfect input control in compositional tasks.
- Unified video models are harder to operate than image models because their inputs include temporal information and multiple modalities.The paper identifies this complexity as a practical usage threshold and proposes exploring language or agent models for guidance.
C.2. Societal impacts
The paper describes productivity benefits from intelligent video creation and editing alongside societal risks involving misinformation, bias, and stereotypes.
- Intelligent video generation and editing can provide creators with innovative tools that spark ideas and enhance the artistic and innovative quality of video content.The paper notes that these technologies are being applied across industries, including marketing and advertising.
- Video generation technology can help businesses produce promotional videos quickly and support more creative marketing strategies.The paper states that this can save labor costs and enhance market competitiveness.
- Wider adoption may spread misinformation and false content while reinforcing biases and stereotypes in generated media.The paper calls for attention to ethics and responsibility and for appropriate regulation to balance innovation with social responsibility.