UNIC: Unified In-Context Video Editing

Zixuan Ye; Xuanhua He; Quande Liu; Qiulin Wang; Xintao Wang; Pengfei Wan; Di Zhang; Kun Gai; Qifeng Chen; Wenhan Luo

Source-linked AI summary

UNIC: Unified In-Context Video Editing

Zixuan Ye, Xuanhua He, Quande Liu, Qiulin Wang, Xintao Wang, Pengfei Wan, Di Zhang, Kun Gai, Qifeng Chen, Wenhan Luo

arXiv:2506.04216v1Jun 4, 2025cs.CV

TL;DR

Existing video editing methods depend on task-specific architectures or dedicated inversion procedures, limiting unified use of diverse conditions. UNIC instead jointly models source-video, noisy-latent, and multi-modal condition tokens in context, using task-aware RoPE and condition bias to unify six tasks. Experiments report superior performance across the tasks and emergent task composition abilities.

Problem
Existing methods rely on task-specific architectures or DDIM inversion, limiting integration of diverse editing conditions and unification across tasks.
Method
UNIC concatenates source-video tokens, noisy video latents, and task-dependent multi-modal condition tokens into one sequence processed by native diffusion-transformer attention.
Results
UNIC achieves superior performance on six representative video editing tasks and exhibits emergent task composition abilities.
Takeaways & Limitations
A single in-context model can flexibly perform diverse video editing tasks and combine them without task-specific architectural changes.
Takeaways & Limitations
The current unification is limited to six tasks; audio modalities such as lip-syncing remain unexplored, and high token counts may increase computational overhead.

Abstract

from arXiv · show

Recent advances in text-to-video generation have sparked interest in generative video editing tasks. Previous methods often rely on task-specific architectures (e.g., additional adapter modules) or dedicated customizations (e.g., DDIM inversion), which limit the integration of versatile editing conditions and the unification of various editing tasks. In this paper, we introduce UNified In-Context Video Editing (UNIC), a simple yet effective framework that unifies diverse video editing tasks within a single model in an in-context manner. To achieve this unification, we represent the inputs of various video editing tasks as three types of tokens: the source video tokens, the noisy video latent, and the multi-modal conditioning tokens that vary according to the specific editing task. Based on this formulation, our key insight is to integrate these three types into a single consecutive token sequence and jointly model them using the native attention operations of DiT, thereby eliminating the need for task-specific adapter designs. Nevertheless, direct task unification under this framework is challenging, leading to severe token collisions and task confusion due to the varying video lengths and diverse condition modalities across tasks. To address these, we introduce task-aware RoPE to facilitate consistent temporal positional encoding, and condition bias that enables the model to clearly differentiate different editing tasks. This allows our approach to adaptively perform different video editing tasks by referring the source video and varying condition tokens "in context", and support flexible task composition. To validate our method, we construct a unified video editing benchmark containing six representative video editing tasks. Results demonstrate that our unified approach achieves superior performance on each task and exhibits emergent task composition abilities.

1 Introduction

UNIC frames diverse video editing inputs as a unified token sequence and jointly models them in context, addressing task ambiguity and positional conflicts. It unifies six representative tasks and reports strong performance across them.

Motivation: Existing video editing methods use DDIM inversion or task-specific adapter designs, limiting efficiency, extensibility, and task unification.DDIM inversion adds an inference stage, while adapter-based methods require separate modules for different condition signals.
UNIC Framework: UNIC represents reference-video tokens, multi-modal condition tokens, and noisy video latents as one jointly modeled token sequence.The sequence is processed with native transformer attention without task-specific architectural changes.
Task Unification: Task-aware RoPE assigns condition-dependent positional indices to handle varying condition lengths and temporal correspondences.Camera poses align frame by frame, whereas style images affect the entire video, creating positional conflicts under direct concatenation.
Task Unification: A learnable condition bias helps distinguish editing tasks that use the same condition modality.For example, an image may encode object identity or style, creating task ambiguity without additional task information.
Evaluation: UNIC constructs a six-task benchmark spanning local editing, global editing, and re-camera control across text, image, and camera-pose conditions.The evaluated tasks include ID Insert/Delete/Swap, stylization, propagation, and re-camera control.
Evaluation: The unified framework delivers superior performance across the evaluated tasks and supports flexible task composition.The introduction reports that the approach successfully unifies the tasks despite their differing modalities and editing areas.

2 Related Work

Prior video editing systems preserve reference information through DDIM inversion or inject controls with specialized modules. These strategies add processing or architectural complexity and limit extension to diverse tasks, motivating unified full-attention approaches.

Video Editing and Re-rendering: Video editing injects motion, style, object attributes, or camera pose to support local and global transformations of reference videos.The conditioning signals span multiple modalities and editing scopes.
Video Editing and Re-rendering: DDIM inversion initializes generation noise from the reference video but introduces additional processing and can double inference steps and cost.Methods such as VideoP2P and FLATTEN use inversion-related features to preserve reference-video structure.
Video Editing and Re-rendering: Auxiliary control modules inject reference or condition information directly into denoising networks to preserve structure, layout, content, or motion.Examples use layout encoders, depth ControlNet, separate feature encoders, or modality-specific encoders.
Video Editing and Re-rendering: Specialized modules for different condition types increase model complexity and limit extensibility to novel video editing tasks.This design is common among methods that use dedicated ControlNet, identity, or style components.
Universal Generative Models: Unified generative models tokenize conditions as direct transformer inputs, replacing external plugins or dedicated control modules with joint processing.Prior image-generation systems motivate extending full-attention unification to video editing.

3 Method

UNIC unifies video editing by representing noisy latents, reference videos, and task-specific multimodal conditions as one token sequence processed with native transformer attention. Task-aware biases and positional indices address task ambiguity and alignment conflicts across diverse editing conditions.

3.1 Multi-modal Driven Video Editing Tasks: UNIC represents noisy tokens, reference video tokens, and multimodal condition tokens as three basic input types for video editing.This classification supports uniform processing across editing tasks.
3.1 Multi-modal Driven Video Editing Tasks: Reference-video tokens provide temporal context and motion information, with strict alignment for tasks such as stylization and ID deletion and softer guidance for re-camera control.Strict alignment preserves frame correspondence, whereas soft reference permits larger deviations while following overall content or motion.
3.2 Unified In-Context Video Editing: The flow-matching video diffusion transformer trains on noisy video states and samples synthesized videos by integrating the learned velocity field from Gaussian noise.The network receives xt = tx1 + (1 −t)x0, where x1 is a video sample and x0 is Gaussian noise.
3.2 Unified In-Context Video Editing: The model concatenates target noisy latents, reference-video tokens, and condition tokens along the frame dimension for joint full 3D attention.This avoids an intricate inversion process and task-specific architectural modifications.
3.2.2 Effective and Flexible Task Unification: Condition bias adds task-specific learnable embeddings before attention, while task-aware RoPE reuses frame indices for aligned conditions and offsets indices for nonaligned conditions.These mechanisms distinguish conditions from different tasks and preserve appropriate temporal alignment when lengths and correspondence patterns vary.

4 Experiments

UNIC is evaluated on six video editing tasks using task-specific, overall-quality, and ablation studies. It performs strongly across tasks, while sequential training and the combined design of condition bias and task-aware RoPE improve unification.

Evaluation setup: The benchmark evaluates six tasks using task-specific metrics and overall video quality measures, including alignment and reconstruction criteria.Metrics include DINO-score and CLIP-score for ID tasks, CSD-score, ArtFID, and CFSD for stylization, and broader text-alignment and quality measures.
Main comparisons: UNIC achieves consistent performance across all six tasks, leading on ID Insert and Re-Camera Control and matching specialized stylization performance.For ID Delete, VideoPainter has better PSNR, but UNIC exceeds it on CLIP-score, Smoothness, and Aesthetic score; UNIC also supports arbitrary resolutions and lengths.
Training strategy: Sequential training from either hard-to-easy or easy-to-hard improves multi-task learning, whereas joint training from scratch struggles on difficult re-camera control.Re-camera control is identified as the most difficult task, while ID-related tasks are comparatively easier to learn.
Single-task performance: The unified model does not significantly degrade single-task performance and improves camera control and stylization similarity, with a small stylization content-preservation trade-off.The trade-off is attributed to retaining more information during stylization because ID training emphasizes content preservation.
Ablation analysis: Condition bias and task-aware RoPE together produce the best overall ablation performance by separating task types and handling complex temporal alignment.Condition bias improves task disambiguation, while task-aware RoPE is especially effective for re-camera control and per-frame alignment.

5 Conclusion

UNIC unifies diverse video editing tasks in one in-context model by jointly modeling task inputs as a unified token sequence. Across six tasks, it achieves superior task performance and emergent task composition abilities.

Conclusion: UNIC represents video-editing inputs as three token types and jointly models them with diffusion-transformer full attention.Task-aware RoPE and conditional bias support flexible editing and task combination.
Conclusion: Experiments on six representative tasks show superior performance on each task and emergent task composition abilities.

A UNIC Benchmark

The UNIC benchmark covers six editing tasks with varied modalities, editing scopes, and evaluation cases. Its cases are constructed from curated videos, references, masks, styles, and camera trajectories.

Benchmark composition: The benchmark contains six tasks: ID insert, ID swap, ID delete, stylization, propagation, and re-camera control.
ID insert: ID insert uses 20 Artgrid source videos with selected identities spanning clean objects and complete images with backgrounds.
ID swap: ID swap uses 20 videos, segments replaceable objects with SAM2, and selects replacement identities from an ID pool.
ID delete: ID delete expands VPBench to 20 videos, segments objects with SAM2, and captions the resulting target videos.
Stylization and propagation: Stylization uses 12 reference styles and 50 Artgrid source videos, while propagation expands GenProp to 50 examples with stylization cases.
Re-camera control: Re-camera control applies 10 camera trajectories to 50 Koala videos, producing 50 trajectory-video pairs.

B Training Dataset Construction

The dataset-construction section introduces the construction process for UNIC’s six-task training datasets without detailing individual task procedures.

Dataset construction: The section details construction of the training datasets for UNIC’s six video editing tasks.

B.1 ID-related Task

The paper constructs training data for ID-related and stylization tasks using segmentation-based inpainting correction and a two-stage stylized-video transformation strategy.

ID-related Task: SAM2 extracts object masks, and cv2.inpaint uses them to create videos for ID-related deletion, swapping, and insertion tasks.A ControlNet conditioned on the original video is trained to remove artifacts introduced by simple inpainting.
ID-related Task: A ControlNet conditioned on the original video eliminates visual artifacts that simple inpainting introduces in ID-related training videos.
Stylization: The stylization dataset is built by generating high-quality stylized videos with a T2V model, converting them into realistic counterparts with tile-based video ControlNet, and creating 10,000 pairs.Direct video-to-video stylization produced temporal inconsistencies, flickering, and lower visual quality.

B.3 Propagation Task Dataset

The propagation dataset reuses paired source-target videos from ID-related and stylization data, while the six-task architecture assigns task-specific input encodings and RoPE indices.

Propagation Task Dataset: Existing ID-related and stylization datasets provide paired source and target videos for constructing propagation data.
Propagation Task Dataset: Propagation triplets contain a source video, a target video, and the target video's first frame, generated from existing paired data in two role configurations.
Model Details for Six Tasks: Re-camera control uses 136,000 videos from the Multi-Cam Video dataset, while Figure S11 documents input encoding and RoPE indexing across six tasks.
Model Details for Six Tasks: ID images use 3D VAE encoding with task-specific RoPE offsets, style tokens begin at index 206, propagation references use index 1, and camera parameters are MLP-embedded into VAE-aligned tokens.Camera parameters are flattened from F × 3 × 4, spatially padded, and embedded into tensors shaped F × emb_dim × H/4 × W/4.

C.2 Training Progress

UNIC evaluates two task-ordering strategies during training and observes that tasks differ substantially in the data volume needed to reach satisfactory performance.

Training Progress: Training progression is tested in both hard-to-easy and easy-to-hard task orders.Task difficulty is classified from empirical observations during in-context training.
Training Progress: Re-camera control requires approximately 100k training examples for satisfactory performance, whereas ID insertion or swapping reaches comparable results with about 20k.This difference motivates categorizing re-camera control as the harder task.
Training Progress: UNIC is finetuned for 16k iterations on 32 H800 GPUs with batch size 64, updating the transformer and new tokenizers while freezing the 3D VAE and Text T5 Tokenizer.

D Limitation and Future Work

The current study unifies six video-editing tasks but leaves broader modality coverage and scalability to future work.

Limitations: UNIC's current unification scope is limited to the six tasks evaluated in this work.
Future Work: Audio-conditioned modalities such as lip-syncing remain unexplored.
Future Work: Future work will test whether more tasks can be unified and develop mechanisms to reduce the computational overhead of high-token-count self-attention.