VideoX-Qwen: Data-Centric Instruction-Based Video Editing

• Video editing requires aligned source–target pairs that show the requested transformation while preserving unrelated content, scene structure, motion, and temporal continuity.Ordinary video–caption data lack aligned before–after relationships, while independently generated videos may differ in pose, composition, geometry, or timing.Source: §1 · pp. 1–2
• A pretrained video generator is not designed to jointly interpret source video and instructions, preserve dense structure, and synthesize only the requested change.The paper identifies a need for an interface between multimodal understanding and video generation, plus a training strategy that retains pretrained capabilities.Source: §1 · p. 2
• VideoX-Qwen combines task-routed data production with a unified Qwen–Wan editor that uses multimodal semantic guidance and dense source-video structure.Progressive image–video training aligns the interface, learns source-conditioned editing, and refines output quality.Source: §1 · p. 2
• The corpus contains more than 1.2 million directional editing records, with over 400,000 in each major task group, under a shared source–instruction–target representation.The shared representation allows task groups to train one general editor.Source: §1 · p. 2
• The editor achieves the best mean result on nine of eleven reported metrics in comparison with UniVideo and Kling O1.The framework combines multimodal instruction understanding, dense source-video guidance, and a pretrained Wan generator through staged image–video adaptation.Source: §1 · p. 2
• The combined data-production and training capabilities offer a practical route toward broader, more controllable video-editing systems.The pipeline can incorporate stronger expert models, while the editor can absorb resulting tasks through the same interface.Source: §1 · p. 2

2.1 Synthetic Paired Supervision for Video Editing

• Prior synthetic-supervision methods explore scalable pair construction, specialist-generated examples, first-frame editing, staged filtering, and reversed addition–removal pairs [1]–[6].The paper’s pipeline builds on these developments by combining source selection, localization, task-specific synthesis, screening, and instruction enrichment [7]–[10].Source: §2.1 · pp. 2–3

2.2 Multimodal and Source-Conditioned Video Editors

• VideoX-Qwen combines a Qwen multimodal encoder with a pretrained Wan generator, using sparse source frames and instructions for semantic conditioning and full-video latents for dense structural context.Related editors include systems combining creation and editing conditions, video understanding and generation, learned queries, or source-latent conditioning [14]–[16], alongside relevant backbones [11]–[13].Source: §2.2 · p. 3

2.3 Our Positioning

• The paper treats data construction and model training as one system, combining constructed video records with Ditto video data and GPT-IMAGE and NHR image-editing supervision.The authors describe image supervision as contributing spatial editing knowledge and video supervision as contributing temporal editing knowledge.Source: §2.3 · p. 3

3 Large-Scale Paired-Video Data Construction

• Each training record uses (S, T, Y) for source video, editing instruction, and target video; construction includes shared source processing, task-specific synthesis routes, visual filtering, and instruction enrichment.Source: §3 · p. 3

3.1 Source Processing and Task Routing

• Videos are selected and parsed before routing addition and removal through masking and video removal, and replacement and attribute editing through first-frame editing and video propagation.Figure 1 identifies SAM3 and Minimax-Remover for addition and removal, and Qwen-Image-Edit and WanAnimate for replacement and attribute editing.Source: §3.1 · p. 4, Figure 1 · p. 4

3.2 Addition and Removal

• For addition and removal, SAM3 turns a target description into a mask sequence and Minimax-Remover generates the object-absent video; an accepted pair supplies both editing directions.The removal instruction describes the disappearing object, while the addition instruction describes the object, appearance, and location introduced.Source: §3.2 · pp. 4–5

3.3 Replacement and Attribute Editing

• For replacement and attribute editing, Qwen-Image-Edit changes the first source frame and Wan-Animate propagates the edited appearance through the video.Replacement instructions specify original and requested objects; attribute-editing instructions specify the target object and attribute to change.Source: §3.3 · p. 5

3.4 Quality Filtering and Instruction Enrichment

• Quality filtering evaluates the complete generated video, and accepted pairs receive multiple instructions only after visual filtering.For addition and removal, both directions must correspond to the visible transition; instructions include direct commands, detailed attributes, and longer natural-language expressions.Source: §3.4 · p. 5

3.5 Constructed Dataset

• The corpus contains more than 1.2 million directional records, including over 400,000 each for addition, removal, and replacement and attribute editing, with an 89% automatic acceptance rate.Records follow the shared (S, T, Y) format.Source: §3.5 · p. 5

4.1 Task Definition

• The editor learns pθ(Y | S, T), encoding source and target videos into latents and decoding generated latents into the output.The target video supplies training supervision; inference uses only the source and instruction and starts from Gaussian noise.Source: §4.1 · p. 6, Figure 2 · p. 7

4.2 Dual-Condition Architecture

• Sparse source frames and the instruction feed a multimodal encoder, while dense source latents enter the DiT alongside noisy target latents.The semantic path identifies the requested change; the structural path retains source context.Source: §4.2 · p. 6, Figure 2 · p. 7

4.3 Multimodal Semantic Condition

• Learnable queries and a connector turn joint visual-language context into compact generator-facing features without an intermediate natural-language description.The MLLM is adapted with LoRA, while the connector and query embeddings are optimized jointly.Source: §4.3 · p. 6

4.4 Dense Source-Video Condition

• At training time, noisy target latents and source-video latents are concatenated by channel before DiT patch embedding.This supplies frame-level source structure while leaving the requested operation to the semantic condition.Source: §4.4 · p. 7

4.5 Flow-Matching Training and Inference

• Flow-matching training predicts latent velocity; inference denoises from Gaussian noise with classifier-free guidance and decodes the final latent, without a target video, mask, or edited first frame.Conditional and unconditional predictions share source-video context.Source: §4.5 · pp. 7–8

5.1 Training Stages

• All stages use global batch size 64 and learning rate 10−5; Stages 2–3 use EMA decay 0.9995.Figure 3 depicts the progression from frozen-DiT semantic alignment to source-latent joint adaptation and selected higher-resolution refinement.Source: Table 2 · p. 8, Figure 3 · p. 9
• Stage 1 trains the queries, MLLM LoRA parameters, and connector through the generation objective while the VAE and DiT remain frozen and dense source latents stay disabled.The stage establishes communication between multimodal understanding and video generation while preserving the pretrained generator.Source: §5.1 · p. 8
• Stage 2 enables source-video latents and unfreezes the DiT, jointly adapting the generator to follow source layout, appearance, and motion.Joint image–video supervision combines spatial transformations with temporally coherent editing examples.Source: §5.1 · pp. 8–9
• Stage 3 applies 500 higher-resolution updates on 30,000 manually selected videos to refine local detail and naturalness.The selected-data refinement follows establishment of broad editing behavior rather than rebuilding the interface.Source: §5.1 · p. 9, Figure 3 · p. 9

6.1 Evaluation Metrics

• The 100-example comparison gives every method the same source videos and instructions and evaluates instruction following, editing quality, preservation, video quality, and target agreement.The eleven metrics include model-judge scores, VBench measures, VFID distances, SSIM, LPIPS, and Kiwi-Edit Score.Source: §6 · p. 9, §6.1 · p. 9

6.2 Main Quantitative Results

• VideoX-Qwen has the favorable numerical value in nine rows, while Kling O1 leads on aesthetic and imaging quality.It has the best mean on all three model-judge editing dimensions, target-agreement metrics, and background consistency.Source: §6.2 · pp. 9–10, Table 3 · p. 10

6.3 Qualitative Evidence and Visible Limits

• Figures 4 and 5 show six representative comparisons of the source video, UniVideo, Kling O1, and Ours, spanning subject replacement, color editing, object addition, compound instructions, and local attribute changes.Examples include replacing a bear with a horse or an elephant with a giraffe, changing a black swan to pink, adding open books and whiteboard text, and recoloring clothing or a motorcycle.Source: §6.3 · p. 10, Figure 4 · p. 11, Figure 5 · p. 12
• Qualitative examples show only the first 49 frames of each result, with full-video metrics complementing the frame-level comparison.Local color edits expose unintended pose, layout, and background changes; compound edits should be checked for complete versus partial execution.Source: §6.3 · p. 10

7 Discussion

• Task-specific operations are normalized into one source–instruction–target representation, allowing task coverage and data volume to grow without redesigning the final interface for each operation.The production pipeline makes specialist models producers of reusable supervision rather than isolated editing tools.Source: §7 · p. 13
• More than 1.2 million directional records, including over 400,000 in each major task group, form a principal training resource; directional reuse and instruction enrichment improve pair utilization and language variation.The enriched instructions express the same visual operation in varied natural-language forms.Source: §7 · p. 13
• Multimodal queries represent source semantics and requested changes, while dense source-video latents preserve spatial and temporal information; progressive training connects both to the pretrained generator.The paper reports improvements across instruction following, preservation, similarity, and video-distribution metrics for the combined system.Source: §7 · p. 13

8 Conclusion

• VideoX-Qwen combines repeatable synthesis routes for addition, removal, replacement, and attribute editing into a common source–instruction–target interface, yielding more than 1.2 million directional records.The unified editor combines multimodal semantic queries with dense source-video latents and is progressively adapted to execute instructions while preserving source content.Source: §8 · p. 13
• The complete system obtains the best mean result on nine of eleven metrics in a 100-example comparison with UniVideo and Kling O1.The conclusion describes the framework as an extensible mechanism for turning specialist visual-model capabilities into reusable data for one general editor.Source: §8 · p. 13