PaperlayerView exact revision on arXiv

Source-linked AI summary

Aurora: Unified Video Editing with a Tool-Using Agent

Yongsheng Yu, Ziyun Zeng, Zhiyuan Xiao, Zhenghong Zhou, Hang Hua, Wei Xiong, Jiebo Luo

arXiv:2605.18748v1cs.CV

TL;DR

Unified video editors require precise text, references, and spatial grounding that raw user requests often omit. Aurora adds a tool-augmented VLM agent that constructs model-ready edit conditions before generation, raising AgentEdit-Bench’s overall score from 74.7 to 87.9 with the same video DiT.

  • Problem

    Unified video editing models require precise instructions, reference images, and spatial grounding that real user requests often omit.

  • Method

    Aurora uses a tool-augmented VLM agent to rewrite requests, assign task labels, and retrieve references or ground masks before unified video generation.

  • Results

    74.7 to 87.9: adding the VLM agent raised Aurora’s overall AgentEdit-Bench score with the same video DiT.

  • Takeaways & Limitations

    AgentEdit-Bench evaluates both the complete agent-enhanced framework and the VLM agent’s contribution under textual and visual underspecification.

  • Takeaways & Limitations

    Large, physically plausible motion for newly inserted subjects remains difficult, partly because of video-DiT capacity and limited large-motion training coverage.

Abstract

from arXiv · show

Recent video editing models have converged on a unified conditioning design: a single diffusion transformer jointly consumes text, source video, and reference images, and one set of weights covers replacement, removal, style transfer, and reference-driven insertion. The design is flexible, but it assumes that the user already provides model-ready text, reference images, and spatial grounding for local edits, which real requests often omit. We present Aurora, an agentic video editing framework that pairs a tool-augmented vision-language model (VLM) agent with a unified video diffusion transformer. The VLM agent maps a raw user request to a structured edit plan aligned with the transformer's conditioning channels, thereby resolving textual and visual underspecification before generation. We train the VLM agent with supervised data for complete edit planning and reference-image selection, together with preference pairs for robust tool use and instruction refinement. We introduce AgentEdit-Bench to evaluate agent-enhanced video editing under textual and visual underspecification. Experiments on AgentEdit-Bench and two existing video editing benchmarks show that Aurora improves over instruction-only baselines and that the VLM agent transfers to compatible frozen video editing models. Project page: https://yeates.github.io/Aurora-Page

1 Introduction

Aurora addresses the condition-construction gap in unified video editing by using a tool-augmented VLM agent to convert raw requests into model-ready edit plans before video generation. It also introduces agent training data and AgentEdit-Bench for evaluating editing under textual and visual underspecification.

  • Motivation: Unified video editing models support replacement, removal, style transfer, and reference-driven insertion, but require appropriate text, reference images, or masks for each request.These inputs specify the edit, preserved source content, appearance, and location.
  • Motivation: Aurora frames the unresolved challenge as constructing the model’s expected conditioning tuple from a raw user request.Missing conditions can include a complete instruction, a reference image, or spatial grounding, all needed before video generation.
  • Framework: Aurora’s tool-augmented VLM agent maps a source video and raw request to a four-field edit plan: rewritten instruction, task label, optional image-search query, and optional mask phrase.Rewriting and task routing address textual underspecification, while image search and mask phrases recover missing visual conditions.
  • Framework: The unified video diffusion transformer consumes the agent-produced conditions through shared multimodal context and latent token sequences, without rewriting instructions or calling tools.The transformer is built on Wan2.2-TI2V-5B and receives rewritten instructions, sampled source frames, reference images, and corresponding tokens.
  • Evaluation: AgentEdit-Bench evaluates the complete framework, VLM transfer across video editing models, and improvement over an instruction-only baseline under textual and visual underspecification.The benchmark is designed to evaluate tool-augmented-agent-enhanced video editing jointly rather than evaluating the framework components only in isolation.
  • Contributions: Aurora contributes training data for complete edit planning, reference-image selection, and DPO-based preference alignment.The training data supports the VLM agent’s planning, reference selection, and preference alignment capabilities.

2 Related Work

Prior video editing work progressed from frame-sharing instruction-conditioned methods and data-construction pipelines toward unified models with richer conditioning inputs. Related image-generation agents externalize planning to retrieve references, decompose requests, select tools, and evaluate intermediate results, while video-domain agents address complementary settings.

  • Video editing models: Early instruction-conditioned video editors adapted image-editing techniques by sharing attention across frames.Senorita-2M scales supervision with task-specialist synthesizers, while OpenVE and Ditto use VLMs to generate instructions and filter low-quality edits.
  • Video editing models: Recent video editing methods move toward unified models that accept richer conditioning inputs, including instructions, source videos, reference images, and masks.
  • Agentic generation: Image-generation agents keep planning outside the generator by retrieving knowledge or references, decomposing requests, choosing tools, and scoring intermediate results with feedback or learned rewards.Existing video-domain agents target complementary settings, including generalist video generation and compositional text-to-video generation.

3 Method

Aurora combines a tool-augmented VLM agent with a unified video DiT, using structured edit plans to resolve textual and visual underspecification before generation. The DiT unifies source, reference, and masked-image conditioning while preserving unedited content through multimodal attention and token-level visual conditioning.

  • System overview: Aurora uses separate Qwen-family models: a LoRA-finetuned Qwen3-VL-8B-Instruct agent creates edit plans, while a frozen Qwen3.5-4B encoder supplies DiT cross-attention context.The two checkpoints share no weights and serve distinct roles.
  • System overview: The agent maps the source video, raw instruction, and optional references into a complete edit plan and concrete conditions for the unified DiT.The plan is assembled into a conditioning tuple while preserving source identity, motion, and background where unchanged.
  • Edit planning and tools: The typed plan rewrites the instruction and optionally specifies a task label, image-search query, and mask phrase under constrained tool-routing rules.User references suppress web search, while search queries are restricted to eligible tasks such as addition, replacement, and background change.
  • Edit planning and tools: Search and mask tools both produce reference-image assets: retrieval selects one web image, while grounded segmentation composites a masked image onto the source video.GroundingDINO and Segment Anything support mask generation, and neither tool requires a separate mask branch in the DiT.
  • Video DiT architecture: The DiT uses two conditioning paths: multimodal cross-attention encodes rewritten text, sampled source frames, and references, while self-attention concatenates noisy, source, and reference tokens.Reference tokens use learnable index embeddings, and source and reference tokens remain fixed across denoising timesteps.
  • Training: Training first fits the unified-condition DiT, then freezes it while supervised fine-tuning trains the agent for complete planning and reference-image selection, followed by plan-level DPO alignment.SFT directly supervises inference decisions rather than natural-language rationales.

4 Data Construction

Aurora constructs two interdependent datasets: one trains the unified video DiT across source, text, reference, and masked-image conditions, while the other trains the VLM agent to select inference-time conditions. The agent data combines supervised edit planning and reference-image selection with preference pairs for boundary-case decisions.

  • Video editing model data: The video DiT training mixture teaches image-edit instruction following, temporal source conditioning, reference-guided identity, and masked-image editing.The mixture uses three corresponding subsets: image editing, instruction video editing, and reference-guided video editing.
  • Video editing model data: 2,388,440 accepted training pairs comprise the listed image-editing sources CrispEdit-2M, UltraEdit, and TextEdit.Table 1 counts accepted training pairs after curation; source annotations mark filtering, re-captioning, and reference-related synthesis operations.
  • Video editing model data: Reference-guided video editing draws on OpenS2V, Ref VIE, SpatialVID, ROSE, EffectErase, and HuMoSet, with filtering or synthesis applied to marked sources.The passage identifies OpenS2V, SpatialVID, and HuMoSet as filtered sources, and OpenS2V, ROSE, and EffectErase as synthesized sources.
  • VLM agent supervised data: 25K planning examples and 10K reference-image selection examples supervise four-field planning and choosing one retrieved candidate for the reference slot.Planning examples are created by degrading accepted editing instructions into shorter or more ambiguous requests.
  • Preference alignment: 1.8K chosen/rejected preference pairs refine VLM-agent decision boundaries across five boundary-case categories.The categories include false triggers, ambiguous mask phrases, false search triggers, constraint-losing rewrites, and task-routing confusions.

5 Experiments

Aurora is evaluated on underspecified requests, standard video-editing benchmarks, agent-training ablations, and transfer across video diffusion models. The experiments show gains from both the unified video DiT and the VLM agent, with the agent remaining effective across compatible models.

  • Existing benchmarks: Aurora also evaluates on EditVerse-Bench’s 120-case source-video subset and OpenVE-Bench to test whether its agentic design remains competitive when instructions are already sufficiently specified.The reported Overall scores are judged by Gemini 2.5 Pro on benchmark-specific scales.
  • Benchmarks: AgentEdit-Bench evaluates 150 underspecified cases spanning five task types, measuring instruction following, edit localization, source preservation, and visual quality.Gemini 2.5 Pro scores each case with a rubric as a VLM-as-judge benchmark.
  • Main results: 74.7 Overall: Aurora’s video DiT without the agent outperforms UniVideo at 67.0 and Kiwi-Edit at 69.7 on identical raw-prompt inputs.These results show that the gain on AgentEdit-Bench is not driven by the agent stage alone.
  • Main results: 87.9 Overall: adding the VLM agent raises Aurora from 74.7 on AgentEdit-Bench, while resolving omitted reference images and spatial grounding before sampling.Raw-prompt baselines often synthesize generic substitutes or edit the wrong region when these conditions are missing.
  • Agent ablations: 2.9 points: DPO improves the supervised-planning score from 85.0 to 87.9, after supervised planning raises the no-agent score from 74.7 to 85.0.Supervised planning supplies most of the gain, while DPO sharpens ambiguous boundary cases such as false image-search triggers and ambiguous mask phases.
  • Agent transfer: 3.29: the VLM agent improves Kiwi-Edit from 3.02 to 3.29 on OpenVE-Bench, and improves Aurora’s video DiT from 3.31 to 3.46.The agent’s image-search and mask tools are disabled on these benchmarks; it only rewrites already specified instructions.

6 Limitations

Aurora’s limitations include offline-only agent alignment, which prevents reward-based optimization over the full editing pipeline under the current compute budget. Like other video editing models, it also poses misuse, copyright, and likeness risks, so it should not be treated as an evidence-producing system.

  • Technical limitations: Aurora aligns its VLM agent only with offline DPO because online RL over the full editing pipeline is impractical under the available compute budget.Online RL would derive rewards from the actually edited clip rather than from the plan.
  • Technical limitations: Few-step video DiT distillation with joint online RL of the VLM agent and distilled DiT is proposed as a way to close this optimization gap.The proposed future direction jointly trains both components using rewards from the full editing process.
  • Broader Impacts: Aurora can be misused for non-consensual identity manipulation, while retrieved references introduce copyright and likeness risks.The paper recommends treating Aurora as a creative tool rather than an evidence-producing one.

7 Conclusion · A Extended qualitative comparison on AgentEdit-Bench

Aurora addresses underspecified video-editing requests by combining a tool-augmented VLM agent with a unified video-editing model. The paper also introduces AgentEdit-Bench and extends qualitative comparisons against UniVideo and Kiwi-Edit under default raw-prompt deployment.

  • 7 Conclusion: Aurora places a tool-augmented VLM agent before a unified video-editing model.The framework targets requests that omit information required by the video model.
  • 7 Conclusion: The VLM agent rewrites user instructions and assigns task labels before generation.
  • 7 Conclusion: When needed, Aurora retrieves a reference image or grounds a mask before generation.
  • 7 Conclusion: AgentEdit-Bench evaluates video editing under textual and visual underspecification.The benchmark measures the full framework and isolates the c…
  • A Extended qualitative comparison on AgentEdit-Bench: Figures 7, 8, and 9 add AgentEdit-Bench cases comparing Aurora with UniVideo [36] and Kiwi-Edit [19].
  • A Extended qualitative comparison on AgentEdit-Bench: The baselines use their default deployment, applying the raw user prompt without a VLM agent.
  • A Extended qualitative comparison on AgentEdit-Bench: For IP cases, Aurora uses a reference image retrieved by its fine-tuned VLM agent.The retrieved image is shown as a small inset overlaid on the first source frame.

B Qualitative comparison on EditVerse-Bench · C Qualitative comparison on OpenVE-Bench

The qualitative sections compare Aurora with open-source baselines across diverse editing categories on EditVerse-Bench and OpenVE-Bench. Both evaluations use distant frames from each clip to assess temporal consistency, while the EditVerse-Bench analysis also provides per-category quantitative breakdowns.

  • B Qualitative comparison on EditVerse-Bench: Figure 10 compares Aurora with LucyEditDev, VACE, Kiwi-Edit, and UniVideo on reference-conditioned insertion, background change, and reasoning cases.The cases cover three EditVerse-Bench capabilities, including background change (Bkg.).
  • B Qualitative comparison on EditVerse-Bench: The first EditVerse-Bench case uses a landscape clip with frames sampled at 20% and 80% to check temporal consistency.The comparison shows two distant frames from the clip.
  • B Qualitative comparison on EditVerse-Bench: Figure 8 provides an extended qualitative comparison on AgentEdit-Bench.The passage identifies Figure 8 as an extended qualitative comparison.
  • B Qualitative comparison on EditVerse-Bench: Table 6 expands EditVerse-Bench’s Overall result into a full per-category quantitative comparison scored by Gemini 2.5 Pro.Gray rows identify closed-source reference methods, bold marks the best open-source score per column, and Overall scores are summarized in Table 3.
  • C Qualitative comparison on OpenVE-Bench: Figure 11 compares Aurora with UniVideo and Kiwi-Edit on OpenVE-Bench cases covering local removal, local addition, and local change.The comparison includes four cases.
  • C Qualitative comparison on OpenVE-Bench: OpenVE-Bench cases use frames sampled at 25% and 75% of each clip to check temporal consistency.The cases are shown as two distant frames.
  • C Qualitative comparison on OpenVE-Bench: Figure 9 provides an extended qualitative comparison on AgentEdit-Bench in the OpenVE-Bench section.The passage identifies Figure 9 as an extended qualitative comparison.

D Qualitative VLM agent transfer across video DiTs

Aurora’s VLM agent transfers across frozen video diffusion transformers by rewriting raw requests and retrieving reference images before generation. Qualitatively, agent-enabled runs better preserve named-entity identity than instruction-only runs on UniVideo and Kiwi-Edit.

  • Evaluation setup: UniVideo and Kiwi-Edit are evaluated with identical source clips and user requests, comparing raw prompts against VLM-agent rewritten prompts plus retrieved reference images.The receiving video DiT is held fixed while the VLM agent stage is toggled on and off.
  • Qualitative transfer results: Without the agent, runs usually drop named entities or replace them with generic substitutes, whereas agent-enabled runs follow rewritten instructions and use retrieved references.The retrieved reference image is overlaid as a small inset in the first source frame for comparison.
  • Qualitative transfer results: The retrieved reference image helps preserve the identity of the requested real-world entity across agent-enabled transfer examples.This identity-preservation effect is reported for qualitative transfer on both UniVideo and Kiwi-Edit.

E Experimental details · F Additional VLM agent data details

Aurora’s experiments use an 81-frame video diffusion process with benchmark-specific saving protocols, staged training budgets, and guidance branches enabled by prompt and visual dropout. The VLM agent rewrites every input, while web image search and mask overlay are enabled only when benchmarks require them.

  • E Experimental details: The video DiT denoises 81 frames internally, saving all 81 for AgentEdit-Bench and the first 64 for EditVerse-Bench.The 81-frame design follows Wan’s 4k+1 temporal constraint, while saved outputs match each evaluator’s protocol.
  • E Experimental details: Inference evaluates full-condition, negative-condition, and unconditional velocity predictions before combining them into guided velocity.The three predictions are v+ with full text and visual conditions, vvneg with empty or negative conditions, and v∅ with neither.
  • E Experimental details: When λimg = 1, the guidance equation reduces algebraically to the standard two-pass text CFG path.Its unconditional and visual-negative branches correspond exactly to the prompt-dropout and visual-dropout branches trained in Section 3.3.
  • E Experimental details: Training uses two stages: Stage 1 is a low-resolution warmup, and Stage 2 resumes from its checkpoint stream at a higher pixel budget.The video pixel budgets are 399,360 in Stage 1 and 921,600 in Stage 2.
  • E Experimental details: The model processes 81 frames, uses AdamW, and applies a constant learning rate of 1 × 10−5 in the reported training recipe.The architecture and training hyperparameters are summarized in Tables 8–9.
  • E Experimental details: Prompt dropout 0.1 trains the empty-text branch, while visual dropout given prompt 0.5 produces visual-negative and unconditional subsets.These dropout branches support the inference-time guidance formulation.
  • E Experimental details: Table 10 reports per-benchmark inference settings, including cfg_scale, image_cfg_scale, denoising steps, output frames, pixel budgets, and negative prompts.The table defines cfg_scale and image_cfg_scale as λtxt and λimg in the guidance equation.
  • E Experimental details: The VLM agent uses Qwen3-VL-8B-Instruct with an Aurora LoRA adapter and rewrites every input, while web image search and mask overlay are selectively enabled.Both tools are enabled for one benchmark and disabled on the two already-specified benchmarks.

F.1 Data-construction details

Aurora constructs unified video-editing data by filtering source quality and edit fidelity, then trains planning and reference-selection behavior under deliberately underspecified requests. Preference alignment adds targeted examples for plan-level boundary cases that supervised training does not isolate well.

  • Video editing model data: Accepted editing samples preserve instructions, source videos, and reference images in a shared conditioning format after Gemini Flash-Lite quality judgments.Judgments cover source and target quality, localization, identity preservation, motion consistency, edit authenticity, and prompt alignment.
  • Planning SFT and reference-image selection: Planning SFT degrades clean instructions into shorter, ambiguous, colloquial, or identity-hiding requests while supervising the four-field edit plan.The teacher is asked to restore information real users would likely omit rather than provide an explanation.
  • Planning SFT and reference-image selection: The plan activates image_search for identity-bearing targets or named entities, mask for localized removal, and neither tool for global style changes or combined edits when appropriate.These cases are illustrated by fifteen examples pairing source frames, colloquial requests, and supervised four-field plans.
  • Planning SFT and reference-image selection: After a non-empty image_search query, the VLM agent selects one visually grounded and factually consistent image from multiple web-search candidates.Selection must satisfy the named-entity constraints in the search prompt.
  • Preference alignment: 1.8K preference pairs target five plan-level boundary cases: source-entity false triggers, ambiguous masks, false image-search triggers, constraint-losing rewrites, and task-routing confusions.Each pair keeps the source, raw request, and most plan fields fixed while varying one decision axis.

G AgentEdit-Bench · G.1 Scoring rubric and judge prompt

AgentEdit-Bench is a 150-case, five-edit-type benchmark built from Pexels videos, while its rubric scores edit fidelity, localization, preservation, quality, and identity-related criteria. Evaluation uses edit-type-specific axes, a normalized aggregate, and deterministic three-frame judging.

  • G AgentEdit-Bench: AgentEdit-Bench contains 150 cases spanning five edit types, with Pexels stock videos and instructions highlighting targets, counterfactual cues, regions, or physical effects.The benchmark also characterizes instruction length, reasoning sub-style, edit-type composition, and removal-target physical effects.
  • G AgentEdit-Bench: Examples convert underspecified requests into refined instructions specifying targets, appearance, location, scene properties, and requested effects.Examples include replacing a duck with a specifically dressed Joker, changing a stormy cliff to daylight 10 Downing Street, and removing a lifebuoy while restoring tiled-wall continuity.
  • G AgentEdit-Bench: The planning examples cover global styles, combined tasks, color changes, object replacement, and object addition, with image search and masking enabled selectively.Representative subtasks include replacing blocks with a Thomas the Tank Engine toy and adding Dwight Schrute or a child without masks.
  • G.1 Scoring rubric and judge prompt: Each case is scored on seven axes for IP edits and axes 1–5 for reasoning and removal, with maximum scores of 21 and 15, respectively.Removal adds an anti-replacement clause that caps outputs substituting a new object for the target at 1.
  • G.1 Scoring rubric and judge prompt: The rubric evaluates instruction following, edit-region localization, source preservation, visual quality, temporal consistency, IP Presence, and IP Identity Match.IP Presence measures visibility and approximate placement, whereas IP Identity Match checks specific real-world identity cues such as brand colors, logos, or signature shapes.
  • G.1 Scoring rubric and judge prompt: The cross-type aggregate normalizes every case to a 0–21 scale and reports the result as a percentage.This provides a common reporting scale despite different axis counts across edit categories.
  • G.1 Scoring rubric and judge prompt: The deterministic judge uses temperature=0, scores frames at indices 0, n/3, and 2n/3, averages each axis across frames, and stores raw judge text.The prompt requests integer scores from 0 to 3 with brief justifications, comparing before and after frames against the instruction, target entity, and edit region.
Loading 2605.18748v1…