PaperlayerView exact revision on arXiv

Source-linked AI summary

Aurora: Unified Video Editing with a Tool-Using Agent

Yongsheng Yu, Ziyun Zeng, Zhiyuan Xiao, Zhenghong Zhou, Hang Hua, Wei Xiong, Jiebo Luo

arXiv:2605.18748v1cs.CV

TL;DR

Unified video editing models expect users to supply model-ready instructions, references, and spatial grounding, which real requests often omit. Aurora uses a tool-augmented VLM agent to construct edit conditions before generation. On AgentEdit-Bench, adding the VLM agent raises the overall score from 74.7 to 87.9 with the same Aurora video DiT.

  • Problem

    Unified video editing models expect precise instructions, reference images, and spatial grounding, but user requests often omit one or more of these conditioning inputs.

    Source: §1 · p. 2

  • Method

    Aurora pairs a tool-augmented VLM agent with a unified video diffusion transformer to map raw requests to more complete, model-ready conditions before generation.

    Source: §1 · p. 3

  • Results

    74.7 to 87.9 on AgentEdit-Bench: adding the VLM agent raises the overall score with the same Aurora video DiT.

    Source: §7 · p. 10

  • Takeaways & Limitations

    AgentEdit-Bench supports evaluating the full framework and isolating the VLM agent’s contribution on the same source clips.

    Source: §7 · p. 10

    Adding a new subject undergoing large, physically plausible motion remains difficult for Aurora’s 5B video DiT.

    Source: §6 · p. 10

Abstract

from arXiv · show

Recent video editing models have converged on a unified conditioning design: a single diffusion transformer jointly consumes text, source video, and reference images, and one set of weights covers replacement, removal, style transfer, and reference-driven insertion. The design is flexible, but it assumes that the user already provides model-ready text, reference images, and spatial grounding for local edits, which real requests often omit. We present Aurora, an agentic video editing framework that pairs a tool-augmented vision-language model (VLM) agent with a unified video diffusion transformer. The VLM agent maps a raw user request to a structured edit plan aligned with the transformer's conditioning channels, thereby resolving textual and visual underspecification before generation. We train the VLM agent with supervised data for complete edit planning and reference-image selection, together with preference pairs for robust tool use and instruction refinement. We introduce AgentEdit-Bench to evaluate agent-enhanced video editing under textual and visual underspecification. Experiments on AgentEdit-Bench and two existing video editing benchmarks show that Aurora improves over instruction-only baselines and that the VLM agent transfers to compatible frozen video editing models. Project page: https://yeates.github.io/Aurora-Page

1 Introduction

Aurora treats underspecified video editing as condition construction before generation, using a tool-augmented VLM to prepare model-ready inputs for a unified video diffusion transformer. AgentEdit-Bench evaluates the framework and VLM-agent transfer across video editing models.

1 Introduction

  • Unified conditioning models support replacement, removal, style transfer, and reference-driven insertion, but user requests may lack precise text, reference images, or masks.The introduction distinguishes visual underspecification, where a visual input is missing, from textual underspecification, where the concrete editing goal must be inferred.Source: §1 · p. 2
  • Aurora decouples condition construction from video generation because missing instructions, reference images, or spatial grounding must be resolved before generation begins.The unresolved task is constructing the conditioning tuple expected by unified video editing models from a raw user request.Source: §1 · p. 2
  • Aurora maps a source video and raw request to a four-field edit plan that resolves textual underspecification and retrieves or grounds missing visual conditions.The plan contains a rewritten instruction, task label, optional image-search query, and optional mask phrase; tool fields remain empty when the request already specifies the edit.Source: §1 · pp. 2–3
  • The unified video diffusion transformer consumes the agent-produced conditions through shared multimodal context and latent tokens, without rewriting instructions or calling tools.Its inputs include the rewritten instruction, sampled source frames, reference images, and source and reference tokens alongside noisy video tokens.Source: §1 · p. 2
  • AgentEdit-Bench tests underspecified textual and visual requests through protocols for the full framework, VLM-agent transfer, and improvement over raw-instruction editing.The protocols pair the agent with Aurora’s DiT, pair it with other unified editing models, or evaluate a model using only the raw user instruction.Source: §1 · pp. 2–3
  • Aurora’s VLM agent training data covers complete edit planning, reference-image selection, and DPO-based preference alignment.Source: §1 · p. 3

2 Related Work

Aurora addresses the gap between unified video editors’ rich conditioning interfaces and what users provide by turning raw requests into complete model-ready conditions. It distinguishes this planning role from prior image-generation agents and video agents that coordinate model libraries.

2.1 Instruction-conditioned and unified video editing

  • Aurora’s trained VLM agent retrieves references, grounds masks, and rewrites instructions to provide unified video editors with complete conditioning when raw requests omit inputs.Earlier instruction-conditioned methods adapted image-editing techniques across video frames [23] [41] [9], while later unified models accept richer conditions [15] [19] [36] [16] [39]; the burden of supplying missing references or masks remained with users or upstream pipelines. Senorita-2M [48], OpenVE [11], and Ditto [1] address supervision or data construction.Source: §2.1 · p. 3

2.2 Agentic visual generation and video agents

  • Unlike image-generation agents that plan outside generators or video agents that coordinate model libraries, Aurora translates raw instructions into conditions for a single unified video DiT.Related image-generation agents retrieve knowledge or references, decompose requests, choose tools, and score results [7] [42] [20] [40] [45]; video agents address generation, compositional text-to-video, model selection, or multi-step library workflows [44] [18] [13].Source: §2.2 · p. 3

3 Method

Aurora pairs a LoRA-finetuned Qwen3-VL-8B-Instruct agent that plans edits with a unified video DiT whose frozen Qwen3.5-4B encoder processes the resulting conditions. The agent constructs model-ready conditions from the request and optional tools, while the DiT is trained to use them.

3.1 Agent planning and condition construction

  • The agent predicts a four-field plan that rewrites the instruction, routes the task, and optionally specifies an image-search query or mask phrase.User-provided references suppress web search, and search queries are allowed only for eligible tasks; search and segmentation tools produce reference-image assets for the DiT.Source: §3.1 · p. 4

3.2 Unified conditioning in the video DiT

  • The DiT uses a frozen Qwen3.5-4B multimodal encoder for instructions, sampled source frames, and reference images as cross-attention context, while visual conditions are concatenated along the token-sequence dimension.Mask conditioning has no separate branch: a binary mask is composited as a masked image and inserted into the references. Source and reference tokens remain fixed across timesteps, while noisy tokens follow the flow-matching trajectory.Source: §3.2 · p. 5

3.3 Training objectives

  • The DiT learns conditional velocity with flow matching, while the frozen DiT supports agent training through supervised planning and reference selection followed by plan-level DPO.Preference pairs target five boundary cases: unnecessary search for visible entities, vague masks, false search triggers, rewrites that drop constraints, and incorrect task routing.Source: §3.3 · p. 6

4 Data Construction

Aurora constructs interdependent datasets to train the video DiT’s conditioning and the VLM agent’s inference-time condition selection.

Video editing model data

  • The video DiT training mixture uses image-edit pairs for instruction following, video edits for temporal source conditioning, and reference-guided edits for identity and masked-image conditions.A subset of video-edit sources is filtered and relabeled with Gemini Flash-Lite; identity references are synthesized with FLUX.2-klein, and masked images are curated separately. Table 1 reports the mixture and accepted training-pair counts, but no counts are provided in these passages.Source: §4 · p. 6, Table 1 · p. 7

VLM agent supervised data

  • VLM supervised training uses 25K planning examples and 10K reference-image selection examples.Planning SFT degrades clean instructions into shorter or more ambiguous requests and supervises a four-field plan; selection training chooses one reference image from retrieved candidates.Source: §4 · p. 7

Preference alignment

  • The VLM agent’s 1.8K preference pairs target five boundary-case categories and vary one decision axis while holding the source and most plan fields fixed.DPO uses these chosen/rejected pairs to refine decision boundaries for source-entity false triggers, ambiguous mask phrases, false image-search triggers, constraint-losing rewrites, and task-routing confusions.Source: §4 · p. 7

5 Experiments

The experiments test whether Aurora helps with underspecified requests, identify which parts of the framework drive its gains, and assess whether the VLM agent transfers across video editing models. AgentEdit-Bench is the primary test, with existing benchmarks serving as checks on well-specified instructions.

5.2 AgentEdit-Bench

  • 87.9 Overall with the VLM agent exceeds Aurora’s 74.7 without it, UniVideo’s 67.0, and Kiwi-Edit’s 69.7 on AgentEdit-Bench.The agent’s gains are largest on IP replacement and addition, where selecting a concrete reference image resolves a missing visual condition.Source: Table 2 · p. 8, §5.2 · p. 8

5.1 Experiment setup

  • EditVerse-Bench results use its 120-case source-video subset, while OpenVE-Bench results cover six categories because Aurora’s training data omits creative and camera edits.Aurora’s video DiT uses a two-stage, 81-frame training recipe; reported scores use the Stage 2 checkpoint with 50 denoising steps unless noted otherwise.Source: §5.1 · p. 8

5.2 AgentEdit-Bench

  • AgentEdit-Bench contains 150 cases across five task types, each underspecified textually or visually, and scores instruction following, localization, preservation, quality, and temporal consistency.For its three IP tasks, the rubric also scores IP presence and identity match; IP means a named real-world entity such as a brand, product, character, or artwork.Source: §5.2 · p. 8

5.4 Ablation study

  • With its image-search and mask tools disabled, the fixed VLM agent improves Kiwi-Edit from 3.02 to 3.29 and Aurora’s DiT from 3.31 to 3.46 on OpenVE-Bench.It also improves UniVideo from 6.12 to 6.48 and Aurora’s DiT from 7.25 to 7.61 on EditVerse-Bench; the OpenVE averages exclude subtitle removal.Source: Table 5 · p. 9, §5.4 · pp. 9–10

5.3 Results on existing video editing benchmarks

  • Aurora is the strongest open-source method on EditVerse-Bench and OpenVE-Bench; on OpenVE-Bench, Runway Aleph scores 3.51 Overall, 0.13 above Aurora’s 3.38.Aurora also leads the OpenVE subtitle-removal column despite zero-shot generalization, and the like-for-like five-category average excludes subtitle removal.Source: Table 3 · p. 9, §5.3 · p. 9

5.4 Ablation study

  • On AgentEdit-Bench, supervised planning raises Aurora from 74.7 to 85.0, and DPO adds 2.9 points to reach 87.9.Supervised planning teaches conditioning-channel selection; DPO sharpens ambiguous cases such as false image-search triggers, mask phrases, and task routing.Source: Table 4 · p. 9, §5.4 · p. 9

6 Limitations

Aurora’s limitations include offline-only VLM alignment, difficulty inserting subjects with large physically plausible motion, and risks of identity manipulation and retrieved-reference use. The paper presents faster joint training as a path forward and frames Aurora as a creative tool, not an evidence-producing one.

6 Limitations

  • Aurora aligns its VLM only with offline DPO; full-pipeline online RL such as GRPO [30] is impractical under the current compute budget.The proposed direction is few-step video-DiT distillation combined with joint online RL training [3], with rewards from edited clips rather than plans.Source: §6 · p. 10
  • The 5B video DiT handles removal, stylization, and limited-motion insertion reliably, but large, physically plausible subject motion remains difficult.The paper attributes this to the backbone’s motion-modeling capacity and limited training-data coverage of large-motion insertions, and proposes a larger backbone and broader training mixture.Source: §6 · p. 10

Broader Impacts

  • Aurora risks non-consensual identity manipulation, while image search adds copyright and likeness risks from retrieved references.The paper says Aurora should be treated as a creative tool, not an evidence-producing one.Source: §6 · p. 10

7 Conclusion · A Extended qualitative comparison on AgentEdit-Bench

Aurora pairs a tool-augmented VLM agent with a unified video editing model to turn underspecified requests into model-ready conditions. It improves AgentEdit-Bench performance and more faithfully handles edits requiring real-world entity identity.

7 Conclusion

  • Adding the VLM agent raises Aurora’s overall AgentEdit-Bench score from 74.7 to 87.9 with the same video DiT.AgentEdit-Bench measures the full framework and isolates the VLM agent stage on the same source clips.Source: §7 · p. 10
  • Aurora places a tool-augmented VLM agent before a unified video editing model to rewrite instructions, assign task labels, retrieve references, or ground masks.The agent resolves missing information before generation when a request does not provide conditions the video model needs.Source: §7 · p. 10
  • On EditVerse-Bench and OpenVE-Bench, Aurora remains strong on sufficiently specified inputs, and instruction rewriting alone still helps.The conclusion notes these benchmarks as settings where inputs are already sufficiently specified.Source: §7 · p. 10
  • The VLM agent’s benefit transfers to other unified video editing models, rather than depending on Aurora’s video model alone.The conclusion frames reliable conversion of raw requests into model-ready conditions as important alongside stronger video DiTs.Source: §7 · p. 10

A Extended qualitative comparison on AgentEdit-Bench

  • Across the extended AgentEdit-Bench cases, Aurora more faithfully fulfills raw requests than UniVideo [36] and Kiwi-Edit [19], which may drop entities, substitute generic ones, or remove targets poorly.The baselines use their default deployment with raw prompts and no VLM agent; Aurora uses rewritten instructions and, when needed, retrieved references.Source: A Extended qualitative comparison on… · p. 14
  • Aurora’s gains are clearest for edits requiring real-world entity identity, with retrieved reference images shown as insets for IP cases.The examples span portrait and landscape clips and cover replacement, removal, and addition.Source: A Extended qualitative comparison on… · p. 14

B Qualitative comparison on EditVerse-Bench · C Qualitative comparison on OpenVE-Bench

The sections compare Aurora with baseline video editors on EditVerse-Bench and OpenVE-Bench, using selected cases and distant clip frames to assess edits and temporal consistency.

B Qualitative comparison on EditVerse-Bench

  • Figure 10 compares Aurora with LucyEditDev, VACE [15], Kiwi-Edit [19], and UniVideo [36] across three EditVerse-Bench cases.The cases cover reference-conditioned insertion, background change, and reasoning.Source: B Qualitative comparison on… · p. 14
  • The first EditVerse-Bench case shows a landscape clip at 20% and 80% of its duration to check temporal consistency.The two displayed frames are distant samples from the clip.Source: B Qualitative comparison on… · p. 14
  • Figure 8 presents an extended qualitative comparison on AgentEdit-Bench.Source: Figure 8 · p. 15
  • Table 6 expands EditVerse-Bench overall scores into per-category comparisons, with scores from Gemini 2.5 Pro [6].Gray rows denote closed-source methods, bold marks the best open-source score per column, and overall scores are summarized in Table 3.Source: Table 6 · p. 15

C Qualitative comparison on OpenVE-Bench

D Qualitative VLM agent transfer across video DiTs

With the receiving video DiT held fixed, Aurora’s VLM agent helps UniVideo [36] and Kiwi-Edit [19] follow rewritten instructions and preserve requested entity identity using retrieved reference images. Without the agent, runs usually omit the named entity or replace it with a generic substitute.

D Qualitative VLM agent transfer across video DiTs

  • The comparison runs each fixed video DiT on the same source clip and request, toggling only the VLM agent stage on or off.Agent runs use the rewritten prompt and retrieved reference image; the third cell overlays the reference as a small inset on the first source frame.Source: D Qualitative VLM agent transfer… · pp. 16–17
  • With the agent, UniVideo [36] and Kiwi-Edit [19] follow rewritten instructions and use retrieved references, while raw-prompt runs usually omit named entities or substitute generic ones.The reference image helps preserve the identity of the requested real-world entity.Source: D Qualitative VLM agent transfer… · pp. 18–19, Figure 12 · p. 19, Figure 13 · p. 19
  • The qualitative transfer examples test agent-enhanced editing on both UniVideo [36] and Kiwi-Edit [19].The examples are shown in Figures 12 and 13, respectively.Source: Figure 12 · p. 19, Figure 13 · p. 19

E Experimental details · F Additional VLM agent data details

The experimental details specify benchmark-specific output-frame handling, inference-time guidance, model-training settings, and VLM tool toggles. They also describe how guidance relates to the dropout branches used in training.

E Experimental details

  • AgentEdit-Bench saves all 81 denoised frames, while EditVerse-Bench saves the first 64 frames to match its evaluator’s clip-length protocol.The DiT internally denoises 81 frames for both benchmarks.Source: E Experimental details · p. 19
  • For OpenVE-Bench, source clips are resized to 81 frames before denoising, and outputs are resized back to each source’s original frame count.This preserves the input clip length in the saved edited video.Source: E Experimental details · p. 19
  • Inference evaluates full-condition, visual-negative, and unconditional velocity predictions; when λimg = 1, the guidance formula reduces to standard two-pass text CFG.The unconditional and visual-negative branches correspond to the prompt-dropout and visual-dropout branches trained in Sec. 3.3.Source: E Experimental details · p. 19
  • Tables 8–9 report the Aurora video DiT architecture and training parameters, with Stage 1 as a low-resolution warmup and Stage 2 resuming its checkpoint stream at a higher pixel budget.The supplied passages do not provide the tables’ individual hyperparameter values.Source: E Experimental details · p. 19, Table 9 · p. 20, Table 8 · p. 20
  • On the two already-specified benchmarks, web image search and mask overlay are disabled, so the VLM agent only rewrites the instruction.Table 10 also summarizes per-benchmark inference settings and agent tool toggles.Source: E Experimental details · p. 19, Table 10 · p. 20

F.1 Data-construction details

The section describes how Aurora’s editing data, planning supervision, reference-image selection examples, and preference pairs are constructed to address underspecified edit requests.

Video editing model data

  • Open-source editing samples from ReCo [46], Ditto [1], EgoEdit [17], and OpenVE [11] are judged across seven quality criteria, retaining faithful instructions or using judge rewrites.The criteria include source and target quality, localization, identity preservation, motion consistency, edit authenticity, and prompt alignment.Source: §F.1 · p. 21
  • Ditto combined-task data filters out edit pairs that collapse into one change, then retains visually distinct, input-describable transitions, yielding 64,782 pairs.Each training input is one edited variant and its target is another from the same source, which is withheld; Gemini Flash-Lite assesses candidates.Source: §F.1 · p. 21

Planning SFT and reference-image selection

  • Planning SFT degrades clean instructions into ambiguous raw requests and trains the VLM to restore omitted information in the four-field plan, with sampling organized by task control.The target remains the plan defined in Eq. (1); Figure 14 illustrates when identity or entity references trigger image_search, localized removal triggers mask, and global style or combined edits leave both tool fields empty.Source: §F.1 · p. 21
  • Reference-image selection adds 10K supervised examples adapted from [7], training the VLM to select a candidate image index that fits retrieved-entity and web-context constraints.After a non-empty image_search query retrieves candidates, the selection task is to choose the image visually grounded and factually consistent with the named entity.Source: §F.1 · p. 21

Preference alignment

  • Preference alignment uses 1.8K chosen/rejected pairs that hold the source, raw request, and most plan fields fixed while varying one of five plan-level boundary decisions.The cases are source-entity false triggers, ambiguous mask phrases, false image-search triggers, constraint-losing rewrites, and task-routing confusions.Source: §F.1 · p. 21

G AgentEdit-Bench · G.1 Scoring rubric and judge prompt

AgentEdit-Bench contains 150 cases across five video-edit types, and its scoring rubric uses different axis sets for IP edits versus reasoning and removal. The section also describes the judge’s scoring procedure and how scores are aggregated across edit types.

G AgentEdit-Bench

  • AgentEdit-Bench comprises 150 cases across five edit types, with source videos drawn from the Pexels stock library.Figure 18 presents nine cases spanning the five types; Figure 19 shows the instruction vocabulary and distributions of case types, instruction length, reasoning sub-styles, and removal effects.Source: G AgentEdit-Bench · p. 21, Figure 18 · p. 26, Figure 19 · p. 26
  • Figure 18 highlights named targets or counterfactual cues and edit regions or physical effects in benchmark instructions, while Figure 19 summarizes the case mix and subset distributions.Figure 19’s vocabulary is dominated by scene and entity nouns.Source: G AgentEdit-Bench · p. 21, Figure 18 · p. 26, Figure 19 · p. 26

G.1 Scoring rubric and judge prompt

  • IP edits are scored on seven axes, while reasoning and removal use axes 1–5; removal outputs that replace the target object are capped at 1 on axis 1.The rubric separates IP Presence from IP Identity Match, distinguishing placement in roughly the right region from matching a specific real-world identity.Source: §G.1 · p. 22, Table 11 · p. 23
  • Cross-type aggregate scores normalize every case to a 0–21 scale and report the aggregate as a percentage.Per-edit-type numbers use each type’s native axis count.Source: §G.1 · pp. 22–23
  • The judge runs at temperature 0 and scores frames sampled at indices 0, n/3, and 2n/3; per-axis scores are averaged across frames and raw judge text is retained.Figure 20 shows two prompt variants that share a body and differ in axis count and the removal-only anti-replacement clause.Source: §G.1 · p. 23, Figure 20 · p. 28
  • The seven IP axes assess instruction following, region localization, source preservation, visual quality, temporal consistency, IP presence, and IP identity match.Each axis receives an integer score from 0 to 3 with a brief justification, and the judge returns the axis scores and their total.Source: §G.1 · p. 27

Related papers

Loading 2605.18748v1…