PaperlayerView exact revision on arXiv

Source-linked AI summary

EditDuet: A Multi-Agent System for Video Non-Linear Editing

Marcelo Sandoval-Castaneda, Bryan Russell, Josef Sivic, Gregory Shakhnarovich, Fabian Caba Heilbron

arXiv:2509.10761v1cs.CV

TL;DR

Video NLE systems must translate high-level instructions into tool actions while handling complex footage, editing constraints, and iterative feedback. EditDuet uses specialized Editor and Critic agents with exploration-generated communication examples, and the authors report that it outperforms existing approaches across several evaluation dimensions.

  • Problem

    Video editing requires interpreting footage and style, using professional NLE tools, and understanding domain-specific language while translating directives into editing steps.

    Source: §1 · p. 2

  • Method

    EditDuet uses an Editor to build a timeline and a Critic to provide request-based feedback, with exploration-generated examples teaching both agents to communicate effectively.

    Source: §1 · p. 2

  • Results

    EditDuet vastly outperforms existing approaches in failure rate, time constraint satisfaction, and human/automatic preference.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The PreferenceRate based on the automatic judge provides a scalable strategy to quantify NLE performance.

    Source: §3.5 · p. 6

    The setup edits the B-roll timeline while assuming the A-roll is fixed.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Automated tools for video editing and assembly have applications ranging from filmmaking and advertisement to content creation for social media. Previous video editing work has mainly focused on either retrieval or user interfaces, leaving actual editing to the user. In contrast, we propose to automate the core task of video editing, formulating it as sequential decision making process. Ours is a multi-agent approach. We design an Editor agent and a Critic agent. The Editor takes as input a collection of video clips together with natural language instructions and uses tools commonly found in video editing software to produce an edited sequence. On the other hand, the Critic gives natural language feedback to the editor based on the produced sequence or renders it if it is satisfactory. We introduce a learning-based approach for enabling effective communication across specialized agents to address the language-driven video editing task. Finally, we explore an LLM-as-a-judge metric for evaluating the quality of video editing system and compare it with general human preference. We evaluate our system's output video sequences qualitatively and quantitatively through a user study and find that our system vastly outperforms existing approaches in terms of coverage, time constraint satisfaction, and human preference.

1 Introduction

EditDuet targets the challenge of translating high-level language into iterative, tool-mediated video editing across complex footage and constraints. It combines specialized Editor and Critic agents, learned communication examples, and automatic judging; the authors report gains over existing approaches.

1 Introduction

  • Video editing requires interpreting footage, mastering professional NLE tools, and understanding domain-specific natural-language instructions.The system must also decompose directives into steps, map them to tools, and execute them; the footage may include 100× more material than the final edit.Source: §1 · p. 2
  • EditDuet specializes an Editor and a Critic that iteratively revise a B-roll timeline against a high-level request, while the A-roll remains the interview or voiceover.The Editor uses NLE tools to build a draft; the Critic gives feedback and calls the renderer when satisfied.Source: §1 · p. 2
  • Exploration-generated in-context demonstrations teach the Editor to address Critic feedback and the Critic to produce request-aligned feedback.Source: §1 · p. 2
  • Because creative editing admits many valid outputs and ground-truth collection is difficult, the authors propose an LLM-as-a-judge approach for automatic evaluation.User-preference studies can compare quality reliably but are costly and difficult to scale.Source: §1 · p. 2
  • EditDuet vastly outperforms existing approaches in failure rate, time constraint satisfaction, and human/automatic preference.The reported experiments use real-world NLE projects from EditStock.Source: §1 · p. 3

2 Related Work

Prior work addresses constrained editing tasks, specialized editing tools, iterative LLM refinement, agentic workflows, and model-based evaluation. EditDuet combines iterative dual-agent editing with exploration-based collaboration and a video timeline judge.

2.1 AI-assisted Video Non-linear Editing

  • Earlier NLE methods focus on constrained settings, while EditDuet uses extensive raw footage, broader user input, and iterative timeline actions.Prior formulations include selecting views from synchronized multicamera video, transcript-based clip retrieval, and manipulating an existing timeline.Source: §2.1 · p. 3

2.2 Video Editing Tools

  • Video-editing technologies address tasks such as multicamera alignment, cut prediction, transitions, and interactive interfaces; EditDuet builds on visual search and timeline actions.The cited work includes alignment [Arev et al. 2014; Sun et al. 2023; Wang et al. 2014], cut prediction, and interactive editing interfaces [Huber et al. 2019; Huh et al. 2023; Tilekbay et al. 2024; Wang et al. 2024a,b].Source: §2.2 · p. 3

2.3 Iterative Refinement for LLMs

  • Unlike single-agent refinement, EditDuet assigns low-level NLE operations to an Editor and draft-timeline feedback to a Critic during inference.The Critic provides natural-language guidance, unlike reinforcement-learning actor-critic methods [Haarnoja et al. 2018].Source: §2.3 · p. 3

2.4 Agentic Workflows Using LLMs

  • Unlike BAGEL’s single-agent exploration, EditDuet seeks exploration examples that benefit communication between multiple agents and uses self-reflection to find demonstrations faster.Source: §2.4 · p. 3

2.5 LLM-based Evaluation

  • Building on prior evidence that model judgments can correlate with human preferences, the authors propose a VLM judge for NLE timelines and report correlation with human preferences.Source: §2.5 · p. 3

3 Approach

EditDuet frames video editing as iterative Editor–Critic refinement in an NLE environment, supported by synthetic in-context demonstrations and a VLM-based evaluation judge.

3 Approach

  • EditDuet combines an Editor and Critic that iteratively refine an NLE timeline from a video collection, A-roll, and a natural-language user query.The environment exposes A-roll transcription, a collection summary, visual search results, and the current timeline.Source: §3 · pp. 3–4

3.1 Non-Linear Editing Environment

  • The NLE environment represents a fixed A-roll and video collection alongside an editable B-roll timeline, while search exposes up to five results at a time.The collection is segmented into sub-clips and summarized; the timeline can start empty or use randomly selected clips or a previous run’s timeline.Source: §3.1 · p. 4

3.2 Editor Agent

  • The Editor chooses NLE actions from its observations, action history, and Critic feedback, using search and timeline operations before signaling when feedback is satisfied.Available operations include searching the collection, adding or removing clips, rearranging clip positions, and DONE.Source: §3.2 · p. 4

3.3 Critic Agent

  • The Critic checks whether the timeline satisfies the user request, then either returns feedback for another Editor iteration or renders the video.The paper uses Llama3.1-8B-Instruct with structured generation as Critic and does not use a VLM Critic because multiple timeline iterations exceed open-source VLM context sizes.Source: Figure 2 · p. 5, §3.3 · p. 5

3.4 In-Context Learning of Multi-Agent Communication

  • EditDuet’s two-stage in-context learning process gathers Editor demonstrations of tool use and feedback-following, then Critic demonstrations for improving B-roll sequences.Examples are filtered for high scores; the Editor’s examples scoring 4 may be refined, and only score-5 pairs are kept.Source: §3.4 · pp. 5–6

3.5 Automatic Non-Linear Editing Evaluation

  • The VLM judge compares two timelines against a user request using keyframe grids and durations, then PreferenceRate aggregates its pairwise preferences across episodes.The judge is GPT-4o; its prompts include midpoint keyframes from each sub-clip and each sub-clip’s duration.Source: §3.5 · p. 6
  • 80.6% agreement between the automatic judge and human preferences, versus 78.7% agreement among human evaluators, supports the judge’s evaluation reliability.PABAK was 0.61 for judge–human comparison and 0.57 among humans.Source: §3.5.1 · p. 6

4 Results

The evaluation uses annotated documentary B-roll tasks and compares EditDuet with retrieval-based, program-synthesis, and agentic baselines. EditDuet leads on key editing measures, while its qualitative results and remaining failures show both the value and limits of exploration and collaboration.

4.1 Evaluation Dataset

  • The evaluation uses EditStock footage, focusing on five documentaries and annotating B-roll sequences with high-level style and semantic prompts.The five projects contribute 1,458 minutes of video to be edited into 21.5 minutes of final cuts; sequences contain up to 27 sub-clips.Source: §4.1 · p. 6

4.2 Evaluation Criteria

  • Evaluation reports time coverage, repeated sub-clips, and failure rate, including runs that fail to produce valid actions or timelines.Repetition means at least 80% overlap between sub-clips; failure examples include invalid filenames and out-of-bounds indices.Source: §4.2 · pp. 6–7

4.4 Results and Analysis

  • 8.2% failure rate, 89.8% time coverage, and 0.174 repetitions: EditDuet beats competitors on quantitative metrics and automatic preference.The table compares the method with baselines and ablations across editing metrics and preferences.Source: Table 1 · p. 7, §4.4 · p. 7

4.3 Baselines and Competing Methods

  • Baselines include retrieval-based Transcript2Video, adapted VisProg, and BAGEL; ablations test single-agent editing and collaboration without exploration.Transcript2Video uses CLIP similarity and segmented footage, while VisProg executes programs in the editing environment.Source: §4.3 · p. 7

4.5 Qualitative results

  • In examples, EditDuet produces the best-quality footage, pacing, and match to requests; stepwise execution supports self-correction unlike one-pass systems.The qualitative comparison also reports repetitions and empty B-roll sequences for VisProg, and low-aesthetic-quality footage for Transcript2Video.Source: §4.5 · p. 7

4.6 Limitations

  • Exploration reduces system-breaking failures from 19.5% to 8.2%, but function hallucination and unsupported feedback remain the most prevalent failures.Other reported failures include file hallucination, index errors, and out-of-bounds sub-clips.Source: §4.6 · pp. 7–8

5 Conclusion

The paper concludes that EditDuet combines multi-agent editing with a communication method and reports gains on real-world data. It identifies broader editing operations and larger video contexts as directions for future work.

5 Conclusion

  • EditDuet combines a multi-agent architecture with a method for enabling communication between agents, and is formulated as a decision process aligned with human interaction with NLE software.The authors report improvements in quantitative metrics, human preference, and automatic preference judgments on real-world data.Source: §5 · p. 8
  • Future work could extend the action space or add specialized agents for transitions, pixel-level editing, and audio editing.The authors also identify larger vision contexts and reinforcement-learning-based editing agents as future directions.Source: §5 · p. 8
  • The authors describe AI-based evaluation and agent-based editing as openings for reinforcement-learning approaches to non-linear video editing.Source: §5 · p. 8

A Ethical Considerations

The authors frame automated editing as assistance for human editors, not a replacement. Their system leaves the final result under human control.

A Ethical Considerations

B Example Failures

The appendix illustrates recurrent failure modes in the editing systems and notes that exploration reduces their occurrence. It also lists the prompts crafted for the method.

B Example Failures

  • Common failures include function and file hallucinations, unsupported feedback, and out-of-bounds video sub-clips; exploration reduces their occurrence significantly.Source: B Example Failures · p. 12
  • Less common failures include index errors and seemingly nonsensical search queries.Source: B Example Failures · p. 12

C Prompts

D Baseline Implementation Details

The baselines use different adaptations: Transcript2Video tracks target duration, while VisProg uses the same LLM backbone as EditDuet and a modified timeline-addition function.

D Baseline Implementation Details

E Running Time

EditDuet's exploration and inference take substantially longer than the inference times reported for the comparison methods.

E Running Time

  • EditDuet inference takes approximately 2.6 minutes on 8 H100s for a single B-roll sequence.Source: E Running Time · p. 12
  • Exploration takes approximately 82 minutes, and synthesized demonstrations average 17 Editor steps and 4 Critic steps.Source: E Running Time · p. 12
  • T2V and VisProg take less than half a minute on average, while BAGEL takes about 1.5 minutes.Source: E Running Time · p. 12

F Comparison with Human Editors

Professional editors were preferred over every automatic method, though EditDuet was preferred over other automatic editing methods in the authors' comparison.

F Comparison with Human Editors

G Filtered Comparisons

After filtering repetitions and empty sequences, EditDuet remained preferred, while competing methods showed the largest decreases in performance.

G Filtered Comparisons

  • After filtering repetitions and empty sequences, T2V's automatic preference against EditDuet was 8.70%, VisProg's was 9.09%, and BAGEL's was 4.35%.Source: G Filtered Comparisons · p. 12
  • In the filtered setting, Editor-only achieved 18.18% automatic preference and Editor Critic achieved 30.77%.Source: G Filtered Comparisons · p. 12
  • The authors report that performance differences remained similar after filtering, with the largest decreases coming from competing methods.Source: G Filtered Comparisons · p. 12

Related papers

Loading 2509.10761v1…