Source-linked AI summary
Unified Agentic Video Editing Across Levels of Complexity and Creativity
Surabhi S. Nath, Kim Ferres, Milan Petrović, Lion Schulz
TL;DR
Video-editing tasks differ in editorial goals, complexity, and creative freedom, while sharing a need for structured source understanding. The paper evaluates agentic workflows using shared textual metadata across previews, summaries, and trailers. The outputs exhibit different trade-offs in clickability, spoiler control, and information retention, though the study covers only five episodes of one scripted show.
Problem
Editing tasks vary in editorial goal, complexity, and creative freedom, while all require structured understanding of source material.
Source: §1 · p. 1Method
The authors evaluate agentic workflows for previews, video summaries, and trailers using a shared structured textual metadata backbone.
Source: §1 · p. 1, §2.1 · p. 2, §3 · p. 4Results
Full-prompt previews outperformed random picks in clickability, summaries supported 96% quiz accuracy versus 69% without context, and trailers were unanimously preferred to summaries for clickability.
Source: §2.2 · p. 2, §2.3 · p. 3, §2.4 · p. 4Takeaways & Limitations
Structured textual metadata can support diverse agentic workflows tailored to editing complexity and creative freedom.
Source: §3 · p. 4The study covers five episodes from one fictional scripted TV show; broader studies are needed to test workflow generality.
Source: §3 · p. 4
Abstract
from arXiv · showhide
Editing is a core component of video production, requiring creative planning and decisions under multiple constraints. Here, we report methods for agentic tooling for automated video editing across three tasks varying in editorial goal, complexity and creativity, namely scene previews, video summaries and cinematic trailers. We evaluate the outputs and discuss implications for automation and agency.
1 Introduction
Video editing spans tasks with different editorial goals, complexity, and creative freedom, yet all require structured understanding of source material. This work tests whether one hierarchical data structure can support editing from previews to cinematic trailers.
1 Introduction
- Editing tasks vary in editorial goal, complexity, and creative freedom, but all depend on structured understanding of source material.The paper tests one hierarchical data structure across previews, summaries, and cinematic trailers.Source: §1 · p. 1
- Prior work addresses editing subtasks and agentic workflows, but real-world post-production benchmarks and comparisons across orchestration levels remain limited.The paper frames video editing as a multimodal benchmark relevant to creative AI and agentic AI.Source: §1 · p. 1
2 Methods and Results
The paper evaluates three agentic editing tasks using structured text metadata, with workflows tailored to their differing complexity and creative freedom. Results show distinct trade-offs: previews balance clickability and spoiler control, summaries retain narrative information, and trailers are more clickable than summaries.
2.1 Task, system, data and evaluation overview
- Across five episodes, agents used hierarchically linked episode-, scene-, and shot-level text metadata, with workflows matched to each task’s complexity and creative freedom.Previews use a two-call scene-role and clip-selection workflow; summaries sequence plot lines, character introductions, and narrated beats; trailers use goal-directed planning with reusable editing tools, producing 24 steps and 50 API calls.Source: §2.1 · p. 2, §2.2 · p. 2, §2.3 · pp. 2–3
2.4 Task 3: Cinematic trailer editing
- Trailers were unanimously preferred to summaries for clickability, while quiz performance with trailer context was above no-context but below summary-context performance.These comparisons support the intended distinction between trailers’ hookiness and summaries’ informativeness.Source: Figure 2 · p. 3, §2.4 · p. 4
2.3 Task 2: Narrative video-to-video summarisation
- With summaries in context, the multimodal LLM answered 96% of quiz questions correctly, versus 69% without context.The no-context baseline reflects prior knowledge of the popular series; questions missed with summaries involved finer event details omitted by the edit.Source: Figure 2 · p. 3, §2.3 · p. 3
2.2 Task 1: Preview generation
- Full-prompt previews were substantially more clickable than random picks, while unguided selections’ marginal clickability edge was nonsignificant and came at the expense of spoiler control.The full prompt also produced superior understandability and edit quality.Source: §2.2 · p. 2, Figure 2 · p. 3
2.4 Task 3: Cinematic trailer editing
- Trailer edits combined dialogue, voiceover, music, sound design, title cards, silence beats, and over 100 clips on average, with coordinated pacing and creative plot re-narration.The agent chose its own sequence of steps and tool calls; outputs were 2–3 minutes long.Source: §2.4 · pp. 3–4
3 Discussion
The discussion presents structured textual metadata as a backbone for agentic workflows tailored to editing complexity and creative freedom. It also highlights implications for editorial agency, multidimensional evaluation, and the study’s limited scope.
3 Discussion
- A structured textual metadata backbone supports diverse workflows, orchestrated according to edit complexity and creative freedom.The tasks range from engagement-focused previews to narrative-coverage summaries and trailers that creatively balance the two.Source: §3 · p. 4
- Agentic editing may shift editors from manual searching toward specifying intent and creative direction, with systems serving as support while editors retain final control.The paper emphasizes exploration, refinement, and inspiration as support roles.Source: §3 · p. 4
- Trailer evaluation is challenging because open-ended edits depend on audience, enterprise needs, and subjective taste beyond verifiable properties.The study covers five episodes from one fictional scripted show; broader production, genre, and content-type studies are needed, and long agentic chains’ risks were not systematically studied.Source: §3 · p. 4
A.1 Video Understanding System
The video-understanding system produces structured textual metadata at episode, scene, and shot levels. The pipeline is described as applicable across varied content types, genres, and languages.
A.1 Video Understanding System
- The in-house system combines vision-language models with classical machine-learning analysis to extract dense textual metadata at multiple temporal resolutions.Episode metadata covers plot, themes, scenes, tone, and genres; scene and shot metadata include narrative descriptions, characters, and timestamps.Source: §A.1 · p. 8
- The pipeline is described as general-purpose across scripted and unscripted formats, genres, and languages.Examples include TV shows, documentaries, and movies; action, romance, comedy, thriller, and horror; English and German.Source: §A.1 · p. 8
A.2.1 Preview Prompts [blue part not in ablated prompt]
Preview generation used gemini-3.1-pro-preview, averaging 180k tokens.
A.2.1 Preview Prompts [blue part not in ablated prompt]
- Preview generation used gemini-3.1-pro-preview, averaging 180k tokens.Source: §A.2.1 · p. 9
1. Narrative structure (once per video)
The prompt asks for analysis of how a video's story is constructed, beyond simply describing what happens.
1. Narrative structure (once per video)
- The analyst examines the video's narrative architecture, focusing on how the story is constructed rather than only what happens.Source: §1 · p. 9
Task
The task is to analyze narrative structure by identifying the pattern, turning points, act boundaries, and each scene's structural role.
Task
- Identify the overall structural pattern and the narrative's major turning points.Source: Task · p. 9
- Define act boundaries when applicable and classify the structural role of every scene.Source: Task · p. 9
Output
The output assigns every scene exactly one structural role and is requested in JSON format.
Output
- Assign every scene exactly one role: setup, development, escalation, climax, resolution, breather, or transition.The prompt characterizes these roles by their function in establishing or advancing the story, varying tension, resolving conflict, offering respite, or moving between story threads.Source: Output · p. 9
- Return the narrative-structure analysis in JSON format.Source: Output · p. 9
2. Generate preview (once per preview)
The methods cover constrained preview selection, video-to-video summaries, and cinematic trailer generation, with evaluations of preview quality and edit informativeness.
2. Generate preview (once per preview)
- Preview candidates are continuous, genre-representative windows intended to offer broad appeal and an intriguing hook.Source: §2 · p. 9
Requirements
- Each preview must be suspenseful but spoiler-free, self-contained, dialogue-complete, and close to its target duration.Source: Requirements · p. 9
Input
- Preview selection uses plot summaries, key scenes, and scene metadata as input.Source: Input · p. 9
Output
- The preview output consists of candidate-window start and end timestamps plus one short rationale for why the window works.Source: Output · p. 9
A.2.2 Video-to-video summarisation
- Video-to-video summaries use plot-line, scene, and shot metadata to produce a 90–110-second video.The workflow uses three Gemini calls per plot line plus one to order plot lines; a four-plot-line episode takes 13 calls.Source: §A.2.2 · p. 10
A.2.3 Trailer Generation
- Trailer generation gives an agent creative goals, dense metadata, and editing tools, but the resulting workflow remains a deterministic fixed chain.The workflow is broken down into 24 steps and 51 API calls.Source: §A.2.3 · p. 10, Table 1 · p. 10
A.3.1 2-AFC
- Full-prompt previews were more understandable and cut no dialogue; trailers were more clickable but less understandable than summaries despite their longer duration.The preview evaluation compared full and ablated prompts with random picks using LLM-as-a-judge 2-AFC votes, with order flips treated as ties.Source: §A.3.1 · p. 11, Figure 4 · p. 11
A.3.2 Quiz Benchmark
- Across the quiz benchmark, summary context ranked first overall and across question bands; trailer context ranked second overall but underperformed no context in two of five bands.The benchmark included 22 questions per episode; no-context performance was still significantly above random chance.Source: §A.3.2 · p. 12, Figure 5 · p. 12