CutClaw: Agentic Hours-Long Video Editing via Music Synchronization

Introduction

• Prior template-based methods, highlight detection, and text-based approaches neglect holistic alignment across music, visuals, and narrative.Templates impose rigid slots, highlight detection focuses on local visual salience, and transcript alignment neglects musical structure.Source: §1 · p. 2
• Practical audio-visual storytelling faces context-window limits, difficulty grounding creative instructions in source context, and the need for fine-grained temporal alignment of music and visual plot.Hours-long footage exceeds current MLLM context windows, while synchronized editing must account for plot, aesthetics, and emotion.Source: §1 · pp. 2–3
• CutClaw combines bottom-up multimodal footage deconstruction with Playwriter-led, music-anchored narrative planning and collaborative Editor–Reviewer segment selection.The Reviewer checks plot relevance, visual aesthetics, and instruction following.Source: §1 · p. 3
• The paper defines audio-driven video editing as joint optimization of instruction-driven storytelling and fine-grained rhythmic harmony.The target is to select and assemble sparse salient segments from hours of footage while maintaining global narrative coherence and local audio-visual harmony.Source: §1 · pp. 2–3
• The authors report that CutClaw significantly outperforms state-of-the-art baselines in visual quality, instruction following, and rhythmic harmony.The introduction says the evidence comes from extensive experiments and user studies.Source: §1 · p. 3

AI-assisted Video Editing

• Unlike prior methods that assemble pre-segmented clips or rely on explicit scripts, CutClaw edits raw, untrimmed footage without manual scripts while combining semantic storytelling with fine-grained audio-visual harmony.Earlier work includes energy-minimization approaches such as Write-A-Video and ESA [7], and generative methods driven by high-level instructions or subtitle narratives [8] [18].Source: §2 · pp. 3–4

Video Temporal Grounding and Highlight Detection

• Temporal-grounding and highlight-detection methods struggle with long-term context and precise retrieval duration, limiting their suitability for cuts synchronized to musical beats and rhythms.The literature spans query-based localization using pretrained encoders [10] or MLLMs [25], and highlight detection based on visual saliency or textual prompts; CutClaw handles hours-long footage with textual and musical input.Source: §2 · p. 4

Agents for Video Generation and Editing

• Existing video-agent systems, including ViMax, EditDuet, and targeted trimming [30], face context-window and coarse-planning limits; CutClaw pairs hierarchical decomposition with audio-anchor alignment.These limits hinder scalability and audio-visual synchronization on hours-long footage.Source: §2 · p. 4

3.1 Problem Formulation

• CutClaw treats video editing as agent-driven clip extraction and assembly from raw footage, target music, and text instructions.The assembled clips are intended to follow the narrative instruction while maintaining organic audio-visual harmony.Source: §3.1 · p. 4
• The target edit is a trimmed timeline E = (c1, . . . , cN), with each ci representing a continuous segment extracted from the original footage.The formal input is raw footage V, background music M, and user instructions I.Source: §3.1 · p. 4
• The framework optimizes a joint objective combining visual quality, narrative flow, semantic alignment to instructions, and rhythmic alignment to music.The objective weights these four criteria with λ_v, λ_n, λ_c, and λ_s, respectively.Source: §3.1 · p. 5
• Visual quality rewards aesthetic appeal and protagonist prominence, while narrative flow requires coherent storytelling between adjacent clips.These criteria are Q_vis and Q_narr in the joint objective.Source: §3.1 · p. 5
• Semantic alignment measures fidelity to the instructions, and rhythmic alignment encourages cuts to synchronize with musical beats.These criteria are Q_cond(E, I) and Q_sync(E, M) in the joint objective.Source: §3.1 · p. 5
• Rather than brute-force search, CutClaw uses hierarchical search-space analysis and pruning to constrain candidate choices and finalize the timeline.It discretizes footage into structured semantics, uses Playwriter audio-visual correlations to localize candidate pools, then has Editor retrieve clips and Reviewer apply rejection sampling.Source: §3.1 · p. 5, Figure 2 · p. 5

3.2 Bottom-Up Multimodal Footage Deconstruction

• Bottom-up deconstruction turns continuous, high-dimensional footage and music into structured semantic units, creating a finite searchable candidate space for later planning.The passage motivates this step by noting that direct timeline optimization is computationally intractable.Source: §3.2 · p. 5
• The illustrated workflow detects shots across the video, captions them for detailed understanding, and uses an LLM to aggregate similar content into scene-level descriptions.Fig. 3 also depicts Playwriter generating a storyline and detailed shot plan for each music shot.Source: Figure 3 · p. 6
• The method organizes footage hierarchically, treating camera-cut-bounded shots as atomic units and aggregating them into contiguous, spatio-temporally coherent scenes.This hierarchy supports both fine-grained and coarse-grained narrative comprehension within MLLM context-window limits.Source: §3.2 · p. 6
• For each shot, an MLLM extracts cinematography, character dynamics, and environment attributes, which support semantic shot descriptions and scene grouping.Fig. 3 describes shot detection and captioning followed by LLM aggregation of similar content into a scene-level description.Source: §3.2 · p. 6, Figure 3 · p. 6
• Adjacent-shot attribute similarities are weighted, and a scene boundary is placed whenever the resulting transition similarity falls below threshold τ.The similarity vector is derived from LLM features, with α balancing attribute importance.Source: §3.2 · p. 6
• Dialogue-derived character names and roles are injected as textual MLLM conditioning to ground scene descriptions and track recurring protagonists across scenes.The passage illustrates replacing a generic description such as “a man” with a specific identity such as “Joker.”Source: §3.2 · p. 6

Video Shots Aggregation: From Shot to Scene

• To maximize rhythmic alignment, the method begins converting the continuous music waveform into a discrete grid of potential units.The supplied passage ends before specifying what the grid contains or how it is used.Source: Structural Audio Parsing · p. 6

• The hierarchical strategy bridges micro-level beat rhythm and macro-level musical form, giving the Playwriter rigid temporal anchors.Source: Structural Audio Parsing · p. 7
• It detects downbeats, pitch changes, and spectral-energy changes, then temporally filters their combined candidates into robust boundaries [4].Source: Structural Audio Parsing · p. 7
• An MLLM partitions the track into coarse units such as verses and choruses [28], then retains each unit’s keypoints with the highest weighted cue-intensity scores.The score weights downbeat, pitch-change, and spectral-energy intensities.Source: Structural Audio Parsing · p. 7
• The system generates structure-aligned captions describing local rhythm, emotion, and energy to guide visual matching.Source: Structural Audio Parsing · p. 7

3.3 Playwriter: Music-Anchored Script Synthesis

• Playwriter maps semantic scenes to musical units through structural allocation and keypoint-aligned shot planning, optimizing rhythmic alignment, instruction fidelity, and narrative flow.The musical structure serves as the invariant temporal anchor for the visual narrative.Source: §3.3 · p. 7
• The allocation and timing constraints formalize execution rules intended to ensure a valid shot plan.The rules include disjoint scene assignment and durations tied to audio intervals.Source: §3.3 · p. 7
• Scene allocation is disjoint across musical units, preventing source material from being reused across narrative blocks.Each unit receives a subset of scenes, with overlap between units prohibited.Source: §3.3 · p. 7, Structural Scene Allocation · p. 8
• Each unit’s shot-plan duration is anchored to the length of its audio interval.The plan inherits the audio’s fixed temporal topology.Source: §3.3 · p. 7

Structural Scene Allocation

• Structural scene allocation generates a proposal assigning candidate scenes to musical units, conditioned on the user instruction and scene set.The allocation is formulated as conditional generation using a planning function.Source: Structural Scene Allocation · p. 8
• If a proposal reuses a scene across musical sections, the system rejects it and regenerates with negative constraints.This handles violations of the strict disjoint-set requirement.Source: Structural Scene Allocation · p. 8

Keypoint-Aligned Shot Planning

• Each musical segment within a unit receives a shot specification that guides retrieval instead of assigning final timestamps.The plan refines the allocation into executable specifications for the fine-grained segments in each unit.Source: Keypoint-Aligned Shot Planning · p. 8
• Each specification combines a target duration derived from the audio segment, a source scene index, and a semantic visual description.The duration supports rhythmic synchronization, the scene index constrains retrieval to the allocated narrative block, and the description guides visual content matching within that scene.Source: Keypoint-Aligned Shot Planning · p. 8
• Binding each shot to a specific scene converts global optimization into local retrieval tasks and prunes the downstream Editor’s search space.The scene binding helps ensure final clip selection is conducted within the allocated narrative block.Source: Keypoint-Aligned Shot Planning · p. 8

3.4 Editor: Top-Down Hierarchical Visual Grounding

• The Editor is a ReAct agent that iteratively optimizes the joint objective’s local visual-quality term while grounding precise timeline coordinates.It operates within the structural shot plan constrained by the Playwriter.Source: §3.4 · p. 8
• When a candidate pool is insufficient, the Editor retrieves shots from adjacent structural units, then densely grounds and trims candidates for visual quality and protagonist presence.Protagonist presence is measured against the character identity set established in Sec. 3.2; VLM feedback can guide temporal-window shifts.Source: §3.4 · p. 9
• The Editor submits each trimmed candidate to the Reviewer, committing approved clips to the timeline and backtracking to alternative intervals when rejected.Source: §3.4 · p. 9

3.5 Reviewer: Multi-Criteria Validity Gate

• The Reviewer audits every Editor-proposed clip, rejecting those that violate protagonist identity, non-overlap, music-grid duration fidelity, or visual-quality requirements.It uses hierarchical MLLM sampling to calculate a Protagonist Presence Ratio, and rejects shots with significant quality degradation.Source: §3.5 · p. 10
• Identity checks ensure the target subject in H remains the primary visual focus, while temporal checks prevent duplication and align cut points with the music track’s rhythmic grid.The Reviewer rejects clips where the character is only a background extra, occluded, or unrecognizable; any structural violation triggers rejection.Source: §3.5 · p. 10
• When a candidate violates a requirement, the Reviewer returns structured feedback so the Editor can backtrack and explore alternative intervals in the semantic neighborhood.The Editor is directed to select a relevant scene around the current one.Source: §3.5 · p. 10
• Reviewing each candidate clip in the optimized timeline produces the final edited video.Source: §3.5 · p. 10

4.1 Evaluation and Implementation

• The benchmark contains 10 film/VLOG source pairs totaling about 24 hours of footage, 10 music tracks, and 20 cases spanning character-centric and narrative-centric instructions.Raw footage runs 1–3 hours, with target edits of 20 seconds to one minute; evaluation covers Visual Quality and Instruction Follow scored by GPT-5.2, AV Harmony based on temporal offset, and user-rated Human-Likeness.Source: §4.1 · pp. 10–11
• Baselines include subtitle-driven NarratoAI, which is inapplicable to VLOGs, and moment-retrieval UVCOM and temporal-grounding Time-R1 [25], adapted by segmenting long videos and selecting top-5 clips.Selected segments are trimmed to the target duration, discarding excess frames.Source: §4.1 · p. 11
• CutClaw uses MiniMax-M2.1 for Editor and Reviewer agents, Gemini3-Pro for Playwriter, and Qwen3-VL-30B-A3B [2] and Qwen3-Omni-30B-A3B [28] for visual and music captioning.Preprocessing uses PySceneDetect and Whisper-v3-turbo; inference downsamples video to 360p short-side resolution at 2 FPS.Source: §4.1 · p. 11

4.2 Main Results

• Baseline methods exhibit rigid segment selection and fail to align with musical structure; NarratoAI [12] loosely follows instructions but degrades visuals, while UVCOM [26] and Time-R1 [25] preserve visual quality but lack cross-shot narrative connections.The comparison uses full-length footage from Paprika and La La Land paired with music tracks.Source: §4.2 · p. 12, Figure 6 · p. 13
• CutClaw achieves superior performance across all reported quantitative metrics: Visual Quality, Instruction Follow, and AV Harmony.The paper attributes its Visual Quality results to coarse-to-fine editing, Instruction Follow results to precise visual content localization, and AV Harmony results to rhythmic alignment with input music.Source: §4.2 · p. 11, Table 1 · p. 13

Ablation Study

• 86.5 to 77.2: replacing beat-aware audio analysis with fixed-length segmentation reduces AV Harmony; removing the Reviewer lowers Visual Quality from 77.6 to 76.0.Replacing the Editor with a random clip selector reduces the average Visual Quality and Instruction Following score from 70.0 to 65.6, which the authors attribute to loss of hierarchical structuring.Source: Table 2 · p. 13, Ablation Study · p. 14

User Study

• CutClaw receives 49.8% of Visual Quality votes and 53.0% of Audio-Visual Harmony votes, more than double second-best Time-R1’s 21.4% and 20.0%, respectively.It also receives 48.8% of Human-Likeness votes. The study involved 25 participants, 80 evaluation items, and 2,000 opinions; Table 3 reports results across four metrics, instruction types, and video types.Source: User Study · p. 14, Table 3 · p. 14

4.3 Limitation

• The framework lacks advanced visual hooks such as generated effects or monologue highlights, and its multi-stage processing of extensive footage causes high inference latency.The paper identifies generative video models and faster or coarse-to-fine processing for real-time feedback as future directions.Source: §4.3 · pp. 14–15

5 Conclusion

• CutClaw significantly outperforms state-of-the-art baselines across Visual Quality, Instruction Following, and AV Harmony.The conclusion does not provide metric values.Source: §5 · p. 15
• Hierarchical decomposition structures continuous high-dimensional data into semantic units for collaboration among specialized agents.The Playwriter anchors narrative to musical structure, the Editor performs fine-grained visual grounding, and the Reviewer enforces aesthetic and continuity constraints.Source: §5 · p. 15
• CutClaw targets long-context processing and precise audio-visual consistency to enable instruction-aligned, music-driven storytelling from hours-long footage.Source: §5 · p. 15