Source-linked AI summary
CutClaw: Agentic Hours-Long Video Editing via Music Synchronization
Shifang Zhao, Yihan Hu, Ying Shan, Yunchao Wei, Xiaodong Cun
TL;DR
Hours-long video editing must reduce massive multimodal footage into concise, instruction-following stories synchronized with music. CutClaw addresses this with hierarchical decomposition and Playwriter, Editor, and Reviewer agents, and significantly outperforms state-of-the-art baselines in visual quality, instruction following, and rhythmic harmony.
Problem
Hours-long video editing requires labor-intensive selection of sparse segments that preserve narrative coherence while achieving fine-grained visual-audio harmony.
Method
CutClaw hierarchically decomposes video and audio, then uses Playwriter, Editor, and Reviewer agents to plan narratives and select, assess, and assemble synchronized clips.
Results
CutClaw significantly outperforms state-of-the-art baselines in visual quality, instruction following, and rhythmic harmony.
Takeaways & Limitations
The framework supports instruction-aligned, music-driven storytelling from hours-long raw footage through coordinated multimodal agent collaboration.
Takeaways & Limitations
The multi-stage pipeline has high inference latency, and the system lacks advanced visual hooks such as generated effects or specific monologue highlights.
Abstract
from arXiv · showhide
Editing the video content with audio alignment forms a digital human-made art in current social media. However, the time-consuming and repetitive nature of manual video editing has long been a challenge for filmmakers and professional content creators alike. In this paper, we introduce CutClaw, an autonomous multi-agent framework designed to edit hours-long raw footage into meaningful short videos that leverages the capabilities of multiple Multimodal Language Models~(MLLMs) as an agent system. It produces videos with synchronized music, followed by instructions, and a visually appealing appearance. In detail, our approach begins by employing a hierarchical multimodal decomposition that captures both fine-grained details and global structures across visual and audio footage. Then, to ensure narrative consistency, a Playwriter Agent orchestrates the whole storytelling flow and structures the long-term narrative, anchoring visual scenes to musical shifts. Finally, to construct a short edited video, Editor and Reviewer Agents collaboratively optimize the final cut via selecting fine-grained visual content based on rigorous aesthetic and semantic criteria. We conduct detailed experiments to demonstrate that CutClaw significantly outperforms state-of-the-art baselines in generating high-quality, rhythm-aligned videos. The code is available at: https://github.com/GVCLab/CutClaw.
1 Introduction
Audio-driven editing requires selecting sparse segments from hours of footage while preserving both global narrative coherence and fine-grained visual-audio harmony. CutClaw addresses this challenge with hierarchical multimodal analysis and collaborating agents, outperforming state-of-the-art baselines.
- Motivation: Audio-driven editing treats hours-long footage reduction as a complex multimodal alignment problem involving narrative coherence and local auditory dynamics.Existing editing approaches often neglect audio's role, producing outputs without synchronization, semantic awareness, or narrative progression.
- Method: CutClaw uses bottom-up multimodal deconstruction to represent raw video and audio as structured visual scenes and musical sections.This coarse-to-fine abstraction addresses the context-length challenge of hours-long footage for current MLLMs.
- Method: A Playwriter agent plans music-anchored narratives, while Editor and Reviewer agents collaboratively perform precise segment selection.The workflow separates global storytelling from fine-grained visual grounding and review.
- Task and Contribution: CutClaw models audio-driven video editing as joint optimization of instruction-driven storytelling and fine-grained rhythmic harmony.The framework targets both creative intent and synchronization with music.
- Results: CutClaw significantly outperforms state-of-the-art baselines in visual quality, instruction following, and rhythmic harmony.The introduction attributes this result to extensive experiments and user studies.
2 Related Work
Prior video-editing systems assemble pre-segmented clips or rely on scripts and generally neglect rhythmic guidance from music. CutClaw instead processes raw footage without manual scripts for hierarchical, instruction-aligned, music-driven editing.
- AI-assisted Video Editing: Earlier editing methods formulated shot selection as energy minimization, while recent generative methods construct sequences from high-level instructions or subtitle narratives.These approaches represent a shift from optimization-based heuristics toward instruction-driven generation.
- AI-assisted Video Editing: Existing approaches are limited by pre-segmented clip assembly, explicit script dependence, and insufficient rhythmic guidance from music.These limitations leave music underused as a structural signal for editing.
- CutClaw: CutClaw directly processes raw, untrimmed footage without manual scripts, jointly targeting semantic storytelling and fine-grained audio-visual harmony.It formulates editing as hierarchical narrative construction rather than isolated temporal grounding or highlight detection.
3 Method
CutClaw formulates editing as agent-driven segment extraction and assembly, using hierarchical multimodal analysis to reduce hours-long footage into searchable semantic units. Playwriter anchors narrative planning to musical structure, while Editor and Reviewer select, validate, and commit clips under visual, narrative, instruction, and synchronization constraints.
- Problem Formulation: CutClaw extracts and synchronizes video clips from raw footage, music, and text instructions to construct an edited timeline.The target timeline is a sequence of continuous clips extracted from the original footage.
- Problem Formulation: A joint objective combines visual quality, narrative flow, instruction fidelity, and rhythmic alignment for timeline optimization.The framework approximates optimization through hierarchical search-space analysis and pruning rather than brute-force search.
- Bottom-Up Multimodal Footage Deconstruction: Bottom-up deconstruction discretizes continuous video and music into structured semantic units, creating a finite candidate space for hierarchical planning.The process includes shot detection, detailed captioning, scene-level aggregation, and structural audio parsing.
- Bottom-Up Multimodal Footage Deconstruction: Music analysis detects downbeats, pitch changes, and spectral-energy changes, filters them temporally, and organizes them within coarse musical sections.An MLLM partitions the track into structural units, while weighted cue intensities identify significant boundaries and guide structure-aligned captions.
- Playwriter: Music-Anchored Script Synthesis: Playwriter maps decomposed visual scenes onto musical units, generating keypoint-aligned shot specifications that guide later retrieval rather than final timestamps.Each specification includes an audio-derived duration constraint and a semantic visual description.
- Playwriter: Music-Anchored Script Synthesis: Playwriter enforces non-overlapping scene assignments across musical units and anchors each planned block to the corresponding audio interval.Violating the no-reuse condition triggers proposal rejection and regeneration with negative constraints.
- Editor and Reviewer: The Editor evaluates candidate intervals using aesthetic quality and protagonist presence, then submits approved clips to the Reviewer or backtracks after rejection.The Reviewer acts as a discriminatory gate auditing candidate clips for semantic identity consistency.
4 Experiment
CutClaw is evaluated on long-form film and VLOG footage using quantitative metrics, qualitative comparisons, ablations, and user preferences. It outperforms baselines overall, while component removals reveal the importance of audio context, reviewer feedback, and hierarchical editing.
- Benchmark: The benchmark contains 20 evaluation cases from 10 source pairs spanning five feature films and five long-duration VLOGs, with approximately 24 hours of footage.The footage ranges from 1 to 3 hours per source pair and includes character-centric and narrative-centric instructions.
- Metrics: CutClaw is evaluated using Visual Quality, Instruction Follow, and AV Harmony, with a user study additionally measuring Human-Likeness.AV Harmony measures minimum temporal offset between audio onsets and video scenes, rewarding alignments within thresholds such as ∆t ≤0.1s.
- Main Results: CutClaw achieves superior performance across all quantitative metrics, including Visual Quality, Instruction Follow, and AV Harmony.Its Instruction Follow performance particularly excels for object-oriented instructions, while AV Harmony indicates rhythmically aligned cuts.
- Main Results: Qualitative comparisons show baselines making rigid or musically misaligned selections, while NarratoAI, UVCOM, and Time-R1 exhibit different narrative or visual shortcomings.NarratoAI loosely follows instructions but degrades visual quality; UVCOM and Time-R1 preserve visual quality but lack logical connections across shots.
- Ablation Study: 86.5 to 77.2: replacing beat-aware audio analysis with fixed-length segmentation substantially reduces AV Harmony.Removing the Reviewer reduces Visual Quality from 77.6 to 76.0, while replacing the Editor with random selection lowers the average score from 70.0 to 65.6.
- User Study: CutClaw receives 49.8% of Visual Quality votes and 53.0% of Audio-Visual Harmony votes, more than double Time-R1’s corresponding 21.4% and 20.0%.It also receives 48.8% of Human-Likeness preferences from 25 participants evaluating 80 items.
- Limitations: The framework lacks advanced visual hooks and has high inference latency when processing extensive raw footage.The authors identify generated effects, monologue highlights, speed optimization, and real-time feedback as future directions.
5 Conclusion
CutClaw automates professional editing from hours-long raw footage through hierarchical decomposition and collaboration among specialized agents. The reported experiments show gains across Visual Quality, Instruction Following, and AV Harmony.
- Conclusion: CutClaw automates professional video editing from hours-long raw footage using hierarchical decomposition and specialized collaborating agents.The Playwriter anchors narrative structure to music, the Editor performs fine-grained visual grounding, and the Reviewer enforces aesthetic and continuity constraints.
- Conclusion: CutClaw significantly outperforms state-of-the-art baselines across Visual Quality, Instruction Following, and AV Harmony.These results support its approach to long-context processing and precise audio-visual consistency.