Source-linked AI summary
Beyond Coherence: Benchmarking Professional Editing-Technique Execution in Multi-Shot Audio-Video Generation
Tianyi Zeng, Junchao Liao, Yujie Wei, Ziying Zhang, Litao Li, Tianyi Wang, Zhichao Wei, Shuyao Xu, Wenwen Qiang, Siyu Zhu, Zhenghao Zhang, Long Qin
TL;DR
Multi-shot generators can produce coherent videos yet remain unreliable at executing professional editing instructions, which existing benchmarks do not directly measure. CutCraft introduces structured editing specifications and a hierarchical hybrid evaluation framework, alongside an agentic plan-generate-compose baseline. Across 13 systems, the benchmark exposes a persistent gap between coherence and editing-technique execution, while the baseline improves selected discrete controls.
Problem
Existing benchmarks largely evaluate quality, synchronization, or plausibility rather than editing-technique execution in multi-shot audio-video generation.
Method
CutCraft combines structured editing specifications, shot alignment, hybrid expert and multimodal evaluation, and an agentic planning, shot-generation, and composition baseline.
Results
Across 13 state-of-the-art systems, coherent and plausible videos remain unreliable at executing editorial instructions, while aesthetic quality is only weakly correlated with compliance.
Takeaways & Limitations
Editing-aware evaluation and decomposition expose editorial-control failures that coherence and visual quality alone do not capture.
Takeaways & Limitations
The evaluation pipeline requires over 120 seconds per sample and may accumulate errors across its multiple stages.
Abstract
from arXiv · showhide
Recent multi-shot audio-video generators can produce increasingly coherent and cinematic outputs, but coherence does not imply the ability to execute editing techniques. Professional editing depends on shot structure, transition grammar, audio-video cut relations, and montage, yet existing benchmarks largely rely on proxies such as content quality, synchronization, or physical plausibility, systematically missing whether such editing instructions are actually executed. We introduce CutCraft, the first benchmark for editing-technique execution in multi-shot audio-video generation. CutCraft extends structured multi-shot prompts with explicit editing specifications and is paired with a hierarchical hybrid evaluation framework that combines shot-structure alignment, expert-model metrics, tool-grounded multimodal judgment, and rubric-based question answering. Beyond evaluation, we design an agentic editing baseline that decomposes generation into planning, shot-level synthesis, and post-hoc composition, explicitly realizing editing semantics such as J-cuts, L-cuts, and transition timing. Across 13 state-of-the-art closed- and open-source models, CutCraft reveals a consistent gap between coherence and editing-technique execution: current systems often produce plausible multi-shot videos yet fail to execute editorial instructions reliably. We find unstable shot structures, weak control of transition execution, and sharp degradation on higher-order montage, while aesthetic quality is only weakly correlated with editing-technique compliance. The benchmark and metrics, and the editing agent baseline are available at https://github.com/AlibabaResearch/cut-craft-bench.
1 INTRODUCTION
Multi-shot video generation is becoming coherent and cinematic, but existing evaluation misses whether systems execute the editing decisions that create structure, rhythm, and meaning. CutCraft addresses this gap with explicit editing specifications, hierarchical evaluation, and an agentic generation baseline, revealing a persistent separation between coherence and editorial control.
- Professional editing gives sequences structure and meaning through shot arrangement, cut timing, audio-video relations, continuity, and montage.
- Existing evaluations largely measure plausible content while overlooking unstable shot layouts, montage logic, fine-grained transitions, and controlled J-cuts or L-cuts.
- CutCraft extends structured multi-shot prompts with explicit shot structure, transition type, audio-video cut relation, transition timing, and montage specifications.
- Its hierarchical hybrid framework resolves shot mismatches and combines expert-model measurements, tool-grounded multimodal judgment, and rubric-based question answering.
- The agentic baseline decomposes generation into planning, shot-level synthesis, and post-hoc composition to realize optical transitions, J-cuts, L-cuts, and audio remixing.
- Across 13 state-of-the-art systems, visually plausible and coherent outputs remain unreliable at following editorial instructions, while aesthetics correlate only weakly with editing-technique compliance.
2 RELATED WORK
Existing benchmarks assess video quality, prompt alignment, cinematic understanding, or editing reasoning, but CutCraft targets execution of editing techniques in generated multi-shot audio-video.
- General video and audio-video benchmarks primarily evaluate perceptual quality, prompt alignment, compositionality, semantic control, speech, or physical reasoning.
- Editing-oriented benchmarks mainly assess cinematic literacy, editing understanding, workflows, creation-and-editing reasoning, story consistency, or narrative structure in existing videos.
- CutCraft differs by evaluating whether generative models execute editing techniques in synthesized multi-shot audio-video outputs.
3 CUTCRAFT
CutCraft combines expert-curated structured data with editing-aware labels and a hierarchical evaluation pipeline designed to measure professional editing execution across narrative, rhythm, montage, transition, cinematography, and audio.
- Data construction: Each sample receives structured prompts, expert questions, montage labels, and shot-level event-coherence labels for evaluating intended editing structure.
- Data construction: Expert review filters 1,500 candidate meta prompts to 295 final structured samples, alongside 1,770 core questions and 933 transition-related survey questions.
- Data analysis: The dataset covers montage subtypes, diverse content and styles, shot counts, and broad video-, transition-, and shot-level editing attributes.
- Evaluation metrics design: The metric suite organizes professional editing into six groups, including narrative execution measures for shot counts, event alignment, causal coherence, and style consistency.
- Evaluation metrics design: Generated videos are first aligned to prompt-defined shot structures before hybrid scoring through direct expert models, VLM arbitration, and VLM judgment.
- Agentic editing baseline: The agentic baseline plans shot-level tasks, preserves shared identity and setting constraints, and composes independently generated shots with explicit transition and audio-offset operations.
4 EXPERIMENTS
Experiments across closed-source, open-source, and agentic settings show that models are stronger on coherence than on editing-specific execution. Discontinuous montage, increasing shot complexity, and visual quality expose distinct limitations, while agentic decomposition improves some discrete controls.
- Across closed-source, open-source, and agentic settings, models remain substantially stronger on coherence-related dimensions than on editing-specific dimensions.
- Discontinuous event chains produce lower montage execution scores than continuous event chains across all models.
- Transition performance generally degrades as the number of shots increases, indicating weaker control in more complex multi-shot structures.
- Agentic generation substantially improves shot structure, pacing control, and optical transition execution through explicit planning and post-hoc composition.
- Visual quality is only weakly and statistically insignificantly correlated with editing-related dimensions, so aesthetic quality is a poor proxy for editing execution.
- The evaluation uses expert judgments from eleven film and video editing experts over 640 samples to validate six editing-aware metrics and compares automated MLLM evaluators.
5 CONCLUSION
The paper constructs CutCraft from curated montage categories, video-content and visual-style attributes, and explicitly constrained multi-shot prompt drafts. Its evaluation materials include structured generation requirements and cross-shot physical-consistency questions.
- Prompt construction: The dataset construction expands curated meta combinations into 15-second drafts with exact shot counts, temporal coverage, continuous per-shot scenes, and plausible audio-video events.The drafting instructions require one continuous take per shot and prohibit internal cuts or interleaving within a shot description.
- Prompt construction: Structured prompt generation enforces shot-level formatting, tiled time ranges, substantial descriptions, and one unbroken camera scene per shot.The requirements also specify that the final shot must end at 15.00 seconds and that shot-description arrays match the requested shot count.
- Evaluation questions: The question bank evaluates cross-shot physical consistency by asking whether phenomena such as appearance, lighting, motion, or spatial relations satisfy physical laws.The E3 protocol uses a binary Yes/No response and requires the phenomenon to involve at least two shots.
A.2 DATA ANALYSIS DETAILS
The dataset analysis characterizes CutCraft across video, transition, and shot levels, while the evaluation pipeline aligns generated shots before scoring narrative, pacing, montage, transition, and plausibility dimensions.
- Dataset analysis: CutCraft analyzes the dataset at video, transition, and shot levels.The supplied analysis covers dataset composition and the metric-processing pipeline across these levels.
- Dataset analysis: The dataset contains 295 videos, 933 transitions, and 1,228 shots spanning montage forms, transition logic, framing, camera motion, and optical motion.Narrative, expressive, and intellectual montage account for 43.1%, 35.9%, and 21.0% of videos, respectively.
- Transition-level analysis: At the transition level, J-Cut accounts for 40.3%, L-Cut for 34.4%, and Straight Cut for 25.3% of audio-video relations.Dissolve is the most common optical effect at 38.0%, followed by Hard Cut at 32.3%.
- Evaluation pipeline: The evaluation first detects raw shot boundaries and aligns generated segments with prompt-defined shots before computing metrics.TransNetV2 supplies raw boundaries, followed by VLM-based shot alignment.
- Narrative and montage metrics: A2 scores event-order correctness, shot presence, and event fidelity, while A3 evaluates editing logic and counterfactual sensitivity to shot-order reversal.A2 uses complete, partial, failed, and missing-shot outcomes; A3 combines editing-logic judgment with original-versus-reversed-order plausibility.
- Pacing, style, and plausibility metrics: B1 measures shot-duration accuracy without additionally penalizing missing shots, while A4 combines target-style matching with cross-shot style consistency.The framework also includes rhythm and pacing, signal-based audio-video relation arbitration, and a binary physical-consistency dimension.
B.2.1 SHOTS PRE-PROCESS ABLATION AND HUMAN ALIGNMENT
The shot-preprocessing studies test whether automated shot alignment agrees with human judgments and whether alignment is necessary for downstream evaluation. The results support alignment as a required preprocessing step.
- Human alignment: Automated shot alignment is compared against human evaluation using precision, recall, and F1 scores.The study uses a questionnaire in which annotators select shot-to-plot matching relationships.
- Human alignment: The reported results show that the automated shot-alignment procedure is highly consistent with human judgment.Figure 7 presents the alignment questionnaire used for comparison.
- Alignment ablation: Ablation results demonstrate the necessity of shot alignment for subsequent evaluation.Without alignment, fragmented or missing shots can misalign event chains and affect event-chain, montage, and transition metrics.
B.2.2 METRIC-WISE HUMAN ALIGNMENT
The metric-wise human-alignment study uses professionally experienced annotators to assess core editing-related dimensions and examines agreement across those judgments. Agreement exceeds 70% for every evaluated dimension.
- Study design: Human annotators with film-editing or related audiovisual-production experience evaluate the core editing-related metrics.All annotators have at least three years of practical experience in editing, post-production, or cinematographic work.
- Annotation protocol: The human protocol evaluates shot count, event-chain alignment, and other editing dimensions using procedures consistent with the automated evaluation.For A1, annotators count visually perceived shot changes; for A2, they select the matching event chain and assess event execution.
- Agreement results: Human annotator agreement exceeds 70% across all evaluated dimensions.C1 montage type and D1 cinematographic transition type have relatively lower agreement but still reach majority consensus.
B.2.3 TESTING WITH POSITIVE AND NEGATIVE SAMPLES
The validation tests use displaced prompt-video pairs and manually constructed positive and negative transition samples to examine metric sensitivity and transition recognition. The reported results show substantial score drops under misalignment and accurate detection of optical and audio-video transition changes.
- Sample displacement test: The sample-displacement test shifts Seedance 2.0 videos by 30 samples before reevaluation.This tests whether metrics respond to mismatches between videos and their corresponding prompts.
- Sample displacement test: Misalignment causes substantial score drops in dimensions tied to shot prompts and transition localization.Affected dimensions include A1, A2, A3, A4, B1, C1, D1, D2, D3, and E3, while prompt-independent dimensions are less affected.
- Transition metric tests: Manual positive and negative samples test transition-related metrics for optical effects and audio-video track overlap.The tests construct clips using editing-software operations at shot junctions.
- Optical transition tests: The D2 metric accurately identifies hard cuts, dissolves, flashes, and wipes and determines transition timing from content changes near the cut.The optical-transition tests are reported in Figure 8.
- Audio-video relation tests: The D3 metric accurately identifies changes in audio-video relationships created by adjusting track overlap across transitions.The audio-video relation tests are reported in Figure 9.
B.2.4 METRIC-WISE SENSITIVITY ANALYSIS
The sensitivity analysis tests alternative scoring parameters for event execution and causal-chain metrics. Although these settings change absolute scores, they have little effect on model rankings within each dimension.
- The analysis varies A2 event-execution scores and the A3 λ0 parameter across alternative settings.The current A2 scheme is [1.0, 0.5, 0.1, 0], while alternatives are [1.0, 0.6, 0.2, 0] and [1.0, 0.7, 0.4, 0.1]; A3 currently uses λ0=0.3.
- Different parameter settings change absolute scores but have little effect on relative model rankings within each dimension.
- Increasing intermediate A2 execution scores raises all models’ overall scores while largely preserving their ranking.
B.2.5 OVERALL SCORE AGGREGATION STRATEGY
The paper compares aggregation strategies for combining editing and supporting-quality metrics, while also describing an agentic baseline that operationalizes editing plans through shot-level synthesis and composition. Across aggregations, transition execution and fine-grained editability remain difficult.
- Aggregation strategy: The benchmark aggregates editing and general video-quality dimensions using both “Six Groups” and “Two Groups” strategies.The main text uses a single aggregate score, while these strategies expose capabilities at different levels.
- Six Groups: Minimax H3, Seedance2.5, Seedance2.0, Happyhorse1.1, and Kling V3 perform consistently strongly across most groups.Minimax H3 leads group A, while Seedance2.0 is slightly stronger in group D, indicating complementary strengths among leading models.
- Six Groups: Group D is the most challenging for nearly all models, indicating that transition execution remains difficult.
- Two Groups: Many mid- and lower-tier models score much higher on Supporting than Editability, showing that general video quality is easier to achieve than fine-grained editing control.Top models have editability scores around 0.60 and similarly strong supporting scores; Seedance 2.5 leads Supporting but not Editability.
- Agentic editing baseline: The agentic baseline segments prompts into shots, plans each shot independently, and composes explicit transitions and audio-video relations post hoc.Its planner specifies prompts, generation modes, references, durations, and editing headroom; composition maps labels to operations such as cuts, dissolves, wipes, and flashes.
D.2 RESULT ANALYSIS
Bootstrap analysis shows that CutCraft reliably separates most models, although the strongest systems form a competitive cluster that is harder to rank with large statistical margins.
- 71 out of 78 pairwise comparisons are statistically significant after Holm-Bonferroni correction.The score distributions of models from different performance tiers are clearly separated.
- CutCraft is strongly discriminative at the benchmark level, while top models are easier to group into a leading cluster than to rank with large statistical margins.
D.3 MORE CASES
Additional cases expose recurring failures in shot structure, event and causal execution, montage, transitions, audio-video relations, and physical consistency. The examples also show that specialized evaluation models support selected montage and transition assessments.
- Shot-count failure: Shot-count failures mainly involve fragmented shots or omission of later shots relative to the ground truth.Examples include seven total shots caused by frame-skipping fragmentation and missing or split shots in six-shot cases.
- Event execution failure: Complex event instructions are often incompletely executed, with only some models preserving detailed actions, scene elements, and camera movement.
- Causal coherence failure: Causal-coherence failures occur when models lose steps from a previous shot, breaking the logical chain of a continuous process.This dimension focuses on event logic rather than physical or other visual errors.
- Montage type failure: Psychological montage is difficult because non-sequential editing requires features outside the main event chain, and only selected models realize it successfully.Happyhorse1.1 and Minimax H3 successfully realize the psychological montage example.
- Transition failure: Transition failures include missed wipe-by execution and collapse of varied optical transitions into hard cuts.The wipe-by example is accurately generated only by Seedance2.0, while complex six-shot prompts often lose other transition types.
- Audio-video relation failure: Nearly all models fail the first audio-video transition, where sound continuity must span shots.
- Physical consistency failure: Rubik’s Cube cases reveal widespread artifacts, distortions, and abrupt shape changes, with Kling V3 preserving structure relatively well but not common-sense colors.
- Open evaluator: The open evaluator uses task-specific experts distilled into Qwen3-VL-8B for montage, cinematographic-transition, and audio-video-relation classification.The evaluated tasks derive from the human annotation protocol and include evidence from expert models for selected pathways.
F LIMITATIONS
CutCraft’s high-fidelity evaluation is costly and may accumulate errors across its multi-stage pipeline. Some metrics also depend on VLM judge quality, while the editing agent remains an illustrative baseline rather than a SOTA solution.
- Evaluation pipeline: Evaluation requires over 120s per sample on average, and its multi-stage design may introduce cumulative error.The pipeline includes shot alignment, expert-model extraction, and editing-aware multimodal judgment.
- Evaluation pipeline: VLM-based metrics remain sensitive to judge-model quality, especially for montage, causal editing logic, and audio-video transition relations.The authors report human-correlation studies and train an OPD-based open evaluator to approximate a stronger closed-source judge.
- Agentic baseline: The agentic editing system is only a baseline illustrating one optimization direction, not a SOTA generation solution.Improving editing-aware generation strategies remains future work.