Source-linked AI summary
Thinking on Shots: Consistent Multi-Shot Video Editing with Agentic Reasoning
Chenyang Wu, Fuchen Long, Binyuan Huang, Xinlong Sun, Xi Chen, Chun-Le Guo, Chongyi Li
TL;DR
Long-video editing with multiple heterogeneous instructions is underexplored, and naive chunking can cause hallucinations, inconsistency, and temporal disruption. The paper formalizes MMLVE, proposes MMLVE-Agent with LLM/VLM-driven shot reasoning and feedback, and constructs MMLVE-Bench with tailored metrics; experiments report superiority over existing state-of-the-art models.
Problem
Editing long, multi-shot videos with multiple heterogeneous instructions remains underexplored, while fixed-duration chunking can cause hallucinations, identity inconsistency, and disrupted spatiotemporal structure.
Method
MMLVE-Agent uses physical shot detection, LLM instruction decoupling, VLM analysis, retrieval-based editing, a Global Memory Card, and Pos-Neg Editing Feedback.
Results
MMLVE-Agent significantly outperforms existing state-of-the-art models on multi-shot video editing quality.
Takeaways & Limitations
MMLVE and MMLVE-Bench establish a task, dataset, and three metrics for evaluating cross-shot consistency, instruction decoupling, and spatiotemporal preservation.
Abstract
from arXiv · showhide
While generative AI has significantly advanced video editing, existing methods primarily focus on single-shot or short video clips. Editing long videos with multiple instructions remains a formidable challenge. Naive chunking strategies, e.g., fixed-duration segmentation, often lead to entity fragmentation, severe editing hallucinations, and disrupted temporal continuity. To bridge this gap, we introduce the Multi-Instruction Multi-Shot Long-Video Editing (MMLVE) task, which is structured around three core objectives: Cross-Shot Editing Consistency (CSEC), Multi-Instruction Decoupling (MID), and Zero-Destruction on Spatiotemporal Structure (ZDSS). To tackle these three unique challenges, we introduce an agentic editing framework that leverages the synergy of Large Language Models (LLMs) and Vision-Language Models (VLMs) to achieve shot-level video decoupling and precise instruction parsing. Furthermore, to comprehensively evaluate this task, we construct MMLVE-Bench, which is an MMLVE-focused dataset characterized by complex real-world spatiotemporal dynamics, high-density heterogeneous instructions, and sparse, random entity distributions. Three MMLVE-focused evaluation metrics are further exploited to assess the quality of the editing results. Extensive experiments demonstrate that our MMLVE-Agent outperforms existing closed-source SOTA approaches (e.g., Seedance 2.0), successfully eliminating editing hallucinations, preserving cross-shot editing consistency, and attaining seamless spatiotemporal transitions.
Introduction
MMLVE addresses the underexplored problem of editing long, multi-shot videos with heterogeneous instructions by enforcing consistency, instruction independence, and preservation of spatiotemporal structure. The paper proposes an agentic framework, dataset, and evaluation metrics for this setting.
- Motivation: Existing video editing methods mainly target short, single-shot clips, leaving complex long-video editing with multiple heterogeneous instructions underexplored.Real-world videos involve long durations, shot transitions, complex spatiotemporal dynamics, and sparse entity distributions.
- Motivation: Naive fixed-duration chunking can hallucinate sparse entities, break cross-shot identity consistency, and disrupt temporal continuity at chunk boundaries.Applying every instruction to every chunk may introduce entities where they do not appear and produce flickering or abrupt transitions.
- Task Definition: MMLVE defines three objectives: CSEC preserves entity appearance across shots, MID separates commands without interference, and ZDSS preserves backgrounds, camera motion, and temporal continuity.These objectives directly target the principal failure modes of chunking-based editing.
- Approach: MMLVE-Agent combines shot detection, LLM instruction parsing, VLM analysis, retrieval-based on-demand editing, a Global Memory Card, and closed-loop P-NEF feedback.The Global Memory Card provides a cross-shot visual anchor, while feedback supports instruction decoupling and spatiotemporal preservation.
- Benchmark and Contributions: MMLVE-Bench supplies long videos with complex dynamics, dense heterogeneous instructions, and random entity distributions, alongside three metrics targeting CSEC, MID, and ZDSS.The benchmark and metrics provide a standardized testbed for evaluating multi-shot editing quality.
Related Work
Prior agent research has explored LLM- and VLM-based autonomy, but complex generative video editing remains largely unexplored. MMLVE-Agent extends this direction with heterogeneous agents and Pos-Neg Editing Feedback for instruction decoupling and self-correction.
- Multi-Agent Systems: LLM- and VLM-based agents have advanced autonomous reasoning and tool orchestration, while VLM agents have demonstrated strength in video understanding.The paper positions these capabilities as foundations for its editing framework.
- Multi-Agent Systems: Complex generative video editing with VLM-based agents remains largely unexplored compared with video understanding.MMLVE-Agent addresses this gap through a heterogeneous multi-agent architecture.
- Multi-Agent Systems: MMLVE-Agent introduces Pos-Neg Editing Feedback as a closed-loop mechanism for autonomous self-correction during multi-shot editing.The framework applies feedback at image and video levels within its multi-module pipeline.
Methodology
MMLVE-Agent replaces blind chunking with shot-level analysis, instruction decoupling, global visual memory, and on-demand editing to address consistency, interference, and structure preservation.
- Framework Overview: MMLVE-Agent defines a three-module pipeline spanning Instruction & Video Analysis, Global Memory Card Making, and Multi-Shot Video Editing.The framework is designed around the MMLVE task and its core objectives.
- Instruction & Video Analysis: Physical shot detection and LLM parsing separate the video into shots and resolve complex heterogeneous user instructions.The analysis stage decouples spatiotemporal video dynamics from the instruction set before editing.
- Global Memory Card Making: P-NEF combines Negative Prompts that correct artifacts with Positive Prompts that preserve successful attributes and balance attention.Negative-only feedback can alter correctly edited regions, whereas the positive component acts as a semantic anchor during iterative refinement.
- Global Memory Card Making: The framework generates a validated before-and-after Global Memory Card and enriches the entity description for subsequent shot editing.A VLM evaluator checks generated references, retries failed generations up to Tmax = 3, and selects the best candidate if needed.
- Multi-Shot Video Editing: Retrieval-based editing uses three-time voting and edits a shot only after at least 2 positive votes; otherwise, the original shot is preserved.Edited shots are verified for task success and preservation of non-edited regions before concatenation with untouched shots.
- Benchmark Construction: MMLVE-Bench contains approximately one-minute multi-shot clips with complex camera movements, dense heterogeneous instructions, and diverse entity-scene distributions.The benchmark is manually curated from 25 high-quality clips and supports evaluation of CSEC, MID, and ZDSS through dedicated metrics.
Experiments
Experiments compare MMLVE-Agent with adapted closed-source baselines using qualitative, quantitative, and P-NEF ablation evaluations on MMLVE-Bench. MMLVE-Agent preserves consistency and structure while addressing instruction interference and hallucinations.
- Experimental Setup: MMLVE-Agent uses Gemini 3.5 Flash as the unified VLM backbone for its heterogeneous multi-agent architecture.The system assigns the model across LLM Agent, VLM Agent, and VLM QA Evaluator roles.
- Experimental Setup: Baselines are adapted through uniform fixed-duration chunking, with the full complex prompt applied independently to every short clip.The comparison includes Seedance 2.0, Kling o3, and HappyHorse 1.0.
- Qualitative Evaluation: Only MMLVE-Agent achieves CSEC, MID, and ZDSS simultaneously in the qualitative comparison.The three objectives assess cross-shot consistency, instruction separation, and preservation of spatiotemporal structure.
- Qualitative Evaluation: MMLVE-Agent preserves the red Christmas hat and entity identity across shots, whereas baselines forget attributes or morph entities.The Global Memory Card provides a global visual anchor for consistent edited entities.
- Qualitative Evaluation: Baselines exhibit instruction leakage, hallucinated entities, background destruction, and chronological scrambling, while MMLVE-Agent delivers coherent edits.These failures arise under simultaneous application of complex prompts to all chunks.
- Quantitative Evaluation: 81.84 is MMLVE-Agent’s highest average score, including 84.80 CSEC and 79.04 MID.Seedance 2.0 reaches 82.25 ZDSS through a conservative strategy that also produces missed edits.
- Ablation Study on P-NEF: P-NEF reinforces successful features with a positive prompt, balancing attention, preventing catastrophic forgetting, and reducing trial-and-error attempts.Negative-feedback-only editing accumulates avoidance constraints and can alter previously correct edits.
Conclusions
The paper formalizes MMLVE for complex multi-shot long-video editing and introduces MMLVE-Agent, MMLVE-Bench, and task-specific evaluation metrics.
- MMLVE addresses the underexplored challenge of editing real-world multi-shot long videos with complex instructions.
- The task is governed by CSEC, MID, and ZDSS, while the proposed framework, benchmark, and metrics target these constraints.
User Study
The user study uses an expert-review protocol with anonymized, randomized side-by-side comparisons of MMLVE-Agent and three baselines. Reviewers preferred MMLVE-Agent across all reported head-to-head comparisons.
- Evaluation Protocol: Evaluators compare anonymized videos in randomized order using the original video, prompt, and synchronized playback controls.
- Results: All three preference margins are statistically significant under a two-sided Wilcoxon signed-rank test with Bonferroni correction.The reported significance threshold is p < 0.001.
- Results: Seedance 2.0 and Kling O3 have similar average ranks but different distributions: Seedance is usually second, whereas Kling O3 is more polarized.
All Cases shown in MMLVE-Bench
MMLVE-Bench presents 25 curated multi-shot long-video cases pairing representative original-video frames with complex editing prompts.
- The cases contain high-density heterogeneous ADD, MODIFY, and DELETE instructions targeting multiple entities distributed sparsely across physical shots.
Global Memory Card Show
The Global Memory Card provides a side-by-side visual anchor showing target entities before and after editing. Conditioning the video editor on this reference helps MMLVE-Agent maintain cross-shot consistency and prevent attribute interference.
- The Global Memory Card explicitly shows target entities’ “before-and-after” states as a global visual anchor.
- Conditioning the video editor on the side-by-side reference helps maintain Cross-Shot Editing Consistency (CSEC).
- The mechanism prevents attribute interference during multi-shot editing.
VLM-based Judge Evaluation Metrics
The paper evaluates long-video editing with an automated VLM-based Judge across CSEC, MID, and ZDSS. Its 15-dimension matrix scores each sub-dimension from 0 to 20, while keyframe alignment supports scene-level comparison and results expose baseline failures.
- Traditional CLIP and PSNR metrics are inadequate for judging instruction decoupling and long-term spatiotemporal consistency in complex long videos.
- The automated VLM-based Judge evaluates generated videos across Cross-Shot Editing Consistency, Multi-Instruction Decoupling, and Zero-Destruction on Spatiotemporal Structure.
- The evaluation segments original videos into 15-second intervals, identifies critical transition timestamps with a VLM, and extracts corresponding keyframes.
- Each of the three main indicators is decomposed into five sub-dimensions, with scores of 0, 10, or 20 and a maximum of 100 points per indicator.
- Cross-Shot Editing Consistency (CSEC): CSEC measures identity, attributes, color and texture, structure and geometry, and lighting across physical shots.
- Multi-Instruction Decoupling (MID): MID measures target execution, distractor isolation, hallucination suppression, entity localization, and semantic conflict resolution.
- Zero-Destruction on Spatiotemporal Structure (ZDSS): ZDSS inspects background and non-target preservation, chronological shot alignment, intra-shot motion, and artifact or flicker reduction.
- The 15-dimension matrix provides a comprehensive and interpretable quantitative assessment of multi-shot long-video editing.
Other Results
Qualitative comparisons show that baseline methods struggle with the three core MMLVE challenges, including inconsistent edits, hallucinations, instruction interference, and spatiotemporal destruction. The benchmark includes diverse multi-instruction cases and detailed evaluations across competing methods.
- Baseline methods consistently struggle with Cross-Shot Editing Consistency, Multi-Instruction Decoupling, and Zero-Destruction on Spatiotemporal Structure in complex videos.
- Kling o3 exhibits clock-shape inconsistency, fails to remove the man, hallucinates the man into an unrelated shot, and inserts the clock into another shot.
- HappyHorse 1.0 shows catastrophic spatiotemporal destruction and instruction interference while removing the man.
- MMLVE-Bench contains cases involving heterogeneous instructions for changing colors, modifying clothing or objects, adding designs, and removing entities.
- The benchmark reports detailed evaluation results for MMLVE-Agent, HappyHorse 1.0, Kling o3, and Seedance 2.0.
- Seedance 2.0 and Kling o3 may reject processing for scenes because of their safety mechanisms.