PaperlayerView exact revision on arXiv

Source-linked AI summary

Thinking on Shots: Consistent Multi-Shot Video Editing with Agentic Reasoning

Chenyang Wu, Fuchen Long, Binyuan Huang, Xinlong Sun, Xi Chen, Chun-Le Guo, Chongyi Li

arXiv:2608.26809v1cs.CVcs.MM

TL;DR

Long-video editing with heterogeneous instructions remains underexplored, while existing methods largely target short, single-shot clips. This paper proposes MMLVE-Agent and MMLVE-Bench, and reports that its framework significantly outperforms existing state-of-the-art models.

  • Problem

    Editing long videos with multiple heterogeneous instructions remains largely underexplored, while existing methods predominantly focus on short, single-shot clips.

    Source: Introduction · p. 1

  • Method

    MMLVE-Agent combines LLMs and VLMs for shot-level video decoupling and instruction parsing, with retrieval-based editing and a Global Memory Card for entity consistency.

    Source: Introduction · p. 2

  • Results

    The authors report that MMLVE-Agent significantly outperforms existing state-of-the-art models in experiments.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    MMLVE-Bench and its task-specific metrics provide a standardized testbed for evaluating multi-shot video editing.

    Source: Introduction · p. 2

Abstract

from arXiv · show

While generative AI has significantly advanced video editing, existing methods primarily focus on single-shot or short video clips. Editing long videos with multiple instructions remains a formidable challenge. Naive chunking strategies, e.g., fixed-duration segmentation, often lead to entity fragmentation, severe editing hallucinations, and disrupted temporal continuity. To bridge this gap, we introduce the Multi-Instruction Multi-Shot Long-Video Editing (MMLVE) task, which is structured around three core objectives: Cross-Shot Editing Consistency (CSEC), Multi-Instruction Decoupling (MID), and Zero-Destruction on Spatiotemporal Structure (ZDSS). To tackle these three unique challenges, we introduce an agentic editing framework that leverages the synergy of Large Language Models (LLMs) and Vision-Language Models (VLMs) to achieve shot-level video decoupling and precise instruction parsing. Furthermore, to comprehensively evaluate this task, we construct MMLVE-Bench, which is an MMLVE-focused dataset characterized by complex real-world spatiotemporal dynamics, high-density heterogeneous instructions, and sparse, random entity distributions. Three MMLVE-focused evaluation metrics are further exploited to assess the quality of the editing results. Extensive experiments demonstrate that our MMLVE-Agent outperforms existing closed-source SOTA approaches (e.g., Seedance 2.0), successfully eliminating editing hallucinations, preserving cross-shot editing consistency, and attaining seamless spatiotemporal transitions.

Introduction

The paper formalizes multi-instruction, multi-shot long-video editing around consistency, instruction decoupling, and preservation of spatiotemporal structure. It proposes MMLVE-Agent and introduces MMLVE-Bench with metrics tailored to those objectives.

Motivation

  • Prior long-video editing remains underexplored: existing successes mainly address clips under 15 seconds with simple instructions, unlike complex real-world videos with shot transitions and sparse entities (Song et al. 2026).Source: Introduction · p. 1
  • Fixed-duration chunking applies all instructions to every segment, causing entity hallucinations, inconsistent identities across shots, and flickering or abrupt transitions (Seedance et al. 2026; Team et al. 2023).An instruction for a dog appearing only in the final 10 seconds can cause the model to hallucinate a dog in an earlier chunk.Source: Introduction · pp. 1–2

Task definition

  • MMLVE defines CSEC, MID, and ZDSS: preserving edited-entity appearance across shots, executing commands independently, and protecting backgrounds, camera movement, and temporal continuity.Figure 1 summarizes the three constraints as cross-shot consistency, independent instruction execution, and preservation of non-edited backgrounds and temporal continuity.Source: Figure 1 · p. 1, Introduction · p. 2

MMLVE-Agent

  • MMLVE-Agent combines shot detection and LLM instruction parsing with VLM-voted, retrieval-based editing only in shots containing the target entity.A Global Memory Card provides a reference-to-edited visual anchor across shots; image- and video-level VLM quality checks drive closed-loop Pos-Neg Editing Feedback to refine corrective prompts.Source: Introduction · p. 2

Benchmark and evaluation

  • MMLVE-Bench comprises long videos with complex camera movements, heterogeneous ADD, DELETE, and MODIFY instructions, and randomly distributed entities, alongside three metrics for CSEC, MID, and ZDSS.The authors report that MMLVE-Agent significantly outperforms existing state-of-the-art models in experiments.Source: Introduction · p. 2

Related Work

Prior work addresses video understanding, short-clip editing, and multi-shot generation, but leaves complex long-video editing with heterogeneous instructions unresolved. MMLVE-Agent responds with shot-level analysis, a shared visual anchor, and feedback-driven editing.

Multi-Agent System

  • MMLVE-Agent introduces a heterogeneous multi-agent architecture for decoupling complex editing instructions, with Pos-Neg Editing Feedback (P-NEF) for autonomous self-correction and monotonic visual-generation improvement.Its LLM and VLM agents handle instruction parsing and video analysis, while feedback refines the editing process.Source: Related Work · p. 2, Figure 2 · p. 3

Long-Video Editing

  • Prior long-video methods use sliding windows (Li et al. 2025) or latent interpolation (Ge et al. 2022; Team et al. 2025), but blind chunking can cause editing hallucinations and temporal disruption.The paper shifts to “Thinking on Shots” for precise temporal localization and Zero-Destruction on Spatiotemporal Structure (ZDSS).Source: Related Work · p. 2

Multi-Shot Video Editing

  • Multi-shot generation methods such as StoryDiffusion (Zhou et al. 2024) and Soap2soap (Song et al. 2026) target consistent characters in new scenes, not editing existing complex videos.Existing multi-shot editing must preserve entity identity across transitions while executing diverse instructions independently.Source: Related Work · p. 2
  • A Global Memory Card provides a unified visual anchor for Cross-Shot Editing Consistency (CSEC) and Multi-Instruction Decoupling (MID) across physical shots.The framework retrieves target entities through VLM voting, iteratively refines the card with image-level P-NEF, then uses retrieval-based on-demand editing and video-level feedback to support MID and ZDSS before combining shots.Source: Related Work · p. 2, Figure 2 · p. 3

Methodology

MMLVE-Agent defines editing around cross-shot consistency, instruction decoupling, and preservation of spatiotemporal structure, and combines agentic video editing with a benchmark for evaluating these challenges.

Task Definition of MMLVE

  • MMLVE requires consistent edited entities across shots, independent instruction-to-entity and temporal mapping without hallucinating absent entities, and localized edits that preserve other regions and motion.The task addresses long videos with non-overlapping shots and entities that may appear only in sparse, random subsets of shots.Source: Task Definition of MMLVE · p. 3

Overall Architecture of MMLVE-Agent

  • MMLVE-Agent analyzes shots and instructions, builds global entity memory cards with positive-negative feedback, edits only detected entities, and checks edits while preserving untouched shots.A shot is edited only after at least 2 of 3 entity-presence votes; video-level feedback checks task execution and preservation of non-edited regions. The workflow concatenates edited shots with skipped shots, which remain unchanged.Source: Overall Architecture of MMLVE-Agent · pp. 4–5, Figure 3 · p. 4

MMLVE-Bench

  • MMLVE-Bench contains 25 approximately one-minute multi-shot clips, with about 5 manually verified editing instructions per video and complex scenes with dense, temporally sparse targets.The benchmark is derived from UniVA-Bench and uses VLM-generated candidate prompts followed by human correction; its scene dynamics and camera movements probe structural preservation, while heterogeneous instructions probe decoupling and localization.Source: MMLVE-Bench · p. 5

Experiments

MMLVE-Agent achieves the highest reported average score on MMLVE-Bench and, in qualitative comparisons, avoids the identity loss, instruction interference, and structural disruption observed in baseline edits.

Implementation Details

  • The MMLVE-Agent combines Gemini 3.5 Flash for language, vision-language, and evaluation tasks, Nano Banana 2 for reference-image generation, and HappyHorse for localized video edits.HappyHorse edits are conditioned on the Global Memory Card and decoupled prompts.Source: Implementation Details · p. 5

Baselines

  • Because no framework explicitly targets MMLVE, the study compares against Seedance 2.0, Kling o3, and HappyHorse 1.0 using naive fixed-duration chunking.Each chunk receives the full prompt containing all instructions; HappyHorse 1.0 is also evaluated alone to assess the agentic workflow’s contribution.Source: Baselines · p. 5

Qualitative Evaluation

  • MMLVE-Agent maintains edited entity identity across shots, while Seedance 2.0 and Kling o3 forget the red hat and HappyHorse 1.0 morphs the entities.The authors attribute the agent’s consistency to the Global Memory Card, which provides a global visual anchor.Source: Qualitative Evaluation · p. 6, Figure 5 · p. 6
  • Applying all instructions to every chunk causes interference and attribute leakage in baselines, whereas MMLVE-Agent decouples instructions to avoid those errors.Examples include a person incorrectly turned into a green alien, white paper dyed blue, and unrequested sponges inserted by HappyHorse 1.0.Source: Qualitative Evaluation · pp. 6–7
  • Kling o3 and HappyHorse 1.0 disrupt backgrounds and scramble shot chronology, while MMLVE-Agent preserves unedited frames and operates on parsed physical shots.The qualitative comparison reports these baseline failures as violations of Zero-Destruction on Spatiotemporal Structure (ZDSS).Source: Figure 5 · p. 6, Qualitative Evaluation · p. 7

Quantitative Evaluation

  • 81.84 average, 84.80 CSEC, and 79.04 MID: MMLVE-Agent achieves the highest average score and significantly outperforms all baselines in CSEC and MID.Seedance 2.0 scores 82.25 on ZDSS, which the authors attribute to conservatively ignoring instructions; MMLVE-Agent’s negligible ZDSS drop accompanies more comprehensive edit execution.Source: Table 1 · p. 7, Quantitative Evaluation · p. 7
  • The evaluation scores CSEC, MID, and ZDSS using Gemini 3.5 Flash as a VLM judge, with five sub-dimensions scored 0–20 per indicator.The paper avoids frame-level CLIP scores because they do not capture complex instruction decoupling and long-term spatiotemporal consistency.Source: Quantitative Evaluation · p. 7

Ablation Study on P-NEF

  • The P-NEF ablation indicates that positive prompts anchor successful edits, while NEF-only constraint accumulation can divert attention and cause catastrophic forgetting.The passage reports that P-NEF balances attention, supports monotonic improvement, and reduces attempts needed for a robust edit.Source: Ablation Study on P-NEF · p. 7, Figure 6 · p. 7

Conclusions

The paper addresses editing real-world, multi-shot long videos driven by complex instructions. It formalizes MMLVE around three constraints and presents an agent, benchmark, and task-focused evaluation metrics.

Conclusions

  • MMLVE targets the underexplored challenge of editing real-world multi-shot long videos driven by complex instructions.Source: Conclusions · pp. 7–8
  • The paper formalizes MMLVE around Cross-Shot Editing Consistency (CSEC), Multi-Instruction Decoupling (MID), and Zero-Destruction on Spatiotemporal Structure (ZDSS).Source: Conclusions · p. 8
  • The paper proposes MMLVE-Agent, constructs MMLVE-Bench, and introduces MMLVE-focused evaluation metrics.Source: Conclusions · p. 8

User Study

An expert-reviewed user study assessed the perceptual quality of MMLVE videos; MMLVE-Agent achieved the strongest human rankings against three baselines.

Custom-Built Evaluation Platform

  • A custom web platform presented anonymized, randomized outputs side by side and enabled synchronized playback and scrubbing of the input and four results for fine-grained comparison.Adjustable playback speeds supported inspection of transient flickering and subtle attribute bleeding.Source: User Study · p. 10

Evaluation Criteria and Ranking Mechanism

  • Evaluators ranked four anonymized videos using execution accuracy, visual quality and naturalness, temporal consistency, and artifact reduction; rankings were logged for statistical aggregation.The platform was designed to complement automated VLM-based evaluation with expert assessment of perceptual quality.Source: User Study · p. 10

Participants and Protocol

  • Nine expert evaluators, experienced in video generation and visual assessment, each ranked five randomly assigned cases to manage the workload of complex multi-shot videos.The authors designed this controlled workload to help evaluators maintain focus while checking decoupled instructions and spatiotemporal dynamics.Source: User Study · p. 10

Results and Analysis

  • MMLVE-Agent was preferred over Seedance 2.0, Kling O3, and HappyHorse in 88.4%, 80.5%, and 93.3% of co-rated cases, respectively; each margin was significant (p < 0.001, Bonferroni-corrected).Seedance 2.0 and Kling O3 had similar average ranks (2.67 vs. 2.63) but different distributions: Seedance was rarely first (4.7%) and often second (48.8%), while Kling was more polarized (17.1% first; 19.5% last).Source: User Study · p. 10
  • Across 45 rankings, MMLVE-Agent ranked first in 75.6% and in the top two in 93.3%, with best average rank 1.36 versus baselines’ 2.63–3.16.Table 2 summarizes the collected rankings.Source: Table 2 · p. 10, User Study · p. 10

All Cases shown in MMLVE-Bench

The section showcases all 25 curated MMLVE-Bench cases through original-video frames paired with their editing prompts. The examples demonstrate the dataset’s high-density, heterogeneous instructions and sparse entity distributions across physical shots.

All Cases shown in MMLVE-Bench

Global Memory Card Show

The Global Memory Card provides a side-by-side visual anchor of target entities’ before-and-after states, helping MMLVE-Agent maintain cross-shot consistency and avoid attribute interference.

Global Memory Card Show

  • The Global Memory Card explicitly shows target entities’ exact before-and-after states as a global visual anchor.The passage identifies this as the role illustrated in Fig. 14.Source: Global Memory Card Show · p. 10
  • Conditioning the editor on this reference card maintains strict Cross-Shot Editing Consistency (CSEC) and prevents attribute interference.The passage reports this behavior even amid drastic camera movements, varying scales, and complex backgrounds.Source: Global Memory Card Show · p. 10
  • The card supports consistent editing across scenes with drastic camera movements, varying scales, and complex backgrounds.Source: Global Memory Card Show · p. 10

VLM-based Judge Evaluation Metrics

The section introduces a VLM-judge framework that evaluates multi-shot long-video editing across CSEC, MID, and ZDSS, then reports strong MMLVE-Agent scores alongside baseline failures on key dimensions.

VLM-based Judge Evaluation Metrics

  • The VLM-based judge addresses frame-level metrics’ inability to assess instruction decoupling and long-term spatiotemporal consistency, evaluating editing quality across CSEC, MID, and ZDSS.The authors describe its 15-dimension matrix as a comprehensive and interpretable quantitative assessment of multi-shot long-video editing.Source: VLM-based Judge Evaluation Metrics · pp. 10, 12
  • The pipeline aligns keyframes by segmenting original videos into 15-second intervals, identifying plot-transition timestamps, and jointly prompting the judge with original and edited keyframes plus the editing instruction.Missing generations are recorded and skipped; scores across 15 sub-dimensions and the main dimensions are exported for statistical analysis.Source: VLM-based Judge Evaluation Metrics · p. 11
  • CSEC assesses cross-shot entity unity through identity, attribute, color and texture, geometric alignment, and lighting and shadow consistency.Its criteria include identity preservation and attributes remaining present across shots with varying angles, scales, and lighting.Source: VLM-based Judge Evaluation Metrics · p. 11
  • MID assesses precise execution of dense instructions without interference, while ZDSS checks preservation of non-edited regions and chronological order.MID includes target accuracy, distractor isolation, hallucination suppression, localization, and conflict resolution; ZDSS includes background and non-target preservation, shot alignment, motion consistency, and artifact reduction.Source: VLM-based Judge Evaluation Metrics · pp. 11–12

Evaluation results for each method

  • MMLVE-Agent scores highly across almost all scenes and sub-dimensions, especially Identity Preservation and Target Execution Accuracy, while HappyHorse 1.0 and Kling o3 often fail on Distractor Isolation and Chronological Shot Alignment.Tab. 4 also marks Seedance 2.0 and Kling o3 cases where safety mechanisms rejected processing, denoted by “/”.Source: Evaluation results for each method · p. 12

Other Results

Figure 15’s qualitative comparisons show MMLVE-Agent completing heterogeneous edits in their intended shots, while Seedance 2.0, Kling o3, and HappyHorse 1.0 exhibit missed edits, hallucinations, or spatiotemporal disruption. The two cases illustrate instruction isolation and preservation of the original scenes.

  • In the first case, Seedance 2.0 misses several edits, Kling o3 moves entities into unrelated shots, and HappyHorse 1.0 deletes or overwrites scene content.MMLVE-Agent changes the clock, removes the man, and adds a cap to the skeleton in their intended temporal windows without background destruction.Source: Other Results · pp. 12–13
  • In the second case, MMLVE-Agent uses its Global Memory Card and retrieval-based on-demand editing to isolate the artisan, skeleton, and soldier edits to their respective shots.The baselines instead miss requested edits, erase original shots, or introduce entities and attributes into unrelated scenes.Source: Other Results · p. 13
  • Across both cases, MMLVE-Agent performs the requested edits in their correct shots, unlike baselines that miss edits, hallucinate entities, or disrupt scenes.The comparisons cover clock, person, skeleton, clothing, and soldier edits across multi-shot videos.Source: Other Results · pp. 12–13, Figure 15 · p. 17

Related papers

Loading 2608.26809v1…