PaperlayerView exact revision on arXiv

Source-linked AI summary

MoviEdit: Long-form Multi-Shot Video Editing Made Easy

Suhyeon Lee, Zhengang Li, Soo Ye Kim, Peihao Zhu, Jong Chul Ye, Chun-Hao Paul Huang, Duygu Ceylan, Yan Kang, Hyeonho Jeong

arXiv:2611.02737v1cs.CVcs.AIcs.LG

TL;DR

Long-form multi-shot editing must keep intended edits coherent across scene and appearance changes, while existing systems largely focus on short, single-shot videos. MoviEdit combines paired-data generation, cascaded bidirectional–autoregressive editing, and a dedicated benchmark; it reports results above open-source models and on par with strong proprietary systems across text- and image-guided editing.

  • Problem

    Existing editing methods largely target short, single-shot videos, while long-form multi-shot editing requires edits to remain coherent across cuts and changing scenes, viewpoints, motion, and appearance.

    Source: §1 · p. 2

  • Method

    MoviEdit combines scalable paired-data generation, a cascaded bidirectional–autoregressive editing architecture, and a benchmark for long-form multi-shot editing.

    Source: §1 · p. 2

  • Results

    MoviEdit substantially outperforms existing open-source models and performs on par with strong proprietary systems across text-guided and image-guided editing.

    Source: §1 · p. 2, §5 · p. 14

  • Takeaways & Limitations

    The framework brings supervision, modeling, and evaluation components to long-form editing beyond isolated clips.

    Source: §1 · p. 2

    Target localization can fail with visually similar subjects or partial target visibility, producing wrong, incomplete, or incorrect edits.

    Source: §4.2 · pp. 13–14

Abstract

from arXiv · show

Long-form multi-shot video editing requires an intended edit to remain coherent across abrupt changes in scene, viewpoint, and subject appearance, while processing substantially longer visual context than conventional single-shot editors. We introduce MoviEdit, an end-to-end framework for scalable long-form multi-shot video editing that jointly addresses the challenges of data, modeling, and evaluation. To provide cross-shot supervision data at scale, we develop automatic data-generation pipelines spanning synchronized multi-view footage, real-world multi-shot videos, and generated multi-shot content, yielding 87K paired editing videos without manual annotation. Architecturally, MoviEdit employs a cascaded bidirectional-autoregressive design: a lightweight bidirectional stage first establishes a globally coherent editing draft over a spatiotemporally compressed representation, which is then autoregressively propagated and refined over the full video. We further introduce MoviEdit-Bench, a dedicated benchmark for systematic evaluation of long-form multi-shot video editing. Extensive experiments show that MoviEdit substantially outperforms existing open-source models and performs on par with industry frontier models across text-guided and image-guided editing. Visit our project page for complete results and to access the benchmark: https://multi-shot-edit.github.io/

1 Introduction

Long-form multi-shot editing must preserve an intended edit across cuts, but existing editors, data, and benchmarks largely target short, single-shot videos. MoviEdit pairs scalable supervision and a cascaded editing architecture with a dedicated benchmark, and reports performance above open-source systems and on par with proprietary systems.

1 Introduction

  • MoviEdit substantially outperforms existing open-source solutions and performs on par with strong proprietary systems across text-guided and image-guided editing.The framework is evaluated on both editing tasks.Source: §1 · p. 2
  • The framework combines multi-view-aware image editing, single-shot video editing, and VLM-based quality filtering to create paired examples with explicit cross-shot subject correspondence.Its contributions also include a paired-data generation pipeline, an editing architecture, and an evaluation suite.Source: §1 · p. 2
  • A lightweight bidirectional stage drafts a globally coherent edit before a causal stage autoregressively propagates and refines it across the sequence.The decomposition avoids jointly processing the complete source and target videos while preserving temporal and appearance consistency across shots.Source: §1 · p. 2
  • MoviEdit-Bench evaluates local subject-specific edits and global scene-level transformations, including cross-shot target localization and edit consistency.The benchmark includes a curated collection of copyright-free multi-shot videos.Source: §1 · p. 2

2 Related Work

Prior video-editing work has shifted from inference-time manipulation of pretrained priors toward paired-data learning, while long-form generation research explores efficient ways to manage context. MoviEdit situates its work around cross-shot supervision, long-context modeling, and dedicated evaluation for editing.

Diffusion-based Video Editing

  • Earlier diffusion-based editors use inversion, feature or attention manipulation, visual conditioning, and per-video optimization, but rely on pretrained priors rather than learning generalizable editing behavior from data.The passage notes that this reliance can limit robustness, scalability, and flexibility while adding inference overhead.Source: §2 · p. 3
  • Data-driven editors learn from paired source–target videos and vary in source conditioning and edit specification; MoviEdit supports edited reference frames from arbitrary video positions.Prior reference-image-guided methods typically condition on an edited first frame (Ouyang et al., 2024; Ku et al., 2024; Liu et al., 2025).Source: §2 · p. 3

Related Work

  • MoviEdit distinguishes its focus from short, single-shot editing through cross-shot supervision, long-context modeling, and dedicated evaluation.Figure 2 depicts annotation-free paired-data pipelines, including pixel-aligned spatial–temporal decomposition and semi-pixel-aligned coupled generation.Source: §2 · p. 3, Figure 2 · p. 4

Long-Form and Multi-Shot Video Synthesis

3 MoviEdit: Long-Form Multi-Shot Video Editing

MoviEdit combines scalable paired-data generation, a cascaded long-form editing architecture, inference optimizations, and a benchmark designed to evaluate cross-shot editing. Its data pipelines pair source and target videos across pixel-aligned, localization, and semi-pixel-aligned editing tasks.

3.1 Scalable Paired Data Generation for Multi-Shot Video Editing

  • Three data sources—synchronized multi-camera footage, real-world cinematic clips, and generated multi-shot videos—support pixel-aligned editing, subject localization, and semi-pixel-aligned editing.For pixel-aligned tasks, shot grouping or camera-view selection yields multi-shot clips; first frames are edited jointly and then propagated within each shot.Source: §3.1 · pp. 4, 6
  • Quality checks assess cross-shot and within-shot consistency, and reject 17%/38% of multi-camera pairs and 32%/47% of cinematic pairs at two stages, respectively.For subject localization, generated pairs are filtered for consistent localization and correct effect application, rejecting 24%.Source: §3.1 · p. 6
  • Subject localization tracks queried subjects across shots, creates dense per-frame masks, and applies effects such as highlighting, outlining, or color filling.The pipeline uses VLM filtering to check localization consistency and effect application.Source: Figure 3 · p. 5, §3.1 · p. 6
  • Semi-pixel-aligned generation preserves spatial structure, temporal composition, and narrative correspondence while allowing broader edits, including environment, medium, and character-appearance changes.Source and target share per-shot captions and initial noise; task-specific conditioning includes a shared denoising trajectory before branching for character-appearance change, and ControlNet structure guidance for environment and medium changes.Source: §3.1 · p. 6

3.2 Cascaded Architecture for Long-Form Video Editing

  • The Render Editor is trained with causal diffusion forcing, which predicts each chunk using itself and preceding chunks retained in the context window.The Draft Editor uses full bidirectional attention and a conventional flow-matching objective, while the Render Editor uses the causal diffusion forcing objective.Source: §3.2 · pp. 7–8
  • The Draft Editor uses bidirectional diffusion on a subsampled full sequence to establish a global edit, then the Render Editor applies causal diffusion to full-resolution chunks conditioned on that draft.The draft supplies global appearance anchors and cross-shot consistency; each rendered chunk also uses recent generated context and its corresponding source chunk. The stages share weights, differing in attention pattern and spatiotemporal resolution.Source: §3.2 · pp. 7–8

3.3 Inference Optimization

  • Inference optimization distills the Draft Editor to 12 denoising steps and the Render Editor to 4, and uses KV caching to avoid recomputing reusable tokens.The asymmetric schedule preserves capacity for global consistency while reducing the more computationally expensive rendering stage; reusable cached tokens include text, draft, source video, and prior chunks.Source: §3.3 · p. 8

3.4 MoviEdit-Bench: Evaluating Long-Form Multi-Shot Video Editing

  • MoviEdit-Bench targets cross-shot localization and consistency absent from short, single-shot benchmarks, with 64 videos and 128 scenarios spanning eight categories and text-guided and edited-frame-guided editing.Evaluation covers seven dimensions, including cross-shot consistency, and combines pixel-level metrics, embedding similarity, and VLM judgments; temporal metrics are computed within each shot.Source: §3.4 · pp. 8–9

4 Experiments

MoviEdit is evaluated on text-guided and edited-frame-guided multi-shot editing, with comparisons against public and proprietary systems. It leads overall on V2V and performs on par with Aleph 2.0 on IV2V, while also showing efficient inference, data-recipe benefits, and generalization beyond its training horizon.

4.1 Experimental Setup

  • Evaluation covers V2V and IV2V on MoviEdit-Bench, with IV2V models assessed on first-frame and, when supported, arbitrary-frame settings.Training uses 87K paired multi-shot videos and a two-stage V2V-then-IV2V schedule.Source: §4.1 · p. 10

4.2 Results

  • MoviEdit achieves the strongest overall V2V performance and highest overall IV2V success, while performing on par with Aleph 2.0 across individual IV2V dimensions.The radar plot highlights balanced performance across major benchmark criteria.Source: §4.2 · p. 12, Figure 8 · p. 12
  • Qualitative comparisons favor MoviEdit in source preservation, edit localization, and prompt alignment, while task-specific data improves localization and handling of larger structural changes.Public baselines show cross-shot artifacts and localization errors; HappyHorse and Gemini Omni also deviate from source timing and motion.Source: Figure 7 · p. 11, §4.2 · pp. 12–13, Figure 11 · p. 13
  • 11.8 s denoising latency is below the 15 s input-video duration, after optimizations deliver a 30.9× speedup over the full-attention baseline.Sliding-window attention and KV caching reduce latency before few-step distillation is added.Source: Table 4 · p. 12, §4.2 · p. 13
  • Increasing either training duration or dataset size consistently improves performance across metrics.The scaling study varies training from 0% to 100% of 17K steps and dataset size from 0% to 100% of 87K pairs.Source: Figure 10 · p. 12, §4.2 · p. 13
  • MoviEdit supports multi-reference editing and video set-to-set editing, and generalizes from training videos up to 15 seconds to 25-second inputs with as many as 17 shots.It also generalizes to long-take single-shot videos.Source: Figure 9 · p. 12, §4.2 · p. 13
  • Target localization can fail when visually similar subjects create identity ambiguity or when the intended target is only partially visible.These conditions can lead to edits applied to the wrong subject, or to incomplete or incorrect edits.Source: §4.2 · pp. 13–14, Figure 12 · p. 14

5 Conclusion

MoviEdit combines scalable paired-data generation, cascaded bidirectional–autoregressive modeling, and a dedicated benchmark for long-form multi-shot editing. It outperforms existing open-source models and matches strong proprietary systems across text- and image-guided editing, while generalizing to set-to-set editing and videos beyond its training horizon.

5 Conclusion

  • MoviEdit unifies paired multi-shot editing data generation, cascaded bidirectional–autoregressive modeling, and dedicated evaluation.The framework jointly addresses scalable supervision, efficient modeling, and evaluation.Source: §5 · p. 14
  • Across text- and image-guided editing, MoviEdit substantially outperforms existing open-source models and matches strong proprietary systems.Source: §5 · p. 14
  • MoviEdit generalizes to set-to-set editing and videos beyond the training horizon, but can struggle to localize targets under strong identity ambiguity or partial visibility.The paper focuses on correspondence-preserving edits and identifies free-form motion and narrative transformations as future work.Source: §5 · p. 14

Supplementary Material

The supplementary material adds details on training-dataset construction and MoviEdit-Bench, then provides additional external-benchmark and beyond-training-horizon results.

Supplementary Material

A Training Dataset Construction

The training dataset combines synchronized multi-camera, cinematic, and generated videos into paired multi-shot editing examples. The supplement also details MoviEdit-Bench source videos and scenarios, including manually reviewed edits and edited-frame references.

A Training Dataset Construction

  • The training data uses synchronized multi-camera footage, cinematic clips, and generative videos, yielding 86,619 paired multi-shot videos at 540p, approximately 15 seconds, and 24 fps.The sources comprise 1,147 synchronized multi-camera clips, 32,474 cinematic clips, and 27,630 generative videos.Source: A Training Dataset Construction · p. 19, Table 5 · p. 20

B.1 Source Multi-Shot Videos

  • MoviEdit-Bench contains 64 story-driven source videos spanning varied subjects, settings, poses, viewpoints, visibility, and framing.Its eight categories range from persistent single subjects to crowded scenes, non-photorealistic media, and global scene edits.Source: §B.1 · p. 19, Figure 13 · p. 19
  • Benchmark videos are generated from scratch with four to six shots and 361 frames, then manually reviewed for coherence, identity, correspondence, and editing suitability.Source: §B.1 · pp. 19–20

B.2 Editing Scenarios

  • The benchmark defines 128 scenarios, with local edits in 75% and global edits in 25%, and manually reviews instructions and edited-frame references.It evenly divides edited-frame-guided scenarios between first-frame and non-first-frame references.Source: §B.2 · p. 20

B.3 Evaluation Metadata

MoviEdit-Bench supplies evaluation metadata for local edits, shot-level metrics, and target-video conditioning. Its masks and descriptions are paired with source videos and checked for consistency.

B.3 Evaluation Metadata

  • Local-edit metadata includes per-frame target masks produced with video segmentation models, then manually verified and corrected.The masks are temporally aligned with the source video and paired with the edit instruction.Source: §B.3 · p. 20
  • Shot-boundary metadata records each detected shot’s start and end frame indices using OmniShotCut.These boundaries support shot-level metric computation.Source: §B.3 · p. 21
  • Each scenario includes a manually reviewed target-video prompt with a global scene description and per-shot descriptions.The prompt describes the fully edited target video as a self-contained description, rather than a relative change to the source prompt.Source: §B.3 · p. 21
  • Target-video prompts also support evaluating baselines that use target-video conditioning, including Wan-Edit and SDEdit.The passage identifies Wan-Edit and SDEdit (Meng et al., 2022).Source: §B.3 · p. 21

B.4 Evaluation Metrics

MoviEdit-Bench evaluates editing along complementary dimensions, using shot-aware protocols and both automated metrics and VLM judgments. The overall success score aggregates strict frame-level editing criteria.

B.4 Evaluation Metrics

  • The seven evaluation dimensions are non-subject preservation, edit localization, subject preservation, cross-shot edit consistency, edit-prompt alignment, video quality, and edited-frame fidelity.An overall success score jointly considers multiple editing criteria; model-level results average scenario scores where applicable.Source: §B.4 · p. 21, Table 8 · p. 23
  • Evaluation resizes videos and target masks to 960 × 544, aligns them to the 361-frame source, and evaluates shot-level computations independently within each shot.Shorter outputs are assessed over their overlapping range; mask-based metrics dilate target masks by 9 pixels and omit frames or shots where the target is not visible.Source: §B.4 · p. 21
  • The VLM protocol samples four frames per shot and judges edit presence, target correctness, preservation, and coverage against the source, edit instruction, and target description.Sampling includes first and last frames and shots where the target is absent, allowing detection of edits leaking to other subjects or shots.Source: §B.4 · pp. 21–22
  • Edit localization measures target coverage and confinement, while preservation metrics assess unchanged regions, target attributes, and motion.Localization uses VLM judgments for unintended changes and complete coverage; preservation includes masked LPIPS, target-region LPIPS, MFS tracking, and VLM judgments as applicable.Source: §B.4 · p. 22
  • Cross-shot consistency judges whether edits match across target-bearing shots, while edited-frame fidelity checks whether IV2V results retain the reference identity and attributes.Consistency is defined when at least two relevant shots are available and does not apply to removal edits; fidelity allows natural viewpoint, pose, lighting, and motion variation.Source: §B.4 · p. 22
  • Edit-prompt alignment compares shots with their target descriptions, and video quality evaluates coherence, image quality, prompt fit, and visual artifacts.CLIP and ViCLIP use eight sampled frames per shot; quality measures include DINO, NIQE, PickScore, and VLM ratings.Source: §B.4 · pp. 22–23
  • Overall edit success is the fraction of sampled frames meeting applicable criteria for edit presence, correct targeting, and preservation of unrelated content.Figure 14 illustrates a progression from incorrect localization or incomplete edits to successful cross-shot-consistent editing.Source: §B.4 · p. 23

C.1 External Benchmark Validation

MoviEdit is evaluated on FiVE-Bench and VPBench to test its performance in conventional short-form, single-shot editing. It is reported as comparable to or competitive with proprietary models across both benchmarks.

C.1 External Benchmark Validation

  • MoviEdit is evaluated on the FiVE-Bench and VPBench short-form, single-shot video-editing benchmarks.The reported evaluation covers text-guided video editing (V2V).Source: §C.1 · p. 24
  • Across both benchmarks, MoviEdit performs comparably to or competitively with proprietary models.The quantitative evaluation is summarized in Table 9.Source: §C.1 · p. 24
  • The results complement MoviEdit-Bench evaluation and indicate strong editing capabilities in conventional single-shot settings.Source: §C.1 · p. 24

C.2 Beyond-Training-Horizon Editing

MoviEdit is tested beyond its 15-second training horizon on 25-second videos with different temporal structures. The qualitative results show editing across a 17-shot sequence and an extended continuous shot.

C.2 Beyond-Training-Horizon Editing

  • 25-second tests extend beyond MoviEdit’s 15-second training horizon, covering both a 17-shot sequence and a continuous single-shot video.Figure 15 presents both qualitative examples.Source: §C.2 · p. 24, Figure 15 · p. 26
  • The 17-shot example demonstrates consistent editing across frequent transitions, beyond the training duration and shot count.Source: §C.2 · p. 24, Figure 15 · p. 26
  • The continuous single-shot example shows sustained editing consistency over an extended sequence.Source: §C.2 · p. 24, Figure 15 · p. 26
  • Figure 14’s cross-model examples progress from incorrect localization or incomplete edits to successful cross-shot-consistent editing.Source: Figure 14 · p. 25

Related papers

Loading 2611.02737v1…