PaperlayerView exact revision on arXiv

Source-linked AI summary

CoVEBench: Can Video Editing Models Handle Complex Instructions?

Jiangtao Wu, Jiaming Wang, Yiwen He, Yuanxing Zhang, Shihao Li, Dunyuan Liu, Xuedong Zhao, Jialu Chen, Zekun Moore Wang, Jiaheng Liu

arXiv:2606.08415v2cs.CVcs.AI

TL;DR

Existing video-editing benchmarks focus on isolated edits and coarse metrics, limiting assessment of realistic compositional workflows. CoVEBench introduces a fine-grained benchmark and evaluation framework, revealing that current models still struggle to complete multiple edits while preserving physical plausibility and unedited content.

  • Problem

    Existing benchmarks emphasize isolated edits and coarse global metrics, inadequately assessing interacting multi-point edits and preservation of unrelated scene content.

  • Method

    CoVEBench combines 416 source videos, 626 complex instructions, and 9,990 checklist items with automated and MLLM-based measures of compliance, quality, and fidelity.

  • Results

    Current models struggle with high-quality instruction completion, particularly executing all required edits while maintaining physical plausibility and content preservation.

  • Takeaways & Limitations

    CoVEBench exposes persistent weaknesses in compositional video editing, including edit omissions, physical violations, and unwanted background shifts.

  • Takeaways & Limitations

    CoVEBench is restricted primarily to text-guided instructions, excludes multimodal control signals, and provides neither a corresponding solution nor a large-scale paired training dataset.

Abstract

from arXiv · show

While recent text-guided video editing models excel at elementary tasks (e.g., style transfer, object insertion), real-world user requests are highly compositional. A single prompt often demands multiple coupled edits, such as modifying subjects, actions, and camera views, while strictly preserving unrelated spatiotemporal content. Existing benchmarks, heavily constrained by isolated edits and coarse global metrics, fail to diagnose how models handle such complex workflows. To address this gap, we introduce CoVEBench, a compositional video editing benchmark comprising 416 curated source videos, 626 multi-point editing instructions, and 9,990 fine-grained checklist items. Covering diverse editing dimensions, CoVEBench evaluates models via MLLM-judged instruction compliance and video fidelity, alongside automated metrics for video quality. Extensive experiments reveal that compositional editing remains a profound challenge: current models frequently omit edits, violate preservation constraints, or introduce artifacts when handling multiple operations simultaneously. CoVEBench provides a challenging, diagnostic testbed to advance video editing toward realistic user workflows.

1 Introduction

CoVEBench addresses the gap between rapidly advancing video editing models and evaluations centered on simple, isolated edits. It benchmarks realistic compositional workflows with fine-grained diagnostics for execution accuracy, modification realism, and semantic preservation.

  • Motivation: Current models have advanced in style transfer and object replacement, but existing benchmarks focus on simple isolated edits and lag behind their capabilities.
  • Evaluation gap: Coarse global metrics such as CLIP scores cannot assess interactions and interference among multiple editing operations or diagnose specific failures.
  • Benchmark: The benchmark covers subject, background, camera, style, motion, positional, and special-effects edits, decomposed into 9,990 verifiable checklist items.
  • Benchmark: CoVEBench comprises 416 curated source videos and 626 complex editing instructions averaging approximately 3 atomic edit operations each.
  • Findings: Evaluations across leading open-source and proprietary models reveal significant performance gaps against the demands of complex compositional editing.

2 Related Work

Video editing research has shifted from prompt-based adaptation toward instruction-driven systems, but existing evaluations still emphasize simple edits rather than realistic compositional workflows. Current benchmarks therefore provide limited coverage of coupled edits that must preserve unrelated source content.

  • Evolution of Video Editing Models: Early video editing methods added temporal mechanisms to pretrained text-to-image models, but per-video optimization and rigid source-target prompt pairs limited flexibility.Examples include cross-frame attention and optical-flow mechanisms for maintaining frame consistency.
  • Evolution of Video Editing Models: Recent methods use large-scale triplets and unified frameworks to support more open-ended instruction-driven video editing.This shift follows earlier prompt-based adaptation approaches.
  • Video Editing Benchmarks: Existing models are mostly evaluated on simple, isolated edits, leaving their ability to perform multiple coupled edits while preserving irrelevant source content insufficiently explored.Compositional instructions combine several editing operations and preservation constraints within one request.
  • Video Editing Benchmarks: Existing benchmarks focus on isolated edits, while compositional instructions are rare and generally limited to trivial attribute modifications.This creates a mismatch with realistic workflows requiring broader compositional editing capabilities.

3 CoVEBench

CoVEBench is a three-stage benchmark built from curated videos, compositional editing instructions, and refined fine-grained checklists. It covers diverse editing dimensions and evaluates instruction compliance, video quality, and preservation of unedited source content.

  • Dataset Statistics: 416 curated videos and 195,625 sampled instruction pairs yield low TF-IDF cosine similarity, with mean 0.0168 and median 0.0106.The videos have varied durations and resolutions, while token and keyword distributions indicate broad semantic richness.
  • Benchmark Construction: CoVEBench follows a three-stage pipeline covering source video collection and filtering, editing instruction generation, and checklist generation with refinement.
  • Benchmark Construction: The benchmark uses a hierarchical taxonomy spanning seven editing dimensions and 83 manually formulated category combinations designed to reflect realistic workflows.
  • Evaluation Framework: Evaluation spans Instruction Compliance, Video Quality, and Video Fidelity, respectively measuring edit adherence, visual quality and artifacts, and preservation of unedited content.The dimensions cover realism of executed edits, aesthetic and temporal properties, low-level quality, and source-content preservation.
  • Evaluation Framework: Fine-grained MLLM checklists use multiple-choice and yes/no questions to assess whether requested edits are executed and whether unnatural artifacts occur.Checklist-derived metrics distinguish instruction execution accuracy from editing realism through the Instruction Following Score and Video Realism Score.

4 Experiment

Experiments across 10 video editing models show that compositional instruction completion remains difficult, with trade-offs between edit execution and preservation. Additional analyses establish metric reliability, reveal declining performance under temporal and editing complexity, and show joint editing outperforming sequential decomposition.

  • Main Results: Closed-source models achieve stronger checklist-based instruction compliance, especially UAS, although this finding relies on only two proprietary models.The comparison covers 10 popular models, including two evaluated closed-source systems.
  • Main Results: Current models’ UAS remains substantially below their individual instruction-following and realism scores, indicating difficulty completing all edits while preserving physical plausibility.The union accuracy gap suggests that compositional completion is harder than satisfying isolated dimensions.
  • Main Results: Ditto combines competitive edit-execution scores with much weaker SEM, illustrating a trade-off between performing edits and preserving unrelated content.Stronger edits may introduce unintended changes to content that should remain preserved.
  • Metric Validity: Checklist score variations on OmniWeaving and Kiwi stay within 0.5 points across repeated evaluations, supporting high metric reliability.Human-alignment analysis additionally compares metric answers with human evaluators on 100 Wan and Kiwi cases.
  • Scalability and Complexity: Performance declines as generated sequences, source videos, edit-point counts, or instruction lengths increase, showing that temporal and compositional complexity remains challenging.Temporal scalability is evaluated on InsV2V, Kiwi, and Lucy using 300 sampled instances per model; editing complexity is evaluated across the entire dataset.
  • Editing Strategy: Joint editing outperforms sequential decomposition in UAS (30.63% vs. 23.70%), IFS (56.56% vs. 48.68%), and VRS (56.48% vs. 51.62%).Sequential editing degrades because new edits can overwrite previous modifications and insufficiently preserve content.

5 Conclusion

CoVEBench is a rigorous framework for evaluating compositional video editing through automated metrics and nearly 10,000 fine-grained MLLM checklists. Its assessment exposes current models’ weaknesses in edit execution, physical validity, and content preservation.

  • CoVEBench probes the limits of compositional video editing with a rigorous evaluation framework.It assesses execution accuracy, visual quality, and content preservation.
  • Nearly 10,000 fine-grained MLLM checklists and automated metrics enable holistic assessment.
  • Current models exhibit edit omissions, physical violations, and unwanted background shifts.These weaknesses reveal fundamental bottlenecks in their internal logic.

Limitations · Appendix · A The Practical Need for Compositional Video Editing

CoVEBench addresses the practical need for compositional video editing, where multiple edits must be performed within one clip while unrelated content is preserved. Its scope remains limited because it currently supports primarily text-guided instructions and excludes multimodal control signals needed for precise spatial-temporal alignment.

  • Limitations: The current benchmark scope is primarily restricted to text-guided instructions.This limitation leaves more advanced real-world workflows outside the current framework.
  • Limitations: Reference images, spatial bounding boxes, and audio cues are not accommodated, despite supporting precise spatial-temporal alignment in advanced workflows.These are examples of supplementary multimodal control signals used by creators.
  • A The Practical Need for Compositional Video Editing: CoVEBench targets editing scenarios involving multiple objects, attributes, regions, and stages within the same clip.These scenarios require preserving everything else faithfully.
  • A The Practical Need for Compositional Video Editing: Single-edit benchmarks cannot expose these practical compositional requirements.CoVEBench is positioned as a benchmark for the complex scenarios underlying these applications.
  • A The Practical Need for Compositional Video Editing: Short-form creators may chain background replacement, wardrobe restyling, and visual-effect insertion while preserving a subject’s identity, expression, and motion trajectory.Partial preservation failures can make the entire output unusable.
  • A The Practical Need for Compositional Video Editing: Film and TV post-production likewise require handling multiple edits within a single shot rather than applying only one isolated operation.The passage introduces this application as another setting where compositional editing matters.
  • A The Practical Need for Compositional Video Editing: Every CoVEBench task, prompt, and metric is constructed to stress-test models under multifaceted editing conditions.The benchmark contrasts these conditions with simplified single-edit protocols prevalent in prior benchmarks.

B Comparison with other Benchmarks

CoVEBench is designed specifically for complex compositional video editing, unlike broad unified benchmarks such as UniVBench. It offers targeted, fine-grained, diagnostic evaluation through multi-point instructions and verifiable checklist-based assessment.

  • Benchmark scope: CoVEBench is constructed specifically for complex compositional video editing, whereas UniVBench is a broad unified video benchmark covering six tasks.The comparison spans benchmark scope and task coverage.
  • Dataset and instructions: 416 curated source videos, 626 multi-point editing instructions, and 9,990 fine-grained checklist items comprise CoVEBench.Each instruction involves approximately three atomic edit operations on average.
  • Evaluation framework: CoVEBench decomposes complex instructions into verifiable checklist questions and separately evaluates instruction compliance and video quality.This design enables more targeted and fine-grained diagnosis than broad benchmark evaluation.
  • Evaluation framework: CoVEBench provides a more targeted, fine-grained, and diagnostic evaluation framework for complex compositional video editing than UniVBench.UniVBench remains valuable as a unified video benchmark.

C Dataset Samples · D Construction of the Test Set · E Topic Distribution of Source Videos

The benchmark illustrates compositional video-editing evaluation through frame-based checklist questions that test both requested semantic changes and preservation of untargeted attributes. Its test-set construction uses structured, multi-stage annotation, while the source videos span diverse domains, settings, actions, subjects, perspectives, and styles.

  • C Dataset Samples: A representative dataset sample combines five key frames from one video with checklist questions organized by specific editing points.The checklist evaluates whether target semantic changes are applied correctly and untargeted attributes remain faithful.
  • C Dataset Samples: Checklist questions assess both editing accuracy and preservation of unrelated video attributes.Questions are designed around target semantic changes and untargeted attributes.
  • D Construction of the Test Set: The annotation interface unifies source videos, editing instructions, target descriptions, evaluation groups, question categories, answer options, and expected answers.The system structures each video-editing case into multiple evaluation groups according to the intended editing points.
  • D Construction of the Test Set: Six trained annotators follow a rigorous workflow to verify instructions, refine checklists, and conduct sequential review using unified criteria.They revise wording, category labels, answer choices, and expected answers, while targeting factual errors and related issues.
  • E Topic Distribution of Source Videos: The dataset covers diverse video domains, including perspective attributes, scene settings, actions, subjects, and thematic styles.Its distribution spans real-world, people-centered, indoor, outdoor, nature, urban, documentary, cinematic, commercial, and educational content.
  • E Topic Distribution of Source Videos: Source videos also vary in camera motion patterns and environmental contexts, including indoor and outdoor environments, nature, and urban scenarios.The topic distribution is presented as a sunburst chart.

F Detailed Evaluation Methodology … F.3 Video Fidelity

The evaluation methodology combines MLLM-judged instruction-compliance metrics, automated video-quality measures, and fidelity metrics for preserving unchanged content and motion. It emphasizes complete edit correctness, perceptual and technical quality, semantic preservation, structural similarity, motion trajectories, and static-region identity.

  • F Detailed Evaluation Methodology: Together, the metrics distinguish semantic edit completion, edited-region realism, overall video quality, and preservation of unchanged spatial, temporal, and visual content.The methodology evaluates both what should change and what should remain unchanged, using complementary MLLM-judged and automated measures.
  • F.1 Instruction Compliance: Instruction compliance uses IFS, VRS, and UAS to assess edit execution, edited-region realism, and complete correctness.IFS averages checklist answers for fundamental transformations, VRS evaluates naturalness and artifacts, and UAS requires every associated instruction and realism query to be correct.
  • F.1 Instruction Compliance: UAS assigns 1 only when all associated IFS and VRS questions are correct; otherwise, the edit receives 0.For the “Addition and Placement of Side Cups” example, all four corresponding questions must be answered correctly.
  • F.2 Video Quality: Video quality combines VQR, AES, MSM, and TQ to measure perceptual quality, aesthetics, temporal smoothness, and technical distortions.VQR averages VisualQuality-R1 scores over 10 sampled frames, AES averages keyframe and sampled-frame aesthetic scores, MSM uses optical-flow stability across T = 10 frames, and TQ uses DOVER++ technical scores.
  • F.3 Video Fidelity: SEM uses preservation-oriented checklists to verify that non-target objects, backgrounds, attributes, actions, camera motion, and spatial relationships remain consistent.The metric evaluates content that should remain unchanged after editing, including checklist items such as Q11–Q13 in Fig. 7.
  • F.3 Video Fidelity: SSIM compares 10 corresponding grayscale frame pairs to measure preservation of global layout, shapes, and spatial structure.Background-change cases are excluded because SSIM is highly sensitive to background alterations.
  • F.3 Video Fidelity: MF uses CoTracker trajectories, position-velocity matching, and Hungarian assignment to measure source-video motion preservation.A higher MF indicates better preservation of source motion trajectories; the matching formulation uses α = 0.5 and image diagonal dmax.
  • F.3 Video Fidelity: SRC compares masked unchanged regions with DINOv2 feature cosine similarity to evaluate whether preserved static entities retain their visual identity.Masks for preserved static entities are obtained using Grounding DINO and SAM2, with an introduced filtering mechanism for mask-based similarity.

F.4 Discussion on metrics · G Experiment · G.1 Experiment Settings.

The paper argues that global similarity metrics are inadequate for complex, instruction-driven video editing and instead separates intended changes from preservation through an instruction-aware checklist. Experiments evaluate eight open-source models using their official inference pipelines on a single NVIDIA H200 GPU, with model-specific frame and resolution settings.

  • F.4 Discussion on metrics: Text–Video CLIP scores vary narrowly from 22.15 to 23.31, limiting their ability to distinguish true performance gaps.
  • F.4 Discussion on metrics: Text–Video CLIP rankings can contradict human judgments: VACE scores 22.57 versus ReCo’s 22.15 despite VACE largely failing to execute edits.
  • G Experiment: Inference settings vary by model, including output or maximum resolution, valid preprocessed frame ranges, default or used frame counts, and additional frame constraints.The table specifically notes requirements such as the 4k + 1 frame-count rule.
  • G Experiment: The benchmark reports category-level union accuracy for open-source video editing models across subject, background, and visual-effects editing categories.Categories include subject addition, removal, replacement, attribute modification, background editing, and VFX.
  • F.4 Discussion on metrics: Global similarity metrics can penalize intentional, instruction-driven modifications, especially when complex edits cause drastic visual and structural transformations.The evaluation paradigm is reconstructed to distinguish intended changes from expected preservation.
  • F.4 Discussion on metrics: The proposed semantic evaluation uses an instruction-aware checklist, termed SEM, to parse editing prompts.
  • G.1 Experiment Settings.: The experiments evaluate eight representative open-source models on a single NVIDIA H200 GPU using publicly released checkpoints and each model’s official pipeline.Recommended default hyperparameters are used for each model.

G.2 Agreement Experiment Settings.

The agreement experiment tests whether automatic metric preferences align with human judgments across 60 blinded pairwise comparisons among four video editing models. Metric-specific tie thresholds are adaptively calibrated from the proportion of human Tie judgments to accommodate differing score scales.

  • Study design: 60 pairwise comparisons evaluate preference alignment among VACE, OmniWeaving, Kiwi, and Ditto.Annotators compare anonymized outputs side by side while viewing the source video and editing instruction, without access to metric scores.
  • Agreement definition: A metric prefers the output with the higher score, and agreement is counted when it matches a human’s clear A-or-B preference.Contradictory preferences are counted as disagreements; human Tie selections are handled separately.
  • Tie handling: The tie threshold τ is adaptively set for each metric from the overall proportion of human Tie judgments rather than manually fixed.If X% of judgments are Tie, τ is defined as the X-th percentile of that metric’s pairwise absolute score differences.

G.3 Category Analysis. · G.4 Error Analysis.

Open-source video editing models show sharply uneven, complementary category-level capabilities rather than uniformly reliable performance. They perform better on global transformations but struggle with precise spatiotemporal control, object-level manipulation, and compositional localized edits.

  • G.3 Category Analysis.: Models show substantial performance variation across editing categories.Figure 10 uses min-max normalization by UAS to emphasize relative inter-model differences rather than absolute accuracy.
  • G.3 Category Analysis.: Style transfer, background editing, and visual effects generally yield higher accuracy than camera control and motion-related edits.
  • G.3 Category Analysis.: ReCo and ICVE perform competitively on subject replacement and removal, while VACE consistently underperforms across nearly all categories.
  • G.3 Category Analysis.: Current open-source models remain far from uniformly reliable, especially for precise spatiotemporal control, object-level manipulation, and compositional multi-edit instructions.
  • G.3 Category Analysis.: OmniWeaving and Ditto establish the upper bounds for background and style edits.
  • G.3 Category Analysis.: ICVE and ReCo lead challenging object-level manipulations, including subject removal and replacement.

G.4.1 Qualitative error analysis. … Error Analysis — AI Artifact Forensics

The paper combines qualitative and quantitative error analysis with a seven-dimension taxonomy, compositional instruction generation, checklist-based MLLM judging, and open-ended forensic diagnosis. These procedures expose failures in instruction execution, preservation, physical realism, and AI-generated artifacts.

  • G.4.1 Qualitative error analysis.: A balcony-editing example requires simultaneous action, position, object-removal, camera, and preservation edits, but HappyHorse 1.0 fails the standing and leaning actions.The model only accomplishes the portion visible in the supplied passage, which ends mid-sentence.
  • G.4.2 Quantitative error analysis.: Execution Inadequacies count incomplete editing points when any related Instruction Following Score question is uncompleted.Each problematic editing point increases the error count by 1.
  • G.4.2 Quantitative error analysis.: Spatial Entanglement counts unintended changes in non-target regions when a region’s preservation score is less than or equal to 5.A severe spatial entanglement issue adds 1 to the error count.
  • H Category taxonomy: The benchmark taxonomy covers subject manipulation, background modification, style transfer, motion control, spatial repositioning, camera editing, and visual effects.The categories are designed around mainstream operations and practical editing intents.
  • I Prompts: Compositional instructions are generated from 83 category combinations, each specifying 2–4 fine-grained operations, then dynamically sampled and selected for each source video.Five combinations and five corresponding few-shot examples are injected per model call, after which the MLLM selects one suitable combination and outputs undergo manual review.
  • I.5 Error Analysis Prompts: Each instruction is decomposed into edit and preservation points, with checklist questions spanning Execution Accuracy, Physical Logic, and Semantic Preservation.Four question formats are matched to visibility conditions: Video B only for Single-TF and AB-MCQ, and both videos for Dual-TF and Score-MCQ; Qwen3.5-122B-A10B judges Instruction Compliance and Physical Logic.
  • Error Analysis — Physics Forensics: Open-ended forensic analysis separates physical realism violations from AI-generated artifacts, using the editing prompt to avoid treating explicitly requested stylized or exaggerated behavior as a physics error.Physical analysis covers gravity, collisions, inertia, lighting, materials, and fluids, while artifacts include generation or rendering instability such as ghosting and anatomical errors.
  • Error Analysis — AI Artifact Forensics: The AI artifact forensics component acts as an expert artifact detector using an edited video and the editing prompt as input context.The supplied prompt specifies the detector’s role and its two required inputs.
Loading 2606.08415v2…