PaperlayerView exact revision on arXiv

Source-linked AI summary

AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities

Yuqing Wen, Yukai Huang, Qianqian Xie, Jiangtao Wu, Yibin Lin, Yikai Gu, Jialu Chen, Yuanxing Zhang, Jiaheng Liu

arXiv:2607.24821v1cs.MMcs.CVcs.SD

TL;DR

Existing benchmarks underrepresent coordinated audio-visual editing and cross-modal consistency in realistic videos. AVE-Compass evaluates these abilities with fine-grained checklists and multimodal metrics, while AVE-Agent uses structured planning and self-reflection to improve editing. AVE-Agent achieves the strongest edit-execution performance while maintaining competitive preservation and perceptual quality.

  • Problem

    Existing benchmarks largely assess visual edits on silent videos or isolated audio tasks, with coarse metrics for diagnosing cross-modal editing failures.

  • Method

    AVE-Compass benchmarks coordinated audio-visual editing with fine-grained checklists, multimodal quality metrics, and four diagnostic evaluation dimensions; AVE-Agent adds structured planning and self-reflection.

  • Results

    AVE-Agent achieves the strongest edit-execution performance while maintaining competitive preservation and perceptual quality.

  • Takeaways & Limitations

    AVE-Compass exposes persistent cross-modal editing failures and supports evaluation of coordinated edit execution, content preservation, realism, and synchronization.

  • Takeaways & Limitations

    Benchmark evaluation is expensive, and AVE-Agent depends on third-party tools whose behavior may change over time, affecting long-term reproducibility.

Abstract

from arXiv · show

While instruction-based video editing has advanced rapidly, real-world videos contain tightly coupled audio and visual signals, and editing one modality often requires coordinated changes in the other. Existing benchmarks primarily evaluate visual transformations on silent clips or isolated audio editing, leaving complex audio-visual editing and cross-modal consistency underexplored. We introduce AVE-Compass, a comprehensive benchmark with 145 curated source videos, 196 audio-visually coupled editing instructions, and 2,688 fine-grained checklist items. It evaluates Instruction Following, Fidelity Preserving, Realism, and Editing Intent through checklist-based MLLM judging and a dedicated realism rubric, complemented by automated cross-modal, video, and audio metrics. Extensive evaluation shows that state-of-the-art models still struggle to execute cross-modal instructions while preserving non-target content. We further propose AVE-Agent, a modular agent framework that decomposes complex instructions into dependent subtasks and iteratively improves editing results through self-reflection and evaluator feedback. AVE-Agent improves instruction execution, Fidelity Preserving, and audio-visual alignment in joint editing while maintaining competitive perceptual quality.

1 Introduction

AVE-Compass addresses the gap in evaluating realistic audio-visual editing, where visual dynamics, sounds, ambience, and speech are tightly coupled. It combines cross-modal editing scenarios with diagnostic checklist-based evaluation and shows that current systems remain unreliable, especially for audio preservation and synchronization.

  • Benchmark: AVE-Compass covers local visual or acoustic modifications through joint audiovisual reconstruction, emphasizing cross-modal dependencies and implicit synchronization constraints.Its instructions span diverse real-world editing scenarios rather than isolated modality-specific transformations.
  • Evaluation: The evaluation protocol decouples performance into Instruction Following, Fidelity Preserving, Realism, and Editing Intent.These signals are designed to provide diagnostic evaluation of audio-visual editing capabilities.
  • Findings: Current open-source and closed-source systems remain unreliable, often distorting background sounds or generating audio temporally inconsistent with edited visual events.The prominent failures indicate difficulty planning editing operations across modalities while jointly enforcing fidelity.
  • Benchmark: 145 source videos, 196 audio-visually coupled editing instructions, and 2,688 fine-grained checklist items comprise AVE-Compass.The benchmark targets realistic scenarios requiring synchronized cross-modal editing and preservation of non-target content.

2 Related Work

Related work has progressed from diffusion-based zero-shot visual video editing toward instruction-guided and unified video generation/editing frameworks. Existing benchmarks mainly assess visual modifications in silent videos, while audio evaluation follows a different focus.

  • Audio-Visual Editing Methods: Video editing methods evolved from diffusion-based zero-shot visual editing toward instruction-guided editing and unified video generation/editing frameworks.Early approaches extended image editing to videos through attention injection, feature propagation, or temporal consistency constraints.
  • Audio-Visual Editing Benchmarks: Visual editing benchmarks evaluate modifications to subjects, attributes, actions, styles, and visual effects in silent videos.They have also introduced multidimensional quality assessment and fine-grained object-level control.

3 AVE-Compass: Benchmark and Evaluation

AVE-Compass is a broad audio-visual editing benchmark spanning four editing branches and 28 fine-grained operation types, designed to test cross-modal causality, non-target preservation, multi-shot editing, and speech-targeted tasks. It combines checklist-based MLLM judging across four dimensions with reproducible automated metrics for cross-modal synchronization, video quality, temporal consistency, audio quality, and speech quality.

  • Benchmark Design: AVE-Compass organizes 28 fine-grained operation types across joint audio-visual, speech, video-only, and audio-only editing branches.The video-only and audio-only branches test non-target modality preservation, while joint and speech branches emphasize cross-modal synchronization and speech editing.
  • Benchmark Design: The benchmark requires cross-modal causality across diverse edit types and additionally supports multi-shot clips, speech-targeted editing, and three-axis difficulty stratification.These design choices address limitations of prior benchmarks that isolate modalities or cover only basic object-level audio-visual operations.
  • Evaluation Framework: AVE-Compass evaluates edited clips with complementary MLLM judging and automated metrics, separating instruction-level behavior from low-level audio-visual measurements.MLLM judging assesses semantic correctness, source preservation, and perceptual plausibility; automated metrics measure synchronization, visual quality, temporal consistency, audio quality, and speech quality.
  • Evaluation Framework: The subjective evaluation reports Editing Intent, Instruction Following, Fidelity Preserving, and Realism, each with overall, video, and audio scores.Editing Intent jointly rewards executing the requested edit and preserving non-target content, while the other dimensions use checklists or a realism rubric for targeted assessment.
  • Automated Metrics: Seven automated metrics are grouped into Cross-Modal, Video, and Audio categories, with larger normalized values indicating better performance.Cross-modal metrics assess lip synchronization and audio-visual event alignment; video metrics assess aesthetics, subject consistency, and motion smoothness; audio metrics assess audio aesthetics and speech quality.

4 Audio-Visual Editing Agent

AVE-Agent is a modular audio-visual editing pipeline that decomposes free-form instructions into dependent subtasks, executes them with self-check reflection, and evaluates the assembled clip for global coherence. Its evaluator-driven replanning applies targeted corrections to cross-modal, structural, and mixing failures while avoiding unnecessary recomputation.

  • Pipeline Overview: AVE-Agent uses planner, executor, and mixed evaluator agents to compile natural-language instructions into edited audio-visual clips.The planner analyzes and decomposes intent, the executor performs subtasks with self-check reflection, and the evaluator assesses the assembled result before replanning.
  • Planner Agent: The planner constructs an executable plan with explicit dependencies by profiling scenes, subjects, actions, and audio context, then identifying unstated cross-modal consequences.A Gemini-based planner conditions intent decomposition on the structured input profile.
  • Executor Agent: The executor routes each subtask to video, audio, or speech tools, compiles high-level intent into tool-specific guidance, and performs local error correction.Each subtask is grounded in concrete editing operations and checked through a self-reflection loop.
  • Mixed Evaluator Agent: The mixed evaluator checks the complete clip for instruction following, fidelity preservation, quality, and cross-modal conflicts that subtask-level checks may miss.Examples include sound effects obscuring dialogue or generated audio no longer matching a modified scene.
  • Feedback and Correction: The agent selects the least costly corrective action: remixing mixing problems, regenerating faulty audio, or replanning structural and dependency errors.Evaluator feedback is returned to the planner for full replanning when local correction is insufficient, avoiding unnecessary recomputation.

5 Experiments

Experiments on AVE-Compass show that AVE-Agent is the strongest edit executor, while current models struggle to balance instruction execution with preservation and realism across modalities. Additional analyses examine response-gated evaluation, difficulty robustness, metric validity, single-modality editing, failure patterns, and agent-component ablations.

  • Main Results: AVE-Agent achieves the best Editing Intent and Instruction Following, with particularly large gains in audio-side instruction following and audio-visual synchronization.It also outperforms baselines on most remaining metrics, supporting the effectiveness of its agent-based workflow.
  • Response-Gated Evaluation: High Fidelity Preserving or Realism can reflect leaving the input unchanged, so response-gated scores isolate preservation quality after the target modality changes.Non-response remains an Instruction Following failure but is excluded from gated Fidelity Preserving and Realism.
  • Robustness under Difficulty: AVE-Agent remains comparatively stable as source videos lengthen and across object-localization hardness, audio-source complexity, and cross-modal linkage degree.These conditions are evaluated in Figure 4 using modality-matched metrics.
  • Metric Analysis: Objective and subjective metrics show generally weak Spearman correlations, indicating largely distinct yet complementary aspects of editing quality.Automated scores vary by less than 0.01 and MLLM-as-Judge scores by less than 1% across five repeated scoring runs.
  • Error Analysis and Case Study: AVE-Agent produces the fewest errors overall, while non-responsive models mainly fail Instruction Following and highly regenerative models mainly fail Fidelity Preserving.The error taxonomy contains five concrete categories, and AVE-Agent is not dominated by any single failure type.
  • Ablation Study: Removing prompt enhancement decreases Editing Intent by 7.76 and Instruction Following by 6.91, while removing retry refinement causes an even larger decline.Prompt enhancement also improves Realism by translating user intent into executable tool prompts.

6 Conclusion

The paper introduces AVE-Compass for holistic evaluation of coordinated audio-visual editing and AVE-Agent for structured, iterative execution of complex cross-modal instructions. Experiments identify current-model failure modes and show AVE-Agent achieves strong edit execution while preserving competitive perceptual quality.

  • 6 Conclusion: AVE-Compass evaluates coordinated audio-visual editing across Instruction Following, Fidelity Preserving, Realism, and Editing Intent.Its framework combines checklist-based MLLM judging, a dedicated realism rubric, and automated cross-modal, video, and audio metrics.
  • 6 Conclusion: AVE-Agent addresses complex cross-modal editing through structured task planning and iterative self-reflection.The framework is modular and targets coordinated execution across editing subtasks.
  • 6 Conclusion: AVE-Agent achieves the strongest edit-execution performance while maintaining competitive preservation and perceptual quality.The experiments also uncover characteristic failure modes of current models.
  • 6 Conclusion: AVE-Compass aims to advance multimodal editing beyond isolated pixel-level modifications toward physically and perceptually coherent results.The benchmark is intended to support progress in coordinated audio-visual editing.

A Benchmark Construction Details · A.1 Objective Metric Formulas · A.2 Subjective Evaluation Formulas

AVE-Compass combines category-aware objective metrics with checklist-based subjective judging for instruction following, fidelity, edit response, and realism. Its formulas normalize modality-specific scores, separate modality evaluation, and compute editing intent from both execution and preservation.

  • A.1 Objective Metric Formulas: Each sample is assigned audio-only, video-only, joint, or speech categories, activating only relevant objective metrics without penalizing inactive modalities.Scores are generally oriented so larger values are better, and bounded predictor outputs are mapped to [0, 1].
  • A.1 Objective Metric Formulas: The objective branch reports seven reference-free scores, supplemented by non-target-stream preservation similarities for single-modality edits.videoclip_video_similarity evaluates visual preservation for audio-only edits, while audio_similarity evaluates audio preservation for video-only edits.
  • A.1 Objective Metric Formulas: Audio and speech quality are measured with AudioBox Aesthetics and NISQA, while lip_sync and av_sync assess speech-mouth alignment and audio-visual temporal alignment.speech_quality is enabled only for speech edits; av_sync uses tolerance τ = 2.0 s, assigning 1.0 to exact alignment and 0 once the offset reaches 2 seconds.
  • A.1 Objective Metric Formulas: Video quality is evaluated through video_aesthetic, subject_consistency, and motion_smoothness, targeting visual appeal, identity stability, and temporal continuity.video_aesthetic uniformly samples T = 10 frames, subject_consistency uses adjacent-frame DINOv2 cosine similarity, and motion_smoothness compares interpolated with actual middle frames.
  • A.2 Subjective Evaluation Formulas: The subjective protocol uses branch-specific atomic Yes/No checklists for Instruction Following and Fidelity Preserving, with separate Edit Response questions and standalone Realism evaluation.Checklist items are tagged by dimension, subdimension, and modality, while response questions record whether the target changed rather than whether the edit was correct.
  • A.2 Subjective Evaluation Formulas: The Absolute Affirmative Paradigm requires “Yes” only when evidence clearly satisfies the success condition, rejecting absent, ambiguous, partial, weak, or approximate matches.Modality-split judging gives audio questions audio evidence, video questions video evidence, and general questions integrated audio-video evidence.
  • A.2 Subjective Evaluation Formulas: Per-modality and overall scores are Yes-rates over applicable checklist items, with empty modality-dimension cells omitted rather than imputed.Human annotators conservatively deduplicate only items receiving the same answer under the same evidence, retaining the more specific question.
  • A.2 Subjective Evaluation Formulas: Editing Intent is computed per case as a product of Instruction Following and Fidelity Preserving, then macro-averaged over cases where both factors are defined.Audio and video intent use M = {audio} and M = {video}, respectively, and the same macro-averaging rule applies to Instruction Following, Fidelity Preserving, and Edit Response.

B Implementation Details of AVE-Agent … C Experiment Details

AVE-Agent is implemented as a modular pipeline that analyzes videos, plans typed cross-modal intents, dispatches modality-specific tools, and iteratively evaluates and improves edits. The supplied passages describe these implementation components but provide no substantive details for C Experiment Details.

  • B Implementation Details of AVE-Agent: The appendix specifies a Video Analyzer, typed planner schemas, tool registry, self-check loop, and Mixed Evaluator with actionable retry signals.These are the three-module implementation areas and optimization mechanisms documented in the appendix.
  • B.1 Video Analyzer: Preprocessing separates audio and visual tracks, tone-maps HDR→SDR, normalises width to 720–2160 px with square pixels, and extracts 1–3 keyframes.Gemini 2.5 Flash captions the full clip with visual, audio, and shot-list fields.
  • B.2 Planner Schema: The planner emits ordered typed intents with actions, shot or global scope, visual→audio dependencies, and explicit audio inventory fields.PlanValidator enforces audio↔visual consistency, including naming removal targets and pairing vocal actions with visual motion edits.
  • B.2 Planner Schema: Validated intents are translated in parallel into modality-specific prompts under strict word and format constraints, with MMAudio negatives derived from deleted inventory sounds.The fields include video_prompt, sam_prompt, mmaudio_prompt, speech_*, and optional per-step evaluation criteria.
  • B.3 Tool Registry: The ToolRegistry dispatches video edits to Wan 2.7 by default or Seedance 2.0 alternatively, while routing audio edits into separate, generate, and volume steps.Shots shorter than 3 s are freeze-padded and re-sliced, and audio uses fal.ai SAM Audio for source separation.
  • B.4 Self-Check / Optimization Loop: Three sub-evaluators score separation, generation, and post-branch results against the audio inventory and return failure signals for optimization.Separation fidelity below 0.4 imposes a hard score ceiling, while generation reports missing and unwanted content fields.
  • B.4 Self-Check / Optimization Loop: Improvers preserve at least 70% of original prompts, apply SAM and MMAudio word caps, maintain bounded negative lists, and allow up to three Subtask attempts.SAM adds at most 1–2 descriptors, while MMAudio adjusts only terms identified as missing or unwanted.
  • B.5 Mixed Evaluator: The Mixed Evaluator scores instruction following, fidelity, and quality, then applies volume-only re-muxing, audio regeneration, or structural escalation through bounded retries.Volume adjustment permits up to two retries without model re-invocation, and needs_regenerate permits one audio-Subtask retry.

C.1 Metric Similarity and Redundancy Analysis · C.2 Human Consistency Calculation

C.1 analyzes alignment and redundancy among automated metrics, MLLM-as-Judge dimensions, and human judgments, while C.2 defines a human consistency study for validating automatic judgments against annotations.

  • C.1 Metric Similarity and Redundancy Analysis: Metric similarity analysis uses Spearman correlation to compare automated metrics with MLLM-as-Judge dimensions.The analysis provides an intuitive view of how metrics align with one another and with human judgments.
  • C.1 Metric Similarity and Redundancy Analysis: Some objective metrics meaningfully align with subjective human preferences, indicating correspondence between automated evaluation and human assessment.
  • C.1 Metric Similarity and Redundancy Analysis: Several objective metrics exhibit exceptionally high inter-metric correlations, revealing significant redundancy in the evaluation framework.These metrics capture highly overlapping characteristics when assessing generated outputs.
  • C.1 Metric Similarity and Redundancy Analysis: The metric-similarity results are visualized as a heatmap in the left panel of Figure 5 in the main text.
  • C.2 Human Consistency Calculation: The human consistency study tests whether MLLM-as-Judge results remain stable relative to human annotations for sampled questions.Automatic judgments are counted as correct when consistent with human annotations and incorrect otherwise.
  • C.2 Human Consistency Calculation: Human consistency scores are computed for each evaluation dimension and modality, then reported as averaged percentages in Table 5.Ncorrect and Nincorrect denote matched and mismatched judgments, respectively.

C.3 Difficulty-Stratified Analysis · C.4 Single-Modality Editing

The section presents a difficulty-stratified breakdown with modality-matched metrics and reports objective results for single-modality editing. These analyses correspond to Tables 7 and 8 and the radar visualization in Figure 5.

  • C.3 Difficulty-Stratified Analysis: Table 7 provides the full difficulty-stratified breakdown for the analysis of robustness under difficulty.The breakdown is referenced in Section 5.2.
  • C.3 Difficulty-Stratified Analysis: Each difficulty-stratified axis is paired with a modality-matched metric.
  • C.3 Difficulty-Stratified Analysis: The analysis is organized around robustness under difficulty.This framing is explicitly associated with Section 5.2.
  • The section combines difficulty-stratified evaluation with objective single-modality reporting.The two analyses are presented through Tables 7 and 8, respectively.
  • C.4 Single-Modality Editing: Table 8 reports objective numbers for single-modality editing.
  • C.4 Single-Modality Editing: The single-modality results are summarized by the radar in the middle panel of Figure 5.

C.5 Error Analysis

The error analysis categorizes generation defects across five failure categories covering video/audio Fidelity Preserving, video/audio Instruction Following, and audio-visual quality. AVE-Agent incurs the fewest total errors, while non-response baselines fail predominantly on Instruction Following.

  • Error-analysis protocol: Errors are counted when checklist yes-rates fall below 0.5, or when audio-visual Realism falls below 3/5.The analysis spans five failure categories across all six models.
  • Overall errors: 175 total errors make AVE-Agent the best-performing approach, compared with 248–406 for the baselines.Lower error counts are better.
  • Characteristic failure modes: Audio-IF contributes 144 of Gemini’s errors and 133 of Seedance’s errors, reflecting frequent returns of the source video unedited.The non-response baselines fail predominantly on Instruction Following.
  • Characteristic failure modes: Gemini’s 5 Audio-Fidelity-Preserving errors reflect non-response rather than preservation quality.Its low count is a direct consequence of returning the source unedited.

C.6 Case Analysis · C.7 Effect of the Agent Backbone · D Broader Impact

Case analyses expose baseline failures in audio fidelity, visual realism, responsiveness, physical logic, and temporal consistency, while backbone ablations show agent augmentation can improve intent and instruction following with trade-offs. AVE-Compass and AVE-Agent broaden access to audio-visual creation but retain dual-use risks, especially for speech-targeted editing.

  • C.6 Case Analysis: Wan2.7 substantially degrades audio during conductor removal and transforms a requested hot stone slab into a lava-like, AI-looking effect.These examples demonstrate Audio Fidelity Preserving and visual realism failures in representative cases.
  • C.6 Case Analysis: LTX2 sometimes leaves clips unedited, while HappyHorse replaces a bicycle with a motorcycle but retains bicycle-like pedaling.Other cited failures include unrequested captions and temporally inconsistent or physically implausible changes.
  • C.6 Case Analysis: Baseline cases reveal audio degradation, visually over-synthetic effects, non-response, unsupported insertions, physical-logic errors, and temporal inconsistencies.Figure 7 identifies these as dominant failures across selected outputs, while Wan2.7 and LTX2 illustrate several specific weaknesses.
  • C.7 Effect of the Agent Backbone: For Seedance, AVE-Agent (sd2) improves Editing Intent and Instruction Following but reduces Fidelity Preserving and Realism.The reported trade-off is attributed mainly to standalone Seedance often leaving the input unchanged, thereby preserving realism without executing edits.
  • D Broader Impact: AVE-Compass and AVE-Agent lower barriers to free-form audio-visual creation for education, independent creators, and accessibility-oriented uses.The cited examples include redubbing and content localisation.
  • D Broader Impact: Improved cross-modal coherence carries dual-use risks including deepfakes, disinformation, and non-consensual voice cloning, particularly in the speech-targeted branch.The passage states that the benchmark uses only publicly licens…; the supplied text is truncated before completing the mitigation details.

E Limitations

AVE-Compass prioritizes broad, fine-grained coverage over raw benchmark scale because per-instance evaluation is expensive. A full 196-instruction generation pass can cost about $220 in video-generation fees alone, with MLLM judging adding further expense.

  • Evaluation Cost: Per-instance scoring is expensive because it runs closed-source video and audio editing APIs alongside MLLM-based checklist judging.The benchmark deliberately maximizes coverage rather than raw count within this evaluation budget.
  • Coverage–Cost Trade-off: 28 fine-grained edit subcategories broaden AVE-Compass beyond subject-level operations.This broader coverage is part of the benchmark’s deliberate trade-off against evaluation cost.
  • Evaluation Cost: $220 is the estimated generation cost for one full 196-instruction pass using the official BytePlus Seedance 2.0 API plan.The estimate assumes average clip duration of roughly 7.4 seconds and about $0.15 per generated second.

F Prompt Templates

Appendix F documents AVE-Compass’s prompt templates for source-video captioning, checklist generation, checklist evaluation, and realism evaluation. The templates use structured, source-grounded outputs to describe audiovisual content and generate modality-specific evaluation questions.

  • F Prompt Templates: The appendix collects prompts for captioning, checklist generation, checklist evaluation, and realism evaluation within the benchmark pipeline.These templates are accompanied by short descriptions of their roles.
  • F.1 Source-Video Captioning Prompt: The captioning stage runs a multimodal model on the original video with audio and requests structured JSON for downstream prompt generation and evaluation.The requested description is source-video grounded and supports later benchmark processing.
  • F.1.2 Captioning User Prompt: Captioning prompts require independent visual and audio descriptions, chronological shot-level coverage, and explicit identification of shot transitions.The visual caption excludes audio, while the audio caption describes heard content without transcribing speech.
  • F.1.2 Captioning User Prompt: Captioning instructions emphasize objectively verifiable audiovisual facts, instructing the model to mark uncertainty as “unclear” or “unknown” and avoid guessing or fabrication.They also require valid JSON without extra text, markdown, or code fences.
  • F.1.2 Captioning User Prompt: The captioning schema records caption, audio_caption, speech_or_non_speech, shot_count, and visual_style as structured fields.The prompt specifies shot_count as the number of distinct shots or cuts and speech_or_non_speech according to whether human speech is audible.
  • F.1.2 Captioning User Prompt: The prompt’s visual checklist covers setting, subjects, actions, framing, camera angle, camera motion, depth of field, and transitions when multiple shots occur.Its audio checklist covers speech, sound effects, ambient sound, music, and their relationships to visual events.
  • F.2 Checklist Generation Prompts: Four V2 generators produce modality-specific, source-grounded checklist questions for video-only, audio-only, joint audio-video, and speech editing tasks.The video-only generator uses dimensions including Edit Response, Instruction Following, and Fidelity, with subdimensions for correctness, preservation, and hallucination control.
  • F.2.1 Video-Only Checklist Generator: Video-only checklist prompts require positive “Yes” answers, concise checklists typically containing 8-16 questions, and no audio response question.Q01 is the sole Edit Response question and asks whether any visible change appears compared with the source video.
Loading 2607.24821v1…