PaperlayerView exact revision on arXiv

Source-linked AI summary

AVENUE: Audio-Video EditiNg Understanding and Evaluation

Hayeon Kim, Yoojin Jang, Jaejun Yoo

arXiv:2609.04253v1cs.MMcs.CVcs.SD

TL;DR

AV editing requires models to determine which modality a prompt should change and which it should preserve, a capability not adequately covered by existing benchmarks or evaluation systems. AVENUE provides a broader benchmark and sample-specific, modality-aware evaluation, and its assessment finds that no existing editing paradigm fully resolves modality-selective control.

  • Problem

    Existing AV editing benchmarks have limited edit-type and modality coverage, while evaluation systems often lack modality-aware, sample-specific criteria for assessing preservation.

    Source: §1 · pp. 1–2

  • Method

    AVENUE combines a VGGSound benchmark of 1,291 clips and 7,957 instructions with human-verified per-sample annotations of intended changes and preserved content.

    Source: §1 · p. 2

  • Results

    No evaluated paradigm fully resolves modality-selective editing; each exhibits a distinct trade-off between edit fidelity and modality preservation, with performance varying across edit types.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Modality-selective controllability remains an open challenge, and the conclusion calls for models designed with explicit awareness of what to change and what to leave intact.

    Source: §5 · p. 10

    Instructions come from one multimodal LLM and require manual verification, which makes curation difficult to scale; VGGSound may also leave residual quality artifacts.

    Source: F Limitations · p. 24

Abstract

from arXiv · show

Audio-video (AV) editing aims to modify audio and video content according to a target prompt. Unlike single-modality editing, AV editing requires models to infer a modality-selective edit scope from the prompt alone: determining not only what should change, but also which modality should be preserved. Faithfully evaluating such models therefore requires both (i) benchmarks that span diverse edit types and modality categories, and (ii) evaluation that is itself modality-aware and sample-specific. However, existing AV editing benchmarks provide limited coverage of edit types and modality combinations, while current evaluation systems are often modality-blind and sample-agnostic, making it difficult to assess whether models faithfully preserve the unintended modality. To address these gaps, we introduce AVENUE, Audio-Video EditiNg Understanding and Evaluation, comprising two contributions: (1) a benchmark of 1,291 source clips and 7,957 editing instructions across audio-targeted, video-targeted, and AV-coupled edit types, curated and human-verified from VGGSound; and (2) a sample-specific, modality-aware evaluation framework that specifies, for each sample, both the intended change and the content that must remain intact. We evaluate representative AV editing models spanning three editing paradigms : joint, sequential, and separate, providing the first systematic analysis of modality-selectivity across paradigms. Our findings reveal a fundamental open challenge: when editing one modality, existing models frequently induce unintended changes in the other, regardless of paradigm. AVENUE provides a benchmark and modality-aware evaluation framework to drive progress toward more controllable AV editing models. Our dataset is publicly available on Hugging Face: https://huggingface.co/datasets/AVENUE-dataset/AVENUE.

1 Introduction

AV editing requires models to infer from the prompt which modality to change and which to preserve, but existing benchmarks and evaluation frameworks do not adequately assess that selectivity. AVENUE pairs broader edit coverage with per-sample preservation annotations and evaluates existing models across three editing paradigms.

1 Introduction

  • Existing benchmarks cover only 1.3% of the defined edit taxonomy and AV-coupled edits, leaving audio- and video-targeted selectivity unassessed.The cited prior benchmark covers 110 samples, focused on object-level manipulation.Source: §1 · p. 2
  • AVENUE contains 1,291 source clips and 7,957 instructions across 12 edit types spanning audio-targeted, video-targeted, and AV-coupled settings.The clips are curated from VGGSound, and the instructions have human-verified annotations.Source: §1 · p. 2
  • Per-sample annotations specify both the intended edit and what must remain intact, grounding automated MLLM judges in sample-specific criteria.The framework includes Selective Controllability alongside Edit Accuracy, Modality Selectivity, Perceptual Quality, and AV Consistency.Source: §1 · pp. 2–3
  • The study examines joint, sequential, and separate AV editing models, diagnosing their robustness to unintended cross-modal effects.The authors present this as a comprehensive study across the three paradigms.Source: §1 · p. 3

2 Related Works

Prior work includes modality-specific editing methods and unified AV editors, but benchmark coverage and evaluation remain limited. The literature also explores foundation models as judges, while AV editing calls for broader cross-modal assessment.

Single modal Generation and Editing

  • Independent visual and audio editing methods can break AV coherence when their outputs no longer share consistent semantics, motivating unified editors.Unified AV approaches coordinate edits across modalities.Source: §2 · p. 3

Unified Audio-Video Editing and Benchmarks

  • Unified AV editors coordinate synchronized streams through joint or sequential approaches, but benchmarks remain small or narrow and metrics do not assess per-sample preservation.Existing protocols largely rely on embedding-based alignment metrics.Source: §2 · p. 3

MLLM as-a-Judge

  • Foundation-model judges show promise in other generative tasks, but MLLM-as-a-judge remains largely unexplored for AV editing's cross-modal consistency demands.Prior work reports gaps in MLLM scoring and ranking despite human-like discernment in pairwise comparisons.Source: §2 · p. 3

3 AVENUE-Bench

AVENUE organizes AV editing by modality scope and semantic category, then builds taxonomy-aligned instructions and a sample-specific evaluation framework. Its design distinguishes edits that should affect one modality from edits that should change both coherently.

3 AVENUE-Bench

  • AVENUE organizes editing tasks by modality scope and semantic category, addressing the fact that many edits should leave one modality unchanged.The benchmark is curated from VGGSound with instructions for taxonomy combinations.Source: §3 · p. 3

3.1 Dataset Curation

  • The curation pipeline filters VGGSound clips and generates taxonomy-aligned instructions, using the test set to avoid leakage with models pretrained on its training data.The pipeline applies category and audio-video quality filtering.Source: §3.1 · p. 4

3.2 Editing Taxonomy

  • The taxonomy separates audio-targeted, video-targeted, and AV-coupled edits; category-aware templates restrict instructions to edits meaningful for each semantic category.For example, Action Change is inapplicable to Nature scenes, while Weather Change is meaningful for Nature and Animals.Source: §3.2 · pp. 4–5

3.3 Selective Controllability Evaluation

  • Four evaluation dimensions use human-verified what_changed and what_preserved annotations to assess each sample against its intended edit scope.Edit Accuracy and Modality Selectivity use source and edited outputs; Perceptual Quality uses only the edited output, and evaluator inputs align with the edit category.Source: §3.3 · pp. 5–6

4 Experiments

Across editing paradigms, performance varies by edit type and preservation competes with edit strength: no paradigm leads on every metric. The experiments compare separate, sequential, and joint models, including null-edit preservation and human–judge alignment.

4.1.1 Audio-Video Editing Paradigms

  • The study compares separate single-modality editing, sequential video-to-audio editing, and joint editing, using RAVE-edited video as shared input for the sequential and separate paradigms.The shared video input isolates the effect of audio-video integration strategy from video-editing capability.Source: §4.1.1 · p. 6

4.1.2 Self-prompt preservation of AV Editing models

  • Under null edits, Joint has the best preservation scores across both modalities, while Separated models structurally prevent cross-modal interference and Sequential models show audio instability.For AvED, Modality Selectivity is 76.3 for video and 85.0 for audio; the authors attribute Sequential audio instability to generation conditioned on video without an anchor to the original audio.Source: §4.1.2 · pp. 6–7, Table 2 · p. 7

4.2.1 AV-coupled Edit Results

  • On Category C, AV Consistency is nearly identical across paradigms (∼78.5), while Joint trails Sequential and Separate by 7–8 Edit Accuracy points and ∼15 Perceptual Quality points.Sequential methods have the most balanced profile, while Separate methods show a slight Edit Accuracy edge.Source: §4.2.1 · p. 7, Figure 3 · p. 8
  • Within Category C, rankings depend on edit subtype: Sequential and Separate outperform Joint by a large margin on C1 and C4, Joint reverses that gap on C3, and all three converge on C5 speed changes.On C3, Joint leads in Edit Accuracy (53.5) and AV Consistency (70.5), while Sequential leads in Perceptual Quality (72.3 vs. 55.7).Source: §4.2.1 · pp. 7–8

4.2.2 Results on Audio-targeted and Video-targeted Editing

  • Inversion-based ZETA combines competitive Edit Accuracy (79.7) with Modality Selectivity (93.0), whereas SDEdit scores 78.7 and 51.4, respectively.The authors associate ZETA's inversion with retaining source-audio structure, while SDEdit's noising step loses some source information; MMAudio has Perceptual Quality 89.5 but Modality Selectivity 39.7.Source: §4.2.2 · p. 9
  • No paradigm dominates audio editing: joint editing excels at preservation but under-edits, while noise-based and generative approaches edit more strongly but preserve less.With a shared RAVE video backbone, AVED scores 97.5 for Modality Selectivity and 64.1 for Edit Accuracy; separated and sequential models reach 76–80 Edit Accuracy, while ZETA scores 93.0 and SDEdit 51.4 in audio preservation.Source: §4.2.2 · p. 9

4.3 Human validation of the automated judge

  • Qwen3-Omni achieves 67.0% pairwise agreement with humans, compared with 71.1% human–human agreement, and the other tested judges show similar alignment trends.The study used 13 human annotators; Qwen3-Omni's leave-one-out agreement is 72.5% versus 79.2%, and ranking-level Kendall’s τb is 0.353 versus 0.546.Source: §4.3 · p. 9

5 Conclusion

AVENUE enables systematic assessment of modality-selective editing across diverse edit types and model paradigms. The evaluation finds no existing paradigm fully resolves the challenge, while inversion-based mechanisms show promise on edit fidelity and modality preservation.

5 Conclusion

  • AVENUE enables the first systematic assessment of modality-selective editing across diverse edit types and model paradigms.Source: §5 · pp. 9–10
  • No existing editing paradigm fully resolves modality-selective controllability, and performance varies across edit types, with no single architecture generalizing across the spectrum.Inversion-based mechanisms show the most promise for competitive performance on both edit fidelity and modality preservation.Source: §5 · p. 10

A.1 Dataset Access

  • AVENUE dataset annotation files and evaluation code are available for download.Source: §A.1 · p. 14

A.2 Dataset Statistics for AVENUE

AVENUE includes diverse, category-aware editing instructions across many source clips, and reports results at the video–instruction level. Its prompts span distinct lexical forms and semantic requests, while category counts reflect VGGSound’s underlying distribution.

A.2 Dataset Statistics for AVENUE

  • AVENUE contains 1,291 unique source clips and 7,957 editing instances, averaging 6.16 instructions per clip, with a median of 6 and a range of 3–9.Source: §A.2 · p. 14
  • AVENUE results are reported per video–instruction pair because one source clip may support semantically distinct editing operations.Multiple instructions also permit behavior comparisons while controlling for source-content difficulty.Source: §A.2 · p. 14
  • AVENUE has more unique source clips than OAVE (44), AvED (110), and VGG-Edit (450).Source: §A.2 · p. 14
  • In AVENUE, Instruments has 481 clips, Animals 265, and Domestic 181, while Nature and Explosions have 14 and 18, respectively.The category imbalance reflects the underlying VGGSound distribution [5].Source: §A.2 · p. 14
  • Edit-specific terms are distinctly distributed across audio-targeted, video-targeted, and AV-coupled prompt word clouds.The word clouds use targeted nouns and adjectives extracted by aligning source descriptions and target prompts, excluding domain-generic terms.Source: §A.2 · p. 14, Figure A1 · p. 15
  • AVENUE’s prompts are more diverse than AvED’s: TTR is 0.305 vs. 0.078, distinct-1 is 0.305 vs. 0.078, and distinct-2 is 0.640 vs. 0.127.Unique-prompt ratio is approximately 100% vs. 20.9%; self-BLEU is 0.240 vs. 0.867, and mean pairwise cosine similarity is 0.160 vs. 0.235.Source: §A.2 · pp. 14–15, Table A1 · p. 15

B.1 Dataset Filtering

Embedding-based evaluation finds the strongest preservation for joint models, while separated and sequential models show audio instability under self-prompt conditions. The curation pipeline combines filtering and human verification, and the human annotators show substantial agreement.

B.1 Dataset Filtering

  • The overall curation pipeline has three stages—categorization, filtering, and category-aware instruction generation—with human verification at two points rather than as another filtering stage.The two-stage filtering denotes the quality filters within the filtering stage.Source: §B.1 · p. 15, Figure A2 · p. 16

B.1.1 Audio-Video Quality Filtering.

  • Audio-video quality filtering retains clips only when ImageBind audio-video similarity and CLAP audio-language alignment both exceed 0.3.This removes poorly correlated streams that would undermine AV-coupled edit evaluation.Source: §B.1.1 · p. 15

B.1.2 Editing Suitable Filtering.

  • Editing-suitability filters screen for scene consistency, coherent subjects, person density, and temporal dynamics, with remaining evaluation clips manually checked for static content.The criteria include mean inter-frame CLIP similarity of at least 0.75, a dominant object in at least 9 of 10 sampled frames, and mean consecutive-frame SSIM no higher than 0.85.Source: §B.1.2 · pp. 15–16

B.2 Human Verification

  • Two expert annotators reviewed generated instructions jointly across source prompt, target prompt, changed content, and preserved content; they achieved 90.6% raw agreement, κ = 0.690, and AC1 = 0.865.Flagged issues included already-present targets, cross-modal effects in audio-targeted edits, ambiguity, static videos, and inaccurate change or preservation descriptions.Source: §B.2 · pp. 16–17

B.3 Human alignment with automated MLLM Judge

  • The human–judge alignment protocol compares rankings from 13 evaluators with five automated judges using pairwise preferences, leave-one-out agreement, and Kendall’s τb.It samples 60 edit instances and produces 260 instance-level system pairs; order-inconsistent or tied direct comparisons are excluded.Source: §B.3 · pp. 18–19

C.1 Self-Prompt Preservation under Embedding Metrics

  • Under null edits, joint models show the strongest preservation (V-CLIP: 0.995, LPIPS: 0.110, CLAP: 0.886), while separated and sequential models exhibit audio deviations.The reported embedding-based trends are consistent with the MLLM-based preservation results.Source: §C.1 · p. 19

C.2 Cross-Modality-Conditioned Targeted Editing

The pilot tests whether single-modality edits can be timed to trigger events in the other modality, extending evaluation beyond cleanly separated edits. Models often produce the requested content but fail to align it with the trigger, and neither evaluated paradigm resolves this setting.

C.2 Cross-Modality-Conditioned Targeted Editing

  • CMTG scores were uniformly low (20–29/100), although edit accuracy remained comparable to or mostly slightly above the non-conditioned setting; no paradigm resolves trigger alignment.Edit Accuracy measures whether the requested content was produced, while CMTG measures whether it was timed to the trigger event.Source: Table A3 · p. 20, §C.2 · p. 20
  • For Source → CAVE, the source video is retained directly, so video-preservation metrics are not separately evaluated; audio preservation is scored separately for each complete pipeline.The reporting note also states that identical video-preservation scores are merged across corresponding rows.Source: §C.2 · p. 20
  • The pilot uses 70 human-curated samples—40 audio-targeted and 30 video-targeted—with each edit conditioned on a trigger in the other modality.The setup compares AvED with RAVE + ZETA; MMAudio is excluded for the same reason as in the main evaluation.Source: §C.2 · pp. 19–20
  • The separated pipeline led audio-targeted Edit Accuracy at 87.0, but its CMTG score was 24.6 versus 28.6 for the joint model.The joint model’s CMTG edge reflects being less poor, not competence; its cross-attention masks and contrastive objective were not designed for temporal grounding.Source: §C.2 · p. 20, Table A3 · p. 20
  • Cross-modality-conditioned edits add a cross-modal reasoning test to cleanly separated edits, which measure modality-selective preservation; the authors identify a dedicated track and CMTG reporting as direct extensions.The pilot is presented as an underexplored axis that can sit alongside the existing categories.Source: §C.2 · p. 20

C.3 Consistency with Embedding-based Evaluation

The appendix checks agreement between MLLM-based evaluation and embedding metrics, then presents overall AV-editing scores across Cat. C subcategories. MLLM scores correlate positively and significantly with embedding-based metrics across the reported dimensions and categories.

C.3 Consistency with Embedding-based Evaluation

  • The correlation analysis averages Spearman ρ across evaluated AV-editing models for each evaluation dimension and edit category.Figure A6 reports mean ρ and gives the min–max range across generation models; its ta, tv, and av suffixes denote text–audio, text–video, and audio–video alignment.Source: §C.3 · pp. 20–21, Figure A6 · p. 21
  • All 12 MLLM-score correlations with embedding metrics were positive and significant (p < 0.05), with Modality Selectivity showing the strongest ρ across categories.Edit Accuracy in Cat. B also has relatively strong correlation, while AV Consistency correlations are modest but consistent.Source: Figure A6 · p. 21, §C.3 · p. 21

C.4 Overall scores of AV editing models for each sub-category Cat. C

  • Figure A7 visualizes overall model scores for each Cat. C subcategory, while Table A5 reports Edit Accuracy by AV-coupled subcategory.The supplied passages identify the scope of these displays but do not provide their score values.Source: §C.4 · p. 21, Figure A7 · p. 22, Table A5 · p. 22

C.5 Audio-conditioned Video Editing with AVIEdit

The AVIEdit evaluation adds an audio-conditioned video-editing pipeline to AVENUE. It produces plausible video but does not consistently improve instruction fidelity or preservation of non-target visual content.

C.5 Audio-conditioned Video Editing with AVIEdit

  • AVIEdit requires an edited audio track and a spatial video mask; the pipeline uses Qwen3 to extract a target region, Grounded SAM-2 to localize it, and a full-frame mask if localization fails.ZETA supplies the edited audio.Source: §C.5 · p. 22
  • AVIEdit scored 39.2 on both Scene Change and Mood Change, but achieved 60.4 on Action Change, 50.5 on Object Replacement, and 77.0 on Speed Change.Its localized masks are less suited to global scene-level edits; performance was comparable to other evaluated models on the object-, action-, and speed-related categories.Source: §C.5 · p. 22
  • AVIEdit broadens evaluation to explicit audio-conditioned video editing and highlights a trade-off among edit accuracy, perceptual quality, and preservation.The authors conclude that its outputs are perceptually plausible but do not consistently improve instruction fidelity or preservation of non-target visual content.Source: §C.5 · p. 22
  • ZETA→AVIEdit achieved 73.9 Perceptual Quality for Video-targeted editing, above RAVE+ZETA (69.5) and AvED (53.2), but its Edit Accuracy was 65.0 versus 79.1 and 77.3.Under audio-targeted editing, its video Modality Selectivity was 84.6, below RAVE+ZETA (86.2) and AvED (87.5).Source: §C.5 · p. 22, Table A6 · p. 23

D Implementation Details

The implementation details specify default model configurations, output matching constraints, and the conditions used to assess audio-side editing performance. Evaluation time varies substantially across model families.

D Implementation Details

  • Models run under their proposed default configurations; AvED uses 4 fps, RAVE matches the source fps, and MMAudio inputs are truncated to 8 seconds.The 4 fps setting follows AvED’s 2×2 image-grid temporal shuffling, while the truncation matches MMAudio’s training duration.Source: D Implementation Details · p. 23
  • All outputs retain the source fps and are trimmed to the shortest output duration among models for the same sample.ZETA, SDEdit, MMAudio’s small_16k variant, and CoherentAVEdit operate at 16,000 Hz.Source: D Implementation Details · p. 23
  • On one NVIDIA A6000 GPU with batch size 1, audio-based models finish inference within 1 minute per sample, compared with about 20 minutes for AvED and 35 for RAVE.AvED’s timing is attributed to its 2×2 grid-based temporal shuffling mechanism.Source: D Implementation Details · p. 23
  • The audio-modality evaluation uses Cat. A for edit accuracy and perceptual quality, and Cat. B to assess audio preservation during video-targeted editing.For sequential paradigms, an original-source-video condition bypasses video editing to isolate the audio model’s capability from upstream artifacts.Source: D Implementation Details · p. 23

E Dataset License

VGGSound is released under CC BY 4.0, and AVENUE states that its use complies with the license terms.

E Dataset License

F Limitations

This section outlines AVENUE’s limitations, including possible LLM and source-data biases, manual verification that is difficult to scale, and restricted clip lengths and instruction styles. It also describes accompanying materials and sample-specific evaluation prompts that rate modality selectivity, perceptual quality, and audio-video consistency while distinguishing intended edits and preserved content from technical quality and cross-modal coherence.

F Limitations

  • AVENUE’s instructions rely on one multimodal LLM and manual verification, while noisy, roughly 10-second VGGSound clips limit quality and temporal-coherence assessment.The authors also note declarative-only prompts and analysis limited to sequential, joint, and separated architectures; automating verification, adding longer-form sources, and broadening paradigm coverage remain future work.Source: F Limitations · p. 24

G Broader Impacts

  • The authors release AVENUE and evaluation code to support reproducible comparisons across audio-targeted, video-targeted, and AV-coupled editing, while cautioning that dataset biases may limit real-world applicability.They report efforts to diversify scenarios and encourage care when applying evaluated models in real-world settings.Source: G Broader Impacts · p. 24

H Ethics Statement

  • The dataset uses VGGSound’s CC BY 4.0 test split, collects no personal data, and contains LLM-generated instructions without sensitive or harmful content.The authors acknowledge potential misuse of AV editing to create deceptive content and encourage responsible research use.Source: H Ethics Statement · p. 24

I LLM Usage

  • A multimodal LLM serves as the evaluation judge; LLMs also assisted with writing edits, result visualization, and data processing and filtering..Source: I LLM Usage · p. 24

J.1 Instruction Generation

  • Instruction generation asks an LLM to produce only plausible scene-specific edits in a structured format that records the source, target, intended changes, and preserved content.The target description is required to cover the full relevant modality or modalities, not merely the change.Source: §J.1 · p. 25

J.2 Category-aware editing taxonomy (category_taxonomies.txt)

  • The category-aware taxonomy distinguishes audio-only edits that preserve video, video-only edits that preserve audio, and coupled edits that change both modalities.Examples span sound-layer changes, visual attributes or style, and coordinated scene, object, or action changes across domains.Source: §J.2 · pp. 26–27

J.3.1 Edit Accuracy

  • Edit Accuracy prompts score whether intended changes were achieved and preserved elements remained intact, with separate rubrics for audio, video, and coupled edits.The coupled rubric considers whether both modalities were edited as intended and whether preserved elements were damaged.Source: §J.3.1 · pp. 29–31

J.3.2 Modality Selectivity

  • Modality Selectivity prompts check whether the unintended modality remains unchanged using general preservation questions and, for audio-targeted edits, a sample-specific check of human-verified preserved elements.The supplied audio-targeted visual checklist covers subject, motion, background, artifacts, and color or style; the audio-targeted prompt also asks whether specified preserved elements remain unaltered.Source: §J.3.2 · p. 32
  • The modality-selectivity prompt checks whether specified audio elements remain unaltered and scores six questions in total.For the preservation check, evaluators identify any specified elements that changed and explain how.Source: §J.3.2 · p. 33

J.3.3 Perceptual Quality

  • Perceptual-quality prompts rate audio, video, and coupled edits on a 1–5 scale, focusing on technical quality rather than edit correctness.Audio criteria include clarity, continuity, natural timbre, and artifacts; video criteria include sharpness, temporal consistency, natural appearance, and visual artifacts.Source: §J.3.3 · pp. 34–35

J.3.4 Audio-Video Consistency

  • The AV-consistency prompt assesses semantic coherence, temporal synchronization, and whether both modalities reflect the intended coupled edit, using a 1–5 rubric.Evaluators describe the audio and video, give specific coherence examples, and assess event timing.Source: §J.3.4 · p. 36

Related papers

Loading 2609.04253v1…