PaperlayerView exact revision on arXiv

Source-linked AI summary

AVIO: Learning to Add and Remove Sounding Objects in Audiovisual Scenes

Weihan Xu, Kan Jen Cheng, Koichi Saito, Jingyu Shi, Tingle Li, Yisi Liu, Liming Wang, Masato Ishii, Takashi Shibuya, Gopala Anumanchipalli, Paul Pu Liang

arXiv:2609.36503v1cs.AI

TL;DR

Localized audiovisual object editing requires changing a sounding object's visual presence and sound together, but paired supervision remains limited. The paper introduces AVIOBench and AVIO, a paired dataset and unified object editor. Evaluations report effective addition and removal, with reference guidance controlling appearance and placement for addition.

  • Problem

    Paired supervision for localized non-speech audiovisual editing is limited because edits must isolate overlapping sounds and target the same object across modalities.

    Source: §1 · p. 1

  • Method

    AVIOBench pairs object-level audiovisual edits, while AVIO adapts a pretrained audiovisual model with source-conditioned feature modulation and a reference-frame curriculum.

    Source: §1 · p. 2

  • Results

    Quantitative and qualitative evaluations demonstrate effective audiovisual object removal and addition, with competitive performance relative to existing editing baselines.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    Optional reference guidance provides appearance and placement control for addition, while the curriculum supports instruction-only editing.

    Source: §1 · p. 2, §7 · p. 9

    AVIOBench uses existing public datasets and inherits their limitations in recording quality, coverage, and diversity.

    Source: G LIMITATIONS AND FUTURE DIRECTIONS · p. 26

Abstract

from arXiv · show

Adding or removing a sounding object requires coordinated changes to visual content and sound while preserving the surrounding scene. Yet paired supervision for localized non-speech audiovisual editing remains limited, as visual and acoustic edits must target the same object and isolate its sound from overlapping sources. To address this gap, we introduce \textit{AVIOBench}, a dataset comprising 37.9 hours of paired audiovisual examples spanning 1{,}878 target-object names. AVIOBench links the visual presence and acoustic contribution of each target object through a shared identity and visual mask. Our automated pipeline uses visual grounding and cross-modal consistency to select target-sound removal candidates, then jointly refines the audiovisual pairs to improve perceptual quality and cross-modal consistency. Building on this dataset, we propose \textit{AVIO}, which adapts a pretrained text-to audiovisual generation model through source-conditioned feature modulation to jointly learn object addition and removal. A reference-frame curriculum gradually reduces reference conditioning during training, enabling one model to perform instruction-only editing with optional visual guidance. Quantitative and qualitative evaluations demonstrate effective audiovisual object removal and addition, with optional reference guidance providing appearance and placement control for addition.

1 INTRODUCTION

Localized audiovisual object editing must change a sounding object's visual presence and associated sound together while preserving unrelated scene content. The paper introduces AVIOBench and AVIO to address the paired-supervision gap and support object addition and removal.

1 INTRODUCTION

  • Paired supervision is limited because edits must isolate overlapping non-speech sounds and change the same object's presence in both modalities without altering unrelated content.Source: §1 · p. 1
  • The AVIOBench pipeline identifies a target object, selects a sound-removal candidate with strong audiovisual correspondence, and jointly refines edited pairs.Source: §1 · p. 2, Figure 2 · p. 3
  • AVIO adapts a pretrained text-to-audiovisual model with source-conditioned feature modulation and a reference-frame curriculum for text-only editing or optional visual guidance.Reference guidance controls appearance and initial placement for addition.Source: §1 · p. 2
  • AVIOBench contains 37.9 hours of paired audiovisual examples for adding and removing objects associated with non-speech sound events, constructed from AudioSetCaps and VGGSound (Bai et al., 2024; Chen et al., 2020).Source: §1 · p. 2

2 RELATED WORK

Prior work spans single-modality editing, audiovisual object editing, and joint audiovisual generation. AVIO focuses on paired non-speech object addition and removal, using source modulation and optional reference guidance.

Object Editing Datasets

  • Visual datasets OpenVE-3M and Goku [He et al., 2026] and audio resource AUDIT provide single-modality supervision, while audiovisual resources cover editing and cross-modal evaluation.AVIOBench focuses on non-speech sounding objects with aligned object-present and object-absent pairs for addition and removal.Source: §2 · p. 3

Audiovisual Joint Generation

  • Audiovisual generation has shifted from conditioning one modality on another toward joint generation, including MM-Diffusion and Transformer methods JavisDiT and JavisDiT++ [Liu et al., 2025; 2026].AVIO adapts a pretrained audiovisual generator through token-aligned source modulation and paired editing supervision.Source: §2 · p. 3

Audiovisual Joint Editing

3 AVIOBENCH DATASET

AVIOBench pairs audiovisual clips with object-level names and masks for sounding-object addition and removal. Its construction pipeline identifies a target, edits audio and video, and jointly refines the resulting pair.

3.1 DATA CONSTRUCTION PIPELINE

  • The dataset draws source videos from AudioSet and VGGSound, processes the first five seconds of each clip, and excludes clips containing only speech.The source datasets consist of 10-second YouTube clips; the pipeline processes 159 hours of video in total.Source: §3.1 · p. 4
  • For each clip, the pipeline obtains an audiovisual caption, identifies a primary sounding object, and uses its name to extract a video mask.Source: §3.1 · p. 4
  • Audio candidates are generated with object-name and visual-mask prompts across random seeds; the selected target stem has the highest ImageBind similarity to the mask-conditioned video.Candidates below an RMS amplitude of 0.005 are discarded, and the corresponding residual becomes the object-absent audio.Source: §3.1 · p. 4
  • The visual pair is formed by inpainting the target region, retaining clips only when the mean detected mask area falls by at least 75%.Source: §3.1 · p. 4
  • Edited audio and video are jointly refined with an empty-prompt SDEdit step, and exchanging the source and refined target forms removal and addition examples.The refined clip is denoted X\O.Source: §3.1 · p. 4

3.2 DATASET STATISTICS

  • AVIOBench contains 37.9 hours of unique paired examples, with object names and target-object masks; it covers 1,878 unique names.Source: §3.2 · p. 4

4 AVIO MODEL

AVIO adapts a pretrained text-to-audiovisual generator for joint object addition and removal, predicting an edited audiovisual target from a source pair and instruction, with optional reference-frame guidance.

4.1 INSTRUCTION-GUIDED AUDIOVISUAL EDITING

  • Source-conditioned feature modulation aligns source features with target tokens and uses them to guide editing while preserving unrelated source content.AVIO encodes video and audio latents, maintains their temporal alignment, and modulates intermediate hidden states with source features.Source: §4.1 · p. 5

4.2 REFERENCE-GUIDED CURRICULUM

  • A reference-guided curriculum progressively reduces reference availability so one model learns editing both with and without visual guidance.The reference is initially retained and phased out over 100K training steps; it specifies the added object's appearance and initial placement.Source: §4.2 · p. 6

4.3 TRAINING AND INFERENCE

  • AVIO jointly trains addition and removal using each pair in both directions at a 1:1 ratio, then jointly generates audio and video at inference.Training predicts modality-specific velocities with equal-weight losses; inference uses 50 Euler steps, with an edited reference optionally controlling addition appearance and placement.Source: §4.3 · p. 6

5 EXPERIMENTS

Experiments assess audiovisual object removal and addition using a standardized test protocol, comparing AVIO with unimodal and audiovisual editors. The qualitative comparison reports coordinated edits while preserving the surrounding scene.

5.1 DATASETS AND IMPLEMENTATION DETAILS

  • The experiments use 24,444 training pairs, 1,389 validation pairs, and 1,425 test pairs, with each training pair used in both directions.This yields 48,888 directed training examples for addition and removal.Source: §5.1 · p. 6

5.2 BASELINES

  • Comparisons include specialized audio and video editors, joint audiovisual editors, and cross-modal editors that use one modality to guide the other.AVI-Edit receives oracle target audio, while CoherentAVEdit receives oracle target video as conditioning.Source: §5.2 · pp. 6–7
  • AVIO edits objects and their sounds while preserving the surrounding scene, and matches the reference object's appearance and placement for addition.Figure 4 compares removal on the left with addition on the right; the audiovisual baselines receive oracle target video or audio as specified.Source: Figure 4 · p. 7

5.3 EVALUATION METRICS

  • Evaluation measures visual quality, audio quality, and audiovisual consistency using reconstruction, distribution, edit-success, and temporal-alignment metrics.The protocol includes PSNR, SSIM, LPIPS, FVD, LSD, FAD, IS, KL divergence, VES, AES, AVC, and DeSyncScore.Source: §5.3 · pp. 7–8

6 RESULTS

AVIO is evaluated on audiovisual object removal and addition, with reference-based visual and audio metrics and joint-editing measures. Results show strong performance across both tasks, alongside qualitative examples of coordinated edits that preserve the surrounding scene.

6.1 QUALITATIVE EXAMPLES

  • AVIO removes the dog with fewer visible remnants while suppressing its sound and preserving the scene; its helicopter addition matches the reference appearance and placement while generating associated sound.Source: §6.1 · p. 8

6.2 QUANTITATIVE RESULTS

  • 1.30 dB higher addition PSNR than the next-best method; AVIO also achieves the best addition PSNR and LPIPS and the best removal PSNR and LPIPS among audiovisual baselines.Source: §6.2 · p. 8
  • 1.36 KL divergence and 12.81 LSD for removal, the best among audiovisual baselines; addition achieves best KL divergence (1.29) and LSD (11.85), outperforming InstructAV2AV on all four audio metrics.Source: §6.2 · p. 8
  • 32.70% VES and 11.30% AES for removal; 40.10% VES, 77.90% AES, and 71.50% AVC for reference-guided addition, with AVIO highest in VES and AES across both tasks.Source: §6.2 · pp. 8–9, Table 1 · p. 9

7 CONCLUSION

The conclusion presents AVIOBench and AVIO for paired object-and-sound editing, while the reference-guidance ablation reports stronger addition quality and edit success with comparable reference-free removal video reconstruction. The paper also notes licensing and synthetic-content labeling considerations.

7 CONCLUSION

  • AVIOBench is a paired dataset for audiovisual object addition and removal, and AVIO is a unified editor for objects and their associated sounds.Source: §7 · p. 9
  • Token-aligned source modulation and a reference-frame curriculum support instruction-only editing and reference-guided addition with appearance and placement control.Source: §7 · p. 9
  • Edited outputs can alter the apparent content of recorded events, so they should be labeled synthetic or modified, particularly when shared outside research settings.Source: ETHICS STATEMENT · p. 11
  • AVIOBench, code, and a LangGraph-based data engine are planned for public release subject to applicable licenses.Source: REPRODUCIBILITY STATEMENT · p. 11

A ABLATIONS

  • The ablation compares reference-guided training with AVIO-NoRef and AdaLN source injection with AVIO-XAttn cross-attention.Source: A ABLATIONS · p. 15

A.1 REFERENCE-GUIDED CURRICULUM.

  • Removal video reconstruction remains comparable: PSNR changes from 20.09 to 20.06 and FVD from 206 to 191, while other audio and evaluator metrics trade off between variants.Source: §A.1 · p. 15
  • 19.22 dB PSNR (vs 16.65) and 317 FVD (vs 538); addition VES rises from 16.5% to 41.5% and AES from 77.0% to 80.0% with AVIO.Source: §A.1 · p. 15

A.2 SOURCE AUDIOVISUAL CONDITIONING.

The ablation favors source-conditioned feature modulation over source cross-attention when audiovisual quality, edit success, and consistency are considered together. However, some removal success scores favor cross-attention despite degraded outputs, so those scores alone do not establish better editing.

A.2 SOURCE AUDIOVISUAL CONDITIONING.

  • For removal, AVIO-XAttn has higher VES and AES than AVIO: 62.5% versus 37.5% and 20.0% versus 9.5%.The table’s Gemini-based VES, AES, and AVC evaluation covers 200 test clips; other metrics use the full test set.Source: §A.2 · p. 15, Table 2 · p. 16
  • Eighty percent of AVIO-XAttn removal clips were judged abstract or degraded textures, and 82% of its removal AVC passes had no checkable event.Degradation can remove unrelated content while passing absence checks, so removal success and consistency scores must be read alongside reconstruction quality.Source: §A.2 · p. 15
  • For addition, AVIO reaches higher VES and AES than AVIO-XAttn: 41.5% versus 10.0% and 80.0% versus 71.0%.AVIO-XAttn retains higher AVC, 85.5% versus 76.0%, but has lower addition success and poorer reconstruction.Source: §A.2 · p. 15, Table 2 · p. 16
  • Source-conditioned feature modulation is more effective than source cross-attention when quality, edit success, and audiovisual consistency are considered jointly.AVIO improves all four video quality metrics and several audio and alignment metrics in both tasks, though not every metric favors AVIO.Source: §A.2 · pp. 15–16, Table 2 · p. 16

B DATASET COMPARISON

The dataset comparison covers duration, modality, audio focus, and editing supervision. AVIOBench is described as the largest audiovisual paired dataset targeting general audio events.

B DATASET COMPARISON

  • AVIOBench is the largest audiovisual paired dataset targeting general audio events.The comparison spans duration, modality, audio focus, and editing supervision; durations count each clip or editing pair once.Source: B DATASET COMPARISON · p. 16, Table 3 · p. 16
  • The comparison marks documented attributes with ✓ and uses – for attributes not established or not applicable.Unverified durations are also denoted by –.Source: Table 3 · p. 16
  • Source and target refer to actual media, although targets may be pseudo-ground truth.Descriptions alone do not constitute editing instructions or paired target media; mask availability as dataset annotations is not established.Source: B DATASET COMPARISON · p. 16

C EVALUATION METRICS

The section defines a common evaluation protocol for audiovisual edits using complementary visual, acoustic, temporal-alignment, and LLM-judged criteria to assess target presence, source attribution, and audiovisual timing. It standardizes baseline checkpoints and text prompts, provides addition methods with a shared edited-frame reference, and uses oracle inputs for one modality in two baselines.

C EVALUATION METRICS

  • For a common comparison, outputs and pseudo-ground-truth targets are resampled to 81 frames at 256×256 and 16 fps, plus 81,000 mono audio samples at 16 kHz.Addition methods receive the same edited first frame as a reference; removal methods receive no target reference frame.Source: C EVALUATION METRICS · pp. 16–17

C.1 VISUAL AND ACOUSTIC QUALITY

  • Visual quality uses PSNR, SSIM, and LPIPS on corresponding frames, plus FVD on video feature distributions; higher PSNR and SSIM and lower LPIPS and FVD indicate better performance.The metrics capture pixel reconstruction, perceptual similarity, and video-distribution fidelity, respectively.Source: §C.1 · p. 17
  • Audio quality uses FAD, IS, KL divergence, and LSD; higher IS and lower FAD, KL divergence, and LSD indicate better performance.IS measures confident and diverse audio-event predictions rather than direct target-clip similarity; classifier-based evaluation uses CNN14 from PANNs.Source: §C.1 · p. 17

C.2 MODEL-BASED TEMPORAL ALIGNMENT

  • DeSyncScore estimates temporal offset in the first and last 4.8-second windows, averages offset magnitudes across clips, and reports seconds, with lower scores indicating better estimated alignment.It complements LLM judgments but does not determine whether the requested edit succeeds.Source: §C.2 · p. 17

C.3 LLM-BASED EDIT SUCCESS AND AUDIOVISUAL CONSISTENCY

  • LLM judgments independently score visual edit success, audio edit success, and audiovisual consistency over the full clip, with Yes rates reported separately for each criterion.Removal and addition are assessed against task-specific requirements; a high AVC score alone does not establish edit success, and clips without a checkable audiovisual event receive Yes to indicate no observed mismatch.Source: §C.3 · pp. 17–18
  • Gemini judges the evaluated clip with its audio and target description, without a source or reference clip; addition judgments therefore assess target presence and within-clip consistency, not appearance matching to a separate reference.Visual, acoustic, and audiovisual observations are assessed independently, and oracle pairing is used only for evaluation, with the oracle-supplied modality's edit-success score marked N/A.Source: §C.3 · pp. 17–18
  • Target-object and source success require the intended target to be present and audibly attributable, while allowing natural pauses and intermittent sounds.Unexplained bursts, incompatible sources, and unnatural dropouts or reappearances count as failures.Source: §C.3 · p. 21
  • Audiovisual timing is judged by matching audible events to visible actions throughout the clip, with clear lead-lag, drift, or unmatched sound-producing actions counting as mismatches.The protocol samples the beginning, middle, and end; one synchronized moment cannot offset a mismatch elsewhere.Source: §C.3 · p. 21
  • Timing may be marked Yes when no event is checkable, but absent targets still fail presence or source checks, and a coherent target with delayed sound can fail timing alone.The protocol distinguishes target presence, sound attribution, and timing rather than treating them as a single judgment.Source: §C.3 · p. 21

D BASELINE IMPLEMENTATION DETAILS

E.1 MODEL ARCHITECTURE AND INITIALIZATION

AVIO adapts a pretrained audiovisual transformer with token-aligned source conditioning, then jointly trains addition and removal using a shared model. Its inference procedure generates audio and video together, optionally conditioning on a reference frame.

E.1 MODEL ARCHITECTURE AND INITIALIZATION

  • AVIO adapts JavisDiT++ using a Wan2.1-1.3B DiT backbone initialized from JavisDiT-v1.0.The backbone has a hidden dimension of 1536, 30 layers, and 12 attention heads.Source: §E.1 · p. 22
  • Video and audio latents are patchified into concatenated tokens, while source patches align index-by-index with target tokens across modalities.If source duration or resolution differs, each modality is resampled to the corresponding token count.Source: §E.1 · p. 22
  • Source-conditioned modulation combines source tokens with the timestep embedding to produce token-wise modulation triples for self-attention, feed-forward, and text cross-attention.The pretrained self-attention and feed-forward heads are retained, and the new cross-attention modulation head is shared across blocks.Source: §E.1 · pp. 22–23
  • At initialization, the model is functionally identical to the loaded weights; the added modules total 7.23M parameters, or 0.32% of the 2.26B-parameter model.All 2.26B parameters are updated during fine-tuning, while the VAEs and text encoder stay frozen.Source: §E.1 · p. 23

E.2 TRAINING CONFIGURATION AND OBJECTIVE

  • AVIO jointly trains addition and removal with shared parameters and uses one checkpoint for both tasks, without task-specific training or checkpoint selection.The video and audio rectified-flow losses are averaged over their latent elements and summed with equal weights.Source: §E.2 · p. 23
  • Training jointly predicts both targets using rectified flow, with one timestep shared across modalities, independent modality noise, and summed per-element mean squared error.Text and source conditions are independently dropped with probability 0.1; when text alone is dropped, targets are replaced by source latents.Source: §E.2 · p. 24

E.3 INFERENCE AND SAMPLING

  • At inference, AVIO uses the conditional prediction directly, optionally prepends a reference frame, and takes 50 explicit Euler steps jointly for video and audio.The outputs are decoded by the pretrained video and audio decoders.Source: §E.3 · p. 24

F.1 DATASET STATISTICS

AVIOBench’s target categories are defined by sound source, with music and instruments, animals, and vehicles making up most samples. The dataset pipeline verifies audiovisual removal candidates through staged filtering and jointly refines edited video and residual audio.

F.1 DATASET STATISTICS

  • Targets are grouped by sound source using ten top-level AudioSet categories, rather than by visual object class.Labels come from the audio-separation pipeline, so names can describe acoustic events as well as on-screen things.Source: §F.1 · p. 24
  • Music and instruments (27.1%), animals (26.5%), and vehicles (20.9%) account for 74.5% of the 27,258 samples and 1,878 distinct target names.Human voice and activity account for 10.0%, machines and tools 6.2%, natural sounds 4.5%, and alarms, household objects, and impacts together 3.7%.Source: §F.1 · p. 25

F.2 DATASET PIPELINE ANALYSIS

  • 46.0% of scored clips are retained at visual removal verification, the pipeline stage with the largest reduction.Table 4 reports stage-wise and cumulative retention; bounds reflect incomplete intermediate records.Source: Table 4 · p. 25, §F.2 · p. 25
  • Visual filtering requires the target in the first frame and at least a 75% decrease in its post-edit SAM3 mask area; audio filtering checks stem RMS and ImageBind consistency.Separation and inpainting must use the same object mask, and the target’s ImageBind similarity must exceed the residual’s by more than 0.05.Source: §F.2 · p. 25
  • Clips at or below 37.6 fps are retained so 189 native frames cover the five-second window.An additional 5,157 inpainted clips could not be scored and were discarded before removal verification; filter bounds were recovered from job logs.Source: §F.2 · pp. 25–26

F.3 AGENTIC DATA ENGINE

  • A modular LangGraph engine orchestrates models for audiovisual description, target identification, visual removal, sound separation, candidate ranking, and joint audiovisual refinement.It supports conditional rejection, GPU-job sharding, checkpointed reuse, replaceable workers, and stage-level agentic optimization with budget checks and human review.Source: §F.3 · p. 26

G LIMITATIONS AND FUTURE DIRECTIONS

The paper identifies limits in evaluation interpretation, pseudo-ground-truth quality, and the source data’s coverage and diversity. It describes the data engine as a reusable workflow for expanding paired supervision.

G LIMITATIONS AND FUTURE DIRECTIONS

  • The Oracle–Oracle removal reference receives an AES of 11.10%, but aggregate results cannot determine whether residual sounds, overlapping sources, or judge errors explain the score.The authors complement LLM judgments with reconstruction, perceptual, distributional, and synchronization metrics.Source: G LIMITATIONS AND FUTURE DIRECTIONS · p. 26
  • Automated filtering and joint refinement do not prevent pseudo-ground-truth targets from containing incomplete removals or editing artifacts, limiting reference-based metrics as measures of editing quality.The proposed generative-prior explanation for examples that appear better than pseudo-targets remains a hypothesis, not a systematic result.Source: G LIMITATIONS AND FUTURE DIRECTIONS · p. 26
  • AVIOBench inherits recording-quality, coverage, and diversity limitations from its existing public datasets.The authors propose using the reusable data engine with newer, higher-quality, and more diverse recordings to expand the dataset.Source: G LIMITATIONS AND FUTURE DIRECTIONS · p. 26
Loading 2609.36503v1…