PaperlayerView exact revision on arXiv

Source-linked AI summary

CrossEdit: Cross-Modal Training Enables Rich Audio-Visual Editing

William Chen, Prem Seetharaman, Ke Chen, Oriol Nieto, Kevin Duarte, Siddharth Srinivasan Iyer, Mamshad Nayeem Rizve, Zhiwen Cao, Shinji Watanabe, Yuanjun Xiong, Jianming Zhang, Zeyu Jin, Justin Salamon

arXiv:2610.10264v1cs.SDeess.ASeess.IV

TL;DR

Because generative models may not match users’ desires in one attempt, iterative AV edits are especially difficult: changes must synchronize picture and sound while preserving everything else. CrossEdit combines synthetic compositional audio-editing pairs with curated cross-modal tasks, including self-supervised AV masked reconstruction, to train an omni-modal editor. CrossEdit outperforms cascaded, sampling-based, agentic, and state-of-the-art end-to-end models on zero-shot AV movie-scene editing while maintaining or improving performance on lip-synced speech, image, and video editing.

  • Problem

    Because generative models may not match users’ desires in one attempt, iterative AV edits are especially difficult: changes must synchronize picture and sound while preserving everything else.

    Source: §1 · p. 1

  • Method

    CrossEdit combines synthetic compositional audio-editing pairs with curated cross-modal tasks, including self-supervised AV masked reconstruction, to train an omni-modal editor.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    CrossEdit outperforms cascaded, sampling-based, agentic, and state-of-the-art end-to-end models on zero-shot AV movie-scene editing while maintaining or improving performance on lip-synced speech, image, and video editing.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The results support zero-shot AV scene editing through cross-modal transfer of instruction following, alongside maintained or improved performance on trained editing tasks.

    Source: §6 · p. 10

    The model was tested on short clips up to 10 seconds, and long-form AV editing remains an open challenge.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

Multimedia editing requires generative models to interpret complex, compositional instructions and modify only the desired elements across modalities, a capability largely beyond existing systems. Current approaches are typically limited to simple, single-attribute edits within one modality, since diverse, high-quality editing pairs are hard to obtain, especially for cross-modal tasks where aligned audiovisual (AV) data is scarce. We present CrossEdit, a unified omni-modal editing model for images, audio, and video that performs AV movie scene edits zero-shot through cross-modal transfer. Our key observation is that complex instruction-following, once learned in any modality, generalizes to others. We exploit the additive nature of acoustic signals to procedurally generate large-scale audio editing pairs with compositional instructions, and show that fine-tuning on this synthetic data and self-supervised AV masked reconstruction, alongside a targeted set of curated cross-modal tasks, induces instruction-following that transfers zero-shot to unseen modality and instruction combinations. To evaluate this capability, we release CrossEditBench, a human-annotated benchmark of AV edits on movie scenes, and propose AV-FES, a metric that jointly scores instruction following and consistency. We show that our proposed techniques improve performance on zero-shot AV movie scene editing, while maintaining or improving performance on lip-synced speech editing and standard image, video, and audio editing benchmarks. Demos are available at https://wanchichen.github.io/crossedit/.

1 INTRODUCTION

Multimedia editing must handle iterative, compositional changes while preserving unrelated content, a challenge especially acute when audio and video must change together and stay synchronized. CrossEdit uses synthetic audio editing data and curated cross-modal tasks to transfer instruction following zero-shot to AV movie-scene edits, evaluated with CrossEditBench and AV-FES.

1 INTRODUCTION

  • Compositional AV edits must change the requested picture and sound, keep them synchronized, and leave everything else untouched.For example, adding a motorbike to a street scene should change both picture and sound.Source: §1 · p. 1
  • Existing editing models are largely siloed by modality, while few multimodal systems support more than simple, single-attribute edits.Compositional AV-to-AV editing pairs are rare in natural data and expensive to curate.Source: §1 · p. 1
  • CrossEdit’s approach rests on the hypotheses that instruction following transfers across modalities and masked AV reconstruction can serve as self-supervised AV editing.Acoustic signals are additive, allowing large-scale simulated audio-editing pairs to be generated.Source: §1 · p. 2
  • CrossEdit combines synthetic audio editing pairs with curated cross-modal tasks to transfer instruction following to zero-shot AV movie-scene editing.The paper introduces CrossEditBench, a human-annotated benchmark, and AV-FES, a metric for editing performance.Source: §1 · p. 2

2 RELATED WORK

Prior work has advanced multimodal generation and instruction-based editing, but AV editing systems remain narrower than CrossEdit’s target of free-form, zero-shot movie-scene edits. CrossEdit positions its approach as transferring instruction following across modalities rather than relying on paired AV editing data for the target task.

2 RELATED WORK

  • Recent synchronized audio-video generators can produce multimodal outputs, but the cited systems are described as generation-only rather than editing systems.The related-work passage names Seedance 2.0, LTX-2, and UniAVGen in this category.Source: §2 · p. 2
  • Earlier instruction-based editing developed within individual modalities, while early AV-to-AV approaches used cascades or sampling tricks applied to unimodal models.The related-work passage describes subsequent end-to-end AV editing systems as targeting narrower settings.Source: §2 · pp. 2–3
  • CrossEdit differs from described concurrent AV editors by targeting free-form zero-shot movie-scene edits without target-task AV editing pairs.The passage says SpongeBob and InstructAV2AV target simpler edits, while CrossEdit is omni-modal; it also identifies MiniMax H3 as a unified image, audio, and video generation-and-editing model.Source: §2 · p. 3

3 TRAINING FOR CROSS-MODAL TRANSFER

CrossEdit is trained for cross-modal transfer using a shared omni-modal editing model and complementary data sources: mined lip-synced speech pairs, synthetic audio editing pairs, and AV masked reconstruction. Its training setup reserves complex AV scene instructions for zero-shot evaluation rather than AV-to-AV training.

3 TRAINING FOR CROSS-MODAL TRANSFER

  • CrossEdit acquires zero-shot AV scene editing from mined lip-synced speech pairs, synthetic audio editing pairs, and AV masked reconstruction on a shared backbone.These are the three data sources identified for training.Source: §3 · p. 3

3.1 ARCHITECTURE

  • The model represents editing as a conditional distribution over output sequences given input media and a natural-language instruction.The input and output sequences can vary in number and modality across examples.Source: §3.1 · p. 3
  • CrossEdit uses separate pretrained VAEs for audio and visuals, semantic embeddings for text and visual inputs, and a DiT mixture-of-experts backbone.The backbone has 3B active and 12B total parameters.Source: §3.1 · p. 4

3.2 MINING LIP-SYNCED SPEECH EDITING PAIRS

  • Mined lip-synced speech pairs are formed by grouping retained video segments in which the same on-screen speaker says different sentences.The mining pipeline filters source video for speech, segments and transcribes it, and tracks visible faces.Source: §3.2 · p. 4

3.3 AUDIO EDITING DATA SYNTHESIS

  • Synthetic audio editing data combines random mixtures of real audio files with LLM-generated scenes to cover compositional instructions and diverse acoustic scenes.The mixtures are acoustically plausible but semantically incoherent; generated scenes are more natural but use synthetic audio.Source: §3.3 · p. 4

3.4 AUDIO-VISUAL MASKED RECONSTRUCTION

  • AV masked reconstruction trains the model to inpaint clips after temporal, spatial, or combined masking because audio and mined pairs alone do not require new visual content within an existing scene.The model reconstructs the unmasked clip given the instruction “inpaint this video”.Source: §3.4 · pp. 4–5

3.5 TRAINING

  • CrossEdit starts from CrossEdit-Base and is fine-tuned for 10K steps with synthetic audio editing and masked reconstruction data.Its zero-shot AV edits use instructions absent from AV-to-AV training; only lip-synced speech editing and masked inpainting are AV-to-AV training types.Source: §3.5 · p. 5

4 ZERO-SHOT AUDIO-VISUAL SCENE EDITING

This section evaluates zero-shot AV scene editing with CrossEditBench and AV-FES. CrossEdit leads the reported systems, while ablations examine the effects of its training components.

4.1 CROSSEDITBENCH

  • CrossEditBench contains 100 public-domain movie clips, with human edit instructions and modality-specific descriptions of intended changes.Its six categories distinguish target modality and whether an edit must be synchronized across modalities.Source: §4.1 · pp. 5–6

4.2 AV-FES: AUDIO-VISUAL FAITHFUL EDIT SCORE

  • AV-FES averages modality-specific harmonic-mean scores balancing instruction gain over the untouched input with consistency to the input.Instruction gain is normalized by the available improvement over the unedited clip, while consistency ratings are mapped to the unit interval.Source: §4.2 · pp. 6–7

4.3 RESULTS

  • CrossEdit leads AV-FES, V-FES, and A-FES, and beats MiniMax H3 in AV-FES (0.30 vs. 0.23) at 12B vs. 33B parameters.Its advantage is driven by the strongest instruction following in both modalities.Source: Table 2 · p. 7, §4.3 · p. 7

4.4 ABLATION OF TRAINING METHODS

  • Synthetic audio editing raises human-rated AV-FES from 0.26 to 0.35; adding AV masked reconstruction improves all four axes, reaching 0.43.Audio editing data also improves visual consistency on lip-synced speech editing.Source: §4.4 · pp. 7–8, Table 3 · p. 8
  • CrossEdit improves every instruction category, with largest gains for synced scenes (0.41 to 0.62), speech+speaker (0.15 to 0.46), and unsynced video (0.23 to 0.48).Audio editing data alone hurts synced audio and video scores (0.31 to 0.18 and 0.19 to 0.06); AV masked reconstruction recovers them to 0.34 and 0.35.Source: Figure 5 · p. 8, §4.4 · p. 8

5 RETAINING OMNI-MODAL PERFORMANCE

CrossEdit’s training improves several editing measures while retaining strong performance on lip-synced speech, video, and image editing. The section compares CrossEdit primarily with CrossEdit-Base, with dedicated models as references.

5 RETAINING OMNI-MODAL PERFORMANCE

  • The section focuses primarily on CrossEdit versus CrossEdit-Base; lip-synced speech editing is evaluated on 600 internal celebrity interviews, with audio-editing results deferred to the appendix.The speech-editing setup measures transcription, speaker similarity, visual consistency, AV synchronization, and human ratings.Source: §5 · p. 8, §5.1 · p. 8

5.1 LIP-SYNCED SPEECH EDITING

  • For lip-synced speech editing, SFT improves speaker similarity, video input consistency, and LSE-D, while human raters strongly prefer both CrossEdit models.F5-TTS remains stronger on WER and speaker similarity, but its cascades cannot preserve the input video; SFT mainly improves visual-background preservation.Source: §5.1 · pp. 8–9, Table 4 · p. 9

5.2 VIDEO EDITING

  • CrossEdit improves 5 of 6 video-editing metrics over CrossEdit-Base and remains in the range of dedicated models such as Lucy Edit.The authors hypothesize that synthetic audio data exposes the model to more diverse instructions.Source: §5.2 · p. 9, Table 5 · p. 9

5.3 IMAGE EDITING

  • Image editing’s overall score rises from 3.69 to 3.72 over CrossEdit-Base, with gains in Action, Adjust, Hybrid, and Remove.Both models remain competitive with dedicated image models of similar or larger size.Source: §5.3 · p. 9, Table 6 · p. 9

6 CONCLUSION

CrossEdit combines synthetic audio editing data and AV masked reconstruction to support zero-shot editing of real movie scenes, while retaining or improving performance on other editing tasks. The authors describe evaluation and reproduction resources, but withhold code and checkpoints because of misuse risks and note that long-form AV editing remains open.

6 CONCLUSION

  • CrossEdit outperforms cascaded, sampling-based, agentic, and state-of-the-art end-to-end models on zero-shot AV scene editing while maintaining or improving performance on other editing tasks.The reported retained tasks include lip-synced speech, image, and video editing.Source: §6 · pp. 9–10
  • Cross-modal transfer from synthetic audio editing data and AV masked reconstruction supports CrossEdit’s zero-shot scene-editing capability.AV masked reconstruction is framed as a self-supervised editing task.Source: §6 · p. 10
  • CrossEditBench and AV-FES provide an evaluation set and metric for instruction-based AV movie-scene editing.The authors also plan to release the annotated benchmark set, annotator instructions, judge prompts, and model outputs.Source: §6 · p. 10, REPRODUCIBILITY STATEMENT · p. 11
  • Evaluations do not cover every scenario, and the model was trained and tested only on clips up to 10 seconds; long-form AV editing remains open.The authors note that training on audio up to 30 seconds does not resolve this open challenge.Source: ETHICS STATEMENT · p. 11
  • The authors will not publicly release code or checkpoints because of realistic-deepfake risks, while describing components needed to reproduce and evaluate CrossEdit.The described materials cover the model, training data and setup, and evaluation protocols.Source: REPRODUCIBILITY STATEMENT · p. 11

A ARCHITECTURE

CrossEdit uses separate pretrained visual and audio encoders to produce latents, with clean input latents and semantic embeddings conditioning a shared latent-diffusion backbone that denoises output latents.

A ARCHITECTURE

  • CrossEdit uses frozen, modality-specific tokenizers and a latent-diffusion backbone; separate pretrained VAEs encode audio, video, and images.Images are treated as single-frame video by the visual VAE.Source: A ARCHITECTURE · p. 19
  • The audio DAC-VAE encodes 48 kHz stereo audio into 256-dimensional continuous latents at 40 Hz.It was pretrained on over 100,000 hours of speech, sound effects, and music.Source: A ARCHITECTURE · p. 19
  • The visual VAE normalizes video to 540p at 24 FPS and maps it to 48-channel latents at 6 FPS; a 540p image yields one 48-channel latent frame.The video latents have spatial dimensions of 60×34.Source: A ARCHITECTURE · p. 19
  • The backbone conditions on clean Qwen3-VL semantic embeddings and input VAE latents, then denoises output VAE latents.It is a diffusion transformer with mixture-of-experts layers.Source: A ARCHITECTURE · p. 19

B TRAINING DETAILS

AV masked reconstruction trains CrossEdit to recover a clean clip from a masked input using the instruction “inpaint this video.” The training mixture also replaces some video-to-video and lip-synced speech samples with this task.

B.1 AUDIO-VISUAL MASKED RECONSTRUCTION

  • The spatial scheme samples up to 12 masks of 200×200 pixels per frame for 540p videos.The authors report that S=12 worked well for their 540p videos.Source: §B.1 · p. 19
  • AV masked reconstruction uses temporal, spatial, or combined spatio-temporal masking, then conditions on the masked clip to generate the clean clip.All three schemes use the same instruction, “inpaint this video”; temporal masking zeros corresponding audio with 50% probability.Source: §B.1 · p. 19

B TRAINING DETAILS

  • Video-to-video and lip-synced speech editing samples are replaced by AV masked reconstruction with 50% probability in the data mixture.The table describes the mixture by total tokens and samples per batch.Source: Table 7 · p. 20

B.2 PRE-TRAINING AND SUPERVISED FINE-TUNING

CrossEdit starts from an omni-modal generation-and-editing checkpoint, adds supervised fine-tuning data, and uses separate guidance for input fidelity and instruction adherence. CrossEditBench evaluates edits on annotated movie clips with varied audiovisual requirements.

B.2 PRE-TRAINING AND SUPERVISED FINE-TUNING

  • CrossEdit-Base is pre-trained for text-to-modality generation, conditional generation, and editing across image, video, audio, and audiovisual tasks.Its editing tasks include image-to-image, video-to-video, audio-to-audio voice cloning, and audiovisual lip-synced speech editing.Source: §B.2 · p. 20
  • CrossEdit is obtained by adding synthetic audio editing and audiovisual masked reconstruction data, then fine-tuning for 10K steps.The fine-tuning run takes around 2.5 days on 64 H100 GPUs or 5 days on 64 A100 GPUs.Source: §B.2 · p. 20

B.3 CLASSIFIER-FREE GUIDANCE AND INFERENCE

  • Independent dropout of the input and instruction enables classifier-free guidance with separate scales for input fidelity and instruction adherence.Inference uses 50 sampling steps, with both scales set to 5.0 for video and 3.0 for audio unless stated otherwise.Source: §B.3 · p. 20

C.1 CROSSEDITBENCH

  • CrossEditBench annotates 100 public-domain movie clips with instructions and modality-specific changes spanning edits with and without audiovisual synchrony.Examples include adding a barking dog across modalities, adding footsteps synchronized with a person walking, and removing music.Source: §C.1 · p. 21

C.2 SUBJECTIVE EVALUATION

The subjective evaluation uses crowdsourced ratings of visual and audio instruction following and consistency. AV-FES combines those dimensions to reward requested edits while penalizing unintended changes.

C.2 SUBJECTIVE EVALUATION

  • 86 of 100 recruited raters remained after excluding 14 who gave the unedited control low consistency scores; each rated four clips per system on four 1–5 axes.The four axes were visual instruction following, visual consistency, audio instruction following, and audio consistency.Source: §C.2 · p. 21

C.3 AV-FES DETAILS

  • AV-FES addresses the weakness of judging instruction following or consistency alone: either an untouched input or a full re-synthesis can score well on one axis.The evaluation included the unedited input as a control to validate rater attention.Source: §C.2 · p. 21, §C.3 · p. 24
  • AV-FES combines instruction gain and consistency with a harmonic mean, which is high only when both are high.An untouched copy scores zero because its instruction gain is zero; full re-synthesis is reduced by low consistency.Source: §C.3 · p. 24
  • For single-modality instructions, AV-FES rewards editing the requested modality and penalizes collateral changes in the other modality.The metric is bounded in [0, 1], but the judge’s consistency scores cap the empirical ceiling for a flawless editor at about 0.86 without affecting rankings.Source: §C.3 · p. 24

C.4 ROBUSTNESS TO THE CHOICE OF JUDGE

CrossEdit’s AV-FES advantage over baselines persists across two LLM judges, whose rankings also broadly agree with human ratings. The appendix notes a specific judge disagreement and illustrates visual-consistency effects in lip-synced speech editing.

C.4 ROBUSTNESS TO THE CHOICE OF JUDGE

  • Without the unedited input, Gemini and Qwen system rankings correlate at AV-FES Pearson 0.87 and Spearman 0.80, but their scores should be compared only within one judge.Qwen scores span roughly two to three times Gemini’s range; both rank CrossEdit first and CrossEdit-Base second among reported systems.Source: §C.4 · p. 24
  • Both LLM judges rank systems much as humans do, but Qwen’s visual-consistency ranking of CrossEdit versus cascades differs from human ratings; the small system samples make correlations coarse.The comparison covers four systems shared with Gemini and five shared with Qwen.Source: §C.4 · p. 24

C.5 CONFIDENCE INTERVALS FOR LLM-JUDGE SCORES

  • CrossEdit’s AV-FES exceeds every baseline under both judges, with margins over most systems outside the confidence intervals and a smaller but consistent margin over MiniMax H3.The intervals are reported over 100 CrossEditBench clips.Source: §C.5 · p. 24, Table 9 · p. 25, Table 10 · p. 25

C.6 LIP-SYNCED SPEECH EDITING

  • Lip-synced speech editing generates new speech with matching lip movements; Figure 9 shows synthetic audio data correcting unwanted visual changes caused by inconsistent mined audiovisual pairs.The illustrated mined pairs differ in text overlay and lighting, prompting hallucinated changes without the correction.Source: §C.6 · p. 25, Figure 9 · p. 25

D AUDIO EDITING RESULTS

CrossEdit is evaluated on synthetic complex scenes, real-world audio scenes, and movie-trailer source separation. It outperforms prior work on all three test sets, with its largest reported gain on AudioCapsEdit.

D AUDIO EDITING RESULTS

  • The evaluation covers free-form open-vocabulary StoryGenEval scenes, real-world AudioCapsEdit scenes without ground-truth outputs, and movie-trailer source separation without ground-truth outputs.Movie Separation tasks extract or remove ambience, music, dialogue, and sound effects.Source: D AUDIO EDITING RESULTS · p. 26
  • The primary metrics are ΔFLAM for input consistency and editFLAM for instruction following; Movie Separation also uses SAM-Audio-Judge.Comparisons include AudioChat and SAM-Audio Large; extraction instructions are inverted for SAM-Audio because it supports sound removal only.Source: D AUDIO EDITING RESULTS · p. 26
  • 40.3 vs. 15.5 editFLAM over AudioChat is CrossEdit’s largest reported gain, and CrossEdit outperforms prior work on all three test sets.The authors attribute the AudioCapsEdit gain to large-scale synthetic mixtures and report that CrossEdit also surpasses SAM-Audio on source separation.Source: D AUDIO EDITING RESULTS · p. 26, Table 14 · p. 27
Loading 2610.10264v1…