PaperlayerView exact revision on arXiv

Source-linked AI summary

SwanWeave:One-Stage Multi-Task Instruction-Guided 3D Spatial Audio Editing

Ke Lei, Chenyuhao Wen, Yu Zhang, Wenxiang Guo, Changhao Pan, Sashuai Zhou, Yongshi Li, Ruiqi Li, Ruofan Hu, Haorui Xu, Xiang Yin, Zhou Zhao

arXiv:2609.04975v1cs.SD

TL;DR

Instruction-guided 3D spatial audio editing must change specified parts of an existing FOA scene while preserving the rest and coordinating events, spatial cues, dynamics, and environmental information. SWANWEAVE addresses this with paired FOA supervision, dual-level expert routing, preference alignment, and staged training, and reports better editing quality than general audio and spatial-audio baselines across all tasks.

  • Problem

    Existing language-guided editors mainly target conventional audio or sequential operations and do not directly support one-stage editing of complex 3D spatial instructions.

  • Method

    SWANWEAVE trains on paired source-instruction-target FOA supervision and combines SE-MoE dual-level routing, SPO, and staged training for heterogeneous spatial edits.

  • Results

    SWANWEAVE achieves better editing quality than existing general audio editors and spatial audio baselines across all tasks.

  • Takeaways & Limitations

    Directly modeling FOA spatial audio supports controllable 3D audio creation across audio-event, spatial, dynamic, and environmental edits.

  • Takeaways & Limitations

    The current scenes are approximately 10 seconds long and contain relatively few sound events, limiting coverage of longer, denser, and more open-ended scenes.

Abstract

from arXiv · show

Spatial audio editing modifies an existing soundfield according to a user's instruction while preserving the rest of the scene. Unlike conventional audio editing, it must reason jointly about audio events, spatial information, dynamic changes, and environmental information in first-order Ambisonic (FOA) waveforms. Existing language-guided editors mainly target conventional audio or rely on sequential operations, and therefore do not directly support one-stage editing for complex 3D spatial instructions. We present SwanWeave, the first one-stage multi-task framework for instruction-guided 3D FOA spatial audio editing. We build paired FOA supervision from open-source speech and sound-effect corpora using controllable room simulation, covering more than ten single-operation and compound tasks across the four editing axes. To handle this heterogeneous edit space, SwanWeave uses Spatial Edit Mixture-of-Experts (SE-MoE) with dual-level routing, selecting task-aware expert combinations for compound instructions and frame-level routed/null experts for local edit decisions. We further introduce Spatial Preference Optimization (SPO), a Direct Preference Optimization (DPO)-based alignment objective with edit-specific negative targets, and adopt staged training to improve natural-language grounding. Experiments show that SwanWeave achieves better editing quality than existing general audio editors and spatial audio baselines across all tasks. Spatial audio editing demos can be found at https://swanaigc.github.io/#swanweave, code can be found at: https://github.com/MM-Speech/SwanWeave.

1 Introduction

SWANWEAVE addresses one-stage instruction-guided editing of existing 3D FOA scenes, where edits must jointly handle events, spatial cues, dynamics, and environmental information while preserving unchanged content. It combines paired FOA supervision, dual-level expert routing, preference optimization, and staged training, achieving better editing quality than existing baselines across all tasks.

  • Motivation: Spatial audio editing modifies an existing scene while preserving untargeted content across audio events, spatial layout, motion, and room impression.Natural-language instructions let users specify targeted changes without describing a complete target scene.
  • Challenges: Existing editors mainly target conventional audio, while SmartDJ uses stereo sequential operations that do not directly learn compound 3D spatial edits.Sequential decomposition can accumulate errors across stages.
  • Approach: SWANWEAVE is the first one-stage multi-task framework for instruction-guided 3D FOA spatial audio editing.Its supervision covers more than ten single-operation and compound tasks spanning audio events, spatial information, dynamic changes, and environmental information.
  • Approach: SE-MoE combines instruction-level task experts with frame-level routed/null experts for compound instructions and local edit decisions.This dual-level routing is designed to improve one-stage editing across heterogeneous tasks and events.
  • Approach: SPO uses edit-specific negative targets in a DPO-based alignment objective, while staged training improves natural-language grounding for complex spatial edits.Training progresses from text-to-FOA pretraining to single-operation and compound-edit training.
  • Results: SWANWEAVE achieves better editing quality than existing general audio editors and spatial audio baselines across all tasks.The result is reported for the proposed one-stage multi-task framework and its combined training strategy.

2 Related Work

Prior work covers conventional audio editing, spatial generation, spatialization, and representation-level manipulation, but does not directly edit an existing 3D sound scene from an instruction. SWANWEAVE instead applies requested event, spatial, dynamic, or environmental changes while preserving the rest of the scene.

  • Conventional audio editing: Language-guided conventional-audio editors generate edited outputs for operations such as adding, dropping, replacement, inpainting, and super-resolution.AUDIT uses triplets of input audio, instruction, and edited audio, while ZETA, ZEUS, and AudioEditor explore inversion-based editing.
  • Spatial audio research: Spatial audio research addresses generation, spatialization, vocoding, localization, and environmental cues using visual, geometric, binaural, or text-conditioned inputs.These methods support spatial understanding or synthesis rather than direct instruction-guided editing of an existing 3D scene.
  • Positioning: SWANWEAVE directly edits an existing spatial mixture and preserves the rest of the scene after applying audio-event, spatial, dynamic, or environmental instructions.This distinguishes it from prior methods that focus on spatial understanding, localization, spatialization, or generation.

3 Method

SwanWeave formulates spatial audio editing as direct source-to-target transformation of FOA scenes, covering coupled edits across audio events, spatial attributes, dynamics, and room acoustics. Its latent flow-matching editor combines staged training with SE-MoE dual-level routing to preserve unchanged regions while handling heterogeneous and compound instructions.

  • Problem formulation: The task axes cover audio events, spatial information, dynamic changes, and environmental information, whose effects are coupled in the waveform.These include event insertion or removal, source location, time-varying motion or gain, and room-acoustic changes.
  • Problem formulation: SwanWeave directly predicts the final edited FOA scene instead of executing compound instructions as sequential operations.Each target contains the complete acoustic consequence of the requested changes.
  • Data construction: Training triplets pair a source FOA waveform, instruction, and final target FOA waveform across more than ten single-operation and compound tasks.The dataset contains about 125K triplets per single-operation task and 250K compound-edit triplets, with approximately 8.9-second examples.
  • Training: Staged training initializes text-to-FOA grounding, then trains single-operation edits before compound edits to build atomic abilities progressively.The curriculum targets grounding for events, spatial layout, motion, and room acoustics before joint editing.
  • Model overview: The model edits compact FOA latents with an encoder, instruction encoder, and flow-matching latent editor rather than reconstructing long four-channel waveforms directly.Source and target waveforms are mapped to latent representations, and the instruction enters through cross-attention.
  • Spatial Edit MoE with dual-level routing: SE-MoE combines instruction-level task experts with frame-level routed and null experts for task-aware combinations and local preservation.Top-p routing selects expert sets adaptively; null experts contribute zero output for unchanged regions.

4 Experiments

Experiments evaluate SwanWeave against general audio and spatial audio baselines using semantic, reconstruction, instruction-following, spatial, efficiency, and human-preference measures. Ablations show benefits from FOA pretraining, dual-level expert routing, SPO, and one-stage compound editing.

  • Evaluation setup: The evaluation combines distributional, event-level, spectral, instruction-following, spatial-fidelity, and human-preference measures, with FOA outputs rendered to stereo for comparison with stereo baselines.FD, FAD, KL, and LSD assess audio quality; CLAP assesses instruction following; GCC, CRW, and FSAD assess spatial fidelity.
  • Main results: 1.17s average inference time is faster than all baselines, while SwanWeave achieves the best overall efficiency, semantic quality, reconstruction quality, and spatial fidelity.The authors attribute the speed advantage to one-stage latent editing rather than iterative diffusion sampling or sequential editing.
  • Main results: SwanWeave consistently outperforms all baselines on GCC, CRW, and FSAD, indicating stronger preservation of directional structure and stereo spatial cues.The evaluation supports the advantage of directly modeling FOA audio instead of relying only on stereo-domain editing.
  • Motion editing: For angle motion, GCC falls from 23.25 to 1.15 and CRW from 50.16 to 14.84 versus SmartDJ; for distance motion, GCC falls from 16.45 to 1.74 and CRW from 28.66 to 11.32.These comparisons target dynamic edits requiring temporal changes in direction and distance.
  • Ablations: Removing FOA-caption pretraining worsens FD from 5.47 to 8.49, KL from 1.67 to 2.78, and CLAP from 0.21 to 0.16.The ablation indicates that text-to-FOA pretraining improves initialization, semantic alignment, reconstruction quality, and spatial accuracy before editing-specific training.
  • Ablations: Removing SE-MoE or SPO degrades performance, while one-stage editing improves FD, FAD, LSD, GCC, CRW, and FSAD over sequential execution on compound edits.One-stage editing changes FD from 7.16 to 5.47, FAD from 1.67 to 1.16, and LSD from 1.82 to 1.07; human evaluations also prefer SwanWeave over every compared baseline.

5 Conclusion

SwanWeave is a one-stage framework for instruction-guided 3D FOA spatial audio editing, trained with paired simulated supervision and combining latent flow-matching editing, SE-MoE, and SPO. It supports diverse edits and shows consistent improvements over general audio editors and spatial audio baselines in objective and human-preference evaluations.

  • Conclusion: SwanWeave combines paired FOA editing supervision, latent flow-matching editing, SE-MoE, and SPO for diverse event, spatial, temporal, and environmental edits.The framework targets instruction-guided 3D FOA spatial audio editing.
  • Conclusion: Experiments show consistent improvements over general audio editors and spatial audio baselines across objective and human-preference metrics.The conclusion highlights direct FOA modeling as promising for controllable 3D audio creation.

Limitations

SWANWEAVE’s evaluation is limited by simulated supervision and relatively simple, short scenes. Real recordings and longer, denser scenes remain important extensions.

  • Simulated room supervision may not capture microphone characteristics, background noise, occlusion, or highly irregular room responses.
  • Current scenes are approximately 10 seconds long and usually contain only a small number of sound events.
  • Longer scenes with denser event mixtures, overlapping trajectories, and open-ended instructions may require stronger long-context modeling and more diverse supervision.

Ethical considerations

The training data combines open-source speech and sound-effect corpora with controllable room simulation. Future real-recorded FOA extensions must address consent, privacy, and licensing.

  • Speech data may contain human voices, so real-recorded FOA extensions should consider consent, privacy, and dataset licensing.
  • Simulated-data construction reduces the need to collect private real-world spatial recordings.

A.1 Detailed implementation of spatial audio

The implementation constructs FOA editing data through controllable room simulation, defines diverse spatial edit actions, and uses routing regularization and preference negatives to train instruction-following edits.

  • Data construction: The dataset combines open-source speech and sound-effect corpora with controllable room simulation to produce FOA spatial-editing examples.
  • Data construction: FOA rendering uses room impulse responses that capture direct paths, early reflections, and late reverberation in W,Y,Z,X channel order.
  • Spatial representation: The dataset represents ten directional categories, including eight azimuth directions plus above and below, with randomized positions around each nominal location.
  • Basic edit actions: Motion edits move a specified event continuously across angle or distance, while room edits alter room size or reverberation while preserving content and layout.
  • SE-MoE routing: The frame-level router uses latent-token logits and probabilities, selected expert sets, load balancing, z-loss, and null routing to regulate expert usage.
  • Preference learning: SPO mixes wrong-edit and no-operation negatives at a 2:1 ratio to distinguish plausible incorrect edits from unchanged outputs.

C More experiments

The authors select source and instruction CFG scales by balancing preservation of unedited content against successful editing. They use 3.0 for both scales in all experiments.

  • The authors evaluate 40 generated samples for each CFG-scale combination, manually assessing source preservation and editing success.
  • Setting ssrc = 3.0 and sinst = 3.0 provides the best overall balance between preserving unedited content and successfully performing edits.

C.2 Instruction robustness and complexity-stratified evaluation

SPATIALEDIT remains moderately robust to colloquial, redundant, and ambiguous instructions, while human preference increases relative to SmartDJ as editing complexity grows.

  • SPATIALEDIT received 62% average human preference over SmartDJ in complexity-stratified evaluation.
  • SPATIALEDIT’s advantage over SmartDJ increased as editing complexity increased.

C.3 Human evaluation details

Human evaluation compared two edited spatial-audio versions using joint judgments of perceptual quality and instruction following. Volunteer graduate students from the research group participated without monetary compensation.

  • Annotators compared an original spatial audio clip, an editing instruction, and two edited versions labeled A and B.
  • Annotators selected the better version by jointly considering perceptual audio quality and instruction adherence.
  • Volunteer graduate students recruited from the research group performed the evaluation without monetary compensation.
Loading 2609.04975v1…