PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatial Audio Generation and Editing with Executable Plans and Self-Distilling Transfusion

Kam Man Wu, Yu Zhang, Pengjun Fang, Mingfei Chen, Weijia Chen, Zeyue Tian, Qifeng Chen

arXiv:2610.34415v1cs.SDcs.AI

TL;DR

Spatial audio authoring must coordinate source decisions with soundfield synthesis, and editing must specify changes while retaining unchanged sources. AMBIT combines executable scene planning, FOA-aware latent supervision, and joint on-policy distillation of planning and rendering. Across speech, music, and sound, the reported evaluations show improved generation, editing, and preservation outcomes, with a scope boundary of roughly 15-second scenes.

  • Problem

    Spatial audio editing must specify requested source changes while preserving unchanged sources’ acoustic context, and source decisions must align with multichannel synthesis.

    Source: §1 · p. 1

  • Method

    AMBIT unifies executable scene planning and FOA rendering with an FOA-aware VAE and joint on-policy self-distillation of planning and rendering.

    Source: §1 · pp. 1–2, §3 · p. 3

  • Results

    Across speech, music, and sound, AMBIT reports improved generation content and spatial quality, editing acoustic fidelity, and unedited-source preservation over compared baselines.

    Source: §1 · p. 2, §4.3 · p. 7, §4.4 · p. 8

  • Takeaways & Limitations

    The results support jointly learning scene control and acoustic execution for spatial audio authoring.

    Source: §5 · p. 9

    The current SCENEPLAN configuration supports scenes up to approximately 15 s; longer soundscapes and consistency across successive edits remain future work.

    Source: §5 · p. 9

Abstract

from arXiv · show

Spatial audio authoring coordinates source decisions with soundfield synthesis. For editing, plans specify what to change and retain, while reference audio supplies further acoustic detail. We present AMBIT, a unified framework for native First-Order Ambisonics (FOA) generation and editing with three connected contributions. First, a shared Transfusion backbone couples autoregressive planning with rectified-flow rendering through executable ScenePlans that specify source content, activity, and 3D trajectories. For editing, the plan describes the full target scene, including retained sources; contrastive features ground the requested change, while full reference latents supply residual acoustic detail for preservation. Second, an FOA-aware continuous VAE combines shared-decoder omnidirectional recovery, asymmetric grouped KL, and spatial covariance matching to preserve content and inter-channel structure. Third, joint Transfusion on-policy self-distillation (OPSD) refines planning on student-generated prefixes and rendering at sampling states reached under student plans, with target-conditioned teachers supervising both modes of the shared student. Evaluations across speech, music, and sound show improved generation content and spatial quality, acoustic fidelity during instruction-guided editing, and unedited-source preservation over the compared baselines. Project page: https://wjm-george.github.io/AMBIT/

1 INTRODUCTION

Spatial audio generation and editing must coordinate source content, timing, and motion, while editing also needs to preserve unchanged sources. AMBIT unifies planning and rendering, adds FOA-aware latent supervision, and jointly refines planning and rendering; the reported evaluations show gains across generation, editing, and preservation.

1 INTRODUCTION

  • Spatial authoring must align source content, activity, and 3D motion with coherent soundfields; editing must also retain surrounding sources.Executable plans can state changes and retained sources, while reference audio supplies residual acoustic detail.Source: §1 · p. 1
  • Prior approaches leave gaps in FOA source-level control, explicit inter-channel supervision, and joint refinement on student-generated planning and rendering contexts.Channel-wise reconstruction does not explicitly supervise inter-channel energy, coherence, and phase.Source: §1 · p. 1
  • AMBIT uses executable SCENEPLANs to connect autoregressive planning and rectified-flow rendering, with complete target-scene plans and reference latents for editing.Contrastive features ground requested changes, while reference latents supply residual detail for unchanged sources.Source: §1 · pp. 1–2
  • Its FOA-aware VAE combines omnidirectional recovery, asymmetric grouped KL, and spatial covariance matching to supervise inter-channel structure.The covariance objective targets relative energy, coherence, and phase between channels.Source: §1 · p. 2
  • Joint Transfusion OPSD trains on student-generated prefixes and rendering states, using target-conditioned teachers for both planning and rendering.Token and clean-latent losses update shared and task-specific parameters.Source: §1 · p. 2
  • AMBIT improves generation content and spatial quality, editing fidelity and accuracy, and unedited-source preservation over the compared baselines.Joint OPSD leads continued SFT, ReFL, and GRPO across all nine reported metrics.Source: §1 · p. 2

2 RELATED WORK

Related work develops spatial audio generation, editing, and on-policy refinement, but AMBIT combines source-level 3D control and native FOA editing with acoustic and spatial supervision. Its joint OPSD applies student-context distillation to both the discrete planner and continuous renderer.

Spatial audio generation and representation

  • Prior work spans multimodal and timing-aware generation, spatial conditioning, trajectory-aware FOA synthesis, and compact audio representations.AMBIT combines source-level 3D control with acoustic fidelity across native FOA generation and editing.Source: §2 · p. 2

Audio editing and executable scene planning

  • Existing editing methods include text-guided spatialization, diffusion editing, language-guided editing, and native FOA editing, while structured plans support compositional generation.AMBIT binds source content, activity, and 3D paths in executable SCENEPLANs and uses FOA features for edits and reference latents for preservation.Source: §2 · pp. 2–3

On-policy refinement and self-distillation

  • Prior on-policy methods refine generated sequences or continuous generators, whereas Transfusion OPSD jointly refines a shared FOA editor’s planner and renderer.Frozen teachers use target plan memory and target audio to supervise student prefixes and flow states reached under student plans.Source: §2 · p. 3

3 METHODOLOGY

AMBIT uses a shared Transfusion Transformer to plan executable FOA scenes and render audio latents, with reference audio supporting editing. Its VAE supervises omnidirectional recovery and inter-channel structure, while joint OPSD distills both planning and rendering on student-generated contexts.

3 METHODOLOGY

  • A shared Transfusion Transformer predicts executable scene plans and renders native FOA through rectified-flow latent integration.Generation conditions on instructions; editing also uses source latents and AMBIT CLAP features for grounding.Source: §3 · p. 3

3.1 UNIFIED TRANSFUSION PLANNING AND RENDERING

  • SCENEPLANs specify source-indexed content, activity, and 3D motion, then compile deterministically into semantic tokens and frame-aligned spatial controls.Editing plans describe the complete target scene, including modified and retained source attributes.Source: §3.1 · p. 3
  • The autoregressive planner serializes plans under grammar constraints and uses source-aware heads and editing memory to predict edit targets and updated attributes.Paired rendering uses ground-truth target plans, whereas inference uses student-predicted plans.Source: §3.1 · p. 4
  • For editing, compiled plans specify scene structure while full reference latents provide retained sources’ acoustic detail to the flow renderer.The rendering objective perturbs clean target latents with noise at sampled flow times.Source: §3.1 · p. 4

3.2 AMBIT VARIATIONAL AUTOENCODER

  • The FOA VAE combines auxiliary W-channel reconstruction through a shared decoder, asymmetric grouped KL, and spatial covariance matching.Covariance matching supervises relative channel energy, coherence, and phase; weaker KL on the W latent supports its recovery role without enforcing exact content-space disentanglement.Source: §3.2 · p. 5

3.3 JOINT TRANSFUSION ON-POLICY SELF-DISTILLATION

  • Joint OPSD addresses the mismatch between paired training and inference by supervising planning and rendering on contexts encountered by the student.The planner teacher uses target plan memory, while the renderer teacher uses target-audio reference tokens; both are frozen.Source: §3.3 · p. 5
  • Planning distillation matches teacher token distributions on student-generated prefixes, while rendering distillation matches clean-latent predictions at visited student-plan sampling states.The losses update shared Transformer and active task modules through shared updates and fresh rollouts; sampled plans and states remain detached.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

Across codec evaluation, generation and editing, spatial evaluation, and post-training, AMBIT shows gains in reconstruction, content, spatial accuracy, and preservation, with joint OPSD leading all reported post-training metrics. Results also distinguish semantic and lexical fidelity from perceptual or distributional rankings, and target-direction accuracy from retention of unedited sources.

4.1 EXPERIMENTAL SETUP

  • Experiments use 8,000 generation requests, 5,000 editing pairs, and 2,000 VAE test clips across speech, music, and environmental sounds.Training uses 1.6M synthetic and recorded FOA examples and 1M editing pairs; AMBIT uses 332.6M trainable parameters.Source: §4.1 · p. 6

4.2 AMBIT VAE EVALUATION

  • AMBIT outperforms DAC, EnCodec, and WavTokenizer across all four FOA reconstruction metrics.In the cumulative ablation, SCM reduces direction error from 6.51° to 5.14° and raises W SI-SDR from 5.63 to 6.08 dB; grouped KL with auxiliary W recovery then improves all four metrics.Source: §4.2 · pp. 6–7, Table 1 · p. 6

4.3 GENERATION AND EDITING QUALITY

  • AMBIT leads paired CLAP, classifier KL, and FAD in music and sound, and has lower speech WER/CER but lower UTMOS than Qwen3-TTS.AudioX-Turbo retains lower FD-PaSST/FD-OpenL3; semantic and lexical fidelity improve, while perceptual and distributional rankings vary.Source: §4.3 · p. 7, Table 2 · p. 7
  • AMBIT improves all six editing metrics over SwanWeave, with higher paired CLAP and lower LSD extending gains from semantic agreement to spectral reconstruction.Spatial metrics are reported separately in Table 5.Source: §4.3 · p. 7, Table 3 · p. 7

4.4 SPATIAL AUDIO EVALUATION

  • AMBIT leads both scene- and object-level spatial metric groups; Qwen3.5-4B + DiT improves GCC, CRW, and wCS over spatialization pipelines but has higher direction errors.With the same renderer, AMBIT’s gains over Qwen3.5-4B + DiT support the learned planner’s spatial controls at both levels.Source: §4.4 · p. 8, Table 4 · p. 8
  • AMBIT leads scene, object-direction, and preservation metric groups, improving target direction accuracy while retaining unchanged-source direction, waveform, and level.The preservation evaluation compares output with original source audio on eligible unchanged-source intervals; listener ratings also favor edit success and preservation.Source: §4.4 · p. 8, Table 5 · p. 8

4.5 POST-TRAINING WITH OPSD

  • Joint OPSD leads all reported post-training metrics, while DiT OPSD improves every metric over SFT and AR OPSD improves only LSD and FSAD.Further reductions in classifier KL and CRW over DiT OPSD support combining planning distillation on student prefixes with rendering distillation at states visited under student plans.Source: §4.5 · p. 9, Table 6 · p. 9

5 CONCLUSION AND LIMITATIONS

AMBIT unifies native FOA generation and editing through executable SCENEPLANs, an FOA-aware VAE, and joint Transfusion OPSD. Its current SCENEPLAN configuration supports scenes up to approximately 15 seconds; longer soundscapes and consistency across successive edits remain future directions.

5 CONCLUSION AND LIMITATIONS

  • AMBIT combines executable SCENEPLANs, an FOA-aware VAE, and joint Transfusion OPSD in one framework for native FOA generation and editing.Plans connect source content, timing, and motion to rendering, while specifying requested edits and retained sources.Source: §5 · p. 9
  • Experiments across speech, music, and sound show improvements in content fidelity, spatial accuracy, and unedited-source preservation.Joint OPSD further improves all nine reported metrics over continued supervised training, ReFL, GRPO, and both branch supervision variants.Source: §5 · p. 9
  • The current SCENEPLAN configuration supports scenes up to approximately 15 seconds; longer soundscapes and acoustic consistency across successive edits remain future directions.Source: §5 · p. 9

A DATASET DETAILS

The appendices describe the datasets, conditioning interfaces, and training details supporting AMBIT’s spatial-audio generation and editing framework. They specify how synthetic and real recordings are assembled, how SCENEPLAN controls are encoded, and how the VAE and AMBIT CLAP features are trained and used.

A DATASET DETAILS

  • The datasets combine synthetic spatial scenes and real FOA clips, with editing pairs retaining source correspondence and partitions assigned by original recording or video.The generation test set includes 50 ten-second FOA clips from the annotated STARSS23 dev-test split; source assets and editing additions follow disjoint partitions.Source: A DATASET DETAILS · pp. 16–17

B.1 SCENEPLAN CONDITIONING

  • SCENEPLAN conditioning projects frozen Qwen3.5-0.8B text features into cross-attention tokens and encodes activity and interpolated pose controls for up to four sources.Azimuth interpolation uses the shortest signed increment, and inactive source blocks are zeroed.Source: §B.1 · p. 17

B.2 AMBIT VARIATIONAL AUTOENCODER

  • The VAE uses 64 latent channels split into zW and zR, with auxiliary W recovery, grouped KL regularization, and spatial covariance matching.Its reconstruction setup combines spectral, phase, adversarial, cepstral, and SI-SDR terms; covariance matching supervises inter-channel statistics.Source: §B.2 · pp. 17–18

B.3 AMBIT CLAP REFERENCE FEATURES

  • AMBIT CLAP learns semantic and scene features from paired audio and captions, distinguishing content matches from matches of spatial and temporal attributes.Swapped trajectories or activity intervals provide hard negatives for source-attribute binding, and the pretrained encoder supplies editing-planner features.Source: §B.3 · pp. 18–19

B.4 GENERATION PLANNING AND RENDERING

Generation uses an autoregressive planner to produce executable SCENEPLANs, whose compiled conditions guide rendering. Editing conditions planning and rendering on source audio and predicts a complete target scene that includes both retained and modified sources.

B.4 GENERATION PLANNING AND RENDERING

  • Generation rendering combines noisy latents with spatial conditions and zeroed reference channels, while speech-active frames receive greater residual-loss weight.Sound-only clips also use a weighted latent-difference loss at lags 1, 2, and 4.Source: §B.4 · p. 19
  • The generation planner completes source inventory, timing, and motion fields into executable SCENEPLANs that condition rendering.The codec serializes scene and source fields, while grammar-constrained decoding produces the plan.Source: §B.4 · p. 19, Table 10 · p. 19
  • Qualitative spatial and temporal controls are deterministically converted into numeric azimuth, distance, motion, onset, and offset values.Spatial defaults use seed 42; timing controls select feasible phase pairs and enforce a minimum ten-frame gap when feasible.Source: §B.4 · p. 20

B.5 EDITING PLANNER AND RENDERER

  • Editing conditions the planner and renderer on source audio and targets a complete SCENEPLAN that describes retained and modified sources.AMBIT CLAP features additionally condition the planner; field masks distinguish preserved attributes from requested changes.Source: §B.5 · p. 20
  • Canonical target ordering aligns autoregressive targets, rendering conditions, and edit annotations without changing source identity.A shared injective source assignment supervises recognition, edit binding, preservation, modification, and target presence.Source: §B.5 · p. 20

C TRAINING AND INFERENCE SETTINGS

AMBIT uses one Transfusion model for generation and editing, sharing Transformer blocks across autoregressive planning and DiT rendering. Training combines VAE pretraining with joint optimization, while inference uses task-specific conditioning and sampling settings.

C TRAINING AND INFERENCE SETTINGS

  • AMBIT shares Transformer blocks across autoregressive planning and DiT rendering for generation and editing.The complete student has 332.6M trainable parameters; generation uses language instructions, while editing additionally conditions on source audio.Source: C TRAINING AND INFERENCE SETTINGS · p. 20
  • The VAE is pretrained on approximately 1M FOA clips, and cumulative ablations use matched optimization steps and the same data split.Source: C TRAINING AND INFERENCE SETTINGS · p. 20
  • Joint optimization updates shared model components using editing pairs, with compiled ground-truth target plans for paired rendering and student predictions at inference.The pretrained text backbones, AMBIT CLAP, and AMBIT VAE remain frozen.Source: C TRAINING AND INFERENCE SETTINGS · pp. 20–21
  • Generation and editing use different flow-time sampling and inference settings, including 100 Euler steps for generation and 20 for editing.Generation uses guidance scale 3.0 and rescaling 0.4; editing uses guidance scale 1.0.Source: C TRAINING AND INFERENCE SETTINGS · p. 21
  • Planner comparisons hold the VAE, renderer weights, sampler settings, and noise seeds fixed within each task.OPSD further refines the shared student through joint planning and rendering supervision.Source: C TRAINING AND INFERENCE SETTINGS · p. 21

D.1 EVALUATION PROTOCOL AND BASELINE INTERFACES

Evaluation uses separate generation and editing benchmarks, shared valid observations for audio comparisons, and baseline-specific interfaces. External spatialization systems receive common executable plans and are compared with a shared target waveform.

D.1 EVALUATION PROTOCOL AND BASELINE INTERFACES

  • Generation and editing results use 8k and 5k benchmarks, respectively; audio comparisons use executable plans and metric-valid observations from the relevant benchmark, excluding missing measurements rather than assigning zero error.The AMBIT VAE is evaluated on 2,000 FOA clips.Source: §D.1 · p. 21
  • External generators receive semantic descriptions, whereas AMBIT receives qualitative requests; editing baselines use source audio and instructions or target descriptions according to their native interfaces.Qwen3-Omni is a planner for the DiT comparison, distinct from the Audio-Omni editor; its source observation is the W waveform, while AMBIT uses native FOA features.Source: §D.1 · p. 21

D.2 EXTERNAL AUDIO SPATIALIZATION

  • For BEWO + AS and TangoFlux + AS, Qwen3.5-4B supplies both systems’ executable plans, while Qwen3.5-4B + DiT uses AMBIT’s compiler and neural renderer.The external systems use mono source signals placed within predicted activity intervals and Pyroomacoustics rendering with fixed inference geometry; comparisons use a common target waveform.Source: §D.2 · p. 21

D.3 METRICS

The metrics evaluate content, speech, reconstruction, spatial direction, trajectories, and source preservation under shared test examples and task-specific validity rules. Several spatial measures use common valid frames or activity-exclusive intervals, which defines the scope of their comparisons.

D.3 METRICS

  • All systems use the same task examples, while directional errors use frames valid in the reference and every compared output, and object metrics additionally use isolated source intervals.The test sets contain 8,000 generation requests, 5,000 editing requests, and 2,000 VAE clips.Source: §D.3 · p. 21
  • Paired CLAP measures output-reference similarity, FAD and feature-based Fréchet distances compare embedding distributions, and PANNs/PaSST KL compares predicted acoustic classes.Generation uses native W or arithmetic-mean mono; editing uses mono downmixes of binauralized FOA without gain normalization.Source: §D.3 · p. 21
  • LSD measures framewise RMS log-power difference, while GCC measures squared interaural-delay difference and CRW and FSAD compare predicted delay and spatial-feature distributions.FOA reconstruction is decoded with fixed MIT KEMAR filters using eight virtual speakers; native stereo is unchanged.Source: §D.3 · p. 22
  • Scene direction and angular-trajectory metrics compare output with ground-truth FOA directions over common valid frames, with editing additionally restricted to target-plan activity.Angular trajectory RMSE measures directional deviations over time but does not evaluate source distance; overlapping sources contribute mixture directions.Source: §D.3 · p. 22
  • Object metrics compare actual output and ground-truth directions on eligible single-source intervals, then average valid objects within scenes and eligible scenes equally.They use ground-truth energy weights and do not apply source separation, temporal alignment, or angular reassignment; exclusive activity masks can retain reverberant tails.Source: §D.3 · p. 22
  • Unedited-source preservation uses intervals where the unchanged source is the only annotated active source, excluding edited activity and two frames at retained-interval boundaries.The masks establish exclusivity from activity annotations without acoustic source separation, so reverberant tails may remain.Source: §D.3 · p. 22
  • The VAE comparison evaluates omnidirectional reconstruction with W SI-SDR and paired CLAP, and spatial reconstruction with direction and intensity-coherence errors.FOA signals are resampled to 24 kHz after one delay estimated from W is applied across channels, retaining their common temporal overlap.Source: §D.3 · p. 23
  • Intensity coherence error measures normalized intensity-vector magnitude in the time domain, whereas spatial covariance matching uses pairwise STFT covariance statistics.Intensity coherence uses all active reference frames, including low-coherence frames; direction error additionally requires reference coherence of at least 0.1.Source: §D.3 · p. 23

D.4 SUBJECTIVE LISTENING STUDY

The listening study reports higher AMBIT mean ratings than both baselines across seven measures, with the largest generation advantage in spatial adherence. The included planner and OPSD material describes the study setup, planning validity checks, teacher preparation, student-conditioned rollouts, and supervision variants.

D.4 SUBJECTIVE LISTENING STUDY

  • The study compares three systems per case across 400 distinct cases per task, with participants rating all three outputs for each case under the same request and playback setup.Generation and editing use separate pools of 1,000 matched FOA cases, partitioned into non-overlapping sets of 20.Source: §D.4 · pp. 23–24
  • AMBIT exceeds both baselines across all seven listener-rated measures, with a 0.54-point advantage over the strongest baseline in generation spatial adherence.For editing, AMBIT exceeds Qwen3-Omni + DiT by 0.13, 0.31, and 0.27 points in overall quality, edit success, and unedited-content preservation, respectively.Source: §D.4 · p. 24

D.5 PLANNER VALIDITY

  • Planner validity checks schema and compiler compatibility for executability and requested inventory for source-count accuracy, but neither measure establishes full instruction satisfaction or correct attribute binding.Qwen uses its first response without repair or retries, while AMBIT uses grammar-constrained decoding; failed outputs remain in each panel’s denominator.Source: §D.5 · p. 24

E TRANSFUSION OPSD

  • OPSD is applied as post-training on editing pairs, jointly distilling planning and rendering on student-generated contexts while target-conditioned teachers supervise the shared student.AR OPSD and DiT OPSD replace distillation in the other mode with paired ground-truth supervision.Source: E TRANSFUSION OPSD · p. 24

E.1 Teacher construction and reference tokens

  • The planning teacher receives complete target-plan memory, while the rendering teacher receives temporally aligned clean target latents; during refinement, target tokens go only to teachers.Both teachers start from the same supervised model and are frozen after preparation; the student retains its supervised initialization.Source: §E.1 · pp. 24–25

E.2 Planning targets and reference constraints

  • Planning distillation queries the teacher on student-generated prefixes, using decision-family eligibility and reference constraints for selected planning facts.The frozen supervised reference supplies ordinary-condition distributions for room, event kind, text, and activity; source indices alone do not establish identity.Source: §E.2 · p. 25

E.3 Student rollouts and rendering targets

  • Student plans determine rendering conditions and the states at which the rendering teacher is queried, and the sampled objective estimates the full-grid mismatch used in the finite-step tracking bound.The teacher trajectory is an analytical comparison; training queries the teacher at student states, and target-audio accuracy depends on teacher quality.Source: §E.3 · pp. 25–26

E.4 Shared gradients and supervision variants

  • Joint OPSD updates shared parameters without paired ground-truth token, flow, or structured losses, while AR OPSD and DiT OPSD each replace one distillation term with its paired counterpart.Both branch variants retain trainability in both modes, and shared updates allow either loss to affect both modes in later rollouts.Source: §E.4 · pp. 26–27

E.5 Post-training controls and comparison protocol

Post-training comparisons use the same test requests, noise seeds, and sampling settings. The passage describes SFT, reward-based controls, and GRPO, including the narrower scope of the semantic reward used by ReFL and GRPO.

E.5 Post-training controls and comparison protocol

  • The comparison evaluates continued SFT, ReFL, GRPO, AR OPSD, DiT OPSD, and joint OPSD under identical test requests, noise seeds, and sampling settings.SFT continues paired training; ReFL uses CLAP semantic similarity between the generated terminal latent and raw edit instruction.Source: §E.5 · p. 27
  • ReFL and GRPO reward instruction compatibility but do not directly evaluate direction, timing, or unedited-source preservation, and a relative instruction need not describe the full target scene.Both retain paired spatial and preservation supervision, while their reward lacks the target-plan and target-audio information supplied to OPSD teachers.Source: §E.5 · p. 27
  • GRPO samples six plans and trajectories per request using the ReFL reward, assigns undecodable plans a reward of −1, and uses detached group-normalized advantages.Its two branches use categorical policy optimization for plans and Flow-GRPO transitions for rendering.Source: §E.5 · p. 27
  • GRPO’s rendering branch uses a dimension-normalized surrogate based on Gaussian transition log densities rather than a full-vector likelihood ratio.Each candidate queries up to eight non-forced planning decisions and two rendering transitions; behavior groups are reused for two optimizer updates.Source: §E.5 · pp. 27–28

Related papers

Loading 2610.34415v1…