PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysWave: Physics-Guided Latent Diffusion Models for Controllable Spatial Audio Generation

Lingfeng Yao, Chenpei Huang, Xingke Yang, Ziye Geng, Changqing Luo, Hao Wang, Jiang Liu, Miao Pan

arXiv:2608.29549v1cs.SDcs.AIcs.MM

TL;DR

Existing text-to-FOA methods may violate acoustic relations and separate ease of use from precise spatial control. PhysWave combines waypoint-caption conditioning with differentiable direction and distance priors, supported by a 300K-clip dynamic FOA dataset. It improves spatial consistency while maintaining competitive audio quality, and the priors also support training-free inference-time refinement.

  • Problem

    Existing text-to-FOA methods learn spatial relations mainly from data and separate descriptive from parametric control, limiting physical consistency and control flexibility.

  • Method

    PhysWave uses a latent diffusion model with shared waypoint-caption conditioning, spherical-harmonic direction and inverse-square distance priors, and a 300K-clip dynamic FOA dataset.

  • Results

    PhysWave improves spatial consistency while maintaining competitive audio quality, and its acoustic priors enable training-free inference-time spatial refinement.

  • Takeaways & Limitations

    Explicit acoustic priors can support spatially consistent controllable FOA generation during training and inference.

  • Takeaways & Limitations

    PhysWave focuses on single-source trajectories in free-field settings and does not model reverberation, occlusion, multipath propagation, or multi-source scenes.

Abstract

from arXiv · show

Text-to-spatial audio generation, such as text-to-First-Order Ambisonics (FOA), provides a convenient way to create spatial audio for billion-dollar gaming and film industries. However, existing text-to-FOA methods are largely data-driven and may produce audio that violates acoustic relations between source direction and distance. They also separate descriptive and parametric control, forcing users to trade usability for precision. In this paper, we present PhysWave, a physics-guided latent diffusion model for controllable text-to-FOA generation. PhysWave unifies natural-language and trajectory control through a shared waypoint-caption representation, and augments diffusion training with two differentiable acoustic priors: spherical-harmonic direction consistency and inverse-square distance consistency. To support dynamic spatial generation, we further construct a 300K-clip FOA dataset with diverse sound categories and source trajectories. Extensive results show that the proposed priors help PhysWave generate spatially consistent FOA audio while maintaining competitive audio quality. Further analyses show that these physics priors improve spatial consistency during training and can also be used as inference-time guidance for training-free spatial refinement.

1 Introduction

PhysWave addresses physical-consistency and control limitations in text-to-FOA generation with differentiable acoustic priors, unified waypoint-caption conditioning, and a 300K-clip dynamic dataset.

  • FOA generation is difficult for non-experts because it requires specialized tools, manual scene design, and spatial-audio expertise.
  • Existing FOA methods may violate direction and distance relations because they learn acoustic structure implicitly from data.
  • Current systems separate descriptive and parametric interfaces, forcing a trade-off between ease of use and fine-grained spatial control.
  • PhysWave adds spherical-harmonic direction and inverse-square distance losses to guide FOA generation toward requested spatial trajectories.
  • A waypoint-caption representation unifies natural-language descriptions and editable spatial trajectories within one generation model.
  • The authors construct a 300K-clip dynamic FOA dataset with diverse sound categories and source trajectories for training and evaluation.

2 Related Work

Related work spans monaural, stereo, and FOA generation, while PhysWave distinguishes itself by explicitly incorporating acoustic relations and unifying descriptive and parametric control.

  • Monaural methods improve audio fidelity and semantic alignment but do not encode source direction, distance, or motion.
  • Stereo methods introduce spatial cues through two-channel audio, whereas FOA supports device-agnostic spatial representation.
  • Recent FOA work covers sound-category and location conditioning, static scenes, and moving sound sources.
  • Existing FOA generators mainly learn spatial structure from paired data without explicitly using acoustic relations in training objectives.
  • PhysWave combines differentiable acoustic priors with a unified waypoint-caption representation for natural-language and trajectory-based control.
  • Physics-guided learning commonly incorporates known physical relations through residual losses or explicit constraints in generative models.

3 Spatial Audio Dataset and Representation

The dataset pairs captioned monaural audio with sampled listener-relative trajectories, simulates FOA waveforms, and represents source motion and sound fields through trajectories and spherical-harmonic channels.

  • Because real FOA recordings with accurate trajectories are difficult to collect at scale, the authors construct a synthetic dataset by spatializing captioned monaural audio.
  • Figure 1’s pipeline pairs captioned monaural audio with sampled trajectories and renders FOA audio through simulation.
  • Data preparation resamples audio to 16 kHz, selects active 10-second segments, normalizes loudness, and filters weak caption-audio matches.
  • FOA represents a three-dimensional sound field using four spherical-harmonic channels.
  • The W channel is omnidirectional, while X, Y, and Z encode directional projections along front–back, left–right, and up–down axes.
  • For a point source, FOA channel values are determined by the source pressure and its azimuth and elevation.
  • The four FOA channels are coupled because their cross-channel relations encode spatial direction.
  • Source motion is represented by listener-relative azimuth, elevation, and distance trajectories.

4 Physics-Guided Latent Diffusion Model

PhysWave combines waypoint-caption conditioning with FOA latent diffusion and differentiable physical constraints to generate audio aligned with both sound descriptions and source trajectories. Its training applies direction and distance consistency losses to decoded FOA waveforms.

  • Unified Waypoint-Caption Conditioning: PhysWave unifies text and trajectory control by parsing spatial captions into sound-event text, source waypoints, and temporal features.These representations are encoded into joint conditioning tokens for the diffusion model.
  • FOA Latent Diffusion Backbone: The model compresses four-channel FOA waveforms into continuous latents with an FOA VAE, then denoises them using a DiT under joint conditioning.The VAE is pretrained with waveform reconstruction and direction-consistency losses, then frozen during diffusion training.
  • Physics-Guided Training Objectives: A spherical-harmonic direction prior constrains FOA cross-channel structure so the recovered direction matches the trajectory’s per-frame target.The target direction is represented as a listener-relative unit vector derived from azimuth and elevation.
  • Physics-Guided Training Objectives: An inverse-square distance prior constrains the generated signal’s relative temporal energy trend using the trajectory-conditioned profile 1/r(t)^2.Absolute source loudness remains unconstrained because the loss compares normalized log-energy shape.
  • Physics-Guided Training Objectives: During training, predicted clean latents are decoded into FOA waveforms before applying the direction and distance consistency losses.The final objective combines latent denoising with weighted direction and distance constraints controlled by λ_dir and λ_dist.

5 Experiments

Experiments evaluate PhysWave on audio quality, spatial fidelity, controllability, parser quality, and subjective perception. Results indicate strong spatial localization and distance consistency, competitive audio quality, and useful inference-time physics refinement.

  • Evaluation: Experiments assess generated FOA audio along audio quality and spatial fidelity, including semantic alignment, perceptual quality, direction consistency, and distance consistency.Audio metrics are computed on the omnidirectional W channel, while spatial metrics use estimated directions and distance-attenuation profiles.
  • Spatial fidelity: 1.73° static and 5.78° circular-motion DoA errors are the lowest reported under the compared settings.SonicMotion reports 31.09° and 19.22° for circular motion, while ImmerseDiffusion reports 7.07° and 2.79° for static sources under descriptive and parametric conditioning.
  • Audio quality: PhysWave achieves the best KL score of 1.66, while FD 21.22, FADCLAP 0.21, CLAP 0.33, and IS 8.31 remain comparable to strong monaural baselines.The comparison evaluates the W channel after spatializing monaural baseline outputs into FOA.
  • Ablation study: Waypoint conditioning improves spatial fidelity over trajectory text, reducing static angular error from 7.33° to 2.05° and moving angular error from 17.62° to 6.50°.The LLM parser bridges natural-language descriptions and structured waypoint inputs.
  • Ablation study: Direction and distance losses target different spatial errors: direction loss lowers angular error, while distance loss raises InvSq Corr. from 0.48 to 0.74 and lowers InvSq Err. from 5.15 dB to 3.78 dB.Audio quality remains stable across variants, indicating that the auxiliary losses improve spatial fidelity without degrading content quality.
  • Subjective evaluation: PhysWave receives substantially higher subjective scores than compared methods across acoustic-event consistency, spatial-trajectory consistency, and audio realism.The study uses 14 participants and 100 FOA clips, with real recordings serving as a perceptual upper-bound reference.
  • Inference-time guidance: Inference-time physics guidance generally reduces static and moving angular errors and improves InvSq Corr., but larger guidance scales mildly increase FADCLAP.This exposes a trade-off between spatial correction and audio quality while enabling training-free spatial refinement.

6 Conclusion

PhysWave combines waypoint-caption conditioning with differentiable direction and distance priors for controllable text-to-FOA generation. The priors improve spatial consistency while maintaining competitive audio quality and support training-free inference-time refinement.

  • PhysWave combines waypoint-caption conditioning with differentiable direction and distance priors for controllable text-to-FOA generation.The priors explicitly guide the model’s spatial generation.
  • The proposed priors improve spatial consistency while maintaining competitive audio quality.
  • The same priors support training-free inference-time refinement of generated spatial audio.

Limitations

PhysWave is scoped to single-source trajectory control in a free-field setting and does not model several room-acoustic effects. Future extensions include multi-source scenes and realistic room acoustics.

  • PhysWave focuses on single-source trajectory control in a free-field setting.
  • The model does not represent reverberation, occlusion, or multi-path propagation.These effects require additional acoustic modeling beyond the direct-path simulation used in this work.
  • Extending the approach to multi-source scenes and realistic room acoustics is identified as future work.

B Dataset Construction Details

The dataset construction pipeline selects captioned monaural clips, renders static and moving FOA versions, and stores trajectories for waypoint conditioning. An LLM parser converts spatial language into fixed-length waypoint inputs.

  • Dataset Construction: The source pool uses captioned audio from AudioCaps, WavCaps, and Clotho, with clips selected for dominant acoustic events.Retained clips are converted to mono and resampled to 16 kHz.
  • Trajectory Sampling: Each retained monaural clip produces one static-source and one moving-source spatial version.The static version keeps azimuth, elevation, and distance fixed throughout the clip.
  • Trajectory Sampling: Moving trajectories use linear pass-by, circular, and approach/recede motion families.
  • Trajectory Sampling: Trajectories are stored as frame-level samples (t, θ, ϕ, r) at 0.1 s intervals, with r denoting source-listener distance.
  • Trajectory Sampling: Static sources comprise 50% of rendered FOA samples, while each moving family contributes about 16.7%.
  • Spatial Parsing: An LLM spatial parser separates acoustic content from spatial trajectory attributes and emits a JSON representation.
  • Spatial Parsing: Deterministic post-processing converts the parsed trajectory into the M = 10 waypoint condition used by the diffusion model.Figure 5 presents the parser prompt used for this conversion.

D LLM-as-a-Judge Evaluation

The spatial parser is evaluated with an LLM-as-a-judge protocol that scores trajectory faithfulness and semantic preservation on a common caption set. Each example receives one pair of integer scores and a short inspection-only justification.

  • Evaluation Protocol: An LLM judge evaluates parsed trajectory representations after deterministic conversion into fixed-length waypoint conditions.
  • Evaluation Protocol: The judge assigns spatial faithfulness and semantic preservation scores from 1 to 5.Spatial faithfulness measures agreement with the described layout and motion.
  • Evaluation Protocol: All candidate parsers use the same caption set and judge prompt, with each example scored once.
  • Evaluation Protocol: The judge’s short justification is used only for inspection and excluded from reported metrics.

E.1 FOA VAE Training

The FOA VAE compresses four-channel waveforms into a frozen latent representation for diffusion while combining reconstruction, adversarial, regularization, and spatial-consistency objectives. It achieves low angular reconstruction error while preserving waveform quality.

  • VAE architecture: The VAE compresses 16 kHz FOA waveforms with four channels—W, X, Y, and Z—into a continuous latent space for diffusion modeling.It uses a DAC-style 1D convolutional autoencoder and is later frozen as the diffusion model’s encoder and decoder.
  • Training objective: The VAE objective combines multi-resolution STFT reconstruction, adversarial and feature-matching losses, KL regularization, and FOA direction consistency.The direction term is added to better preserve spatial structure during reconstruction.
  • Optimization: The autoencoder is trained with AdamW on 2.05-second crops for 120K steps using two NVIDIA RTX 3090 GPUs and batch size 12.The autoencoder learning rate is 1×10^-4, while the discriminator uses 2×10^-4.
  • Reconstruction quality: 1.92° angular reconstruction error is reported for the FOA VAE while preserving waveform quality on the test split.The result is reported in the FOA VAE reconstruction-quality evaluation.
  • Diffusion interface: The diffusion model is trained on the frozen FOA VAE’s 64-channel latent sequences using a DiT conditioned on text, waypoint, and temporal tokens.Text uses T5-base, while trajectory features are encoded with a Q-Former-style encoder before conditioning the DiT.
  • Physics-guided training: The physics-consistency loss is applied to decoded waveform estimates using direction and distance terms, sampled denoising examples, 40 ms frames, and denoising-SNR weighting.The reported weights are λdir = 1.0 and λdist = 0.05.

F Evaluation Metric Definitions

The evaluation separates audio quality from spatial fidelity, measuring generated audio against references and trajectory-conditioned direction and distance behavior. Additional visualizations compare trajectories and inverse-square energy envelopes over time.

  • Audio quality: Audio quality is evaluated with FD, FADCLAP, IS, KL, and CLAP score using pretrained PANNs and CLAP embedding spaces.These metrics are computed on the generated omnidirectional W channel because standard text-to-audio metrics target single-channel or general audio.
  • Direction consistency: Direction consistency estimates frame-level direction of arrival from the generated FOA signal and compares it with ground-truth direction using spherical angular error.The angular error uses the haversine formula over predicted and target azimuth and elevation.
  • Distance consistency: Distance consistency measures whether generated W-channel energy follows the trajectory-conditioned inverse-square attenuation profile defined by ground-truth source-listener distance.Generated W-channel energy is computed frame by frame, with active frames separately defined.
  • Distance metrics: Lower InvSqErr and higher InvSqCorr indicate better agreement with the specified distance attenuation cue.These metrics quantify agreement with the target inverse-square distance behavior.
  • Visual analysis: Figures 8 and 9 provide qualitative spatial analyses by comparing motion trajectories and generated log-energy envelopes with target inverse-square profiles over time.Figure 9 reports that the physics-loss model follows the target profile more closely than the model without physics losses.
Loading 2608.29549v1…