Source-linked AI summary
PLACE: Positional Latent Adaptation via Conditioned Embeddings for Binaural Audio Generation
Tiernon Riesenmy, You Zhang, Gautam Bhattacharya, Andrea Fanelli
TL;DR
Binaural generation from multimodal prompts is constrained by limited aligned spatial-audio data. PLACE extends AudioX with spatial conditioning and a learned latent transform supervised by decoded-audio interaural objectives. It improves most metrics over ViSAGe and receives favorable listener ratings for video-to-audio and out-of-distribution text-to-audio generation.
Problem
Aligned data pairing binaural audio, video, and spatially descriptive text is limited, constraining development of any-to-spatial models.
Source: §1 · p. 1Method
PLACE extends AudioX by augmenting video features, aligning text and video spatial cues, and applying an interaural-loss-trained low-rank transformation to the generated latent.
Source: §2 · p. 1, §2.4 · p. 2Results
PLACE improves on ViSAGe for every V2A metric on each FAIR-Play split except ITD distribution distance on Split 2, and is competitive on T2A while outperforming SpatialSonic on SpatialCLAP.
Source: §3.2 · p. 3Takeaways & Limitations
Listener ratings favor PLACE across both V2A groups and out-of-distribution T2A, while it trails both baselines overall on in-distribution T2A.
Source: §3.4 · p. 4PLACE assumes SAO latents preserve binaural spatial cues to some extent, although its autoencoder reconstruction is imperfect.
Source: §2.1 · pp. 1–2
Abstract
from arXiv · showhide
We present PLACE, a method that extends the pretrained any-to-audio model AudioX for binaural generation from arbitrary combinations of text, video, and optional audio prompts. PLACE augments video conditioning with Perception Encoder Core features, aligns text and video representations to derive spatial cues, and applies a conditioning-dependent low-rank transformation to the generated latent. The adapter is supervised via decoded-audio interaural level and time difference objectives. Trained on MRSAudio, PLACE improves most metrics over ViSAGe on FAIR-Play and achieves an improved SpatialCLAP score over SpatialSonic on the BEWO-1M Single Static test split. Listener evaluations favor PLACE for video-to-audio and out-of-distribution text-to-audio generation, demonstrating flexible multimodal control and improved spatial consistency.
1. INTRODUCTION
Binaural audio is an accessible spatial-audio format, but generating it from accompanying video and text requires flexible multimodal conditioning. PLACE addresses limited aligned training data by adapting existing architectures using annotated, aligned binaural audio, video, and spatial text.
1. INTRODUCTION
- Binaural recordings encode directional hearing cues, especially interaural level and time differences, and can be played with headphones.The paper motivates generation conditioned on surrounding video and text so the audio matches the scene.Source: §1 · p. 1
- Prior systems have limitations in multimodal control: ViSAGe lacks flexible controls, FoleySpace lacks text input, and SpatialSonic relies on external LLM queries for optimal performance.Other unified approaches either omit spatially descriptive text or focus predominantly on video [6] [7].Source: §1 · p. 1
- Limited data pairing spatial audio, video, and spatially descriptive text constrains any-to-spatial model development and makes training from scratch or creating annotations costly.Synthetic data may not reflect real-world contexts; human annotation or refinement of LLM-generated annotations is costly.Source: §1 · p. 1
- PLACE supports binaural generation from any combination of video, text, and audio, adapting a pretrained generator’s latent rather than using an intermediate inference stage.The paper evaluates generalization on FAIR-Play and BEWO-1M and reports competitive results across evaluations.Source: §1 · p. 1
2. PROPOSED METHOD: PLACE
PLACE extends AudioX for binaural generation by adding spatial conditioning to video and text representations and adapting the generated latent. Its method builds on AudioX’s stereo latent, then aligns modalities and trains the spatial modules with decoded-audio interaural objectives.
2. PROPOSED METHOD: PLACE
- PLACE extends AudioX with PE-Core video features, spatial text–video alignment, a low-rank latent transform, and interaural level and time difference losses.The encoder and decoder remain frozen; AudioX is fine-tuned via LoRA, as summarized in Fig. 1.Source: §2 · p. 1, Figure 1 · p. 2
2.1. Preliminary Study on AudioX Backbone
- SAO autoencoder decoding largely preserves spatial width and left–right level differences, supporting its use as a base representation for binaural generation.Side-channel energy changed by at most 0.22 dB, and left–right level differences had Pearson correlation 0.95–0.96.Source: §2.1 · pp. 1–2
2.2. Video Feature Augmentation with Perception Encoder Core
- PLACE augments AudioX’s image-based video path with PE-Core features to represent spatial tasks and video context beyond individual-frame object composition.PE-Core is selected over PE-Spatial because its patch grid would require 1024× the storage of a global descriptor at the same dimensionality and precision.Source: §2.2 · p. 2
2.3. Spatial Feature Alignment
- Spatial feature alignment projects Synchformer and PE-Core video features, then uses text–video cross-attention and joint self-attention to produce spatial features Hc-sp.For text-only prompts, learned projection biases and a video-modality embedding initialize a fixed no-video representation that attention makes text-dependent.Source: §2.3 · p. 2
2.4. Spatial Latent Adapter
- An MLP maps temporally resampled Hc-sp to position-dependent low-rank factors and a channel-wise gate that transform the latent before decoding.The adapter uses rank 16 and acts once on the final denoised latent at inference.Source: §2.4 · p. 2
2.5. Interaural Losses and Training
- PLACE trains spatial alignment and the latent adapter using decoded-audio ILD and ITD losses alongside latent matching, with all pretrained encoders and the SAO decoder frozen.The ITD objective supervises only the central part of the physiological ±0.8 ms range; evaluation covers the full range at 16 kHz.Source: §2.5 · p. 3
3. EXPERIMENTS AND RESULTS
The experiments test PLACE’s data efficiency and perceptual quality using out-of-distribution benchmarks, held-out MRSAudio ablations, and listener ratings. Results favor PLACE on most objective comparisons and on video-to-audio and out-of-distribution text-to-audio listener evaluations, with an in-distribution text-to-audio exception.
3.1. Experimental Setup
- The evaluation uses FAIR-Play’s three 187-recording video-to-audio splits and BEWO-1M SS-set’s 4,301 text-to-audio pairs, with AudioX, ViSAGe, and SpatialSonic as task-specific baselines.MRSAudio training data were augmented to 24,114 annotated 10-second clips; 596 disjoint clips were held out for ablations.Source: §3.1 · p. 3
- Four objective metrics cover stereo audio distance, interaural cue distribution distances, and the difference in target-versus-generated SpatialCLAP similarity; lower is better for all.Interaural time difference is extracted with GCC-PHAT, and interaural level difference is computed per clip.Source: §3.1 · p. 3
3.2. Generalization Under Limited Training Data
- PLACE improves on ViSAGe for every video-to-audio metric on each FAIR-Play split except ITD distribution distance on Split 2, and beats SpatialSonic on SpatialCLAP for text-to-audio.PLACE trains on roughly 3.4x less data than ViSAGe and up to 41x less than BEWO while remaining competitive or better on most metrics.Source: §3.2 · p. 3, Table 1 · p. 4
3.3. Ablation Studies
- On held-out MRSAudio, full PLACE with the adapter, both interaural losses, and modality dropout achieves the best performance across all four metrics.The 596 held-out examples each provide a 10-second video and caption; the no-adapter configuration performs best without the adapter when using PE conditioning.Source: Table 2 · p. 4, §3.3 · p. 4
3.4. Subjective Listener Test
- PLACE leads both video-to-audio listener groups across all five dimensions and out-of-distribution text-to-audio across all four, but trails both baselines overall on in-distribution text-to-audio.SpatialSonic leads in in-distribution text-to-audio spatial impression and consistency, while AudioX leads in semantic consistency.Source: Table 3 · p. 4, §3.4 · p. 4
4. CONCLUSION
PLACE extends AudioX with multimodal spatial conditioning and a latent transformation trained using decoded-audio interaural losses. It supports flexible binaural generation, improves on ViSAGe on most metrics, and remains competitive with SpatialSonic on the BEWO-1M SS-set despite a different training distribution; listener ratings favor PLACE in overall averages.
4. CONCLUSION
- PLACE extends AudioX with multimodal spatial conditioning and a latent transformation learned through decoded-audio interaural losses.The method supports flexible binaural generation, improves on ViSAGe on most metrics, remains competitive with SpatialSonic on BEWO-1M SS-set despite a different training distribution, and receives favorable overall listener averages.Source: §4 · p. 4