PaperlayerView exact revision on arXiv

Source-linked AI summary

BinauralVAE: Spatial Audio Reconstruction For World Models

Luis Vitor Zerkowski, Luiz Velho

arXiv:2609.06837v1cs.SDcs.LG

TL;DR

Vision-centric world models leave sound comparatively underrepresented, and magnitude-only spectrograms discard phase needed for ITD-based localization. BinauralVAE explores VAE-based reconstruction of binaural audio, including a complex-valued architecture. The Complex VAE improves substantially over the 4-Channel STFT VAE, while its overall MSE remains higher than the Mel VAE; further refinement and evaluation remain future work.

  • Problem

    World models have been dominated by visual modalities, while magnitude-only approaches discard phase that ITD-based spatial localization relies on.

    Source: §1 · p. 1, §2.2.1 · p. 4

  • Method

    The work provides a pipeline of VAE architectures, including a mathematically grounded complex-valued approach, to learn spatial-audio latent representations through reconstruction.

    Source: CVAE Formulation and KL-Divergence · p. 6, §4 · p. 13

  • Results

    The Complex VAE substantially improves reconstruction over the 4-Channel STFT VAE, but its overall MSE is higher than the Mel VAE.

    Source: §3.3 · p. 13

  • Takeaways & Limitations

    The pipeline and baselines provide a foundation for future audio-based spatial state representations in world models.

    Source: §4 · p. 13

    The Complex VAE requires further refinement, including larger training data, broader architecture and hyperparameter optimization, and investigation of early loss convergence.

    Source: §5 · p. 13

Abstract

from arXiv · show

Embodied artificial intelligence has historically very much relied on visual perception, leading to a proliferation of multiple vision-centric world models. However, this reliance fails to capture spatial understanding in its entirety and can even present vulnerabilities in environments with visual occlusions, low-light conditions, or blackouts-scenarios, where acoustic information becomes a critical alternative for spatial awareness and navigation. Despite its potential, research into realistic spatial audio and particularly the development of audio-centric world models remains sparse. In this technical report, we introduce BinauralVAE: a flexible, open-source pipeline (https://github.com/Luizerko/BinauralVAE) that explores multiple models for spatialized audio reconstruction, progressing from fundamental baselines to advanced, mathematically grounded architectures. Our approach evaluates various Variational Autoencoder architectures -- including complex-valued variants -- to learn robust latent representations of binaural signals. Developed alongside AudioWorldSim, our methodology leverages realistic acoustic data captured as a simulated robot navigates an environment. This pipeline establishes a foundation for state representation in a future audio-based world model, designed to map the direct causal connection between navigational actions and their resulting acoustic consequences, and helping to enable sound as an essential complementary modality for spatial knowledge acquisition.

1 Introduction

World models and embodied AI have largely centered on vision, although sound conveys spatial information unavailable through line of sight. BinauralVAE explores open, phase-aware reconstruction of action-conditioned spatial audio as a foundation for audio-based world models.

1 Introduction

  • World models have been dominated by visual modalities, while sound remains neglected despite its role in spatial understanding.Sound can convey scene geometry, material properties, and out-of-view dynamic events, including around obstacles.Source: §1 · p. 1
  • Visual-only perception has critical failure points under occlusion, low light, or blackouts, when acoustic cues become vital for spatial awareness.Source: §1 · p. 1
  • Magnitude spectrograms discard phase, although ITD-based spatial localization relies on phase awareness.Prior phase-aware approaches are often designed for speech or source separation rather than reverberant, multidirectional navigation sounds.Source: §1 · p. 2
  • Spatial-navigation audio models often use spectrogram-only processing, mismatched encoders, or closed-source auditory feature extraction.The report identifies a scarcity of open-source, phase-aware implementations tailored to spatial audio reconstruction in navigational world models.Source: §1 · p. 2
  • BinauralVAE explores VAE architectures from real-valued spectrogram baselines to a complex-valued model for reconstructing navigation audio.The pipeline is developed alongside AudioWorldSim, which addresses the scarcity of action-conditioned binaural data by providing the dataset used in this work.Source: §1 · p. 2

2 System Design

The system is an end-to-end pipeline for training and evaluating spatial-audio reconstruction models. It supports reproducible experimentation and several training, inference, and latent-space workflows.

2 System Design

  • The pipeline supports training models from scratch, automated hyperparameter grid searches, full-sequence audio inference, and latent-space exploration called “dreaming.”Source: §2 · p. 2
  • Its flexible, end-to-end design processes, trains, and evaluates models for spatialized audio reconstruction.Source: §2 · p. 2
  • The pipeline is heavily parameterized to support reproducible and modular experiments across VAE variants.Source: §2 · p. 3

2.1 Data Specification and Preprocessing

The data pipeline divides simulated binaural audio into short action-aligned segments and prepares distinct spectrogram representations for different model architectures.

2.1 Data Specification and Preprocessing

  • Each “binaural image” represents 0.2 seconds of audio, matching one simulated action.The input is raw acoustic data generated by AudioWorldSim.Source: §2.1 · p. 3
  • The Mel STFT representation is a [2, 128, 60] tensor for the left and right ears, derived from a 2048-sample STFT mapped to 128 Mel bands.It uses global-statistics min-max normalization and preserves reference power for downstream reconstruction and dreaming.Source: §2.1 · p. 3
  • The 4-channel STFT stacks magnitude and phase for both ears, while the complex STFT retains complex outputs in a [2, 513, 60] tensor.The figure contrasts a Mel spectrogram segment with raw STFT output; the complex representation uses maximum-value normalization of the complex norm and 0.3 power-law magnitude compression.Source: §2.1 · p. 3, Figure 1 · p. 3

2.2 Models

The models reconstruct binaural segments for 0.2-second robotic actions as a basis for future audio-world-model latents; VAE training smooths outputs, sacrificing fine high-frequency texture.

2.2.1 Mel Spectrogram VAE

  • Three 2D-convolution encoder layers map binaural Mel inputs to a 32-dimensional Gaussian latent, but 2048-sample framing cannot preserve microsecond ITD; ILD and spectral cues remain.The Mel VAE treats the time-frequency representation as an image; its 2048-sample window yields about 46 ms per frame.Source: Figure 2 · p. 4, §2.2.1 · p. 4

2.2.2 4-Channel STFT VAE

  • The 4-channel STFT stacks both ears’ magnitude and phase using a 1024-sample window, but CNN phase prediction collapses under noisy, wrapped phase.The shorter window favors temporal resolution to recover spatial cues from phase.Source: §2.2.2 · p. 5, Figure 3 · p. 5

2.2.3 Complex VAE

  • The CVAE uses complex-valued networks and complex Gaussian prior, posterior, and likelihood to respect phase’s cyclic topology while aiming to preserve phase integrity.Its stated goal is high-fidelity magnitude reconstruction alongside phase preservation.Source: §2.2.3 · p. 5

Understanding Complex Gaussians

  • Complex Gaussians require both covariance and pseudo-covariance to represent cross-variable relationships and each variable’s real-imaginary elliptical skew.The covariance captures total variance and cross-variable correlations; the pseudo-covariance captures real-imaginary dependence.Source: Understanding Complex Gaussians · p. 6

CVAE Formulation and KL-Divergence

  • The CVAE uses an isotropic complex-normal prior and a diagonal covariance/pseudocovariance posterior, keeping dimensions independent while allowing within-variable elliptical skew.Because latent variables are independent, the total KL divergence sums the divergences for individual complex dimensions, treated as bivariate real Gaussians.Source: CVAE Formulation and KL-Divergence · pp. 6–7

The Reparametrization Trick

  • A Cholesky factor transforms standard bivariate Gaussian noise into each complex variable’s target covariance before adding its mean.The real coordinate depends on εr alone, while the imaginary coordinate depends on both εr and εi.Source: The Reparametrization Trick · p. 8

Reconstruction Loss

  • With a circularly symmetric complex-normal decoder, maximum likelihood reduces to MSE between target and reconstructed complex spectrograms, estimated using one posterior sample.The report states that maximizing the complex log-likelihood is mathematically equivalent to minimizing squared Euclidean distance.Source: Reconstruction Loss · p. 9

Architecture and Implementation

  • The implementation uses complexPyTorch [TBZ+18] and builds on Camara et al., but the CVAE is harder to train and sensitive to hyperparameters.Improper tuning can induce catastrophic latent-space collapse.Source: Architecture and Implementation · p. 9

2.3 Training

Training uses a six-hour spatial-audio dataset with held-out test and validation splits, and the CVAE requires more memory and longer training than the real-valued VAEs. The pipeline uses cyclic KL-weight scheduling and supports configurable architectures, while the CVAE requires manual tuning and is excluded from automated grid search.

2.3 Training

  • The roughly six-hour dataset is split 90/10 for training and testing, then 90/10 for training and validation, with a standard 100-epoch loop.Batch size is 256 for Mel and 4-channel STFT VAEs, but 32 for the CVAE, which uses approximately 24 GB of VRAM.Source: §2.3 · p. 9
  • Mel VAE training takes roughly 5 hours, 4-channel STFT VAE training 10 hours, and CVAE training 40 hours at approximately 25 epochs or 160 hours at 100 epochs.The CVAE runtime assumes early stopping at approximately 25 epochs.Source: §2.3 · pp. 9–10
  • The CVAE encodes binaural complex STFT through five complex convolutional layers and three 256-dimensional branches for means, log-variances, and pseudovariances.A sampled latent vector passes through two complex linear layers and five transposed convolutions; the final real-valued Tanh maps output coefficients to [−1, 1].Source: Figure 4 · p. 10
  • A maximum β weight of 0.8 is linearly warmed up and then adjusted cyclically to balance KL-divergence and reconstruction losses and prevent latent-space collapse.The resulting periodic loss oscillations are expected; training should continue while the macro-level loss trend descends.Source: §2.3 · p. 10
  • The pipeline enforces symmetric encoder-decoder structure and configurable initialization and convolutional parameters, but excludes the manually tuned CVAE from automated grid search.Concurrent grid searches require careful GPU-memory and CPU-worker allocation to avoid bottlenecks and prohibitive training times.Source: §2.3 · p. 10
  • Trained Mel VAE, 4-channel STFT VAE, and CVAE models are publicly accessible on Hugging Face.Source: §2.3 · p. 10

2.4 Inference and Dreaming

Inference reconstructs unseen navigational trajectories and converts model outputs back into spatialized waveforms. Mel reconstruction requires phase estimation, while 4-channel STFT and complex-valued models recover waveforms directly; the dreaming module generates latent interpolations as a qualitative probe of learned representations.

2.4 Inference and Dreaming

  • Inference runs trained VAEs on unseen trajectories and converts reconstructed representations to spatialized waveforms in their original modality dimensions.For Mel spectrograms, waveform synthesis uses either Griffin-Lim [GL84] or BigVGAN-v2.Source: §2.4 · p. 11
  • 4-channel STFT and complex-valued VAEs preserve magnitude and phase, bypass phase estimation, and recover waveforms directly through ISTFT.Source: §2.4 · p. 11
  • Mel reconstruction requires phase estimation: Griffin-Lim produces recognizable audio with metallic artifacts, while untuned BigVGAN-v2 produces less recognizable audio with excessive noise.Griffin-Lim remains the default Mel reconstruction method because BigVGAN-v2 could not be fine-tuned to the study's configurations and dataset within the available time.Source: §2.4 · p. 11
  • The dreaming module synthesizes novel acoustic sequences by linearly interpolating between at least two randomly sampled latent keyframes, using 32 frames by default.This qualitative exercise probes continuity and topological organization of the learned representations.Source: §2.4 · p. 11

3 Results

The three VAEs show a trade-off between reconstruction loss and perceptual spatial fidelity: the Mel VAE has the lowest reported loss, while the Complex VAE outperforms the 4-Channel STFT VAE perceptually and preserves spatialization. The 4-Channel STFT VAE’s phase predictions collapse toward a constant, degrading its synthesized audio.

3 Results

  • Evaluation used holdout-test reconstruction loss (MSE) for output fidelity and KL divergence as an indicator of latent-space regularization.The report states that the empirical results align with theoretical expectations.Source: §3 · p. 11
  • Mel VAE had the lowest reconstruction loss (19.78), while Complex VAE audio was perceptually superior to 4-Channel STFT and preserved spatialization, unlike its counterpart.The 4-Channel STFT VAE regressed toward a near-constant phase value, degrading its audio; the Complex VAE instead struggled with high-frequency content.Source: §3.1 · p. 11, §3.2 · p. 12, Figure 6 · p. 12

3.1 Mel Spectrogram VAE

  • The Mel Spectrogram VAE achieved reconstruction loss 19.78 and KL-divergence 7.43, reproducing macroscopic structure but losing fine spectral detail.Its synthesized output also suffers metallic artifacts from phase estimation during STFT inversion, despite perceptually resembling the input and preserving spatialization.Source: §3.1 · pp. 11–12, Figure 5 · p. 12

3.2 4-Channel STFT VAE

  • The 4-Channel STFT VAE achieved reconstruction loss 5081.71 and KL-divergence 18.67; its near-constant phase predictions caused severe signal degradation and loss of spatialization.Magnitude reconstruction was adequate, but phase prediction was problematic; the resulting audio retained basic amplitude envelopes and had little perceptual resemblance to the input.Source: §3.2 · p. 12, Figure 6 · p. 12

3.3 Complex VAE

  • The Complex VAE achieved reconstruction loss 625.83 and KL-divergence 4.73, reconstructing low-frequency magnitude and phase while struggling with high-frequency content.Its muffled or filtered output retained structural and spatial fidelity, and its perceptual results were far superior to the 4-Channel STFT VAE despite higher overall MSE than the Mel VAE.Source: §3.3 · p. 13, Figure 7 · p. 13

4 Conclusion

The conclusion frames spatial-audio latent representations for navigation and spatial understanding as an underexplored area. It presents an open-source VAE reconstruction pipeline as a foundation for future audio-based spatial state representations.

4 Conclusion

  • Learning latent representations of spatial-audio footprints for navigation and spatial understanding remains underexplored, especially in open-source work and world-model development.The conclusion describes auditory data as a natural progression alongside visual spatial understanding because humans use acoustic cues for spatial awareness and navigation.Source: §4 · p. 13
  • The work presents a pipeline of generalizable VAEs that learns spatial-audio latent representations through reconstruction.The pipeline includes simulated data acquisition via AudioWorldSim, signal processing, model training, latent-space sampling, and pretrained weights.Source: §4 · p. 13
  • The pipeline’s baselines, including a mathematically grounded complex-valued approach, are intended as a foundation for future audio-based spatial state representations in world models and navigation systems.The authors invite the community to build on the open-source ecosystem.Source: §4 · p. 13

5 Limitations and Future Work

The authors identify further development of the Complex VAE and evaluation of learned representations in downstream spatial reasoning as future work. The report also calls for broader data, architecture, and training-dynamics investigation.

5 Limitations and Future Work

  • The Complex VAE, identified as the most promising architecture, requires further investigation and refinement.The authors propose scaling the training dataset and extending architecture and hyperparameter optimization.Source: §5 · p. 13
  • Future work should diagnose why training loss typically converges early, between 25 and 30 epochs.Source: §5 · p. 13
  • The audio representations have not yet been evaluated in a complete reinforcement-learning pipeline or in purely audio-based spatial-reasoning tasks.The authors also propose testing whether existing visual world models benefit from adding auditory spatial representations.Source: §5 · p. 14

6 Ethical Statement and AI Usage Declaration

The report states that its experiments and datasets are generated in synthetic simulation environments and describes the use of Gemini 3.1 chat for coding and writing assistance.

6 Ethical Statement and AI Usage Declaration

  • The authors report no specific ethical, human-subject privacy, or safety concerns, and state that all experiments and datasets use synthetic simulations.Source: §6 · p. 14
  • Gemini 3.1 chat assisted with code generation and writing, while the authors state that core ideas, technical explanations, and the initial draft were human-written and the final text was human-revised.Source: §6 · p. 14
Loading 2609.06837v1…