Source-linked AI summary
AudioWorldSim: Realistic Binaural Audio Datasets For World Models
Luis Vitor Zerkowski, Luiz Velho
TL;DR
Realistic, scalable binaural datasets for spatial-navigation world models remain scarce, while existing simulators are not streamlined for dataset extraction. AudioWorldSim extends SoundSpaces 2.0 with automated navigation rollouts and corrected continuous-audio synthesis, producing artifact-free spatial audio for customizable large-scale datasets.
Problem
Realistic, open-source binaural datasets for spatial navigation are scarce, while existing simulators primarily target physical simulation and reinforcement learning rather than scalable dataset generation.
Method
AudioWorldSim extends SoundSpaces 2.0 with automated agent rollouts, action-conditioned acoustic recording, scalable batch generation, and zero-padding to correct continuous-audio transitions.
Results
The framework generates artifact-free spatial audio while preserving agent action-consequence relationships for audio-based world-model dataset creation.
Takeaways & Limitations
AudioWorldSim provides an open-source, configurable pipeline for generating high-fidelity binaural datasets tailored to audio-based world models.
Takeaways & Limitations
Material-specific sound propagation remains unreliable across simulation seeds, so the authors advise against using it for large-scale dataset generation.
Abstract
from arXiv · showhide
This technical report presents AudioWorldSim, an open-source platform designed to generate realistic binaural audio datasets and advance research in audio-based machine learning, particularly world models. Built as a custom extension of Meta's SoundSpaces 2.0 platform, AudioWorldSim leverages their comprehensive acoustics framework, but focuses on the automatic rollout of random agent navigations, as well as implements crucial fixes to how continuous sound is composed. AudioWorldSim is made publicly available to the research community at https://github.com/Luizerko/AudioWorldSim to facilitate reproducibility.
VISGRAF · 1 Introduction
AudioWorldSim addresses the scarcity of realistic, open binaural audio data for spatial navigation by supporting scalable acoustic-data generation for world models. Its stepwise recordings preserve action–sound consequences in each ear, enabling models to learn acoustic dynamics.
- 1 Introduction: Audio provides essential environmental information for world models, including event localization, occlusions, and material composition.World models learn environment dynamics, predict future states, and plan actions, while human perception depends heavily on multimodal integration.
- 1 Introduction: AudioWorldSim targets the shortage of realistic, open-source binaural datasets designed for spatial navigation rather than speech.Existing datasets are typically static, pre-rendered, and insufficiently generalizable for modeling complex physical interactions [BHYM23].
- 1 Introduction: AudioWorldSim extends existing simulation pipelines to address the practical challenge of generating continuous, massive acoustic data for machine learning.Underlying platforms were primarily designed for physical simulation and reinforcement learning rather than scalable audio-dataset generation.
- 1 Introduction: The framework automatically records agent actions at every simulation step, preserving the action–consequence relationship needed for world-model training.This connects each action with the resulting environmental audio across the rollout.
- 1 Introduction: The platform is publicly available to facilitate reproducible research in audio-based machine learning and world models.The introduction identifies AudioWorldSim as the response to practical data-generation difficulties and provides its public repository.
- 1 Introduction: AudioWorldSim captures how sound changes in each ear as the agent interacts with the environment.The resulting binaural dynamics are intended to support prediction of acoustic consequences during interaction.
2 The Simulator
AudioWorldSim extends SoundSpaces 2.0 with configurable NavMesh-based binaural rollouts, automated feature extraction, scalable generation, and corrected continuous-audio synthesis. Its design supports world-model training while retaining explicit action, trajectory, and spatial-audio outputs.
- Simulator Configuration: The simulator places a source and agent on one NavMesh floor, supporting static impulse-response computation and continuous temporal rollouts for dataset generation.The default audio sampling rate is 44.1 kHz, and rollout mode is the primary dataset-generation configuration.
- Navigation: Navigation supports randomized same-floor start and target points routed by shortest NavMesh paths, alongside manually specified configurations, using tools built on Habitat-Sim 3.Because physics is disabled following SoundSpaces 2.0, movement is kinematic and discrete; the default action advances 0.2 meters per 0.2-second step.
- Outputs and Processing: Each episode outputs continuous binaural audio, the agent’s discrete action sequence, and a top-down trajectory map, with optional navigation video for audio-visual alignment.The framework also triggers automated processing that prepares spectral features for complete audio sequences or discrete action steps.
- Feature Extraction: The pipeline provides full Mel spectrograms, reconstruction reference power, raw STFTs, and action-segmented features, including phase-preserving complex STFT representations for spatial modeling.Mel features use 2048-sample windows and 128 frequency bands, while each action’s raw STFT uses 1024 samples, 512 bands, and produces a 512 × 60 representation.
- Scalability and Limitations: Parallel generation scales to approximately 1500 trajectories and six hours of spatialized audio with 30 workers, while about one percent drift and five percent backend failures limit reliability.Drifted runs still produce shorter valid audio sequences and action logs suitable for training; depth-limited backtracking addresses pathing issues.
- Continuous Audio Correction: Zero-padding impulse responses before crossfading resolves SoundSpaces 2.0’s clicking artifact caused by spatially varying impulse-response lengths and temporal misalignment.The artifact becomes more pronounced and frequent near the sound source as direct sound increases and reverberation decreases.
3 Conclusion
AudioWorldSim addresses the scarcity of open-source binaural datasets with realistic acoustic propagation for spatial navigation by providing a robust, artifact-free generation pipeline. It enables researchers to generate large, customized audio datasets from compatible 3D scenes for audio-based world models.
- Open-source binaural datasets with realistic acoustic propagation for spatial navigation remain exceedingly rare, while existing datasets are often pre-rendered and lack generalizability [BHYM23].
- AudioWorldSim resolves critical dependencies and temporal audio bugs in SoundSpaces 2.0, producing artifact-free spatial audio.
- The platform enables researchers to generate massive, task-specific datasets using any compatible 3D scene.
4 Limitations and Future Work
AudioWorldSim streamlines continuous audio dataset generation but remains limited by unresolved material-based acoustics and licensing restrictions on distributing generated audio. Future work targets interactive simulation and synchronized audio-visual data generation.
- 4 Limitations and Future Work: Material-specific sound propagation in the underlying SoundSpaces 2.0 can compromise acoustic quality and consistency across simulation seeds.The authors advise against using material propagation until this upstream issue is resolved.
- 4 Limitations and Future Work: Licensing agreements for Matterport3D and Replica [SWM+19] prevented distribution of a massive pre-rendered acoustic dataset.As mitigation, the framework patches SoundSpaces 2.0 to automatically download, configure, and use a sample Matterport3D scene.
- 4 Limitations and Future Work: Future work will add an interactive simulator and deeper integration with SoundSpaces 2.0’s native RGB-D sensors for synchronized multimodal datasets.These extensions are intended to facilitate rapid acoustic-configuration testing and support audio-visual world models.
5 Ethical Statement and AI Usage Declaration
The work reports no specific ethical, privacy, or safety concerns because its experiments and datasets were generated in synthetic simulation environments. Gemini 3.1 chat assisted with code generation and report revision, while humans wrote the core ideas, explanations, and initial draft.
- The study identifies no specific ethical concerns, human-subject privacy risks, or safety considerations because all experiments and datasets use synthetic simulation environments.
- Gemini 3.1 chat assisted with code generation, including directly incorporated snippets requested from the AI.
- Gemini 3.1 chat also supported report writing through text restructuring, refinement, and language revision after humans wrote the core ideas, technical explanations, and full draft.