Source-linked AI summary
SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning
Changan Chen, Carl Schissler, Sanchit Garg, Philip Kobernik, Alexander Clegg, Paul Calamia, Dhruv Batra, Philip W Robinson, Kristen Grauman
TL;DR
Visual-acoustic research needs realistic sound tied to 3D environments, but earlier platforms constrain continuous sampling, new-environment use, or configuration. SoundSpaces 2.0 renders acoustics on the fly from meshes, and its benchmarks report sim-to-real promise, including for speech recognition.
Problem
Earlier audio-visual platforms constrain continuous spatial sampling, generalization to novel environments, or configurable microphones and materials.
Source: §1 · p. 2Method
SoundSpaces 2.0 performs on-the-fly geometry-based rendering for arbitrary environments, with configurable source-receiver locations, simulation parameters, materials, and microphones.
Source: §1 · pp. 2–3Results
Across two downstream tasks, benchmarks show encouraging evidence of real-world generalization; in far-field speech recognition, synthetic-IR fine-tuning lowers WER relative to prior simulated IRs.
Source: §1 · p. 3, §5.4 · p. 9Takeaways & Limitations
The publicly released simulator, real measurements, and SoundSpaces-PanoIR dataset provide resources for visual-acoustic learning, including stand-alone dataset use.
Source: §7 · p. 10Accurate rendering requires meshes without large holes, and material acoustics cannot be inferred from meshes alone, so common material properties are assigned to objects.
Source: §6 · p. 10
Abstract
from arXiv · showhide
We introduce SoundSpaces 2.0, a platform for on-the-fly geometry-based audio rendering for 3D environments. Given a 3D mesh of a real-world environment, SoundSpaces can generate highly realistic acoustics for arbitrary sounds captured from arbitrary microphone locations. Together with existing 3D visual assets, it supports an array of audio-visual research tasks, such as audio-visual navigation, mapping, source localization and separation, and acoustic matching. Compared to existing resources, SoundSpaces 2.0 has the advantages of allowing continuous spatial sampling, generalization to novel environments, and configurable microphone and material properties. To our knowledge, this is the first geometry-based acoustic simulation that offers high fidelity and realism while also being fast enough to use for embodied learning. We showcase the simulator's properties and benchmark its performance against real-world audio measurements. In addition, we demonstrate two downstream tasks -- embodied navigation and far-field automatic speech recognition -- and highlight sim2real performance for the latter. SoundSpaces 2.0 is publicly available to facilitate wider research for perceptual systems that can both see and hear.
1 Introduction
Visual and acoustic observations are linked by scene geometry and materials, yet prior audio-visual simulators constrain continuity, environment coverage, or configuration. SoundSpaces 2.0 introduces on-the-fly geometry-based audio rendering and evaluates its realism and sim-to-real use.
1 Introduction
- Modeling visual-acoustic correspondence matters for embodied AI and AR/VR applications such as locating callers, monitoring appliances, and producing acoustically consistent experiences.Materials and geometry transform the sounds that reach listeners.Source: §1 · p. 1
- Earlier platforms restrict audio-visual research through precomputed discrete locations and fixed environments, or continuous rendering limited to simplified shoebox geometry.Precomputed data also makes changing microphone or material configurations difficult.Source: §1 · p. 2
- SoundSpaces 2.0 renders audio on the fly from environment geometry, modeling reflections, reverberation, transmission, diffraction, and frequency-dependent effects.Its binaural output preserves spatial correspondence between visual scenes and received sound.Source: §1 · p. 2, Figure 1 · p. 2
- The platform brings audio to established 3D assets including Gibson, HM3D, and Matterport3D, while allowing users to configure locations, simulation parameters, materials, and microphones.The rendering platform and research codebase are publicly available.Source: §1 · p. 3
- Real-world RIR measurements benchmark accuracy, while navigation and far-field speech recognition tests show sim-to-real generalization, improved by acoustic randomization.The work also releases SoundSpaces-PanoIR, a large-scale dataset of images paired with simulated RIRs.Source: §1 · p. 3
2 Related Work
Prior acoustic simulation spans physical, geometric, and interactive approaches, while newer audio-visual tools support learning tasks but retain limits in realism, flexibility, or real-world evaluation. SoundSpaces 2.0 builds on path tracing and targets those gaps.
Acoustic simulation
- Path-tracing approaches offer high accuracy and efficiency, whereas wave-based methods are computationally expensive and image-source methods have lower accuracy for late reverberation.Other simulators also model impact sounds between objects.Source: §2 · p. 3
- SoundSpaces 2.0 builds on SoundSpaces [14] path tracing, adding on-the-fly rendering, diffraction, and improved reverberation-level accuracy while supporting arbitrary meshes and configurable materials and microphones.The comparison also highlights real-time rendering and generalization to new 3D meshes.Source: §2 · p. 3
Audio-visual learning
- Audio-visual simulation supports navigation and visual-acoustic matching, while acoustic rendering also serves speech recognition, separation, localization, and synthesis.SoundSpaces 2.0 revisits multiple tasks with continuous, configurable, and generalizable simulation.Source: §2 · p. 4
Simulation-to-reality transfer
- Acoustic-simulation sim-to-real transfer has been understudied because real-world benchmarks were lacking; this work benchmarks simulation against real measurements and tests transfer to real data.Prior transfer research has focused mostly on policies using visual input.Source: §2 · p. 4
3 SoundSpaces 2.0 Audio-Visual Rendering Platform
SoundSpaces 2.0 integrates geometry-based audio rendering with Habitat-Sim and offers continuous spatial sampling, configurable simulation properties, and support for arbitrary meshes. Its rendering modes balance audio quality with the speed needs of embodied learning.
3.1 Rendering Pipeline and Simulation Enhancements
- The Habitat-integrated pipeline computes impulse responses from scene meshes and source/receiver locations, modeling reflection, transmission, diffraction, and spatialization before producing binaural or ambisonic audio.The resulting impulse response can be convolved with source audio to generate sound at the receiver.Source: §3.1 · p. 4
3.2 Continuity
- Arbitrary source and receiver placements provide spatial continuity, while time-aligned audio segments and crossfading support continuous sound as listeners move.The earlier approach implicitly treated sound as stopping and restarting at each new listener location.Source: §3.2 · p. 5
3.3 Configurability
- Users can configure simulation parameters, choose built-in or custom microphone setups, and load custom head-related transfer functions.Options include monaural, binaural, and ambisonic microphones, as well as user-defined arrays.Source: §3.3 · p. 5
- Users can map object categories to acoustic materials with frequency-dependent absorption, scattering, and transmission properties.The platform provides 29 built-in materials, and users can specify frequency-dependent damping coefficients for air and materials.Source: §3.3 · pp. 5–6
3.4 Generalizability
- The simulator accepts arbitrary scene meshes, supports material-configurable shoebox rooms, and evaluates generalization to real-world RIRs.It is compatible with existing and future scene datasets, while shoebox-room APIs support simpler setups.Source: §3.4 · p. 6
3.5 Rendering Modes and Rendering Performance
- High-speed mode reuses information from previous frames to reduce ray computations, while high-quality mode maximizes rendering parameters and disables temporal coherence.The two modes target the speed needs of embodied learning and the accuracy needs of high-quality impulse responses, respectively.Source: §3.5 · p. 6
4 Large-scale SoundSpaces-PanoIR Dataset
SoundSpaces-PanoIR addresses the lack of large-scale, accurate image–impulse-response datasets for visual-acoustic learning. It provides panoramic visual and acoustic data rendered across hundreds of environments.
4 Large-scale SoundSpaces-PanoIR Dataset
- SoundSpaces-PanoIR contains 10M panoramic image and impulse-response pairs rendered from 750 Matterport3D, Gibson, and HM3D environments.The dataset was composed with SoundSpaces 2.0 as an offline resource for researchers.Source: §4 · p. 7
- Each example provides panoramic RGB and depth, an impulse response, and polar coordinates of the source relative to the panorama center.Source: §4 · p. 7
- The dataset targets visual-acoustic learning tasks for which large-scale accurate image–impulse-response datasets were lacking because data collection is expensive and complex.Source: §4 · p. 7
5 Evaluation and Benchmarks
The evaluation measures rendering speed and acoustic realism against real measurements, then tests SoundSpaces 2.0 on continuous audio-visual navigation and far-field speech recognition. The results show stronger sim2real ASR performance, alongside improved acoustic matching and navigation under continuous spatial and acoustic conditions.
5.1 Simulation Speed vs. Quality Tradeoff
- 8× faster with one thread and 33× faster with five threads, high-speed rendering loses 9.5% accuracy measured as relative RT60 error against high-quality mode.A navigation model trained in high-speed mode differed by less than 1% when tested in high-quality mode.Source: §5.1 · p. 7
5.2 Validating Simulation Accuracy with Real IRs
- 0.98 dB versus 11.0 dB average DRR error for SoundSpaces 2.0 and SoundSpaces, respectively, while both have 12.4% relative RT60 error.The seven-position Replica comparison used measurements scaled to simulation direct-sound levels, with mesh materials optimized to match measurements.Source: §5.2 · pp. 7–8, Figure 3 · p. 8
5.3 Benchmark 1: Continuous Audio-Visual Navigation
- 64.2% success rate and 27.5% SPL (SoundSpaces) versus 64.7% success rate and 49.3% SPL (SoundSpaces 2.0) when only space is continuous; full continuity further improves success and goal proximity.The authors attribute the baseline drop under continuous sound to its reliance on an inaccurately persistent direct-sound cue.Source: Table 3 · p. 8, §5.3 · p. 9
5.4 Benchmark 2: Far-Field Automatic Speech Recognition
- SoundSpaces 2.0 synthetic-IR finetuning produced lower far-field WER than Pyroomacoustics, SoundSpaces 1.0, and finetuning on real IRs.The evaluation also reports that synthetic-IR finetuning improves substantially over the pretrained model on far-field speech.Source: §5.4 · p. 9, Table 4 · p. 9
- 12.04% WER with acoustic randomization versus 12.48% without; uniform 0–1 coefficient sampling yielded 12.58% WER using equal finetuning data.The randomization strategy varies category-level material assignments and adds N(0, 0.1) noise to material coefficients.Source: §5.4 · pp. 9–10
6 Discussion on Limitations and Future Work
SoundSpaces 2.0 is presented as a general, accessible platform with experiments illustrating its potential. Its accuracy depends on mesh quality and material assumptions, and validation so far uses real impulse responses from one Replica apartment.
Discussion
- The authors see the platform as general and accessible, with experiments providing concrete examples of its potential for embodied AI, multimodal perception, and audio research.Source: §6 · p. 10
Limitations
- Low-quality meshes with large holes can leak rays and produce inaccurate simulations; users can inspect ray leakage and repair meshes.The authors also note that path tracing remains vulnerable to room modes, while their implementation addresses the typical lack of diffraction.Source: §6 · p. 10
- Acoustic material properties cannot be accurately inferred from meshes alone, so the simulator assigns common material properties to objects.The authors suggest that acoustic measurements during environment scanning could support more in-depth material treatment.Source: §6 · p. 10
Future Work
- Validation uses real impulse responses collected in one Replica apartment; future work could measure diverse environments with varying geometry and materials.Source: §6 · p. 10
7 Conclusion
SoundSpaces 2.0 provides on-the-fly geometry-based audio rendering for 3D environments. The authors report real-measurement validation and encouraging evidence of real-world generalization on two tasks, and release the platform and measurements for research.
Conclusion
- The authors report encouraging evidence that systems trained with SoundSpaces 2.0 generalize to the real world across two benchmarked tasks.Source: §7 · p. 10
- SoundSpaces 2.0 performs on-the-fly geometry-based audio rendering for 3D environments, and the authors release the simulation and real measurements.Source: §7 · p. 10
- The authors also release the large-scale SoundSpaces-PanoIR dataset for research.Source: §7 · p. 10
8 Supplementary
The supplementary material lists additional video examples, implementation and interface details, dataset statistics, benchmark comparisons, and a societal-impact discussion.
Supplementary video
- The supplementary video provides qualitative simulation examples, real-measurement comparisons, acoustic-continuity demonstrations, PanoIR examples, navigation videos, and a far-field ASR example.Source: §8 · p. 15
Supplementary details
- Additional material describes RLR-Audio-Propagation and the simulator APIs and interface.Source: §8 · p. 15
- The supplement provides further details on material configuration and PanoIR dataset statistics.Source: §8 · p. 15
- Additional material compares RT60 with real measurements and gives navigation-benchmark training and implementation details.Source: §8 · p. 15
- The supplement also provides training and implementation details for the ASR benchmark.Source: §8 · p. 15
Societal impacts
- A supplementary section discusses the societal impacts of the work.Source: §8 · p. 15
8.1 Supplementary video
The supplementary video demonstrates the simulator and downstream examples. The propagation details describe bidirectional ray tracing, accumulation of path energy by delay, and conversion to spatialized pressure impulse responses for convolution with source audio.
8.1 Supplementary video
- The video demonstrates simulation examples, comparisons with real measurements, acoustic continuity, PanoIR, navigation, and far-field ASR.Source: §8.1 · p. 15
8.2 Details of RLR-Audio-Propagation
- Source rays carry energy spectra in log-spaced frequency bands and propagate through reflection, diffraction, and transmission up to a maximum bounce count.Reflection behavior uses a Phong BRDF whose exponent depends on material scattering; diffraction occurs probabilistically at edge geometry, and transmission depends on material coefficients.Source: §8.2 · p. 15
- Listener rays connect to sources and source-ray vertices; completed paths contribute energy by frequency band to delay histograms and spherical-harmonic coefficients.Direct sound and early reflections of at most two bounces are handled separately, with early-reflection paths clustered by reflecting-surface planes.Source: §8.2 · p. 15
- The energy-based impulse response becomes a pressure response by taking square roots of histogram energy and synthesizing random phase, then spatialization yields ambisonic or binaural responses.Binaural conversion convolves with an ambisonic HRTF representation; direct-sound spatial fidelity uses barycentric interpolation of original HRTF data.Source: §8.2 · p. 15
8.3 Details of APIs and Interface
The interface exposes simulation and microphone settings, while material configuration combines an acoustic-material database with mappings from objects to materials.
8.3 Details of APIs and Interface
- The simulation API exposes controls for sampling, frequency bands, spatialization, computation, IR settings, propagation effects, and material properties.It also includes options such as mesh simplification and temporal coherence for faster computation.Source: §8.3 · p. 16
- Seven built-in microphone types range from mono and stereo to surround formats and ambisonics, and users can configure custom arrays of mono microphones.The channel-layout API accepts the channel type and number of channels.Source: §8.3 · p. 16
8.4 Details of Material Configuration
- Material configuration combines an acoustic-material database with a mapping from objects to materials.Configuration files define material coefficients, default category mappings, one-to-many plausible material mappings for randomization, and the combined input configuration.Source: §8.4 · p. 16
8.5 Details of PanoIR Dataset
The PanoIR dataset provides panoramic images and impulse responses with source-location metadata, and the section compares simulated RT60 with real measurements. RT60 distributions vary across scene datasets, while SoundSpaces and SoundSpaces 2.0 have the same reported average relative RT60 error against measurements.
8.5 Details of PanoIR Dataset
- PanoIR provides panoramic RGB/depth images and IRs, with source coordinates recorded as angle and distance from the panorama center.Panoramas stitch 18 views with 20-degree FoV each into 1152×384 images; source and receiver heights are 1.5m, and their separation is limited to 5m.Source: §8.5 · p. 16
- RT60 distributions vary across Matterport3D, Gibson, and HM3D, with Matterport3D skewing lower and Gibson and HM3D having larger averages.The authors attribute the Matterport3D skew mainly to broken meshes that let rays leak through holes, reducing reverberation; mesh quality, scene types, and materials also contribute to variation.Source: §8.5 · p. 16, Figure 4 · p. 17
8.6 RT60 Comparison with Real IRs
- Both SoundSpaces and SoundSpaces 2.0 have a 12.4% average relative RT60 error against measurements at seven Replica apartment positions in the 250Hz to 4000Hz band.Together with the DRR comparison in Fig.3(b), the authors report higher acoustic realism for SoundSpaces 2.0 than for the original SoundSpaces.Source: §8.6 · p. 16, Figure 5 · p. 17
- The real measurements used for the RT60 comparison are included in supplementary materials and made available for researchers to test their models.The authors report the data as publicly available.Source: §8.6 · p. 17
8.7 Training and Implementation Details of the Navigation Benchmark
The navigation benchmark trains an agent with continuous movement and audio, while the ASR benchmark fine-tunes on speech convolved with simulated impulse responses and evaluates on speech convolved with real responses. The discussion reports no negative societal impact of the work that the authors are aware of.
8.7 Training and Implementation Details of the Navigation Benchmark
- The continuous audio-visual navigation policy is trained for 80 million steps, with success defined as stopping within 1m of the goal.Training uses AV-Nav with DD-PPO on the AudioGoal dataset, with continuous movement and audio, and takes 46 hours on 32 GPUs to converge.Source: §8.7 · p. 17
8.8 Training and Implementation Details of the ASR Benchmark
- The ASR model is fine-tuned on speech convolved with 28,539 randomly configured SoundSpaces IRs and evaluated on test-clean speech convolved with real BUT ReverbDB IRs.Fine-tuning uses 24-example batches for 60 epochs on eight GPUs; the Pyroomacoustics comparison also uses 28,539 generated IRs.Source: §8.8 · p. 17
8.9 Discussion on Societal Impacts
- The authors state that they are not aware of negative societal impacts and believe the work will open possibilities for visual-acoustic learning research.They note potential applications in robotics and AR/VR.Source: §8.9 · p. 17