PaperlayerView exact revision on arXiv

Source-linked AI summary

SoundSpaces: Audio-Visual Navigation in 3D Environments

Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicenc Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, Kristen Grauman

arXiv:1912.11474v3cs.CVcs.HCcs.SDeess.AS

TL;DR

Embodied navigation agents rely on vision despite sound’s potential to reveal space and locate targets outside view. This paper introduces end-to-end audio-visual navigation and SoundSpaces, a simulator for scanned 3D environments. The reported results show that audio and vision together improve navigation and learning.

  • Problem

    Current navigation models integrate seeing and movement but lack sound, which can reveal space geometry and locate sound-emitting targets beyond visual range.

    Source: §1 · p. 2

  • Method

    The authors train end-to-end multimodal deep RL navigation policies and provide precomputed audio renderings for Matterport3D and Replica in Habitat.

    Source: §1 · p. 3

  • Results

    Audio and vision together yield faster learning and faster, more accurate navigation, while audio competes well with goal-displacement vectors without assuming perfect GPS odometry.

    Source: §1 · p. 3, §7 · p. 14

  • Takeaways & Limitations

    The work provides a public audio-rendering platform and benchmark suite for audio-visual navigation research.

    Source: §1 · p. 3

    The tasks assume an idealized GPS sensor where GPS is used, while AudioGoal omits GPS and the target itself has no visual presence.

    Source: §4 · p. 7

Abstract

from arXiv · show

Moving around in the world is naturally a multisensory experience, but today's embodied agents are deaf---restricted to solely their visual perception of the environment. We introduce audio-visual navigation for complex, acoustically and visually realistic 3D environments. By both seeing and hearing, the agent must learn to navigate to a sounding object. We propose a multi-modal deep reinforcement learning approach to train navigation policies end-to-end from a stream of egocentric audio-visual observations, allowing the agent to (1) discover elements of the geometry of the physical space indicated by the reverberating audio and (2) detect and follow sound-emitting targets. We further introduce SoundSpaces: a first-of-its-kind dataset of audio renderings based on geometrical acoustic simulations for two sets of publicly available 3D environments (Matterport3D and Replica), and we instrument Habitat to support the new sensor, making it possible to insert arbitrary sound sources in an array of real-world scanned environments. Our results show that audio greatly benefits embodied visual navigation in 3D spaces, and our work lays groundwork for new research in embodied AI with audio-visual perception.

1 Introduction

Navigation agents integrate vision with movement but remain deaf, although sound can reveal space geometry and locate targets outside visual range. This work introduces audio-visual navigation tasks, an end-to-end learning approach, and a sound-rendering platform, reporting that audio and vision together improve navigation.

1 Introduction

  • Audio can reveal geometry and occlusions while locating sound-emitting targets beyond visual range, addressing a sensory gap in current navigation models.The introduction points to targets such as a caller upstairs or a ringing phone occluded by furniture.Source: §1 · p. 2
  • The paper defines AudioGoal, where sound identifies the target, and AudioPointGoal, where a goal-location cue is also provided.The tasks represent, respectively, an initially unseen sounding target and navigation with a GPS pointer in an unfamiliar environment.Source: §1 · p. 2
  • An end-to-end multimodal deep RL policy learns to navigate in novel, unmapped environments from audio-visual observations.The approach pairs audio renderings for Matterport3D and Replica with Habitat for scalable simulation.Source: §1 · p. 3
  • Audio and vision together yield faster learning and faster, more accurate inference-time navigation, while audio competes with goal-displacement vectors without assuming perfect GPS odometry.The authors also explore generalization to unseen environments and unheard sounds.Source: §1 · p. 3
  • The contributions include a platform instrumenting 103 Matterport3D and Replica environments and a benchmark suite for audio-visual navigation.The platform supports arbitrary sound sources and receiver positions.Source: §1 · p. 3

2 Related Work

Prior work studies visual navigation, human audio navigation, and audio-visual tasks such as localization, but does not establish audio-visual navigation by agents in realistic, unmapped 3D environments. The paper positions its end-to-end agent and realistic audio-visual simulator against those gaps.

2 Related Work

  • Audio-visual learning spans sound synthesis, spatialization, separation, cross-modal features, tracking, and material learning, but prior work often localizes sound-associated pixels in video rather than navigating agents.This paper addresses active navigation in 3D rather than learning from human-captured video.Source: §2 · pp. 3–4
  • Visual navigation research uses egocentric inputs and spatial memory, while this work adds hearing to navigation.The cited visual-navigation work also connects navigation to question answering, recognition, and instruction following.Source: §2 · p. 4
  • Prior autonomous-agent audio navigation studies used game boards, acoustically incorrect models, and the same environment for training and testing.Human-facing virtual environments were non-photorealistic and did not support AI-agent training.Source: §2 · p. 4
  • Concurrent work predicts a goal location from audio before planning, whereas this work uses end-to-end RL in visually and acoustically more realistic environments.The paper reports 5,000× more audio data and 15× more environments than.Source: §2 · p. 4
  • The authors present the first audio-visual simulator for AI-agent training and first study of audio-visual embodied agents in realistic 3D environments.Commonly used realistic 3D environments and simulators lacked audio rendering.Source: §2 · pp. 4–5

3 SoundSpaces: Enabling Audio in Habitat

SoundSpaces augments Habitat with acoustically realistic audio for scanned Matterport3D and Replica environments. It precomputes room responses across sampled source-listener positions, enabling binaural rendering at agent locations while accounting for room geometry and materials.

3 SoundSpaces: Enabling Audio in Habitat

  • The platform augments Habitat, an open-source 3D simulator with fast rendering and support for multiple datasets.Habitat provides RGB, depth, and semantic rendering through its API.Source: §3 · p. 5
  • The platform extends Habitat to 85 Matterport3D environments and 18 Replica scenes, and shares the audio platform and data publicly.Matterport3D environments average 517 m2 of floor space; Replica scenes include apartments, hotels, offices, and rooms.Source: §3 · p. 5
  • SoundSpaces models room reflections with bidirectional path tracing and maps mesh semantics to material properties that affect sound propagation.The model includes absorption, scattering, and transmission, including propagation through walls.Source: §3 · p. 5
  • For each scene, room impulse responses encode the transfer function between a sound source and microphone as a function of geometry, materials, and source location.The responses are precomputed for source-listener placements on a dense grid at 0.5 m resolution in Replica and 1 m in Matterport3D.Source: §3 · p. 6
  • The sampled source and listener positions cover Replica scenes with N from 38 to 566 and Matterport3D scenes with N from 20 to 2,103.The locations are sampled at a fixed height of 1.5 m, and responses are simulated for each source-listener pair.Source: §3 · p. 6
  • At runtime, the simulator renders audio for a source-listener pair by convolving the chosen waveform with its precomputed response.It can render ambisonic audio converted to binaural audio, including multiple simultaneous sounds.Source: §3 · p. 6
  • A connectivity graph removes non-navigable positions and links immediate neighbors only when their geodesic and Euclidean distances match.This excludes links with an obstacle between the sampled locations.Source: §3 · p. 6

4 Task Definitions: Audio-Visual Navigation

The paper defines navigation tasks in which agents use sound, vision, and, in one variant, goal displacement to reach targets without a scene map. AudioGoal relies on sound without direct goal-position information, while AudioPointGoal adds displacement guidance.

Task definitions

  • AudioGoal uses sound from the target without direct goal-position information; AudioPointGoal adds the displacement vector used in PointGoal navigation.All three tasks require reaching the target using sensory inputs alone, without a scene map.Source: §4 · pp. 6–7
  • In AudioGoal and AudioPointGoal, the observed audio changes with the agent’s and goal’s locations and the room’s structure and materials.AudioPointGoal combines audio with the PointGoal displacement vector.Source: §4 · p. 7

Agent and goal embodiment

  • All tasks include vision for detecting and avoiding obstacles, while the target has no visual presence.The target is specified as 0.2 m in diameter and 1.5 m in height.Source: §4 · p. 7

Action space

  • The action set is MoveForward, TurnLeft, TurnRight, and Stop; forward movement is invalid without a connecting graph edge.A valid forward action moves 0.5 m in Replica or 1 m in Matterport, with no actuation noise.Source: §4 · p. 7

Sensors

  • Agents receive binaural sound except in PointGoal, GPS except in AudioGoal, and RGB and depth inputs.The binaural microphones are placed at human height; the GPS sensor is idealized.Source: §4 · p. 7

Episode specification

  • AudioGoal and AudioPointGoal episodes convolve a source waveform with the room impulse response for the scene, goal, and agent pose; success requires stopping at the goal within 500 actions.The source set includes familiar and unfamiliar audio.Source: §4 · p. 7

5 Navigation Network and Training

The navigation policy processes egocentric audio-visual observations and accumulated history to choose actions and estimate state value. Its training uses PPO, while the inputs combine separate audio and visual encoders with goal displacement when available.

Navigation policy

  • The policy maps new multisensory egocentric observations to actions, learning through reward for correct and efficient navigation.The agent accumulates partial observations while entering new, unmapped spaces.Source: §5 · pp. 7–8

Sensory inputs

  • Audio inputs are log-magnitude STFT spectrograms from left and right channels, stacked as a tensor; visual inputs are RGB and/or depth images, and goal displacement is a 2D vector.The spectrograms use the first 1,000 ms of audio; visual inputs have 128 × 128 resolution.Source: §5 · p. 8

Network architecture

  • Separate CNNs encode audio and visual inputs; their features are concatenated with relative goal displacement and passed through a GRU before actor and critic heads estimate actions and state value.The GRU updates its representation using the current input and accumulated history.Source: §5 · pp. 8–9

Training

  • Training uses PPO with rewards for stopping at the goal and reducing geodesic distance, penalties for time and increasing distance, and entropy maximization for exploration.The implementation uses Adam and trains for 30M agent steps on Replica and 60M on Matterport3D.Source: §5 · p. 9

Navigation network

  • Audio and visual cues provide complementary information: audio reflects target intensity and room acoustics, while neither modality alone performs as well as their combination.Room acoustics can reveal geodesic distances where goal displacement does not specify the optimal path.Source: §5 · p. 9

6 Experiments

Adding audio to visual navigation improves performance, while audio features supply spatial cues that can complement GPS and generalize across sound sources. The experiments also show that agents draw dynamically on audio and vision, and that full spectrograms provide useful information beyond sound intensity alone.

O.1: Does audio help navigation?

  • In Replica, AudioGoal and AudioPointGoal leave a starting room toward sound through a door, while PointGoal bumps into the wall; in Matterport, AudioGoal avoids backtracking.The trajectories illustrate how sound can guide agents to targets outside the current room or field of view.Source: Figure 4 · p. 11
  • Adding sound to sight and GPS sensing significantly improves navigation performance.The comparison uses SPL, with higher values indicating better performance.Source: §6 · p. 11, Table 2 · p. 12

O.2: Can audio supplant GPS for an audio target?

  • AudioPointGoal degrades more gracefully than PointGoal as GPS noise increases, while 40dB SNR lowers AudioGoal SPL only from 0.756 to 0.753 on Replica and 0.552 to 0.550 on Matterport.AudioGoal accuracy is independent of GPS failures; the authors suggest Replica's multi-room trajectories may explain why AudioGoal outperforms PointGoal and AudioPointGoal there.Source: §6 · pp. 12–13

O.1: Does audio help navigation?

  • Across sampled timesteps, the agents shift between audio-led turns and vision-led obstacle or free-space decisions, with sound intensity also informing stopping.The modality impact analysis finds that agents draw dynamically on either or both modalities to guide movement.Source: Figure 6 · p. 13, §6 · p. 13

O.2: Can audio supplant GPS for an audio target?

  • With intensity-only audio, AudioGoal-Depth SPL falls to 0.291 and 0.014, versus 0.756 and 0.552 with full spectrograms.The authors interpret this drop as evidence that full spectrograms provide environment information beyond an intensity gradient.Source: §6 · p. 13

O.3: What is the effect of different sound sources?

  • AudioPointGoal almost always outperforms PointGoal, including with unheard sounds, while AudioGoal accuracy declines for varied heard and unheard sounds.AudioPointGoal performs similarly on heard and unheard sounds; the authors expect larger training sound repositories to reduce AudioGoal's decline.Source: §6 · p. 14, Table 3 · p. 14

7 Conclusion

The conclusion presents audio-visual navigation in complex 3D environments and reports that audio enriches both target-direction cues and spatial information when combined with egocentric vision. It also introduces Habitat audio rendering for Replica and Matterport3D, while identifying several directions for future work.

Conclusion

  • The work introduces audio-visual navigation in complex 3D environments and extends a deep RL navigation engine to use audio with egocentric visual observations.The authors report that the model leverages directional cues for sound sources and spatial information about the environment for better navigation.Source: §7 · p. 14
  • The work enables audio rendering in Habitat for the publicly available Replica and Matterport3D environments, which the authors say can facilitate future research.Source: §7 · p. 14
  • The authors identify multi-agent scenarios, sim2real transfer, moving sound-emitting targets, and dynamic audio events as future directions.Source: §7 · p. 14

8 Supplementary Material

The supplement details audio simulation and RL training, then adds visualizations, baseline analysis, sound-split specifications, and trajectory examples.

8.1 Audio Simulation Details

  • Source and listener points use 0.5 m Replica or 1 m Matterport grids, then undergo enclosure and wall-distance pruning; Replica semantic labels inform acoustic material assignments.The usual enclosure threshold is 0.5, but open patio scenes use 0.1; points are also removed within 5 cm of geometry.Source: §8.1 · pp. 21–22
  • Bidirectional path tracing computes source-listener impulse responses across four frequency bands; second-order ambisonics encode spatial energy and are converted to binaural audio.Early reflections are handled separately, and the impulse responses can be convolved with arbitrary monaural source audio.Source: §8.1 · pp. 22–23

8.2 Visualizing Audio Simulations

  • Pressure decreases from the source along geodesic paths, making doors intermediate goals; ambisonics visualize incoming sound direction and intensity at receiver locations.The pressure gradient differs from a straight-line path, while the ambisonics visualization shows greater direct-sound energy at the location closer to the source.Source: §8.2 · p. 23, Figure 7 · p. 24, Figure 8 · p. 25

8.3 Reinforcement Learning Training Details

  • The RL agent receives audio, visual, and displacement-vector observations, chooses among four navigation actions, and optimizes expected discounted cumulative reward using PPO.The maximum horizon is 500 actions; MoveForward, TurnLeft, TurnRight, and Stop form the action set.Source: §8.3 · pp. 24–25

8.4 Audio Intensity Baseline

  • Audio intensity performs significantly worse than spectrogram input, which captures navigation cues beyond intensity, including relative goal angle and major obstacles.The intensity baseline uses the RMS of the left and right waveforms; its comparison includes blind, RGB, and depth visual inputs.Source: §8.4 · p. 25

8.5 Heard/Unheard Dataset Splits

  • The study uses 102 natural sounds split into 73 training, 11 validation, and 18 test sounds; the unheard-sound experiment tests on both unseen scenes and sounds.The varied-heard-sound experiment trains on 78 sounds and evaluates on unseen scenes with the same sounds.Source: §8.5 · p. 26

8.6 Additional Navigation Trajectory Examples

  • AudioGoal and AudioPointGoal agents navigate more efficiently than PointGoal in additional test environments from Replica and Matterport3D.The trajectory examples show three agents; path color fades from dark blue to light blue over time, with the shortest geodesic path in green.Source: §8.6 · p. 26, Figure 9 · p. 27
Loading 1912.11474v3…