Source-linked AI summary
Look, Listen, and Act: Towards Audio-Visual Embodied Navigation
Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, Joshua B. Tenenbaum
TL;DR
The paper addresses how an agent can navigate from a random indoor starting point to a sound source using raw visual and audio observations. It proposes a framework combining spatial mapping, sound-source localization, and dynamic path planning, and reports stronger navigation performance than competitive baselines in the evaluated settings.
Problem
Audio-visual embodied navigation asks an agent to integrate visual and acoustic observations while sequentially navigating to a sound source in an indoor environment.
Method
The framework combines a spatial-memory mapper, sound perception for relative source location, and dynamic path planning, evaluated in explore-and-act and non-exploration settings.
Results
The proposed agent performs better than all baselines across metrics, exceeding 65% average success in exploration and reaching nearly 60% success without exploration.
Takeaways & Limitations
The results support connecting visual and sound information with room-layout reasoning to navigate toward sound sources and find shorter paths.
Abstract
from arXiv · showhide
A crucial ability of mobile intelligent agents is to integrate the evidence from multiple sensory inputs in an environment and to make a sequence of actions to reach their goals. In this paper, we attempt to approach the problem of Audio-Visual Embodied Navigation, the task of planning the shortest path from a random starting location in a scene to the sound source in an indoor environment, given only raw egocentric visual and audio sensory data. To accomplish this task, the agent is required to learn from various modalities, i.e. relating the audio signal to the visual environment. Here we describe an approach to audio-visual embodied navigation that takes advantage of both visual and audio pieces of evidence. Our solution is based on three key ideas: a visual perception mapper module that constructs its spatial memory of the environment, a sound perception module that infers the relative location of the sound source from the agent, and a dynamic path planner that plans a sequence of actions based on the audio-visual observations and the spatial memory of the environment to navigate toward the goal. Experimental results on a newly collected Visual-Audio-Room dataset using the simulated multi-modal environment demonstrate the effectiveness of our approach over several competitive baselines.
I. INTRODUCTION
The paper studies audio-visual embodied navigation, where an agent uses camera and microphone observations to reach a sound source through an indoor environment. It introduces a multi-modal environment and framework combining spatial memory, sound localization, and sequential planning.
- Problem: Audio-visual embodied navigation requires an agent to sequentially relate sound and visual observations while navigating to a sound source.The task begins from a random indoor location and seeks the shortest trajectory to the sound source.
- Challenges: The agent must understand visual surroundings, reason about potential sound-source locations, and handle targets located in different rooms.The paper connects these requirements to practical sound-source-seeking applications such as finding phones or turning off faucets.
- Approach: The proposed approach imitates human navigation by estimating sound location, using visual and audio observations, and updating spatial memory during movement.The paper studies both explore-and-act and non-exploration settings.
- Contributions: The work builds a multi-modal navigation environment with complex apartments and an integrated sound module that follows physical laws.This environment supports systematic evaluation of audio-visual navigation agents.
- Contributions: The paper presents the Visual-Audio Room benchmark and compares its framework with competitive baselines to evaluate multi-modal navigation.The authors report that experiments verify the approach’s effectiveness and expose challenges for future work.
II. RELATED WORK
Prior embodied-navigation work used scene maps, direct deep-learning policies, or language and vision, while audio-visual navigation requires integrating acoustic and visual cues to plan actions toward a sound source.
- Embodied navigation: Early navigation methods built scene maps with SLAM, while later deep-learning approaches planned actions directly from raw sensory data.Related methods include reactive reinforcement learning and mapper-planner architectures.
- Vision-language navigation: Vision-language embodied navigation maps language instructions and visual observations directly to actions.The paper distinguishes this setting from audio-visual navigation, whose goal is locating a sound source.
- Audio-visual navigation: Audio-visual embodied navigation requires integrating visual and acoustic cues to plan a sequence of actions.The paper notes concurrent work posing a similar task on the Habitat platform.
B. Sound Localization
The paper’s environment combines photo-realistic indoor scenes with spatial audio so agents can navigate using egocentric RGB images and sound. It extends isolated rooms into multi-room apartments for more challenging navigation.
- Environment: The multi-modal environment is built on AI2-THOR and augmented with Resonance Audio to provide camera and microphone observations.Agents perceive scenes through egocentric RGB images and sound.
- Environment: AI2-THOR supplies 120 indoor scenes across kitchens, bedrooms, living rooms, and bathrooms.Each category contains 30 rooms with diverse visual appearances.
- Environment: The authors concatenate isolated rooms into seven multi-room apartments to make area-goal navigation more challenging.The apartments are discretized into grid worlds for navigation.
B. Sound
The navigation system uses realistic spatial audio and supports both exploration-assisted and non-exploration settings. Its three-part pipeline maps visual observations, estimates sound direction and distance, and plans actions through the environment.
- Sound: Resonance Audio models spatial sound, acoustic ray tracing, material-dependent reverberation, and interaural auditory effects.The simulation accounts for object and house geometry and maps visual materials to acoustic materials.
- Problem setup: In explore-and-act navigation, random exploration builds an internal world model before goal-directed movement; without exploration, spatial memory is built while navigating.The two settings differ in when the agent constructs its environmental representation.
- Problem setup: At each step, the agent receives egocentric visual and sound observations and selects among backward, forward, left, right, and stop actions.Testing uses randomly selected starting and sound-source positions.
- Method: The algorithm consists of a visual perception mapper, an audio perception module, and a dynamic path planner.These components form the core architecture for audio-visual embodied navigation.
- Method: The mapper supplies either an exploration-based spatial memory or an occupancy-derived partial 2D map, after which sound perception estimates source direction and distance for planning.The planner uses these estimates and the available map to select navigation actions.
A. Visual Perception Mapper
The visual perception mapper builds spatial memory from visual observations or an on-the-fly occupancy map, while the sound perception module estimates the sound source’s relative location. These representations support agent localization and multimodal navigation.
- Explore-and-Act Visual Mapper: The exploration-based mapper stores visual observations as keys and location or action metadata as values in a spatial memory.RGB images are encoded with a ResNet feature extractor and normalized before storage.
- Explore-and-Act Visual Mapper: During goal-oriented navigation, the mapper retrieves the three closest memory slots and averages their coordinates to estimate the agent’s location.Similarity is computed using cosine distance between the current visual query and stored keys.
- Non-Exploration Visual Mapper: The non-exploration mapper constructs a dynamic top-down 2D occupancy map whose cells represent beliefs about traversability.This alternative is designed for settings where random exploration is not allowed.
- Non-Exploration Visual Mapper: A convolutional network predicts free-space probabilities in a 5 × 5 grid window from first-person RGB images.Training uses RGB images paired with ground-truth occupancy maps from two apartments.
- Sound Perception Module: The sound perception module estimates the sound source coordinates relative to the agent using audio-visual navigation evidence.Relative coordinates express the source’s position from the agent’s perspective, accounting for agent orientation.
C. Dynamic Path Planner
The dynamic planner updates its environment representation as actions reveal new information, then uses a graph and estimated sound-source coordinates to select a shortest path.
- Dynamic Planning: The planner dynamically updates memory while responding to observations and changes caused by the agent’s actions.This lets planning incorporate newly acquired environmental information during navigation.
- Graph Construction: In exploration-and-act navigation, the agent builds a partial graph from its trajectory and actions, with locations as nodes and consecutive steps as edges.The graph is represented as G = (V, E).
- Graph Construction: In non-exploration navigation, traversable grids become graph nodes and adjacent nodes become edges, while unexplored and unoccupied grids are assumed traversable.Newly detected obstacles modify the corresponding graph structure.
- Path Planning: At each time step, the planner seeks a shortest path from the agent’s graph node to the graph node nearest the estimated sound-source coordinates.The path is output as a sequence of actions leading toward the target.
- Path Planning: Dijkstra’s algorithm computes the shortest path used to generate the planner’s action sequence.The resulting path is represented as a sequence of actions.
V. EXPERIMENT
The experiments compare the proposed navigation model with competing baselines using trajectories and navigation performance in the audio-visual embodied navigation task.
- Evaluation: The experiment section evaluates the benefits of the proposed method against several competing baselines.The evaluation follows the description of the sound-source-seeking audio-visual embodied navigation task.
- Trajectory Comparison: Figure 4 compares the proposed model’s trajectories with those of A3C (V+A+M+Mapper).The proposed trajectories are shown in red and the baseline trajectories in blue.
A. The Visual-Audio Room Dataset
The Visual-Audio Room benchmark is introduced to evaluate multimodal navigation by combining visual and audio observations in simulated indoor environments.
- Dataset and Environment: The Visual-Audio Room benchmark was collected to systematically evaluate multi-modal navigation performance and support future study.The benchmark uses 3D simulated environments built on the AI-Thor platform because controlled physical-robot experiments are challenging.
B. Experiment Setup
The experiment compares seven navigation baselines using success rate and SPL, including random, sound-only, vision-only, audio-visual, and memory-augmented A3C variants.
- Evaluation: The evaluation uses success rate and Success weighted by Path Length (SPL) across testing scenarios.SPL accounts for success and path efficiency relative to the shortest distance to the goal.
- Baselines: Seven baselines include Random Walk, Greedy Search (A), and A3C variants with vision, audio-visual inputs, explicit maps, or memory.The A3C variants differ in whether they use visual input, audio-visual input, explicit occupancy maps, implicit LSTM memory, or both memory types.
- Baselines: Random Walk selects an action randomly at each time step, providing the simplest navigation heuristic.
- Baselines: Greedy Search (A) uses only sound perception and moves toward the source without observing apartment structure or layout.
- Baselines: A3C (V) is a vision-only, goal-oriented deep reinforcement-learning baseline without memory.A3C denotes asynchronous advantage actor-critic.
D. Results
The proposed audio-visual agent outperforms all baselines across metrics, with strong success rates in both exploration-and-act and non-exploration settings. Results also show that combining visual and sound information supports navigation in unseen layouts.
- Overall results: The proposed audio-visual agent performs better than all baselines across all reported metrics.Table I reports success rates and SPL for three sound sources across five testing apartments.
- Exploration-and-act: Over 65% average success was achieved for every sound source in the exploration-and-act setting, almost doubling the best-performing baseline.This result is reported for the proposed model with exploration, denoted w/Exp.
- Non-exploration: Nearly 60% success was achieved without exploration, exceeding the strongest baseline that used exploration.The non-exploration model is denoted no Exp.
- Baseline analysis: Random-walk approaches have very low success rates, while Greedy Search remains below 15% because it lacks visual knowledge of room layouts.Greedy Search can estimate the goal from sound but confuses geometric distance with the shortest navigable path.
- Generalization: A3C-based approaches perform weaker than previously reported because evaluation requires generalization to previously unseen environments.The authors speculate that separating visual and sound perception from path planning improves generalization.
VI. CONCLUSION
The paper introduces audio-visual embodied navigation in a multimodal virtual environment and uses structured environmental representations to navigate toward targets in unseen environments.
- Contribution: The paper tackles audio-visual embodied navigation in a multimodal virtual environment with complex apartments, realistic layouts, and a sound module.
- Conclusion: The agent uses an internal structured representation of the environment to navigate efficiently to targets in previously unseen environments.
- Generalization: State-of-the-art deep reinforcement-learning approaches encounter difficulties with generalization in this setting.