Source-linked AI summary
Semantic Audio-Visual Navigation
Changan Chen, Ziad Al-Halah, Kristen Grauman
TL;DR
Prior audio-visual navigation assumes a continuously sounding target and uses audio mainly as a positional cue, leaving brief, semantically grounded events outside that setting. This paper introduces a goal-descriptor-based model and semantically grounded SoundSpaces simulations; it reports up to an absolute 8.9% improvement in SPL over state-of-the-art models.
Problem
Method
Results
Takeaways & Limitations
The results show the potential for embodied agents to learn how objects look and sound through interactions with a 3D environment.
Source: §1 · p. 2If sound stops too early, the agent may lack enough spatial or semantic cues; it may also stop near, but not at, the target.
Source: §5 · p. 7
Abstract
from arXiv · showhide
Recent work on audio-visual navigation assumes a constantly-sounding target and restricts the role of audio to signaling the target's position. We introduce semantic audio-visual navigation, where objects in the environment make sounds consistent with their semantic meaning (e.g., toilet flushing, door creaking) and acoustic events are sporadic or short in duration. We propose a transformer-based model to tackle this new semantic AudioGoal task, incorporating an inferred goal descriptor that captures both spatial and semantic properties of the target. Our model's persistent multimodal memory enables it to reach the goal even long after the acoustic event stops. In support of the new task, we also expand the SoundSpaces audio simulations to provide semantically grounded sounds for an array of objects in Matterport3D. Our method strongly outperforms existing audio-visual navigation methods by learning to associate semantic, acoustic, and visual cues.
1. Introduction
Semantic audio-visual navigation extends audio-guided search to semantically grounded targets whose sounds may stop, requiring agents to infer the target and use visual, acoustic, and contextual cues. The paper introduces a goal-descriptor-based model and reports stronger performance than existing audio-visual navigation methods.
1. Introduction
- Prior audio-visual navigation assumes a continuously sounding target without visual embodiment or scene semantics, limiting audio to a positional beacon.Real-world objects may instead emit brief or intermittent sounds.Source: §1 · p. 1
- The new task requires finding a contextually situated object from sporadic acoustic cues, without being given its name or displacement, even after it falls silent.The agent must use semantic priors to infer both the likely object and where to search.Source: §1 · pp. 1–2
- The model predicts a goal descriptor with spatial and semantic properties, then uses a transformer to attend to remembered visual and acoustic observations.The work also extends audio simulation by attaching semantically relevant sounds to relevant objects in real scanned environments.Source: §1 · p. 2
- Up to an absolute 8.9% improvement in SPL over state-of-the-art audio-visual navigation models, with greater robustness to short signals and distractor sounds.The model also more often reaches the goal after acoustic observations end.Source: §1 · p. 2
2. Related work
Prior navigation tasks provide a location or object category, while semantic audio-visual navigation asks agents to infer and locate the specific sounding instance. The paper situates its approach among audio-visual navigation, memory models, and passive audio-visual learning.
Visual navigation
- PointGoal supplies a goal position and ObjectGoal an object label [23] [29] [38] [42] [44] [4] [9], whereas this task supplies no specific goal information and requires finding the particular instance that sounded.ObjectGoal accepts any instance of the category; semantic audio-visual navigation does not.Source: §2 · p. 2
Audio-visual navigation
- Earlier AudioGoal methods assume continuous sound and targets without visual embodiment [11] [19] [12], while this work considers limited-duration cues and semantically grounded object sounds.Its Matterport3D simulations attach object-level sounds to relevant objects rather than inserting a small set of sounds randomly.Source: §2 · p. 2
Memory models for 3D environments
- The scene memory transformer combines multimodal observations with a learned goal descriptor to attend to memory and learn audio-visual associations.This extends navigation memory approaches based on recurrent networks, maps, or stored observations and poses.Source: §2 · p. 2
3. Semantic Audio-Visual Navigation
Semantic AudioGoal requires an agent in an unmapped 3D environment to find the specific object instance associated with a possibly brief, semantically meaningful sound. The section describes the Matterport3D simulation, object-sound data, and episode success criteria.
3. Semantic Audio-Visual Navigation
- Unlike conventional AudioGoal [11] [19], semantic AudioGoal permits nonperiodic sounds of variable duration tied to meaningful objects, so agents must infer the sounding object's category and location.The task requires reaching the specific sounding instance, not simply acting on step-by-step audio cues.Source: §3 · p. 3
- SoundSpaces [11] renders sounds in Matterport3D [6] at source and agent locations on a 1 m uniform grid; Matterport3D is used for its greater scale and complexity.Source: §3 · p. 3
- The dataset grounds sounds in 21 ObjectGoal categories, using reachable Matterport3D object instances and both object-emitted and object-related events.Examples include a toilet flush and food being chopped on a counter; average train/validation/test instance counts are 303/46/80 per category.Source: §3 · p. 3
- Episodes randomize the scene, start pose, goal category and instance, and sound duration; success requires stopping at a designated viewpoint within 1 m of the specific goal object.This is stricter than ObjectGoal, which accepts stopping near any instance of the category.Source: §3 · p. 3
4. Approach
SAVi combines persistent multimodal memory with a transformer policy and an audio-derived goal descriptor containing spatial and semantic information. Its components encode observations, update the descriptor, and use descriptor-conditioned memory to select actions.
4. Approach
- SAVi uses persistent multimodal memory and an explicit goal descriptor to attend to spatial and semantic cues for navigation.The transformer can attend to observations at different temporal distances from the current step.Source: §4 · p. 3
4.1. Observation Encoder
- Each observation includes egocentric RGB and depth, binaural audio, agent pose, and the previous action; the model stores encoded observations in a finite memory window.The memory retains observations from max{0, t − s_M} through t.Source: §4.1 · p. 4
4.2. Goal Descriptor Network
- The goal descriptor estimates the target’s relative location and object category from audio, combining the current estimate with a pose-adjusted prior to reduce single-prediction noise.When sound stops, the latest descriptor is transformed according to the agent’s pose change.Source: §4.2 · pp. 4–5
4.3. Policy Network
- A transformer self-attends over encoded observations, then decodes them conditioned on the goal descriptor to produce a state used by an actor-critic to predict actions.Source: §4.3 · p. 5
4.4. Training
- Goal-category prediction is pretrained on 3.5M spectrogram-category pairs and frozen during policy training, while goal-location prediction is learned on-policy.The category predictor uses cross-entropy loss and the location predictor uses mean squared error.Source: §4.4 · p. 5
- Policy training first uses memory size 1 to train the observation encoder, then freezes it and trains the remaining model with memory size 150.Training uses DD-PPO, with rewards for success and reducing geodesic distance and a per-step time penalty; episodes are filtered for path difficulty.Source: §4.4 · p. 5
5. Experiments
SAVi outperforms existing navigation models across the reported metrics and is more successful after sounds stop. Its inferred goal descriptor and multimodal memory support navigation with sporadic sounds, including under silence and distractor noise.
Navigation results
- SAVi outperforms all compared models on all metrics, with heard-sound SPL gains of 0.3%, 8.9%, 7.2%, 7.2% and unheard-sound gains of 4.9%, 3.8%, 4%, 5.3%, respectively.The comparisons are against Gan et al. [19], Chen et al. [11], AV-WaN [12], and SMT [15], respectively.Source: Table 1 · p. 6, §5 · p. 6
- SAVi improves success when silent by a large margin over the closest competitor, and its explicit goal descriptor helps maintain focus after the target stops sounding.SMT+Audio [15] has a large memory pool but lacks the explicit goal descriptor and goal-driven attention.Source: §5 · p. 6
Navigation trajectories
- SAVi succeeds in example trajectories after both a long water-dripping event and a short door-opening-and-closing event, using spatial and semantic cues with memory.The short-event example uses an inaccurate location estimate as a directional cue and a cabinet category estimate to find the object in the kitchen.Source: §5 · pp. 6–7, Figure 3 · p. 7
- Common failures occur when sound stops before enough spatial or semantic cues accumulate, or when the agent stops near but not at the goal.Source: §5 · p. 7
Distractor sounds
- With unheard distractor sounds, SAVi reaches 11.8% success and 7.4% SPL, respectively 7.6% and 4.5% above the best-performing baseline, though all methods decline.The target and distractor sounds are disjoint and both unheard at test time.Source: §5 · p. 7, Table 2 · p. 7
Analyzing the goal descriptor
- Ablating either goal-descriptor component reduces performance, with location Lt having the larger impact.In 56% of successful episodes SAVi stops at its estimated goal location, while in 44% it stops at a different correct location; randomizing the category label drops success and SPL by up to 8%.Source: §5 · pp. 7–8
Goal descriptor accuracy and aggregation
- Goal-descriptor aggregation improves location prediction error to 8.1 m and category accuracy to 64.5%, versus 8.4 m and 53.6% without aggregation.Without aggregation, navigation success rate and SPL drop about 3 points.Source: §5 · p. 8
Robustness to silence duration
- As silence grows, all models accumulate success more slowly, but SAVi shows a noticeable increase beyond silence ratios of 1 and 2 while Chen et al. [11], AV-WaN [12], and SMT [15] rise only marginally.The silence ratio is the minimum actions required to reach the goal divided by audio duration.Source: §5 · p. 8, Figure 4 · p. 8
6. Conclusions
The paper introduces semantic audio-visual navigation and a transformer-based model that predicts a goal descriptor combining target location and category. It reports stronger performance than existing methods and greater robustness to long silence and acoustic distractors.
Contributions
- The paper introduces semantic audio-visual navigation in complex 3D environments and expands audio simulation with semantically grounded object sounds.The simulations provide sounds for objects in Matterport3D.Source: §6 · p. 8
Model
- Its transformer-based model predicts a goal descriptor with spatial and semantic properties, then conditions observation encoding on that descriptor.The model learns to associate acoustic events with visual observations.Source: §6 · p. 8
Findings and future work
- The authors report that the model outperforms existing state-of-the-art models and is more robust to long silence and acoustic distractors.Future work aims to generalize policies learned in the simulators to the real world.Source: §6 · p. 8
7. Supplementary Material
The supplementary material lists a qualitative video and additional implementation details and baseline analysis.
Supplementary video
- The supplement includes a video with audio for qualitative assessment of agent performance.It asks viewers to use headphones to hear spatial sound properly.Source: §7 · p. 11
Implementation and baselines
- The supplement also lists implementation details and analysis of the baselines.These details are identified as Section 5 material.Source: §7 · p. 11
3. Ablation of the policy network
The supplementary material identifies additional analyses of descriptor accuracy over distance and navigation with distractors.
Prediction accuracy
- The supplement lists a distribution of prediction accuracy over distance to goal.The listed material is associated with Section 5.Source: §3 · p. 11
Distractors
- The supplement also lists an analysis of semantic audio-visual navigation with distractors.The material is identified as an additional analysis.Source: §3 · p. 11
7. Ablation with true goal category and location
The supplementary analyses compare baselines and policy variants, examine goal-descriptor accuracy, and assess distractors, location-predictor training, and true goal information.
7.1. Qualitative Video
- The supplementary video presents the audio simulation platform, model–baseline comparisons, and qualitative failure cases.Headphones are recommended for hearing binaural audio correctly.Source: §7.1 · p. 11
7.2. Implementation Details and Analysis of the Baselines
- On unheard sounds, [19] reaches 15.9% success rate and 12.3% SPL, versus SAVi’s 24.8% success rate and 17.2% SPL.The analysis attributes [19]’s errors partly to not using visual cues to reason about goal location; category knowledge alone also often fails to locate the instance. The [19] predictor is augmented with an update operation for silent goals.Source: §7.2 · p. 11
7.3. Ablation of the Policy Network
- Both SAVi variants benefit significantly from the goal descriptor network, while the Transformer model performs best because it attends to longer observation sequences.The comparison replaces the Transformer with an RNN+MLP for action and value prediction in the unheard-sounds setting.Source: §7.3 · pp. 11–12, Table 4 · p. 12
7.4. Distribution of Goal Descriptor Accuracy
- Location error falls and category accuracy rises as the agent approaches the goal, with temporal aggregation improving both predictions.Location error is Euclidean distance to the true goal location; category accuracy records whether the correct goal is predicted.Source: §7.4 · p. 12, Figure 5 · p. 12
7.5. Analysis of Semantic Audio-Visual Navigation with Distractors
- Performance is better when distractors differ from the goal sound; similar sounds such as Cabinet and Creak, or louder sounds such as Horn, make goal-signal extraction harder.The analysis tests one distractor type per run and describes a subset of goal–distractor combinations.Source: §7.5 · p. 12, Table 5 · p. 12
7.6. On-policy Location Predictor Training
- Using an off-policy location predictor underperforms the on-policy model by 4.8% success rate and 4.7% SPL on the unheard-sound test split.Source: §7.6 · p. 12
7.7. Ablation with True Goal Category and Location
- With true goal category and location, SAVi achieves 65% SPL, compared with 24% SPL using predicted descriptors under the same setting.With the true goal location available at each step, the task becomes PointGoal navigation.Source: §7.7 · p. 12