Source-linked AI summary
Semantic Audio-Visual Navigation in Continuous Environments
Yichen Zeng, Hebaixu Wang, Meng Liu, Yu Zhou, Chen Gao, Kehan Chen, Gongping Huang
TL;DR
Existing audio-visual navigation settings restrict agents to discrete positions and rely on precomputed audio, while intermittent or absent goal sounds create a persistent reasoning challenge. The paper introduces SAVN-CE for continuous 3D navigation and MAGNet for memory-augmented goal inference, with MAGNet substantially outperforming existing methods and improving success rate by up to 12.1%.
Problem
Existing audio-visual navigation relies on precomputed RIRs and discrete positions, while intermittent or absent goal sounds can remove spatial and semantic goal information.
Method
SAVN-CE enables continuous 3D movement with coherent observations, while MAGNet fuses multimodal goal representations, historical context, and self-motion cues.
Results
Up to 12.1% absolute improvement in success rate is achieved by MAGNet over existing methods, with robustness to short-duration sounds and long-distance navigation.
Takeaways & Limitations
Memory-augmented goal reasoning supports navigation toward semantically grounded sound-emitting targets even after auditory signals are no longer available.
Abstract
from arXiv · showhide
Audio-visual navigation enables embodied agents to navigate toward sound-emitting targets by leveraging both auditory and visual cues. However, most existing approaches rely on precomputed room impulse responses (RIRs) for binaural audio rendering, restricting agents to discrete grid positions and leading to spatially discontinuous observations. To establish a more realistic setting, we introduce Semantic Audio-Visual Navigation in Continuous Environments (SAVN-CE), where agents can move freely in 3D spaces and perceive temporally and spatially coherent audio-visual streams. In this setting, targets may intermittently become silent or stop emitting sound entirely, causing agents to lose goal information. To tackle this challenge, we propose MAGNet, a multimodal transformer-based model that jointly encodes spatial and semantic goal representations and integrates historical context with self-motion cues to enable memory-augmented goal reasoning. Comprehensive experiments demonstrate that MAGNet significantly outperforms state-of-the-art methods, achieving up to a 12.1\% absolute improvement in success rate. These results also highlight its robustness to short-duration sounds and long-distance navigation scenarios. The code is available at https://github.com/yichenzeng24/SAVN-CE.
1. Introduction
Audio-visual navigation addresses targets that may be invisible or lack distinctive visual cues, while SAVN-CE introduces continuous movement and intermittent goal sounds that require persistent spatial and semantic reasoning. MAGNet combines multimodal goal representations, historical context, and self-motion cues, outperforming prior methods in challenging navigation scenarios.
- Audio-visual perception complements vision by helping agents infer the locations and categories of otherwise invisible targets.
- Existing AVN and SAVN approaches rely on precomputed RIRs, limiting binaural rendering to discrete positions with spatially discontinuous observations.
- SAVN-CE allows free movement in continuous 3D environments with temporally and spatially coherent observations, while goals may initially be silent and later stop emitting sound.
- MAGNet jointly encodes spatial and semantic goal representations and integrates historical context with self-motion cues for continuous goal inference after auditory signals disappear.
- MAGNet outperforms prior methods particularly in short-duration-sound and long-distance navigation scenarios.
- 12.1% absolute improvement in success rate is achieved by MAGNet over existing methods in the reported experiments.
2. Related Work
Related work shows that semantic audio-visual navigation extends sound localization toward visually grounded goals, but standard settings remain constrained by continuous sound, arbitrary goal placement, and discrete grid positions.
- SELD unifies sound event detection and localization by estimating event boundaries, categories, and spatial positions in a shared representation.
- Standard semantic audio-visual navigation assumes continuously emitting goals, arbitrary goal placement without visual embodiment, and predefined grid locations with available RIRs.
3. SAVN in Continuous Environments
SAVN-CE defines semantic audio-visual navigation in continuous 3D indoor environments, replacing grid-constrained, precomputed-audio settings with fine-grained actions and dynamically coherent observations.
- SAVN-CE requires navigation toward semantically grounded sound-emitting goals in continuous 3D indoor environments using fine-grained low-level actions.
- Unlike prior RIR-based settings, SAVN-CE dynamically renders temporally and spatially coherent audio while agents move freely through continuous spaces.
- The simulator uses SoundSpaces 2.0 with Matterport3D scenes, 16 kHz audio, and a 0.25 s simulation step.
4. Method
MAGNet combines multimodal observation encoding, memory-augmented goal description, and context-aware policy prediction for navigation in continuous environments. Its design maintains spatial and semantic goal representations using current observations, self-motion, and historical memory, including when sound is absent.
- MAGNet uses a multimodal observation encoder, memory-augmented goal descriptor network, and context-aware policy network.The observation encoder updates scene memory; the goal descriptor infers spatial and semantic goal representations; the policy predicts actions from encoded memory.
- Multimodal Observation Encoder: The observation encoder combines RGB-D images, previous actions, agent pose, and binaural audio into observation embeddings.Visual, action, pose, and audio encoders process their respective modalities before concatenation.
- Multimodal Observation Encoder: Scene memory stores the most recent Ns encoded observations for temporally informed policy reasoning.The policy encoder processes accumulated scene memory to capture dependencies across past observations.
- Memory-Augmented Goal Descriptor Network: The goal descriptor fuses binaural features, self-motion cues, and episodic memory to model temporal continuity and spatial dynamics.The episodic memory stores recent goal-relevant embeddings, preserves temporal order with positional encodings, and feeds a transformer with two output branches.
- Memory-Augmented Goal Descriptor Network: The descriptor produces a goal embedding and ACCDDOA-formatted descriptions containing sound activity, direction of arrival, and normalized distance.The ACCDDOA representation includes category and time indices, a unit-norm DOA vector, sound activity status, and normalized distance.
- Memory-Augmented Goal Descriptor Network: Integrating self-motion dynamics with accumulated goal embeddings preserves consistent goal representations without auditory input.Fine-grained actions also limit positional changes between consecutive steps, supporting stable tracking during long-horizon navigation.
- Context-Aware Policy Network: The policy network encodes scene-memory history, decodes a context-aware latent state, and uses actor and critic heads to select actions.The latent representation summarizes historical and current sensory information before action sampling.
5. Experiments
Experiments evaluate navigation across clean and distracted environments, trajectory behavior, difficulty factors, goal-descriptor performance, and component ablations. MAGNet consistently improves robustness to silent goals, short-duration sounds, and long-distance navigation.
- Overall performance: MAGNet consistently outperforms all baselines across navigation metrics in Clean and Distracted Environments.Evaluation averages results over 1,000 test episodes and five independent runs.
- Navigation trajectories: MAGNet successfully completes navigation after goal sounds cease by using historical context and self-motion cues to maintain and update goal information.Other agents may lose their way when auditory signals stop, especially after earlier incorrect movements.
- Factors affecting success: As action ratio or geodesic distance increases, cumulative success rates gradually saturate, with successful episodes concentrated at longer sound durations and closer goal distances.A higher action ratio means more actions occur without goal sound, while greater geodesic distance increases difficulty through sound attenuation and complex layouts.
- Factors affecting success: MAGNet achieves higher cumulative-success upper bounds than baselines, demonstrating greater robustness to short-duration sounds and long-distance navigation.The results also attribute stable goal representations to the memory-augmented GDN.
- Goal descriptor evaluation: Distractor sounds substantially reduce SELD performance, while MAGNet outperforms SAVi on most metrics and remains competitive during silent periods.The evaluation reports error rate, F1-score, localization error, localization recall, and relative distance error under sounding and silent conditions.
- Ablation study: Ablations show that episodic memory and self-motion cues each improve navigation and SELD, while combining all components yields further gains.Removing the GDN degrades navigation, and improved SELD performance consistently translates into better navigation outcomes.
6. Conclusion
The paper introduces SAVN-CE for continuous semantic audio-visual navigation and MAGNet for maintaining goal reasoning when sounds become silent. Experiments show strong robustness to short-duration sounds and long-distance navigation, while future work targets more complex auditory scenarios.
- SAVN-CE extends semantic audio-visual navigation to continuous 3D environments with semantically grounded sound-emitting goals and distractor avoidance.
- MAGNet uses a memory-augmented goal descriptor network to support robust goal reasoning and efficient navigation when goals become silent.
- MAGNet substantially outperforms existing state-of-the-art methods and is robust to short-duration sounds and long-distance navigation.
- Future work will extend the framework to auditory scenarios involving multiple or dynamic goals.