Semantic Audio-Visual Navigation in Continuous Environments

• SAVN-CE enables fine-grained movement in continuous 3D environments with coherent spatial and temporal observations, unlike prior RIR-dependent tasks restricted to discrete grid positions and fixed orientations.AVN [9] [20] and SAVN [10] use precomputed room impulse responses for binaural rendering; the paper notes that these require terabytes of storage and constrain movement to 1 m grid positions and four orientations [9].Source: §1 · p. 1
• Agents explore SAVN-CE without goal information, then undertake long-horizon navigation after the goal begins emitting sound, which may intermittently cease or stop entirely.Source: §1 · pp. 1–2
• Partial observations make it difficult to infer a goal’s location and category: intermittent silence disrupts tracking, while short sound windows can leave sound semantics ambiguous.The simulation uses a finer 0.25 s step, and goal information is lost once sound stops.Source: §1 · p. 2
• MAGNet jointly encodes spatial and semantic goal representations and integrates historical context with self-motion cues to infer goals continuously, including after auditory signals disappear.Source: §1 · p. 2
• MAGNet significantly outperforms prior methods, achieving up to a 12.1% absolute improvement in success rate, particularly with short-duration sounds and long-distance navigation.Source: §1 · p. 2

Sound event localization and detection

• SELD [2] unifies sound event detection and localization [1] [23], but existing approaches largely rely on simulated datasets [2] or single-room recordings.Realistic, acoustically complex multi-room environments remain underexplored.Source: §2 · pp. 2–3

Semantic audio-visual navigation

• AVN extensions cover sound-attacker, multi-goal [30], and moving-source scenarios; SAVN [10] grounds goals in objects, but its grid-location constraint remains unresolved [13] [30].Language instructions [36] [38], large language models, and multiple sound sources have further extended SAVN, while continuous navigation remains unexplored in the SAVN paradigm despite SoundSpaces 2.0 [12]. The challenge includes costly binaural rendering and limited simulation speed.Source: §2 · p. 3

VLN in continuous environments

• VLN-CE [32] replaces discrete graphs [5] with low-level 3D actions without topology, oracle navigation, or perfect localization; waypoint planning [3] [14] [27] [31] and obstacle avoidance [3] address its challenges.In contrast, SAVN-CE infers goal information from partial sensory observations after the goal sound ceases.Source: §2 · p. 3

Task

• SAVN-CE replaces discrete RIR-based navigation with continuous 3D movement and dynamically rendered, temporally and spatially coherent audio.The formulation supports fine-grained low-level actions while requiring agents to integrate multimodal inputs and reason over long horizons.Source: §3 · p. 3

Simulator

• At 16 kHz and 0.25 s simulation steps, SAVN-CE accumulates previous-step residual responses to preserve long-tail reverberation.The simulator uses SoundSpaces 2.0 [12] in realistic Matterport3D scenes [8], extending Habitat [41]; the residual accumulation follows.Source: §3 · p. 3

Actions and observations

• Agents use four actions—move forward 0.25 m, turn left or right 15°, or stop—and receive binaural audio, 128×128 egocentric RGB-D images, and relative pose.The action space is consistent with VLN-CE [32].Source: §3 · p. 3

Dataset construction

• The dataset has 0.5M/500/1,000 train/validation/test episodes, 21 goal categories, and 102 periodic distractor sounds; test scenes and sounds are unseen.Adapted from SAVi [10], the dataset uses distractor candidates from SoundSpaces [9]; test episodes average 78.49 oracle actions versus 26.52 in the discrete setting.Source: §3 · p. 3

Success criterion

• An episode succeeds only when the agent stops within 1 m of the goal sound source, and each episode is capped at 500 actions.Stopping near a distractor or another instance of the same semantic category counts as failure, following prior audio-visual navigation work [9] [10].Source: §3 · p. 3

4. Method

• MAGNet comprises a multimodal observation encoder, memory-augmented goal descriptor network, and context-aware policy network.Together, the modules encode current sensory inputs, retain scene and episodic history, infer spatial and semantic goal representations, and predict actions from scene memory.Source: §4 · pp. 3–4, Figure 3 · p. 4

4.1. Multimodal Observation Encoder

• The observation encoder embeds visual, action, pose, and binaural audio inputs, then stores recent encoded observations in scene memory.It uses separate ResNet-18 backbones for RGB and depth, and audio features comprising mean magnitude, inter-channel phase sine and cosine, and inter-channel level difference.Source: §4.1 · p. 4

4.2. Memory-Augmented Goal Descriptor Network

• MAGNet’s goal descriptor fuses binaural audio, self-motion, and episodic history to track goals consistently when sound is intermittent or absent.It stores recent goal-relevant embeddings with positional encodings, processes them with a transformer, and produces a goal embedding plus ACCDDOA-formatted goal descriptions for training. Fine-grained actions limit step-to-step positional changes, supporting coherent tracking over long-horizon navigation.Source: §4.2 · p. 5

4.3. Context-Aware Policy Network

• The policy transformer encodes accumulated scene memory and decodes a context-aware latent state for separate actor and critic predictions.An action sampler selects the next action from the actor’s predicted distribution, supporting decisions based on current and historical observations in a partially observable continuous environment.Source: §4.3 · p. 5

4.4. Training Strategy

• Training alternates 150-step policy rollouts with updates to the GDN and policy network.The GDN is trained online with oracle ACCDDOA labels and MSE on complete episodes; DD-PPO trains the policy using success, geodesic-distance change, and time-penalty rewards.Source: §4.4 · pp. 5–6

5.1. Experimental Setup

• The experiments compare clean scenes, where only the goal emits sound, with distracted scenes, where a distractor also sounds, using navigation baselines and success, efficiency, distance, silence, and distractor metrics.Learning-based methods train for up to 240M steps with early stopping; evaluation selects the checkpoint with highest validation SPL.Source: §5.1 · p. 6, §5.2 · p. 6

5.2. Experimental Results

• MAGNet outperforms all baselines across metrics in Clean Environments, while its gains decrease in Distracted Environments.Results average 1,000 test episodes across five runs; the checkpoint with the highest validation SPL is selected.Source: §5.2 · p. 6, Table 1 · p. 7
• Oracle2 substantially outperforms Oracle1, showing that limited sound duration constrains navigation; MAGNet maintains estimates after silence but remains below both oracle settings.Oracle1 retains oracle labels only while the goal emits sound, whereas Oracle2 retains them until the episode ends.Source: §5.2 · p. 6, §5.1 · p. 6, Table 1 · p. 7

5.3. Visualization of Navigation Trajectories

• In Clean Environments, MAGNet uses historical context and self-motion cues to maintain and update goal information after sound ceases, unlike agents that lose their way.The trajectory examples show that incorrect movement before sound begins can require compensating actions within the limited sound-emitting window.Source: §5.3 · p. 7, Figure 4 · p. 7

5.4. Factors Affecting Navigation Success

• As action ratio or geodesic distance increases, cumulative success rates saturate; MAGNet has higher upper bounds than baselines, showing robustness to short sounds and long-distance navigation.Higher action ratios mean more actions must occur without goal sound, while greater distances make goal inference harder through sound attenuation and complex layouts.Source: Figure 5 · p. 7, §5.4 · pp. 7–8

5.5. Evaluation of the Goal Descriptor Network

• MAGNet outperforms SAVi on most GDN metrics, but distractors degrade localization and categorization, and silence degrades both methods’ performance.The evaluation includes sound-emitting and silent periods and uses SELD metrics for localization and detection, including measures conditioned on correctly detected events.Source: Table 2 · p. 8, §5.5 · p. 8

5.6. Ablation Study

• Removing the GDN degrades navigation, while memory and self-motion cues each improve navigation and SELD; combining all components yields further gains, and better SELD consistently aligns with better navigation.The ablation compares configurations without the GDN, with no memory (Ng = 1), without self-motion cues, and the full memory-augmented GDN.Source: Table 3 · p. 8, §5.6 · p. 8

6. Conclusion

• MAGNet substantially outperforms existing state-of-the-art methods and is robust to short-duration sounds and long-distance navigation.Ablation studies further validate the effectiveness of the method.Source: §6 · p. 8
• SAVN-CE introduces continuous-environment navigation toward semantically grounded sound-emitting goals while avoiding distractors.Agents navigate freely in this setting.Source: §6 · p. 8
• MAGNet employs a memory-augmented goal descriptor network to support robust goal reasoning and efficient navigation when goals become silent.Source: §6 · p. 8
• Future work will extend the framework to auditory scenarios involving multiple or dynamic goals.Source: §6 · p. 8