RAO-Nav: Probing Omni-Language Models for Zero-shot Semantic Audio-Visual Navigation

• SAVN requires agents to infer what and where a target is from short, ambiguous audio cues while navigating with partial visual observations.Targets may be blocked by walls or other obstacles, so agents must combine sparse auditory cues with current visual observations.Source: §1 · p. 1
• RAO-Nav connects one OLM’s audio understanding, target localization, semantic-map guidance, and action updates in an iterative three-stage pipeline.The stages are AudioGoal Prediction, Global Reasoning, and Real-time Action; a distinct location module guides the frozen OLM with dual-frequency spectrograms.Source: §1 · p. 2, Figure 1 · p. 2
• Test-time Latent Navigation Reasoning encourages the OLM to focus on target-relevant observations during global reasoning for more stable decisions in long-horizon, uncertain environments.The module is intended to elicit the OLM’s reasoning ability during navigation.Source: §1 · p. 2
• RAO-Nav outperforms most existing baselines on public SAVN benchmarks without task-specific training, while real-world natural-language-guided audio-visual navigation remains challenging.The authors also introduce a Global Navigation Instruction setting to evaluate OLMs in more realistic language-guided global perception and navigation scenarios.Source: §1 · pp. 2–3

• Early SAVN work jointly modeled visual and auditory observations, with the task requiring inference of targets from sparse audio cues under partial visual observation.SAVi introduced a benchmark for whether a model can navigate to audible semantic targets.Source: §2 · p. 3
• Later SAVN directions include modular transfer learning, knowledge-enhanced scene priors [25], moving sound sources in unmapped environments, and oracle-language navigation with AVLEN.These extensions move beyond static indoor settings toward more realistic scenarios.Source: §2 · p. 3
• Zero-shot MLLMs show promising spatial and action reasoning, yet studies report difficulty with robust long-horizon decisions and generalist embodied navigation [33] [34].The related work frames strong multimodal understanding as insufficient by itself for reliable embodied control.Source: §2 · p. 3

3.1 Problem Definition

• In unseen 3D indoor scenes, SAVN is formulated as sequential decisions over RGB observations, audio, agent state, and a history of past observations, with four available actions.Source: §3.1 · p. 3

3.2 AudioGoal Prediction

• Because audio may be momentary, the OLM updates category scores from accumulated audio and action history after each action, forming a progressively updated target belief.Source: §3.2 · p. 4
• A plug-in Localization Expert conditions spatial estimates on OLM category scores and binaural audio, supplying geometric source positions while retaining the OLM's audio understanding.Source: §3.2 · pp. 4–5

3.3 Global Reasoning

• In QA-driven global reasoning, the OLM uses candidate waypoints, the semantic map, and recent visual observations to choose a path consistent with target semantics.Source: §3.3 · p. 5

3.4 Real-time Action

• A Geometric Controller compiles the selected path into deterministic turn-then-forward actions, which execute as long-horizon actions rather than local action predictions.Source: §3.4 · p. 5
• After execution, the agent receives updated RGB and audio, smooths point-goal estimates, retains the previous estimate without new audio, and continuously decides whether to stop or replan.Source: §3.4 · pp. 5–6

4 Test-time Latent Navigation Reasoning Module

• LNR accumulates historical RGB embeddings and initializes latent text tokens in the same embedding space as other multimodal tokens.The latent tokens are placeholders, and the module does not update model parameters.Source: §4 · p. 6
• The intrinsic reward uses the conditional likelihood of the predicted audio goal; higher likelihood indicates that latent tokens better summarize goal-relevant observations.Source: §4 · p. 6
• After optimizing the latent tokens, LNR mean-pools them into a query and selects the top-K frame embeddings by similarity for the next reasoning step.The gradient is computed through the frozen OLM, with η as the step size.Source: §4 · p. 6

5.1 Experiment Setup

• Evaluation uses unheard sounds and unseen Matterport3D scenes, comparing specialized trained methods, multi-MLLM collaboration, audio-description VLMs, and unified OLM pipelines.The evaluation reports SR, SPL, SNA, SWS, and DTG.Source: §5.1 · pp. 6–7

5.2 Experimental Results

• 39.2% SR with LNR is the best reported result, surpassing supervised baselines while maintaining competitive SPL and strong SWS; zero-shot methods still lag on SPL, SWS, and DTG.Without LNR, RAO-Nav achieves 37.6% SR with Baichuan-Omni-7B.Source: Table 1 · p. 7, §5.2 · p. 7
• In Global Navigation Instruction tests, audio-category inference drops in the open setting, and most OLMs struggle to match visual observations with language instructions, especially in complex scenes.The evaluation includes 164 closed-set and 172 open-set audio samples, plus 248 visual-navigation samples.Source: §5.2 · p. 8, Table 2 · p. 8

5.3 Ablation Studies

• Removing OLM-predicted audio-category input from the spectrogram localizer reduces SR from 39.2% to 12.6%, supporting the use of both acoustic and semantic information for localization.The ablation also reports that general-purpose OLMs are weak at sound-source localization.Source: §5.3 · p. 8, Table 3 · p. 9
• RGB-only input performs best; adding depth lowers SR and SPL and raises DTG, so RAO-Nav adopts the simpler RGB design.The authors suggest depth adds input complexity and reasoning burden without useful cues.Source: §5.3 · pp. 8–9, Table 3 · p. 9

5.4 Visualization of Navigation Trajectories

• Successful trajectories refine direction through iterative observations, whereas inaccurate sound-source localization often sends failures toward misleading regions that reasoning does not recover from.The visualization links successful correction and unrecovered failures to the accuracy of target localization.Source: Figure 4 · p. 9, §5.4 · p. 9

5.3 Ablation Studies

• In an LNR ablation, compared with K-NN (selecting K nearest observations in embedding space), LNR improves SR and SPL by 1.4% and reduces DTG by 1.5 m, showing target-guided latent optimization outperforms simple observation selection.Source: §5.3 · p. 9

• RAO-Nav achieves strong zero-shot performance on SAVN benchmarks and surpasses existing supervised baselines.Source: §6 · p. 9
• The pipeline decomposes navigation into AudioGoal Prediction, Global Reasoning, and Real-time Action for a single OLM.Source: §6 · p. 9
• Test-time LNR filters redundant historical RGB observations and encourages focus on target-relevant information during iterative reasoning.Source: §6 · p. 9