PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniEcho: Audio-Visual Spatial Understanding for Omni-Modal Embodied Agents

Ruixun Liu, Yuxuan Wang, Jiacheng Xie, Yuhuan You, Donghua Cai, Junming Lin, Xiong-Hui Chen, Zhifang Guo, Yunfei Chu, Qize Yang, Xize Cheng, Jin Xu, Yiwu Zhong

arXiv:2609.23407v2cs.SDcs.AI

TL;DR

Spatial audio can provide embodied agents with cues about off-screen events and target direction, but its evaluation and integration into pretrained omni-modal models remain challenging. OmniEchoBench, scalable spatial-audio rendering, and the OmniEcho model address these gaps; experiments report state-of-the-art spatial-perception performance and navigation close to traditional VLN.

  • Problem

    Spatial audio remains underexplored for embodied agents, and how to evaluate it and integrate it into pretrained omni-modal models remains an open question.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces a real-world benchmark and controllable spatial-audio rendering pipeline, then combines an FOA encoder with a pretrained semantic audio pathway in OmniEcho.

    Source: §1 · p. 2

  • Results

    OmniEcho achieves state-of-the-art performance on spatial perception, while sound-guided navigation performs close to traditional VLN methods.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The results highlight spatial audio's value for embodied scene reasoning and navigation, while identifying fine-grained localization and distance estimation as open challenges.

    Source: §6 · p. 10

    Synthetic training data introduces bias, although aligning simulated source-state distributions with real-world states supports spatial perception and may be corrected with a small amount of real spatial-audio data.

    Source: §5.2.1 · p. 8

Abstract

from arXiv · show

Humans can effortlessly localize the direction of a sound source and integrate it with visual cues for reasoning, yet this remains challenging for embodied agents. In particular, it is still unclear how to effectively evaluate and model spatial audio understanding in embodied settings. To address this gap, we introduce \textbf{OmniEchoBench}, a unified benchmark for spatial audio-visual perception and audio-vision-language navigation. OmniEchoBench comprises six tasks over 197 real-world spatial audio-visual scenes, 2,972 question-answer pairs, and 900 navigation samples with first-order ambisonics (FOA) audio collected from 30 real-world environments. To enable scalable training supervision, we develop a controllable rendering pipeline for spatial audio. It preserves geometric consistency among sound sources, visual observations, and agent trajectories. Building on this, we propose \textbf{OmniEcho}, a spatially aware omni-modal model. It introduces an FOA spatial encoder alongside a pretrained semantic audio pathway. Extensive experiments show that OmniEcho achieves state-of-the-art performance on spatial audio-visual perception. For our sound-guided navigation, OmniEcho reaches a performance level close to that of traditional vision-language navigation. These results demonstrate that spatial audio can serve as a valuable signal for embodied scene reasoning and navigation, while also highlighting fine-grained spatial localization and distance estimation as important open challenges. Our code and data will be available in https://github.com/PKU-VaLuE-Lab/OmniEcho/tree/main

1 INTRODUCTION

Spatial audio is underexplored in embodied agents, while real-world data collection, physically consistent simulation, and integration with pretrained omni-modal models pose challenges. OmniEchoBench and OmniEcho address these challenges, with strong spatial-perception results and sound-guided navigation close to traditional VLN.

1 INTRODUCTION

  • Spatial audio remains underexplored for embodied agents, despite offering cues about events outside the camera view and target direction.Key challenges include costly real-world data collection, physically consistent simulation, and integrating spatial audio into pretrained models.Source: §1 · pp. 1–2
  • OmniEchoBench unifies real-world spatial audio-visual perception and navigation, alongside scalable simulated training data for spatial-audio supervision.The rendering pipeline preserves geometric consistency among sources, visual observations, and agent trajectories.Source: §1 · p. 2
  • OmniEcho achieves state-of-the-art performance on spatial perception, while its more challenging sound-guided navigation performs close to traditional VLN methods.Existing omni-modal models struggle on the benchmark.Source: §1 · p. 2
  • OmniEcho combines an FOA encoder with a pretrained semantic audio pathway and uses three training stages to align spatial and semantic information.The model is built on Qwen3-Omni and jointly processes auditory semantics, spatial information, and environmental observations.Source: §1 · p. 2
  • The authors describe OmniEcho as, to their knowledge, the first work on spatial-audio understanding for omni-modal embodied agents supporting both audio-visual QA and sound-guided navigation.The benchmark includes human annotations in real-world scenes, with simulated data for scalable training supervision.Source: §1 · p. 2

2 RELATED WORK

Prior work has advanced vision-language navigation, audio-visual navigation, and spatial-audio modeling, but spatial acoustic cues and real-world recorded audio remain underrepresented in navigation evaluation. OmniEcho positions its benchmark and model as addressing those gaps across spatial perception and sound-guided navigation.

Traditional Vision-Language Navigation

  • Representative VLN models rely on vision and text rather than treating sound-source direction, distance, and spatial layout as native navigation evidence.OmniEcho introduces a navigation model with spatial-audio understanding.Source: §2 · p. 3

Audio-Visual Navigation

  • Prior audio-visual navigation benchmarks use rendered acoustics and do not evaluate navigation with real-world recorded spatial audio.OmniEcho provides a unified benchmark and simulator using real-world collected FOA audio; Figure 3 describes 900 tasks across 30 scenes.Source: §2 · p. 3, Figure 3 · p. 4

Spatial Audio Understanding

  • Spatial-audio research has developed acoustic encoders, language-aligned representations, and audio-language reasoning, while Spatial-Omni lacks visual-frame training and sound-guided navigation.OmniEcho uses real-world FOA recordings to evaluate both spatial audio-visual understanding and sound-guided navigation.Source: §2 · p. 3

3 BENCHMARK AND DATASETS

OmniEchoBench combines real-world spatial-audio perception and navigation data with a rendering pipeline for training-data synthesis. Its QA scenes emphasize sound sources that are off-screen or cross the visual field, while navigation samples pair dense FOA recordings with real indoor environments.

3.1 OMNIECHOBENCH-QA

  • OmniEchoBench-QA uses synchronized first-person video, panoramic video, and four-channel FOA recordings from human-performed scripts emphasizing off-screen and boundary-crossing sources.The panoramic video serves as an annotation reference; models receive a single-view video or still image with FOA audio.Source: §3.1 · p. 4
  • 2,972 QA pairs include 2,372 temporal-reasoning questions and 600 bird’s-eye localization questions requiring selection among four candidate source locations.Temporal categories cover source direction, 3D localization, source motion, and camera rotation.Source: §3.1 · pp. 4–5

3.2 OMNIECHOBENCH-NAV

  • OmniEchoBench-Nav contains 900 samples across 30 real-scanned indoor environments, with 12,900 four-channel, 48 kHz FOA recordings at dense receiver positions.The samples include non-speech environmental sounds and spoken commands; agents receive audio remapped to their current orientation.Source: §3.2 · p. 5

3.3 SPATIAL AUDIOVISUAL SIMULATION FOR DATA SYNTHESIS

  • A shared rendering framework encodes monophonic source stems into FOA using source trajectories and listener pose, with source direction transformed into the listener frame.The formulation can include distance attenuation and room effects and is shared by perception and navigation data.Source: §3.3 · pp. 5–6
  • Training synthesis combines simulated-scene visuals with trajectory-grounded sound events for QA, and renders destination-conditioned FOA observations along augmented navigation trajectories.The pipelines provide synchronized visual observations, spatial audio, and geometry-grounded supervision.Source: Figure 4 · p. 5, §3.3 · p. 6

4 METHOD

The method unifies spatial audio-visual perception and navigation, adding an FOA spatial pathway to a pretrained semantic audio pathway. On OmniEchoBench-QA it outperforms evaluated models in both audio-only and audio-visual settings, while navigation performance is reported as close to traditional VLN methods.

4.1 TASK FORMULATION

  • Perception predicts an answer from FOA audio, video or an image, and a question; navigation predicts an eight-step trajectory from instruction, video frames, and current-window FOA audio.Past visual observations are represented by uniformly sampled keyframes combined with the current frame.Source: §4.1 · p. 6

4.2 SPATIAL AUDIO ENCODING

  • The FOA encoder is first aligned to text semantics, then fine-tuned with query-conditioned localization to predict a target source’s azimuth, elevation, and distance.Localization training retains the Stage-1 semantic objective.Source: §4.2 · pp. 6–7
  • OmniEcho outperforms most evaluated models in overall accuracy and individual spatial reasoning tasks in both audio-only and audio-visual settings.The table distinguishes single-channel audio, four-channel FOA audio, and visual input.Source: Table 1 · p. 7
  • OmniEcho achieves close performance to previous vision-language navigation methods on OmniEchoBench-Nav, where text instruction is also provided for traditional VLN methods.The benchmark compares sound-guided navigation with text-guided navigation methods.Source: Table 2 · p. 7
  • Integration preserves the frozen semantic audio tower and adds temporally aligned FOA spatial tokens through a trainable projector; only the LLM and projector are updated.Spatial tokens are inserted after corresponding audio tokens.Source: §4.2 · p. 7

5 EXPERIMENTS

OmniEcho leads evaluated models on spatial audio-visual QA, while sound-guided navigation is competitive with earlier text-guided baselines but still faces efficiency and stopping challenges. Ablations favor combining semantic audio with pretrained FOA features and freezing the FOA encoder during instruction tuning.

5.1 SETUP

  • Evaluation covers OmniEchoBench-QA, OmniEchoBench-Nav, and the VLN-CE R2R validation unseen split; training uses 363,193 examples.Stage 3 training used 32 NVIDIA A100 GPUs for approximately four days.Source: §5.1 · p. 7

5.2.1 SPATIAL AUDIO-VISUAL QUESTION ANSWERING RESULTS

  • OmniEcho outperforms evaluated models overall and across individual tasks on OmniEchoBench-QA in both audio-only and audio-visual settings.In audio-only evaluation, it also surpasses its Qwen3-Omni-30B-A3B backbone, which uses a single-channel audio encoder.Source: §5.2.1 · p. 8

5.2.2 SPATIAL AUDIO NAVIGATION RESULTS

  • 16.2% SR and 11.5% SPL on OmniEchoBench-Nav substantially exceed Seq2Seq and CMA, while trailing InternVLA-N1 by 1.6 SR points and 5.2 SPL points.The comparison uses agent stopping; efficient navigation and reliable stopping remain challenges.Source: §5.2.2 · p. 8

5.3 ABLATION STUDY

  • 28.5 Overall is achieved by combining native audio and FOA encoders and freezing the FOA encoder during Stage 3; removing native audio lowers Overall to 26.2.Updating rather than freezing the FOA encoder degrades every subtask.Source: Table 4 · p. 9, §5.3 · p. 9
  • 17.9° azimuth error / 7.70° elevation error / 1.13 distance error with feat5+stage1 outperforms raw4+stage1 (30.5° / 13.8° / 1.65).The passage also reports that removing Stage-1 semantic-alignment pretraining degrades performance, while adding an FOA encoder without Stage 1 and Stage 2 brings no clear gains.Source: Figure 6 · p. 9, §5.3 · p. 9

5.2.2 SPATIAL AUDIO NAVIGATION RESULTS

  • 22.2% SR on VLN-CE R2R slightly exceeds Seq2Seq and CMA, but 16.3% SPL trails CMA (20.0%) and OmniEcho remains behind stronger systems in SR and SPL.Its NE is 7.31, and its trajectory length of 5.54 is the shortest among methods with reported TL.Source: §5.2.2 · p. 9

6 CONCLUSION

The paper presents a real-world benchmark and controllable spatial-audio rendering pipeline, then evaluates an omni-modal model that combines spatial and semantic audio. It reports strong spatial-perception results and identifies fine-grained localization and distance estimation as remaining challenges.

6 CONCLUSION

  • OmniEchoBench unifies real-world spatial audio-visual perception and sound-guided navigation, and the rendering pipeline supports data generation.Source: §6 · p. 10
  • OmniEcho combines an FOA encoder with a pretrained semantic audio pathway to capture auditory semantics and spatial information.Source: §6 · p. 10
  • Experiments report state-of-the-art spatial-audio perception and navigation performance close to traditional text-guided VLN; fine-grained localization and distance estimation remain challenges.Source: §6 · p. 10

A.1.1 SPATIAL AUDIOVISUAL SIMULATION FOR DATA SYNTHESIS.

The simulation pipeline creates structured scenes with source trajectories, generates audiovisual media, then corrects and augments the geometry before rendering FOA and deriving question-answer supervision. It also constructs audio-only spatial data and compares the synthetic training distribution with the real benchmark.

A.1.1 SPATIAL AUDIOVISUAL SIMULATION FOR DATA SYNTHESIS.

  • An LLM director specifies scene sources, timing, loudness, trajectories, and video prompts; validation and revision check spatial ranges, visibility, and consistency between language and trajectories.Invalid scenes are regenerated, and the checking–revision loop runs for up to three rounds.Source: §A.1.1 · pp. 15–16
  • Seedance generates full-scene video and isolated source clips; extracted monophonic stems are duration-adjusted, loudness-normalized, and gated or faded.Source: §A.1.1 · p. 16
  • Because scripted trajectories only approximately match rendered pixels, detected objects are tracked and matched to sources, then image geometry and monocular depth help recover and correct per-timestep trajectories.The corrected scene includes source position and visibility state; off-screen sources retain scripted trajectories or are extrapolated.Source: §A.1.1 · pp. 16–17
  • The pipeline appends 2–4 off-screen sources and adds camera yaw motion, keeping rendered audio rotation consistent with the cropped or panned image.Source: §A.1.1 · p. 17
  • Final stems are rendered into FOA, and QA pairs are generated from trajectories for direction, motion, 3D localization, and camera rotation.The process yields 26,585 training questions from 1,484 synthesized videos, with questions balanced across specified outcomes and direction labels.Source: §A.1.1 · p. 17
  • Audio-only scenes use known source trajectories to generate spatial questions, producing 18,518 questions from 2,000 moving-source scenes.The questions cover direction, distance, motion, and relations between sources; an LLM rewrites questions while answers remain fixed by the rendered scene.Source: §A.1.1 · p. 17
  • The same recordings form 100,000 static scenes for semantic-alignment and query-conditioned localization pretraining; training and real QA distance distributions share a 2.8 m median.Figure 7 reports JSD = 0.14 bits for source type and a Kolmogorov–Smirnov statistic D = 0.20 for distance; non-spatial tasks are also added to training.Source: Figure 7 · p. 15, §A.1.1 · p. 17

A.1.2 SOUND-GUIDED NAVIGATION DATA SYNTHESIS

The six-stage pipeline converts existing vision-language navigation corpora into sound-guided navigation supervision. It proposes destination sounds, synthesizes and spatially renders audio, augments it, and formats examples with trajectory targets and reasoning traces.

Pipeline overview

  • The six-stage pipeline converts R2R, RxR, VLN-PE, and ScaleVLN into sound-guided navigation supervision, from sound-event proposal through CoT annotation.Its stages are destination sound-event proposal, audio synthesis, FOA rendering and sample construction, acoustic augmentation, omni-modal SFT formatting, and CoT annotation.Source: §A.1.2 · p. 17

Destination sound-event proposal

  • Destination sound proposals are balanced roughly 50/50 between spoken calls and non-speech events, while excluding destination location, room, and nameable-furniture clues.A vision–language model uses the last five first-person destination frames to propose a plausible event and task instruction.Source: §A.1.2 · p. 18

Audio synthesis

  • Spoken utterances use custom-voice text-to-speech conditioned on voice style, while non-speech events are synthesized with Seedance.The two synthesis methods are summarized in Table 5.Source: §A.1.2 · p. 19, Table 5 · p. 19

FOA rendering and sample construction

  • Each sound is placed at the episode endpoint, transformed into the listener’s camera frame, inverse-distance attenuated, and encoded as four-channel FOA; targets are the next eight relative waypoints.Attenuation is floored at 0.3 m, and clips are not peak-normalized so loudness preserves a distance cue.Source: §A.1.2 · p. 19, Table 6 · p. 19

Acoustic augmentation

  • Room-geometry corpora receive image-source early reflections and direct-sound attenuation, while geometry-free corpora use direct-path distance attenuation; optional augmentation adds directional noise.The optional augmentation mixes 1–3 plane-wave interferers, placed at least 30° from the source direction, at sampled SNRs from 20 to 0 dB.Source: §A.1.2 · p. 19, Table 6 · p. 19

SFT format and CoT

  • SFT examples combine sampled history frames, the current frame, co-located FOA, and task text to predict eight waypoints; CoT prompts receive geometry-derived direction, distance, and meta-actions.The pipeline also adds in-place turning examples and active-stop supervision for stationary episode endings.Source: §A.1.2 · p. 19

A.2.1 OMNIECHOBENCH-QA: REAL-WORLD CAPTURE AND ANNOTATION

OmniEchoBench-QA uses synchronized real-world video and four-channel FOA recordings, with annotations and question answers derived from source trajectories. Its questions cover geometric and temporal reasoning, with the distribution of direction and distance presented in Figure 8.

Capture rig and formats

  • Each live script produces synchronized first-person video, panoramic video, and four-channel FOA audio recorded at 48 kHz across 197 indoor and outdoor settings.The settings include 50 small indoor, 50 medium indoor, 77 large indoor, and 20 open outdoor scenes.Source: §A.2.1 · p. 21

Annotation

  • Annotators use panoramic recordings to label active-source positions every 0.5 seconds, along with visibility and audibility; camera-moving recordings rotate FOA into the current camera frame.At test time, the model receives main-view video or a single frame together with processed four-channel FOA.Source: §A.2.1 · pp. 21–22

Question construction

  • Question answers are computed deterministically from annotated trajectories, covering source bearings, distance, visibility, and interval changes in short closed-bucket responses.The main paper reports four geometric categories totaling 2,372 QA; visibility is additionally annotated, and the 600-QA Cognitive Map subset uses navigation scenes.Source: §A.2.1 · p. 22, Table 7 · p. 22

Coordinate conventions

  • The coordinate convention fixes the world frame to the camera’s initial heading and defines body azimuth by subtracting recovered camera yaw from world azimuth.Azimuth is measured counter-clockwise: 0° ahead, 90° left, 180° behind, and 270° right.Source: §A.2.1 · p. 22

A.2.2 OMNIECHOBENCH-NAV: REAL-WORLD SCENE AND ACOUSTIC-FIELD COLLECTION

This section describes OmniEchoBench-Nav’s real-world acoustic-field data, OmniEcho’s training and evaluation procedures, and results spanning sim-to-real transfer, navigation failure modes, and visualization examples.

A.2.2 OMNIECHOBENCH-NAV: REAL-WORLD SCENE AND ACOUSTIC-FIELD COLLECTION

  • The navigation benchmark provides 900 samples across 30 real-scanned indoor scenes, with dense grids of world-aligned four-channel FOA recordings.Visuals are mosaic-blurred, and released data are intended for spatial perception and navigation research, not biometric analysis.Source: §A.2.2 · p. 22, Table 8 · p. 23, §A.2.3 · p. 23

A.3 TRAINING DETAILS

  • OmniEcho pretrains a compact FOA encoder for semantic alignment and query-conditioned localization, then integrates the frozen encoder into Qwen3-Omni.Pretraining curves show retrieval Top-1 of ∼37% over 644 labels, mean azimuth error ∼16°, elevation error ∼7°, and distance error ∼1.1 m.Source: §A.3 · pp. 23–25, Figure 10 · p. 25

A.4.1 SPATIAL AUDIO-VISUAL QA EVALUATION

  • QA scoring normalizes free-form answers into structured fields, then applies deterministic task-specific rules and averages correctness across samples.The rules include exact letter matches for Cognitive Map questions and stated angular and distance tolerances.Source: §A.4.1 · pp. 25–26, Table 13 · p. 26

A.4.2 SOUND-GUIDED NAVIGATION EVALUATION

  • Sound-guided navigation uses live-pose acoustic observations, egocentric remapping, and model-planned waypoints, with agent self-stop for all reported experiments.Evaluation reports navigation metrics using a 1 m success radius on OmniEchoBench-Nav and 3 m on VLN-CE R2R Val-Unseen.Source: §A.4.2 · p. 27, Table 14 · p. 27

A.5.1 SIM2REAL ANALYSIS

  • Real-data fine-tuning raises the average score from 28.5 to 34.9, with gains across all evaluated task categories.The largest listed gain is Motion (+10.9), while Cognitive Map improves by +0.5.Source: §A.5.1 · pp. 27–28, Table 15 · p. 27

A.5.2 VLN FAILURE MODES

  • Only 16.22% of episodes terminate successfully, although 52.89% enter the 1 m goal region at least once.Distance-estimation MAE is 2.20 m with Pearson correlation 0.27; multiple actions between decisions mean the success gap cannot be attributed solely to stopping errors.Source: §A.5.2 · p. 28, Table 17 · p. 28

A.5.3 VLN WITH GT DIRECTION

  • With ground-truth direction, SR rises from 16.2% to 23.7% and SPL from 11.5% to 22.1%, while OS falls from 52.89% to 34.33%.OS records entering the goal region; SR additionally requires successful termination there.Source: §A.5.3 · pp. 28–29, Table 18 · p. 29

A.6 VISUALIZATION EXAMPLES

  • QA examples show successes in motion, localization, and camera rotation, while failures retain coarse orientation but miss finer geometric details.Navigation visualizations pair trajectories with intermediate observations and planned actions; a physical-robot example approaches an initially out-of-view alarm around obstacles.Source: §A.6 · p. 29

Related papers

Loading 2609.23407v2…