PaperlayerView exact revision on arXiv

Source-linked AI summary

FloorSAV: Elucidating Spatial Audio-Visual Context with 2D Floormap for AV-LLMs

Kyeong-Rae Kim, Sungnyun Kim, Tae-Hyun Oh

arXiv:2610.11310v1cs.CVcs.LG

TL;DR

AV-LLMs have limited explicit means to represent global geometry from raw egocentric audio-visual streams. FloorSAV adds a synchronized dynamic 2D floormap for joint reasoning, and improves spatial-reasoning scores on SAVED-Bench and SAVVY-Bench. The authors report further potential with ground-truth floormaps.

  • Problem

    AV-LLMs lack explicit mechanisms to internalize global geometry from raw sensory streams, and fine-grained spatial reasoning remains challenging in egocentric settings.

    Source: §1 · pp. 1–2

  • Method

    FloorSAV renders a dynamic 2D floormap integrating camera trajectories, estimated sound positions, and semantic objects, then synchronizes it with egocentric video for joint AV-LLM reasoning.

    Source: §1 · p. 2

  • Results

    FloorSAV improves over the egocentric view-only Gemini baseline from 58.3% to 65.3% overall on SAVED-Bench and from 50.9% to 54.7% on SAVVY-Bench.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Evaluations report benefits across single-viewpoint, cross-agent, regional, and path-aware reasoning, with ground-truth floormaps demonstrating potential when spatial data are accurate.

    Source: §6 · p. 10

    Performance is limited by AV-LLM temporal localization of sound events, relative-position biases that ignore video cues, and difficulty interpreting geometric 2D floormaps.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

While 3D spatial reasoning in dynamic egocentric environments is crucial for embodied intelligence, audio-visual large language models (AV-LLMs) lack explicit mechanisms to process and internalize global geometry directly from raw sensory streams. Existing approaches either require costly fine-tuning or underutilize the model's cross-modal reasoning capacities. In this paper, we propose FloorSAV, a novel framework that explicitly grounds spatial audio-visual context by rendering a dynamic 2D floormap. By integrating 3D point clouds, camera trajectories, spatial audio cues, and semantically grounded object landmarks, we inject this floormap into the AV-LLM as a synchronized stream with an egocentric video. AV-LLMs utilize their multi-modal capabilities to jointly reason over visual, auditory, and geometric cues in a single inference with floormap interpretation guidance. We further introduce SAVED-Bench (Spatial Audio-Visual Egocentric Benchmark with Dynamic Agents), constructing essential tasks of spatial capability in real-world scenarios: dynamic relativity, regional, and path reasoning QAs. FloorSAV improves AV-LLMs' spatial reasoning on various tasks from both SAVED-Bench and SAVVY-Bench. Studies with ground-truth floormaps demonstrate the substantial potential of FloorSAV with accurate spatial information.

1. Introduction

AV-LLMs struggle to build coherent global spatial representations from egocentric audio-visual streams, while existing approaches either require costly fine-tuning or leave spatial reasoning to external pipelines. FloorSAV supplies synchronized floormaps and SAVED-Bench, and improves scores on SAVED-Bench and SAVVY-Bench.

1. Introduction

  • AV-LLMs remain below human-level performance on fine-grained spatial tasks, and raw egocentric streams make global geometry and transient sound sources difficult to track.Moving cameras have narrow fields of view, while monaural audio can discard spatial attributes.Source: §1 · pp. 1–2
  • FloorSAV renders camera trajectories and estimated sound-source positions on a dynamic 2D floormap, then feeds it alongside synchronized egocentric video for joint spatial reasoning.The model jointly reasons over visual appearance, auditory cues, and global scene geometry in one inference.Source: §1 · p. 2, Figure 2 · p. 2
  • SAVED-Bench evaluates dynamic-agent spatial reasoning through dynamic relativity, regional, and path-reasoning questions, extending beyond direction and distance estimation.Its categories target cross-agent, semantic region-aware, and path-aware cognition.Source: §1 · pp. 2–3
  • 65.3% from 58.3% on SAVED-Bench and 54.7% from 50.9% on SAVVY-Bench: FloorSAV improves over the egocentric view-only Gemini baseline.With ground-truth agent positions and objects, FloorSAV reaches 71.3% overall on SAVED-Bench.Source: §1 · p. 3

2. Related Work

Prior work explores spatial understanding in MLLMs and egocentric audio-visual reasoning, but explicit egocentric spatial reasoning with AV-LLMs remains limited.

2.1. Spatial Understanding in MLLMs

  • Vision-based MLLM approaches can suffer visual bias and hallucinate in occluded or out-of-view regions, while audio-centric methods may discard visual semantics and rely heavily on simulated data.Unified AV-LLM progress has largely been measured on benchmarks about what and when, rather than where.Source: §2.1 · p. 3

2.2. Egocentric Audio-Visual Spatial Reasoning

  • Egocentric spatial work identifies SAVVY’s limited use of MLLM cross-modal capabilities and its narrow benchmark scope, while explicit spatial abstractions such as 2D maps are proposed for reasoning.The paper introduces semantic grounding as keyframe selection, VLM object mapping, and filtering to enrich floormaps.Source: §2.2 · pp. 3–4, Figure 3 · p. 4

3. Method: FloorSAV

FloorSAV renders a top-down spatial representation from audio-visual signals and presents it alongside egocentric video so an AV-LLM can reason over both streams.

3. Method: FloorSAV

  • FloorSAV injects a rendered global map alongside corresponding egocentric video, unlike SAVVY’s offline rule-based mapping after preliminary AV-LLM subtasks.The synchronized floormap is an explicit visual input intended to support the model’s cross-modal reasoning.Source: §3 · p. 4

3.1. Floormap Rendering via Explicit Spatial Grounding

  • The rendering pipeline projects point clouds, wearer pose, and estimated sound-source distance and direction onto a metric grid, then overlays camera and sound markers and semantic object labels.Keyframes are selected for map coverage; VLM detections are normalized, filtered, and merged across frames.Source: §3.1 · p. 4

3.2. Floormap-Aware Inference

  • The method synchronizes the floormap and egocentric videos as separate streams sampled at the same frame rate, preserving camera and sound-source motion.A guide explains map marks and directs the model to use egocentric video as the primary reference, consulting the floormap when spatial details are uncertain.Source: §3.2 · pp. 4–5

4. Spatial Reasoning Benchmarks

SAVED-Bench extends spatial QA beyond SAVVY-Bench’s fixed-view direction and distance questions, targeting dynamic-agent reasoning across viewpoints, semantic regions, and paths. Its tasks use sound events and scene context to test these capabilities.

4.1. SAVVY-Bench

  • SAVVY-Bench, sourced from AEA, asks about direction or distance at a target sound event from an egocentric or hypothetical exocentric viewpoint.Source: §4.1 · p. 5

4.2. SAVED-Bench: Towards Dynamic Agents in Real-World Scenarios

  • SAVED-Bench broadens spatial evaluation beyond fixed-view direction and distance questions with cross-agent, region-aware, and path-aware tasks.The tasks assess both the egocentric camera wearer and another agent.Source: §4.2 · pp. 5–6, §4.1 · p. 5

4.2.1. Dynamic Relativity QA Synthesis

  • Dynamic Relativity QAs ask models to transform an imaginary object’s direction or distance between Ego’s and Exo’s viewpoints.Questions use a sound event to set query time and specify Exo’s hypothetical facing direction.Source: §4.2.1 · p. 6

4.2.2. Regional QA Synthesis

  • Regional QAs test agent locations and region-visit histories over time, using semantic regions annotated from key-object arrangements and refined by humans.The initial object-group rectangles are refined into region polygons for QA synthesis, not provided to the model.Source: §4.2.2 · p. 6, Figure 5 · p. 6

4.2.3. Path Reasoning QA Synthesis

  • Path Reasoning QAs test closest-object search along a straight path and distances traveled by both agents during a sound interval.The path settings include hypothetical finite straight paths and observed trajectories constrained by physical surroundings.Source: §4.2.3 · p. 7

5. Experiments and Results

FloorSAV is evaluated on SAVED-Bench and SAVVY-Bench, with analyses of cross-agent, regional, and path reasoning and studies using ground-truth floormaps. The reported results show gains across benchmarks, alongside limits associated with acoustic localization and object mapping.

5.1. Setup

  • SAVVY-Bench has over 1,500 QA pairs, while SAVED-Bench has 1,988 questions from 55 indoor recordings; evaluation is zero-shot without fine-tuning.Most tasks use multiple-choice exact-match accuracy; distance estimation uses AbsMRA with thresholds from 0.1 m to 1.0 m.Source: §5.1 · p. 7

5.2. Results on SAVED-Bench

  • 65.3% vs. 58.3% overall on SAVED-Bench, with FloorSAV improving over baseline across all categories.The comparison uses Gemini-3.6-Flash.Source: Table 1 · p. 8, §5.2 · p. 8

5.2.1. Dynamic Relativity: Cross-Agent Reasoning

  • Direction gains are larger for FoV-out than FoV-in (+15.0 vs. +12.5), while both strategies perform worse on Exo-to-Ego tasks.FoV-out means the other agent is not visible at the sound-event query time.Source: §5.2.1 · p. 8, Figure 7 · p. 8

5.2.2. Regional: Bridge Semantic Region with Dynamic Agents

  • Audio-only regional QA results suggest audio contains regional context, while ground-truth object labels help associate spatial locations with semantic regions.The full ground-truth map improves performance on all regional QAs.Source: Table 2 · p. 9, §5.2.2 · p. 9, §5.4 · p. 10

5.3. Results on SAVVY-Bench

  • FloorSAV improves SAVVY-Bench overall scores for Qwen3-Omni-30B, Gemini-2.5-Flash, and Gemini-2.5-Pro; Gemini-2.5-Pro reaches 52.8%.All baselines fall below the 32.1% random-chance level; distance performance is slightly degraded.Source: Table 4 · p. 9, §5.3 · p. 9

5.2.3. Path Reasoning: Spatial Awareness on Physical Environment

  • FloorSAV gains on all path-reasoning tasks; Trajectory Distance rises from 16.9 to 25.0 AbsMRA, and historical path rendering improves that task.Sparse, inaccurate sound localization magnifies the gap between derived and true distances; rendered key objects support Line-Path Search gains.Source: §5.2.3 · p. 9

5.4. Studies with Ground-Truth 2D Floormap

  • With ground-truth agent positions, distance tasks, dynamic Line-Path Search, and path reasoning improve; ground-truth object names improve all regional QAs.These studies indicate FloorSAV’s potential with more accurate spatial components.Source: §5.4 · pp. 9–10

6. Conclusion

The conclusion presents FloorSAV as a training-free framework that synchronizes 2D floormaps with egocentric video, and reports benefits across spatial reasoning benchmarks. The rendering configuration combines room geometry, camera state and motion, acoustic estimates, and semantically grounded objects.

6. Conclusion

  • FloorSAV uses synchronized 2D floormaps integrating camera tracking, spatial audio, and object grounding; evaluations report benefits across SAVED-Bench and SAVVY-Bench spatial reasoning tasks.Ground-truth floormap studies show potential with accurate data; the authors suggest future training could help models internalize floormaps.Source: §6 · p. 10

A.1. Floormap Rendering Configuration

  • The base map filters 3D point clouds for room structure and object boundaries, then renders a metric XY frame with a 1 m grid.Vertical bounds use the 10th–99th percentiles, XY bounds the 1st–99th percentiles expanded by 0.5 m, and uncertain points may be removed.Source: §A.1 · p. 15
  • Camera position and heading are marked in red, its ±50° field of view in yellow, and its recent motion history in blue.The blue polyline represents the most recent 300 camera frames, approximately 15 seconds.Source: §A.1 · p. 15
  • The acoustic pipeline estimates sound-source direction with SRP-PHAT and distance from coherent-to-diffuse ratio, supplying spatial context to the AV-LLM.Localization uses four of seven microphone channels; estimates are produced from non-overlapping 0.25-second chunks.Source: §A.1 · p. 15
  • SRP-PHAT selects the azimuth with the largest summed pairwise response from phase-normalized microphone cross-correlations.The implementation searches −180° to 180° in 1° increments and restricts lags to ±1 ms.Source: §A.1 · p. 15
  • CDR-based distance conversion uses SAVVY calibration; the median of positive calibration constants is used for each recording.The calibration constants convert CDR values into sound-source distances downstream.Source: §A.1 · p. 15
  • Sound estimates are projected from camera-relative angle and distance into global coordinates, then averaged within ±1 second around each output frame.The synchronized input uniformly samples 128 frames across the RGB video and floormap sequence; markers are omitted when no valid observation is available.Source: §A.1 · p. 16
  • Object grounding selects keyframes, maps detected objects onto floormaps, filters implausible or weak detections, and retains spatially distinct instances of the same class.Components require at least three detections; their mean global coordinates become the mapped object locations.Source: §A.1 · p. 16

A.2. Evaluation Details

This section specifies evaluation procedures and the three SAVED-Bench task families, then contrasts FloorSAV’s direct-map reasoning design with SAVVY’s externally computed, multi-stage approach.

A.2. Evaluation Details

  • Evaluation uses inference-only models and synchronized egocentric video, floormap video, and audio; coordinate instructions are simplified for both baseline and FloorSAV questions.Decoding is deterministic, with temperature 0.0; FloorSAV involves no training.Source: §A.2 · pp. 16–18

A.3. SAVED-Bench Configuration

  • AbsMRA is the primary distance metric, averaging accuracy across ten absolute-error thresholds; RelMRA is complementary and measures error relative to ground-truth distance.AbsMRA applies the same sensitivity across target distances, whereas RelMRA allows proportionally larger absolute errors for longer distances.Source: §A.3 · p. 18

A.4. Dynamic Relativity QAs Details

  • 800 dynamic-relativity QAs test transforming angle-and-distance object descriptions between agents’ reference frames, with answers grounded in AEA trajectories.The set contains 200 questions per task and covers both field-of-view-in and field-of-view-out subsets.Source: §A.4 · p. 19

A.5. Regional QAs Details

  • 592 regional QAs test location and visit-history reasoning at sound events or between events; human-refined region boundaries and labels are omitted from maps to prevent direct matching.Region annotations begin with object categorization and rectangles, which annotators refine into polygons aligned with scene walls.Source: §A.5 · pp. 19–20

A.6. Path reasoning QAs Details

  • 596 path-reasoning QAs cover static and dynamic line-segment search and agent travel; trajectory ground truth smooths paths over 0.75 s and applies RDP simplification with 0.5 m tolerance.Trajectory-distance predictions report both agents’ distances, camera wearer first; the task score averages their AbsMRA scores.Source: §A.6 · pp. 20–21

B.1. Utilization of AV-LLMs

  • FloorSAV gives the AV-LLM the floormap as a direct visual prompt, while SAVVY delegates final spatial calculations to external rule-based modules.SAVVY derives spatial cues through separate modules; FloorSAV synchronizes the map with egocentric video and audio for model reasoning.Source: §B.1 · p. 21

B.2. Information Processing Pipeline

  • FloorSAV synchronizes a pre-rendered floormap video with egocentric RGB footage and audio, enabling cross-modal spatial reasoning in a single inference.The prompt includes a floormap guide; evaluation simplifies coordinate instructions so models can infer spatial relations from video inputs.Source: §A.2 · pp. 17–18, §B.2 · p. 22

B.3. Future Scalability

  • FloorSAV’s performance is directly coupled with AV-LLM spatial reasoning, so it can benefit as foundation models’ map-reading and reasoning capabilities evolve.The authors note that current models’ map-reading and spatial reasoning are not yet ideal.Source: §B.3 · p. 22

B.4. Task Flexibility and Broad Applicability

FloorSAV’s top-down spatial hint supports question formats beyond SAVVY’s rigid event- and object-anchored templates, including cross-agent, regional, and physical-object tasks.

B.4. Task Flexibility and Broad Applicability

  • FloorSAV’s top-down floormap supports broader spatial questions, including cross-agent, semantic-region, and physical-object tasks beyond SAVVY’s rigid templates.SAVVY requires a distinct sound event and explicitly present target objects.Source: §B.4 · p. 22

C. Additional Results

The ablations favor separate synchronized floormap and egocentric streams, while frame count and semantic labels also affect performance. Qualitative examples illustrate how map geometry supports spatial reasoning across SAVVY-Bench and SAVED-Bench tasks.

C.1. Floormap Prompting Strategies

  • 49.6% overall performance is achieved by dual-stream injection, which lets the model cross-reference egocentric appearance with dynamic map geometry without visual interference.The comparison evaluates four prompting formats with Qwen3-Omni-30B.Source: §C.1 · pp. 22–23, Table 5 · p. 23

C.2. Number of Video Frames

  • 32 frames achieve the highest overall accuracy (50.1%), while the default uses 128 frames to prioritize complex exocentric tasks.With 32 frames, egocentric direction reaches 73.9% and distance reaches 63.9%.Source: §C.2 · pp. 23–24, Table 6 · p. 24

C.3. Semantic Grounding Effect

  • Removing semantic grounding degrades overall performance, especially on exocentric direction and static Line-Path Search tasks.The default framework has models generate object labels overlaid on the 2D map.Source: §C.3 · p. 24, Table 7 · p. 24, Table 8 · p. 25

D. Qualitative Examples

  • The qualitative section presents successful floormap-guided cases and analyzes recurring failure modes in current foundation models.The examples are organized around successful cases followed by three categories of failure.Source: D. Qualitative Examples · p. 24

D.1. Successful Examples on SAVVY-Bench

  • In SAVVY-Bench examples, coordinates let FloorSAV infer exocentric direction and compute distance from mapped sound-source and target locations.For the distance case, it estimates 2.5 meters from map coordinates after the baseline overestimates by approximately 1.5 meters.Source: §D.1 · p. 25

D.2. Successful Examples on SAVED-Bench

  • SAVED-Bench examples show floormap coordinates supporting cross-agent viewpoint transformation, semantic region recognition, and distance-to-segment reasoning.The examples cover dynamic relativity, Location Awareness, and static Line-Path Search.Source: §D.2 · pp. 25–27

D.3. Failure Analysis 1: Temporal Localization Error

FloorSAV can produce flawed spatial predictions when the model anchors a query to the wrong time. The resulting map geometry may be accurate for that timestamp but irrelevant to the queried event.

D.3. Failure Analysis 1: Temporal Localization Error

  • Incorrect event-time localization leads the model to retrieve spatial geometry from an irrelevant frame, undermining the final prediction even when map reading and geometric calculation work for that moment.The paper identifies temporal localization as a common failure mode in dynamic environments.Source: §D.3 · p. 27
  • The model anchors “acknowledging the watermelon” to 01:53 instead of 02:19 and predicts “front-right” rather than the true “front-left”.The two moments have different relative positions for the camera wearer and speaker.Source: §D.3 · p. 27

D.4. Failure Analysis 2: Long-Context Understanding

Long video contexts can overwhelm current foundation models, and adding the floormap stream further increases the context burden. Failures can affect basic RGB perception as well as spatial reasoning.

D.4. Failure Analysis 2: Long-Context Understanding

  • A 270-frame input at 1 FPS fails on a simple egocentric direction task, whereas a 128-frame input at approximately 0.5 FPS guides the baseline to the correct answer.The authors link this qualitative example to the frame-count ablation in Table 6.Source: §D.4 · p. 28
  • Adding the synchronized floormap stream approximately doubles visual tokens and can overload the model, which may then misread the RGB view itself.In the illustrated failure, the model predicts the person is on the right although they sit on the left for most of the timeline.Source: §D.4 · p. 28

D.5. Failure Analysis 3: Metric Map Misinterpretation

FloorSAV remains vulnerable to errors in interpreting abstract metric grids. Its guide prompts explain the synchronized video streams and the map’s visual conventions to support interpretation.

D.5. Failure Analysis 3: Metric Map Misinterpretation

  • FloorSAV can misread map coordinates despite correctly locating the event time, and larger visual markers do not resolve the issue.The authors identify explicit map reading as a primary bottleneck and connect it to limited top-down and coordinate-based spatial reasoning in training corpora.Source: §D.5 · pp. 29–30, Figure 20 · p. 35

E. Floormap Guide Prompts

  • The prompt uses Exp1 for the egocentric video and Exp2 for the floormap; the egocentric-only baseline uses Exp1 alone.The question and answer options are inserted before the answer instruction.Source: E. Floormap Guide Prompts · p. 36
  • The floormap guide defines the metric grid, camera position and heading, field of view, object landmarks, and estimated sound-source markers for alignment with the egocentric view.The guide notes that sound-source estimates may be inaccurate or absent and that the map does not show another person’s facing direction.Source: E. Floormap Guide Prompts · p. 37

F. Limitations and Future Work

FloorSAV’s spatial prompts depend on high-fidelity sensors, while current AV-LLMs struggle to read abstract metric maps. Evaluation also depends on temporal sound-event localization and does not test integrated reasoning across task categories.

F. Limitations and Future Work

  • The rendered floormaps depend on clean 3D point clouds, sub-centimeter camera poses, and multi-channel spatial audio; sound localization remains noisy in reverberant or overlapping audio.The authors identify sensor quality as a boundary on geometric fidelity and note that improving datasets and wearable sensors could increase prompt precision.Source: F. Limitations and Future Work · p. 37
  • Current AV-LLMs often struggle to read precise coordinates on top-down metric maps and relate their global frame to the egocentric view.Semantic object labels partially mitigate this mismatch, but the authors describe the representational gap as a fundamental bottleneck.Source: F. Limitations and Future Work · p. 37
  • Because performance reflects both temporal localization and spatial reasoning, errors in locating sound events can obscure spatial reasoning ability.The authors recommend disentangling these errors and extending tasks beyond sound-event temporal grounding.Source: F. Limitations and Future Work · pp. 37–38
  • SAVED-Bench tests its three reasoning categories independently, leaving integrated cross-category interactions unevaluated.The authors suggest adding tasks that combine cross-agent viewpoint transformation, semantic region understanding, and recognition of physical surroundings.Source: F. Limitations and Future Work · p. 38
Loading 2610.11310v1…