PaperlayerView exact revision on arXiv

Source-linked AI summary

Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-Visual LLMs

Kim Sung-Bin, Lee Jung-Mok, Jinwoo Jung, Oh Hyun-Bin, Hyeonggon Ryu, David Harwath, Tae-Hyun Oh

arXiv:2610.21588v1cs.CVcs.SD

TL;DR

Most AV-LLMs treat audio as monaural semantic content, limiting spatial grounding and scene-level reasoning. This paper introduces a hear-localize-reason benchmark and an AV-LLM with binaural spatial audio and final-stage chain-of-thought supervision. HLR-LLM improves spatial grounding and relational reasoning while retaining semantic audio-visual perception.

  • Problem

    Most AV-LLMs focus on monaural audio semantics, and existing spatial benchmarks and direct-answer training leave joint spatial grounding and multi-step scene reasoning difficult to assess.

    Source: §1 · p. 2

  • Method

    The paper introduces HLR-AVSceneQA’s hear-localize-reason evaluation and HLR-LLM, which adds a binaural spatial audio branch and final-stage chain-of-thought supervision to a strong AV-LLM.

    Source: §1 · p. 2

  • Results

    HLR-LLM substantially improves spatial grounding and relational reasoning while preserving strong semantic hearing, including sound-event and speech recognition.

    Source: §1 · p. 2

  • Takeaways & Limitations

    HLR-AVSceneQA pretraining provides useful spatial priors for adaptation to real-world spatial audio-visual recordings, as suggested by transfer experiments [19].

    Source: §1 · p. 2

    HLR-AVSceneQA uses a simulator-based pipeline and focuses on indoor scenes, leaving real-world acoustic and visual diversity, dynamic agents, moving sources, and richer embodied interaction outside its current scope.

    Source: §6 · p. 10

Abstract

from arXiv · show

Audio-visual large language models (AV-LLMs) have made strong progress in multimodal understanding, yet most treat audio as monaural semantic content, limiting their ability to reason about where sounds originate and how they relate to the visual scene. Recent spatial audio-visual studies address parts of this problem, but often focus on specific abilities, such as spatial correspondence or direction and distance reasoning. In this work, we introduce HLR-AVSceneQA, a comprehensive benchmark with a hear-localize-reason hierarchy that evaluates whether models can recognize what is heard, localize sound sources in egocentric and allocentric views, and use these localized sources to reason about how they relate to the visual scene. We further introduce HLR-LLM, which augments a strong audio-visual foundation model with a binaural spatial audio branch alongside its monaural semantic audio pathway. We train HLR-LLM with a curriculum that incorporates final-stage chain-of-thought supervision for multi-step spatial reasoning. Experiments show that existing AV-LLMs remain limited in spatial grounding and relational reasoning, while HLR-LLM substantially improves spatial audio-visual scene understanding without compromising semantic audio-visual perception.

1 Introduction

Most AV-LLMs emphasize monaural audio semantics over sound location, while existing spatial benchmarks and training approaches leave joint scene-level reasoning difficult to assess. The paper introduces a unified benchmark and HLR-LLM, which improves spatial grounding and relational reasoning while retaining semantic perception.

1 Introduction

  • Existing AV-LLMs mainly treat audio as monaural semantic content, limiting reasoning about sources that are off-screen, occluded, in another room, or spatially ambiguous.Binaural or multi-channel audio offers direction and distance cues for spatially grounded understanding.Source: §1 · p. 2
  • Spatial benchmarks focus on specific subproblems, while direct-answer training often lacks structure for decomposing multi-step queries across auditory, spatial, and visual cues.This fragmentation makes joint assessment of semantic understanding, spatial grounding, and relational reasoning difficult.Source: §1 · p. 2
  • HLR-AVSceneQA evaluates what is heard, source localization in egocentric and allocentric views, and how localized sources relate to the visual scene.It uses normal-field-of-view observations and requires reasoning about visible, occluded, and off-screen sources from partial visual evidence.Source: §1 · p. 2
  • HLR-LLM adds explicit spatial audio modeling and final-stage chain-of-thought supervision to connect auditory, spatial, and visual cues during multi-step reasoning.It is built on an existing AV-LLM [4].Source: §1 · p. 2
  • HLR-LLM substantially improves spatial grounding and relational reasoning while preserving strong semantic hearing, including sound-event and speech recognition.Transfer experiments on real-world spatial audio-visual recordings [19] suggest HLR-AVSceneQA pretraining provides useful spatial priors for adaptation.Source: §1 · p. 2
  • The paper reports that final-stage chain-of-thought curriculum training improves multi-step spatial reasoning by connecting auditory, spatial, and visual cues through intermediate steps.The benchmark evaluates spatial hearing and grounding both directly and as evidence for downstream relational reasoning.Source: §1 · p. 2

2 Related work

Related work spans spatial audio understanding, spatial audio-visual benchmarks, and spatially aware AV-LLMs. The paper positions HLR-AVSceneQA and HLR-LLM as addressing fragmented task coverage and limited multi-step spatial reasoning.

Spatial audio understanding

  • Spatial audio work includes sound event localization and detection [28] [29] [30], language-model localization and reasoning [9] [10] [31], contrastive audio-text representations [11] [32], and far-field speech understanding [33].These approaches remain largely audio-only or speech-centric, leaving spatial auditory cues in visual scenes less explored.Source: §2 · p. 3

Spatial audio-visual benchmarks and tasks

  • Spatial audio-visual benchmarks cover different subsets: SAVVY-Bench [13] tests dynamic direction and distance, SpatialSceneQA [14] focuses on RGB-D and FOA grounding, and Hear You Are studies correspondence with panoramic images.HLR-AVSceneQA instead unifies spatial hearing, grounding, multi-source scenes, and scene-level relational reasoning with normal-field-of-view observations.Source: §2 · p. 3

Spatial audio-visual large language models (AV-LLMs)

  • Spatially aware AV-LLMs include Hear You Are LLM, SAVVY [13], and JAEGER [14], but are often tailored to specific tasks and commonly use direct-answer supervision.HLR-LLM extends a strong AV-LLM with a spatial audio branch and chain-of-thought supervision [17] [18] to improve spatial reasoning while preserving general audio-visual capabilities.Source: §2 · p. 3

3 HLR-AVSceneQA: Spatial audio-visual scene understanding benchmark

HLR-AVSceneQA organizes spatial audio-visual understanding as hearing, localizing, and reasoning about sound sources in a scene. A simulator-based pipeline creates scenes and metadata, then derives QA tasks and rationale-augmented examples.

3 HLR-AVSceneQA: Spatial audio-visual scene understanding benchmark

  • HLR-AVSceneQA evaluates spatial hearing, egocentric and allocentric source grounding, and relational reasoning about localized sources and visible, hidden, nearby, or off-screen objects.The benchmark contains 12 tasks, 175,061 training and 21,808 test QA pairs, including multi-source scenes and 94,519 CoT-augmented training pairs.Source: §3 · pp. 3–4

3.1 Stage I: Spatial audio-visual scene synthesis

  • Stage I synthesizes indoor 3D scenes from an agent viewpoint, pairing synchronized RGB observations and binaural audio with object layouts, source assignments, and geometric metadata.SoundSpaces 2.0 [16] renders geometry-aware room acoustics using HM3DSem [39]; 145 scenes are used for training and 36 for testing.Source: §3.1 · p. 4
  • Audio and visual assets are curated from VGGSound [40], LibriSpeech, Objaverse-XL [43], and supplementary generation models [44] [45], with filtering and manual mesh checks.The benchmark excludes some sound classes that are not naturally grounded to a single indoor object.Source: §3.1 · pp. 4–5
  • Scene construction varies candidate objects and source placements, including visible, occluded, adjacent-room, and off-screen sources, with metadata recording visibility and geometry.Some audio-object pairs are spatially aligned but semantically mismatched, discouraging reliance on category matching alone.Source: §3.1 · p. 5

3.2 Stage II: Benchmark task design and QA construction

  • Stage II generates QA pairs from rendered scenes and metadata, varying question templates while keeping answer formats controlled for stable evaluation.Most tasks use multiple choice or discretized bins; speech transcription and scene captioning are open-ended.Source: §3.2 · p. 5
  • The task families cover sound-event and far-field speech recognition, egocentric and allocentric direction and distance grounding, and scene-level relational reasoning.Relational tasks include visibility, correspondence, hidden-source localization, loudness change after movement, proximity, and scene captioning.Source: §3.2 · p. 5
  • CoT variants for allocentric grounding and relational reasoning use metadata-generated rationales that expose intermediate cues such as source identity, location, visibility, and distance comparisons.GPT-4 [46] is used only to paraphrase question wording; the rationales encourage explicit connections among auditory, spatial, and visual cues.Source: §3.2 · pp. 5–6

4 Method

HLR-LLM adds a binaural spatial audio pathway to a strong AV-LLM while retaining monaural semantic audio and visual inputs. Its staged training progresses from spatial-audio alignment to multimodal instruction tuning and final chain-of-thought tuning.

4 Method

  • HLR-LLM combines a base AV-LLM with a spatial audio pathway and curriculum-based chain-of-thought supervision for spatial grounding and reasoning.The design aims to retain the base model’s semantic perception ability while adding binaural spatial cues.Source: §4 · p. 6

4.1 HLR-LLM architecture

  • The architecture keeps monaural audio for semantic hearing and adds a spatial audio encoder and projector for direction- and distance-sensitive cues, which the language model integrates with visual and text tokens.Binaural features are projected into the language model embedding space as spatial audio tokens, and answers are generated conditioned on the combined input sequence.Source: §4.1 · p. 6

4.2 Curriculum training for spatial audio-visual reasoning

  • Training first aligns the spatial audio projector, then tunes on HLR-AVSceneQA to use visual, monaural, and spatial audio tokens across benchmark tasks, and finally tunes on CoT-formatted answers.The final stage trains compact rationales that connect auditory cues, spatial estimates, and visual evidence before a normalized answer.Source: §4.2 · p. 7

4.3 Implementation details

  • Training uses autoregressive cross-entropy; Stage 1 updates only the spatial audio projector, while Stages 2–3 update it and language-model LoRA parameters, with encoders frozen throughout.Implementation uses two epochs on four NVIDIA A6000 GPUs, with per-device batch size 4.Source: §4.2 · p. 7, §4.3 · p. 7

5 Experiment

The experiments evaluate HLR-LLM on HLR-AVSceneQA and STARSS23, finding that explicit binaural spatial modeling improves grounding and relational reasoning while preserving semantic audio-visual perception. Ablations and qualitative examples examine modality contributions, training strategy, adaptation, and localization.

5.1 Experimental setup

  • The evaluation uses accuracy for most tasks, WER for far-field speech recognition, and CIDEr-D, METEOR, and SPIDEr for scene captioning.Zero-shot models struggle with direction and distance grounding and relational reasoning; HLR-LLM performs best across the spatial relational reasoning tasks.Source: §5.1 · p. 7, Table 3 · p. 8, Table 4 · p. 8

5.2 Main results on HLR-AVSceneQA

  • HLR-LLM outperforms dual-channel Qwen2.5-Omni across egocentric and allocentric grounding and spatial relational reasoning tasks.It also achieves the best SPIDEr score, while dual-channel Qwen2.5-Omni leads on CIDEr-D and METEOR.Source: Table 3 · p. 8, Table 4 · p. 8, §5.2 · pp. 8–9

5.3 Ablation study

  • Removing vision reduces allocentric grounding and proximity reasoning, while removing binaural audio causes a large drop in grounding.Final-stage CoT tuning performs best across grounding metrics and most relational tasks, with gains especially evident in allocentric grounding, correspondence, and proximity.Source: §5.3 · p. 9, Table 5 · p. 9

5.4 Real-world adaptation

  • On STARSS23 real-world adaptation, HLR-AVSceneQA pretraining improves most evaluated tasks; among HLR-AVSceneQA-pretrained models, HLR-LLM outperforms Hear You Are LLM across all metrics.Models were fine-tuned on STARSS23; accuracy (%) was reported for sound event recognition and egocentric direction, distance, and visibility, averaged over three seeds.Source: §5.4 · pp. 9–10, Table 6 · p. 9

5.5 Qualitative results

  • In qualitative examples, HLR-LLM correctly localizes a source among similar visible objects and identifies a bird hidden behind a bed.The comparison models misidentify the source direction in the first example and the hidden region in the second.Source: §5.5 · p. 10, Figure 4 · p. 10

6 Conclusion and discussion

The paper presents HLR-AVSceneQA and HLR-LLM as a benchmark and model for spatial audio-visual understanding, reporting improved spatial reasoning and useful STARSS23 adaptation. It notes limits in simulator-based coverage and indoor-scene scope; the implementation uses a staged training curriculum.

6 Conclusion and discussion

  • HLR-AVSceneQA evaluates spatial hearing, egocentric and allocentric grounding, and relational reasoning; HLR-LLM adds a binaural branch and final-stage CoT supervision.The paper reports improved spatial reasoning while retaining strong semantic perception, and useful spatial priors for STARSS23 adaptation.Source: §6 · p. 10
  • The simulator-based benchmark cannot fully capture real-world acoustic complexity or visual diversity, and its scope is limited to indoor scenes.Dynamic agents, moving sound sources, and richer embodied interaction are left for future work.Source: §6 · p. 10

A.1 Implementation details

  • HLR-LLM uses Qwen2.5-Omni with a Spatial-AST binaural branch whose features are mapped into language embeddings by a lightweight projector.Training has three stages: Stage 1 trains spatial alignment modules; Stages 2–3 LoRA-finetune the language model while updating those modules and keeping encoders frozen.Source: §A.1 · p. 15

A.2 Details on comparison model experiments

The comparison experiments adapt existing models to HLR-AVSceneQA, including matched-data baselines and a binaural spatial-AV baseline. The STARSS23 preprocessing constructs a complementary real-world evaluation set from synchronized audio-video recordings.

A.2 Details on comparison model experiments

  • Hear You Are LLM is adapted to normal-FOV inputs and matched QA and evaluation conditions, but the comparison is not fully architecture-matched.Its original architecture is retained without HLR-LLM’s spatial audio branch.Source: §A.2 · p. 16
  • Qwen2.5-Omni provides a monaural baseline, while its dual-channel variant tests whether encoding both binaural channels alone supports spatial reasoning.The dual-channel variant adds no dedicated spatial audio encoder or projector.Source: §A.2 · p. 16

A.3 STARSS23 preprocessing and QA construction

  • From the official STARSS23 development split, recordings are divided into non-overlapping 10-second clips; excluding clips without video yields 1,317 train and 1,116 test clips, with midpoint-frame rectilinear images and CIPIC subject-048 HRTF binaural audio.Source: §A.3 · p. 16
  • STARSS23 QA covers recognition, egocentric direction and distance, and visibility, using static sources and excluding ambiguous items.Static sources stay within 5° angular variation and 20% relative-distance variation; visibility is defined by whether source direction falls inside the camera field of view.Source: §A.3 · p. 16
  • Real-world evaluation is limited to tasks supported by STARSS23 annotations, which lack dense object- and scene-level labels for HLR-style hidden-source and allocentric grounding.The authors avoid pseudo-labels for unsupported tasks.Source: §A.3 · pp. 16–17

B.1 Ablation study

The ablations show that binaural audio and final-stage chain-of-thought tuning support spatial grounding and relational reasoning, while monaural audio preserves semantic audio performance. More training data generally helps spatial reasoning, but some tasks vary non-monotonically.

Modality ablation

  • Removing binaural audio substantially degrades grounding and relational reasoning, while removing monaural audio has a smaller grounding effect but severely degrades far-field speech recognition.The ablations distinguish the spatial pathway’s role from the monaural pathway’s role in semantic audio understanding.Source: §B.1 · p. 17

Data-scale ablation

  • With only 10% of training data, the binaural model substantially outperforms the 100%-data model without binaural audio on egocentric grounding and most relational reasoning tasks.More data generally improves spatial reasoning, particularly relational reasoning and scene captioning, though some grounding tasks vary non-monotonically.Source: §B.1 · p. 17

Training ablation

  • Final-stage CoT tuning improves egocentric and allocentric grounding and performs strongest on most relational reasoning tasks, especially correspondence and proximity.It degrades scene captioning metrics and far-field ASR, indicating a trade-off with direct semantic prediction tasks.Source: §B.1 · pp. 17–18

B.2 Analysis on the STARSS23 dataset

Direct transfer from simulated HLR-AVSceneQA training to real-world STARSS23 recordings improves spatial predictions relative to existing AV-LLMs, though sound recognition remains less competitive.

Analysis on the STARSS23 dataset

  • Without STARSS23 fine-tuning, HLR-LLM outperforms existing AV-LLMs on direction, distance, and visibility prediction, with notable gains in distance estimation.Sound recognition remains less competitive; the results suggest spatial grounding can transfer from simulated scenes to real-world recordings without target-domain supervision.Source: §B.2 · p. 18

B.3 Qualitative results

The qualitative analyses illustrate how HLR-LLM uses localized sound sources for scene-level reasoning and how its spatial pathway relates to visual regions. They also show transfer examples, failure patterns, and the simulated scene-generation setup.

B.3 Qualitative results

  • In qualitative CoT examples, HLR-LLM localizes a source and reasons over candidate objects, including rejecting a visible saxophone that is not spatially aligned with the sound.The proximity example identifies the child, rather than Qwen2.5-Omni’s selected drum, as closest to the source.Source: §B.3 · pp. 18–19
  • STARSS23 examples show HLR-LLM recognizing a target event in multi-source audio and predicting source visibility from spatial audio and visual evidence.The examples are representative qualitative cases, not a quantitative comparison.Source: §B.3 · p. 19, Figure S2 · p. 19
  • Failures arise from fine-grained direction ambiguity, spatially clustered candidate objects, or nearby sources when a louder non-target source dominates overlapping cues.Errors can include confusing adjacent direction labels or reasoning about the wrong source.Source: §B.3 · p. 19, Figure S3 · p. 19
  • With the image fixed, changing binaural source position changes HLR-LLM’s localization and downstream loudness reasoning, providing evidence of reliance on spatial audio cues.The counterfactual examples vary the rendered barking position while the visible dog remains in place.Source: §B.3 · p. 20, Figure S4 · p. 20
  • Spatial-audio tokens attend more selectively to ground-truth sound-source regions than conventional audio tokens in the 18th-layer visualization.Attention is averaged across query tokens and heads for modality-to-visual maps.Source: §B.3 · p. 20, Figure S5 · p. 20

C.1 Scene initialization, agent placement, and scene generation

  • Spatial-reasoning scenarios include occluded sources behind objects or walls, same-category ambiguity, semantically mismatched but spatially aligned sources, and audible sources in adjacent rooms.Placements are rejected for collisions, overlap above a horizontal IoU threshold, or minimum-distance violations; active-source binaural audio is rendered with SoundSpaces 2.0.Source: §C.1 · pp. 21, 23

C.2 Task definitions and details

C.2 organizes spatial audio-visual understanding into hearing, source grounding, and relational reasoning tasks, with paraphrased prompts and controlled answers. C.3 adds structured chain-of-thought examples, while C.4 reports quality checks and human verification.

C.2 Task definitions and details

  • Spatial hearing covers sound-event recognition from 10 candidate categories and far-field speech transcription returned as an exact quotation.Recognition distractors include semantically related categories, scene-object-associated categories, and random categories; prompts may specify source direction and distance.Source: §C.2 · pp. 23–24
  • Grounding measures egocentric source direction and distance, plus allocentric direction and distance after imagining a move to a reference object and facing another object.Direction uses 8-bin labels and distance uses 10-bin labels; allocentric direction is generated only when valid visible reference and facing objects are available.Source: §C.2 · pp. 24–25
  • Relational tasks assess source visibility, spatial correspondence, hidden-source location, movement-related loudness change, proximity to objects, and compositional scene captioning.Hidden-source labels distinguish wall or object occlusion, an adjacent room, and positions outside the field of view; proximity excludes adjacent-room sources and tied cases.Source: §C.2 · pp. 25–26

C.3 Chain-of-thought supervision

  • Chain-of-thought supervision uses metadata- and template-derived rationales to connect sound recognition, spatial estimates, and scene evidence before a normalized final answer.This multi-step format is used for tasks including allocentric reasoning, hidden-source localization, loudness change, and proximity reasoning.Source: §C.3 · pp. 26–28

C.4 Quality control

  • Quality control filters invalid rooms, object placements, unclear visibility, unsuitable meshes, and proximity questions with tied nearest objects.Object meshes and assets from Objaverse-XL are manually inspected by human annotators.Source: §C.4 · p. 28
  • 1,133 of 1,144 QA pairs pass human verification (99.0%), with spatial-hearing and egocentric-grounding questions at 100%.Pass rates for remaining task categories range from 96.8% to 100%.Source: §C.4 · p. 28

C.5 Dataset statistics

The dataset statistics describe split composition, task distributions, and the separation of training and test environments, rooms, and object instances. The accompanying sections state intended application areas, LLM uses, and asset licensing practices.

C.5 Dataset statistics

  • Training and test splits use disjoint environments, rooms, and object mesh instances, while test sound and object categories are represented in training.The benchmark therefore evaluates generalization to unseen environments, scene layouts, and object instances with consistent semantic task coverage.Source: §C.5 · p. 28, Table S7 · p. 30
  • The test visibility set contains 30.7% visible and 69.3% non-visible cases, compared with 53.6% and 46.4% in training.Figures S8 and S9 show distance- and direction-bin distributions across egocentric and allocentric train and test settings; Figures S10 and S11 show frequent object and sound categories.Source: §C.5 · p. 28, Figure S8 · p. 31, Figure S9 · p. 32

D Broader impact

  • The benchmark and model are intended to support embodied AI, assistive agents, robotics, and accessibility applications, including indoor sound localization and reasoning about events outside the camera view.The benchmark is simulator-based and contains no private real-world recordings; future deployment in homes or shared spaces should follow privacy, consent, and data-minimization practices.Source: D Broader impact · p. 29

E LLM usage

  • LLMs are used as HLR-LLM’s reasoning module, to paraphrase question templates, and to assist with editing and figure, table, and data-viewer refinement.The reasoning module integrates visual, monaural audio, and binaural spatial audio tokens.Source: E LLM usage · p. 29

F Licenses and terms of existing assets

  • Existing datasets, simulators, models, and 3D assets are used under their stated licenses and terms, and only compatible generated assets and annotations are slated for release.Generated 3D assets are synthetic and are not intended to reproduce real people, private environments, or sensitive locations.Source: F Licenses and terms of existing assets · p. 30

Related papers

Loading 2610.21588v1…