PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometry-Informed Distributed Acoustic Scene Understanding

Yiyuan Yang, Shitong Xu, Niki Trigoni, Andrew Markham

arXiv:2609.08026v1cs.SD

TL;DR

Multi-room acoustic understanding is challenged by centralized sensing that misses events behind walls and by systems that do not account for room geometry. This paper fuses distributed acoustic features with geometry, decodes semantic triplets, and uses a frozen LLM to synthesize narratives. In simulation, the framework reports higher Triplet F1 than a centralized baseline and improved spatial consistency.

  • Problem

    Centralized sensing can miss events across walls, while distributed systems that ignore topology lack a basis for judging cross-room signals and movement.

    Source: §1 · p. 1

  • Method

    The framework fuses distributed microphone features using topology and geometry, decodes semantic triplets, and prompts a frozen LLM with the observations and environment geometry.

    Source: §1 · pp. 1–2, §3.1 · p. 2

  • Results

    The framework achieves a 0.87 Triplet F1-score, compared with 0.51 for the Centralized-Single baseline, and reports an 88.2% SCS.

    Source: §4.2 · p. 4, §4.3 · p. 4

  • Takeaways & Limitations

    The simulation results support distributed sensing and geometry-informed reasoning as promising for complex indoor scenes and reducing non-physical narrative transitions.

    Source: §5 · p. 4

    Evaluation uses known floor plans, microphone positions, and door states, and generated data do not cover dense overlapping sources in the same room.

    Source: A. Limitations and Future Work · p. 6

Abstract

from arXiv · show

Acoustic scene understanding in multi-room environments is a difficult task. Most existing systems use a single centralized microphone array, and they often fail because walls and doors block sound signals. To address this challenge, we propose a geometry-informed distributed acoustic scene understanding framework. Our system leverages distributed microphones and uses an audio spectrogram transformer and a topology-aware graph neural network to fuse spatio-temporal acoustic features. Then, these features are decoded into discrete semantic triplets. Finally, a frozen large language model combines these symbolic observations with the environmental geometry. This allows the system to perform spatial understanding, infer plausible missing transitions, and generate a physically consistent narrative of the scene. Experiments on a custom multi-room simulator demonstrate that our framework outperforms centralized baselines and improves spatial consistency under simulated occlusion.

1. Introduction

Multi-room acoustic understanding is limited by occlusion and by systems that combine distributed signals without accounting for room topology. The paper formalizes scene narratives as semantic triplets and proposes topology-aware fusion with geometry-informed reasoning.

1. Introduction

  • Single-location systems can miss events behind walls and doors, leaving the multi-room scene fragmented.A central sensor may not hear activity in another room because barriers attenuate sound.Source: §1 · p. 1
  • Distributed sensing broadens coverage, but methods that ignore topology cannot reliably weigh cross-room signals or represent room-to-room movement.They may treat sounds as isolated points without modeling leakage through walls or movement between rooms.Source: §1 · p. 1
  • Missing acoustic cues can produce non-physical room transitions, motivating geometry-aware reasoning to infer plausible intermediate steps.The passage illustrates a missed corridor-footstep cue leading to a predicted jump between rooms.Source: §1 · p. 1
  • The framework formalizes multi-room understanding with ⟨subject, relation, object⟩ triplets to move beyond isolated event detection toward grounded narratives.Source: §1 · p. 1
  • The proposed system combines topology-aware graph fusion with geometry-aware LLM reasoning, and simulator results report good accuracy and improved logical consistency under spatiotemporal discontinuities.The evaluation uses a multi-room acoustic simulator.Source: §1 · p. 2

2. Related Work

Related work spans sound localization, distributed multi-room sensing, and audio-language reasoning. The paper positions explicit floor-plan-constrained narrative reconstruction as underexplored.

2.1. Sound Event Localization and Detection

  • SELD methods advance spatial acoustic representations with Transformers, self-supervision, multi-scale fusion, and consumer stereo, but generally focus on single spaces and coordinate estimation.Source: §2.1 · p. 2

2.2. Distributed and Multi-Room Acoustic Sensing

  • Distributed microphones broaden room coverage, while graph models encode microphone connectivity and can emphasize direct propagation paths over spurious leakage.Source: §2.2 · p. 2

2.3. Geometry-Informed Audio-Language Reasoning

  • Audio-language systems use symbolic representations and pretrained language models for scene reasoning, but explicit floor-plan-constrained narrative reconstruction remains underexplored.Source: §2.3 · p. 2

3. Methodology

The framework combines acoustic signals with environmental geometry, fuses distributed microphone features over a topology graph, and decodes them into semantic triplets. A frozen LLM receives the triplet sequence and floor-plan description to synthesize a narrative.

3. Methodology

  • The framework takes synchronized multi-channel acoustic signals and environmental geometry as inputs for producing a textual scene description.Source: §3 · p. 2, §3.1 · p. 2

3.1. Input Formalization via Geometry-Informed Graph

  • The topological graph links microphone nodes in the same or adjacent rooms, with edge weights encoding distance and barrier-dependent transmission.The weights model non-line-of-sight propagation and signal leakage across room boundaries.Source: §3.1 · p. 2

3.2. Spatial-Temporal Graph for Distributed Fusion

  • A shared AST encoder extracts each node’s acoustic embedding, which is combined with microphone-coordinate encoding to initialize graph states.Source: §3.2 · p. 3
  • Spatial message passing aggregates neighboring node features using geometry-weighted edges, then a GRU models temporal continuity across sounds and movement.The resulting spatio-temporal embeddings feed semantic decoding.Source: §3.2 · p. 3

3.3. Semantic Tokenization via Dynamic Acoustic Scene Graph

  • A decoder converts the spatio-temporal features into time-indexed subject–relation–object triplets, with hierarchical prediction and geometry constraining valid locations.Source: §3.3 · p. 3

3.4. Narrative Synthesis via LLM

  • The frozen LLM receives a floor-plan description and chronological triplet log, then uses room adjacency constraints to synthesize a continuous narrative that can fill in partially occluded transitions.The paper’s example infers a transition between vacuuming in a meeting room and footsteps in a corridor.Source: §3.4 · pp. 3–4

4. Experiment

In controlled multi-room simulation, distributed sensing and geometry-aware reasoning improve acoustic event detection and narrative spatial consistency. The ablations show that explicit geometry and graph-based fusion contribute to those outcomes.

4.1. Experimental Setup

  • The custom pyroomacoustics-based simulator tests 2–4-room layouts with six distributed microphones, RT60 of 0.2–0.6 seconds, and 20–40 dB SNR.Events use speech and environmental sounds; trajectories follow room-constrained random waypoints.Source: §4.1 · p. 4
  • Evaluation uses triplet F1 for event detection, BLEU-4, ROUGE-L and BERTScore for language generation, and SCS for narrative spatial consistency.SCS treats identical or adjacent consecutive locations as valid transitions.Source: §4.1 · p. 4

4.2. Comparative Evaluation of System Variants

  • 0.87 Triplet F1-score for the proposed framework exceeds 0.51 for Centralized-Single and 0.74 for Distributed Acoustic-Only.The comparison evaluates sensing paradigms and system variants.Source: Table 1 · p. 4, §4.2 · p. 4

4.4. Ablation Study on Architectural Components

  • 66.5% SCS without the geometry prior and 0.81 Triplet F1 without graph message passing mark the largest reported ablation losses.The geometry-prior variant removes explicit Ω; without graph message passing, microphones work in isolation.Source: Table 2 · p. 4, §4.4 · p. 4

4.3. Geometry-Informed Reasoning Analysis

  • 88.2% SCS accompanies BLEU-4 of 0.55 and ROUGE-L of 0.62, while geometry-informed reasoning reduces physically impossible narrative transitions.The reported linguistic metrics exceed the Template-Based approach; the evaluation is simulated.Source: §4.3 · p. 4

5. Conclusion

The conclusion presents a distributed, graph-fused framework with geometry-informed LLM reasoning for multi-room acoustic scenes. Controlled simulation results are promising, while real-world validation and faster inference remain future work.

5. Conclusion

  • The framework combines distributed microphones and graph-based fusion with geometry-informed LLM reasoning to reduce occlusion and non-physical narrative transitions.The reported evidence is from controlled simulation in complex indoor environments.Source: §5 · p. 4

A. Limitations and Future Work

The evaluation and framework have simulation, scene-complexity, uncertainty, and topology boundaries.

A. Limitations and Future Work

  • Evaluation uses known floor plans, microphone positions, and door states; real-world buildings add device mismatch, imperfect synchronization, changing furniture, and non-stationary noise.Validation in multi-room physical testbeds is future work.Source: A. Limitations and Future Work · p. 6
  • Generated scenes omit dense overlapping sources, and incorrect or uncertain triplets may yield confident but incorrect narratives.Future systems should propagate uncertainty and distinguish detected events from inferred transitions.Source: A. Limitations and Future Work · p. 6
  • The framework assumes known, static Ω; dynamic doors or temporary partitions require adaptive topology and propagation-prior updates.Source: A. Limitations and Future Work · p. 6

B. Details of the Scene Logic Decoder Architecture

The scene logic decoder turns spatio-temporal graph states into discrete event triplets using query-based detection and hierarchical classification. The LLM prompt uses predicted acoustic observations and environmental geometry at inference.

B. Details of the Scene Logic Decoder Architecture

C. LLM Prompt Construction

  • At inference, the LLM conditions on predicted acoustic observations and environmental geometry; simulator logs and reference narratives are excluded from its prompt.Those logs and narratives are used for training supervision and evaluation, respectively.Source: C. LLM Prompt Construction · p. 6

D. Extended Experimental Configurations

The extended simulator models multi-room layouts, moving sound sources, material-dependent propagation, and distributed microphone types. Its structured outputs provide geometry and event information, while Table 3 illustrates the geometry-informed reasoning prompt.

D. Extended Experimental Configurations

  • The simulator adapts pyroomacoustics for multi-room distributed acoustic sensing.Floor plans use JSON room dimensions, shapes, portal connections, and material properties.Source: §D.1 · p. 6
  • Dynamic sources follow room-constrained trajectories, with propagation modeled by the Image Source Method and frequency-dependent wall absorption.RT60 values are sampled from 0.2–0.6 s, and heterogeneous microphones can be distributed across rooms.Source: §D.1 · p. 6
  • The simulator exports structured event annotations containing subjects, spatial relations, and room locations.The three-room example specifies living-room, corridor, and kitchen dimensions, portal states, and absorption coefficients tied to target RT60 values.Source: §D.1 · p. 6
  • Table 3 illustrates a geometry-informed prompt that gives the LLM environmental geometry and predicted triplets, excluding reference data.Source: Table 3 · p. 7

D.2. Dataset Synthesis and Pre-processing

The dataset pipeline resamples and mixes moving acoustic events, with event logs supervising triplets and reference narratives held out from the LLM prompt. The appendix also defines evaluation measures and reports feature, training, and deployment settings.

D.2. Dataset Synthesis and Pre-processing

  • Environmental clips are downsampled from 44.1 kHz to 16 kHz, and each scene mixes K events convolved with trajectory-dependent room impulse responses across N microphones.Source: §D.2 · p. 7
  • Event logs provide triplet-level supervision, while reference narratives are excluded from the LLM prompt and used only for language-generation metrics.The example log records people walking, preparing food, opening a door, and talking across rooms over ten seconds.Source: §D.2 · p. 7

D.3. Formal Definitions of Evaluation Metrics

  • Triplet F1 uses one-second segments and counts a true positive only when subject, relation, and object all match the ground truth exactly.Sequences with fewer than two locations have no transition and are excluded from SCS; if none remain, SCS is undefined and not reported.Source: §D.3 · pp. 7–8
  • BLEU-4 measures n-gram precision with a brevity penalty, while ROUGE-L evaluates the longest common subsequence between candidate and reference narratives.BLEU uses uniform n-gram weights; ROUGE-L uses LCS-based precision and recall.Source: §D.3 · p. 7
  • BERTScore compares contextual token embeddings using cosine similarity and greedy matching.The measure is described as capturing semantic similarity, unlike n-gram-based metrics.Source: §D.3 · p. 7

D.4. Hyperparameters and Deployment

  • Audio features use 16 kHz input, STFT hop length 128, and 256 frames; training uses Adam for 80 epochs with batch size 32.The framework can process overlapping temporal windows and decompose larger floor plans into acoustically connected subgraphs, but the controlled study uses 2–4 rooms and 6 microphones.Source: §D.4 · p. 8

D.5. Experimental Objectives and Baseline Design

The experimental design compares sensing paradigms, topology-aware fusion, and reasoning-based output, then isolates geometry-prior and graph-fusion contributions. The structured JSON scene format stores the geometry, sensors, portal states, and event metadata used for simulation and supervision.

D.5. Experimental Objectives and Baseline Design

  • The main comparisons test centralized versus distributed sensing, topology-aware fusion, and template verbalization versus geometry-informed reasoning.Distributed Acoustic-Only retains distributed microphones but removes geometry-aware graph construction and geometry-conditioned prompting.Source: §D.5 · p. 8
  • Ablations remove either the explicit geometry prior or ST-Graph message passing while retaining the rest of the proposed pipeline.The geometry-prior ablation isolates its contribution to spatial plausibility; the graph-fusion ablation tests spatio-temporal aggregation.Source: §D.5 · p. 8

E. Structured Data Format

  • Scene JSON stores room geometry, microphone coordinates, portal states, material parameters, and simulator event metadata.A simplified listing is described as a six-microphone scene with physical constraints.Source: E. Structured Data Format · p. 8

F. Qualitative Narrative Comparison

In the example, the full framework preserves most event identities, locations, and temporal order, while the centralized baseline makes several location and event errors. The full output nevertheless omits the barking interval and misstates who enters the living room.

F. Qualitative Narrative Comparison

Loading 2609.08026v1…