PaperlayerView exact revision on arXiv

Source-linked AI summary

MiDashengLM-Spatial: Unifying General Audio Understanding and Spatial Awareness

Jinbo Hu, Hang Su, Lichun Fan, Heinrich Dinkel, Gang Li, Zhanchen Dai, Yiru Zhang, Chang Liu, Peng Wang, Junnan Wu, Jian Luan, Cong Zou, Heng Qu

arXiv:2610.11156v1cs.SDeess.AS

TL;DR

Monaural LALMs generally lack explicit spatial cues, while spatial audio-language models do not fully leverage their general audio understanding. MiDashengLM-Spatial combines a semantic audio pathway with a spatial encoder and synthesized spatial-language data; the authors report strong spatial performance while retaining general audio capabilities.

  • Problem

    Monaural LALMs often do not model inter-channel cues, while spatial models do not fully use the general audio capabilities of monaural LALMs.

    Source: §2.2 · p. 3

  • Method

    MiDashengLM-Spatial adds Spatial-Dasheng to MiDashengLM through hierarchical semantic-to-spatial conditioning and trains with synthesized spatial scenes, descriptions, and question-answer pairs.

    Source: §1 · p. 2

  • Results

    MiDashengLM-Spatial substantially outperforms existing models on spatial audio understanding while retaining general audio understanding, and Spatial-Dasheng performs strongly on real-world scenes.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The reported results indicate that spatial awareness can be integrated with general audio understanding in one model without compromising the latter.

    Source: §1 · p. 2, §6 · p. 10

Abstract

from arXiv · show

Large audio-language models (LALMs) have achieved strong performance in general audio understanding, yet most are designed for monaural input and discard the inter-channel cues essential for spatial perception. In contrast, existing spatial audio-language models are purpose-built for spatial tasks and fail to capitalize on the general understanding capabilities of monaural LALMs. We present MiDashengLM-Spatial, the first open-source end-to-end unified audio-language model, to our knowledge, which supports both general audio understanding and spatial awareness within a single architecture. It extends MiDashengLM with a spatial audio encoder, Spatial-Dasheng, integrated through a hierarchical semantic-to-spatial conditioning module that injects intermediate semantic representations into the spatial branch at multiple depths while preserving the original semantic pathway. To provide spatial audio-language supervision at scale, we develop a data synthesis pipeline that renders diverse spatial acoustic scenes with scene-level spatial descriptions and question-answer pairs. Experiments show that Spatial-Dasheng achieves strong performance on sound event localization and detection in real-world scenes, and that MiDashengLM-Spatial substantially outperforms existing LALMs on spatial understanding and reasoning benchmarks. Meanwhile, it remains competitive with state-of-the-art 8B-scale LALMs on diverse monaural benchmarks, demonstrating that spatial awareness can be acquired without compromising general audio understanding. The source code and model checkpoint are available at https://github.com/xiaomi-research/midashenglm-spatial and https://huggingface.co/mispeech/midashenglm-spatial.

1 INTRODUCTION

General audio-language models often process monaural input, while spatial models do not fully leverage their general audio understanding; unifying both remains open. MiDashengLM-Spatial combines semantic and spatial audio processing with synthesized spatial-language supervision, and reports strong spatial results while retaining general audio capabilities.

1 INTRODUCTION

  • Monaural LALMs often lack explicit inter-channel modeling, while spatial models do not fully leverage their general audio capabilities; unifying both remains open.Conventional SELD systems also rely on predefined event categories and structured outputs, limiting open-ended descriptions of unseen events and spatial relations.Source: §1 · pp. 1–2
  • Hierarchical semantic-to-spatial conditioning connects MiDashengLM and Spatial-Dasheng at multiple layers while preserving separate semantic and spatial branches.Intermediate semantic embeddings condition corresponding spatial-branch layers, supporting joint modeling of semantic and spatial information.Source: §1 · p. 2
  • MiDashengLM-Spatial substantially outperforms existing models on spatial understanding while retaining general audio understanding; Spatial-Dasheng also performs strongly on real-world scenes.The authors additionally report robust spatial perception and sim-to-real generalization for Spatial-Dasheng.Source: §1 · p. 2
  • Spatial-Dasheng is a spatial audio encoder for frame-wise detection and localization of overlapping sound events.It is designed to capture cues for source direction, distance, and motion.Source: §1 · p. 2
  • A scalable synthesis pipeline creates multi-channel scenes spanning environmental sounds, speech, and music, with spatial descriptions and question-answer pairs.The generated annotations provide supervision for aligning spatial audio representations with language and training spatial understanding and reasoning.Source: §1 · p. 2

2 RELATED WORK

Prior work offers structured sound localization and detection, general audio-language interaction, and dedicated spatial audio-language models, but these approaches leave gaps in open-ended spatial reasoning and retained general audio understanding.

2.1 SOUND EVENT LOCALIZATION AND DETECTION

  • Conventional SELD methods such as ACCDOA and EINV2 support structured spatial modeling, but do not provide an open-ended language interface for arbitrary scenes.ACCDOA combines detection and localization in one representation, while EINV2 uses interconnected specialized branches (Shimada et al. (2021; 2022); Cao et al. (2021); Hu et al. (2022; 2025a)).Source: §2.1 · pp. 2–3

2.2 LARGE AUDIO-LANGUAGE MODELS AND SPATIAL EXTENSIONS

  • Most LALMs are trained on monaural data and may collapse multi-channel audio, discarding inter-channel cues needed for spatial perception.The passage gives channel averaging as one example of this collapse (Chu et al. (2023; 2024); Xu et al. (2025a;b); Ghosh et al. (2026b;a)).Source: §2.2 · p. 3
  • Spatial audio-language models include embedding-based encoder-only systems and encoder-decoder systems with dedicated spatial components; many focus on static or clip-level attributes.More recent work has begun modeling moving sources with frame-level trajectories (Zhu et al. (2026); Hyun-Bin et al. (2026)).Source: §2.2 · p. 3
  • Prior spatial extensions may compromise general understanding through substantial architectural changes and spatial-data fine-tuning, and often use closed-set labels.The paper positions its approach as retaining the base LALM encoder with minimal modification and synthesizing scene-level spatial audio-language data at scale.Source: §2.2 · p. 3

3 DATA PIPELINE

The data pipeline addresses scarce spatial audio-language annotations by synthesizing multi-channel scenes and accompanying language supervision. It renders static and moving sources with precise metadata, then produces scene descriptions and question-answer pairs grounded in that metadata.

3 DATA PIPELINE

  • Spatial audio datasets are scarce and often tied to fixed microphone-array configurations, so the paper constructs spatial scenes with descriptions and question-answer pairs for training.The passage contrasts these resources with abundant but mostly monaural audio collections.Source: §3 · p. 3

3.1 SPATIAL ACOUSTIC SCENE GENERATION

  • The pipeline renders static and moving audio events in simulated rooms using room impulse responses along linear and circular trajectories.Parametric simulation provides source and receiver positions, event timestamps, room dimensions, reverberation time, and frame-level azimuth, elevation, and distance annotations.Source: §3.1 · p. 3

3.2 SCENE-LEVEL SPATIAL DESCRIPTION AND QUESTION-ANSWER PAIR GENERATION

  • Scene descriptions convert spatial metadata into directional and motion language, then use an LLM to combine it with event captions or transcriptions into a coherent narrative.General scenes use spatial audio captions, while speech scenes preserve spoken content and identify speakers spatially.Source: §3.2 · p. 4
  • Metadata-derived facts populate manually designed templates for localization, motion, and group relations, while dialogue-based generation handles spatially grounded content questions.Template answers and distractors are verified against extracted scene facts; dialogue speaker references are mapped to spatial directions and QA pairs are rephrased while preserving verified semantics.Source: §3.2 · pp. 4–5

4 METHOD

MiDashengLM-Spatial combines semantic and spatial audio encoders, linking them through hierarchical conditioning while retaining the semantic pathway. A three-stage curriculum develops spatial perception, language alignment, and joint audio understanding.

4 METHOD

  • MiDashengLM-Spatial adds a 12-layer spatial encoder to the 32-layer semantic encoder, which processes monaural audio while the spatial branch processes multichannel audio.The architecture is built on MiDashengLM.Source: §4 · p. 5

4.1 SPATIAL-DASHENG

  • Spatial-Dasheng uses multi-ACCDOA tracks to detect overlapping events and estimate their directions and distances, with permutation-invariant training addressing target-prediction misalignment.Each track detects at most one event; the loss balances ACCDOA and distance errors across possible track permutations.Source: §4.1 · pp. 5–6

4.2 HSSC: HIERARCHICAL SEMANTIC-TO-SPATIAL CONDITIONING

  • HSSC injects intermediate semantic representations into the spatial encoder at multiple depths, associating spatial cues with event semantics, particularly for overlapping sources.Conditioning is unidirectional: the spatial branch receives semantic features, while spatial features do not feed back into the semantic hierarchy.Source: §4.2 · p. 6

4.3 TRAINING CURRICULUM

  • The three-stage curriculum progresses from spatial perception training, to spatial audio-text alignment, to joint fine-tuning on monaural and binaural data with emphasis on spatial QA.The final stage is designed to balance general audio understanding and spatial awareness.Source: §4.3 · p. 6

5 EXPERIMENTS

Experiments assess Spatial-Dasheng on real-scene SELD tasks and MiDashengLM-Spatial on spatial grounding, spatial benchmarks, and monaural audio tasks. Results show strong spatial performance alongside competitive general audio understanding, with weaker performance on fine-grained motion and complex scenes.

5.1 IMPLEMENTATION DETAILS

  • The synthesis pipeline produces 1 million binaural clips and 6 million spatial QA pairs, with no more than three overlapping sources per clip.Clips are spatialized from captioning and speech-transcription corpora using simulated room impulse responses and head-related transfer functions.Source: §5.1 · p. 7

5.2 SPATIAL-DASHENG PERFORMANCE ON SELD TASKS

  • Spatial-Dasheng substantially outperforms both baselines on EasyCom and MRSSound, and is competitive with PSELDNets on STARSS23 and SynthBinaural-Test.Despite training exclusively on binaural data, it generalizes to stereo STARSS23; the authors suggest it exploits interaural level differences.Source: Table 1 · p. 7, §5.2 · pp. 7–8

5.3 SPATIALLY GROUNDED EVALUATION

  • Spatial QA accuracy reaches 80.6% on OV1 and 66.9% on OV2, versus a 28.8% random-guess baseline, while exact motion-trajectory accuracy remains below 20%.The paper reports stronger performance on coarse spatial attributes and simpler scenes than on fine-grained details and scenes with overlapping sources.Source: §5.3 · p. 8

5.4 EVALUATION ON SPATIAL AND GENERAL AUDIO UNDERSTANDING BENCHMARKS

  • MiDashengLM-Spatial substantially outperforms evaluated models on spatial subsets of MMAU-Pro and STAR-Bench, and its channel-swap ACR substantially exceeds all baselines.The channel-swap evaluation tests whether answers remain correct when left and right audio channels are exchanged.Source: §5.4 · pp. 8–9, Table 2 · p. 9
  • MiDashengLM-Spatial performs comparably to MiDashengLM and competitively with similar-scale state-of-the-art systems across audio QA, ASR, and captioning; unidirectional HSSC outperforms bidirectional HSSC on 13 of 18 settings.It also exceeds its monaural counterpart on several benchmarks, including GigaSpeech2 multilingual ASR and MusicCaps captioning.Source: §5.4 · p. 9

6 CONCLUSION

MiDashengLM-Spatial unifies general audio understanding and spatial awareness in one open-source end-to-end model. Its spatial encoder and synthetic training data support strong spatial performance while retaining general audio capabilities.

6 CONCLUSION

  • MiDashengLM-Spatial achieves superior performance across spatial audio understanding and reasoning benchmarks without compromising general audio understanding.It remains on par with state-of-the-art 8B-scale LALMs on monaural audio question answering, ASR, and audio captioning.Source: §6 · p. 10
  • Spatial-Dasheng is a spatial audio encoder pre-trained with a SELD objective, integrated with MiDashengLM through hierarchical semantic-to-spatial conditioning.The integration uses minimal architectural modifications and preserves a unified model for general audio understanding and spatial awareness.Source: §6 · pp. 9–10
  • A data-synthesis pipeline renders diverse spatial acoustic scenes with scene-level descriptions and question-answer pairs for spatial-understanding training.Source: §6 · p. 10
  • Spatial-Dasheng exhibits strong spatial perception and generalizes from simulated to real-world acoustic scenes.Source: §6 · p. 10

A DATASET DETAILS

This section details how ground-truth spatial metadata is translated into listener-centered scene descriptions and how spatial question-answer pairs are constructed and diversified. It also describes sound and speech pipelines that constrain generated language to preserve verified spatial and temporal facts.

A.1 MAPPING FROM SPATIAL ATTRIBUTES TO TEXTUAL DESCRIPTIONS

  • Azimuth and elevation are mapped to listener-centered direction labels, with elevation terms added outside the horizontal band and moving sources described by trajectory and quantized endpoints.Azimuth uses four cardinal sectors within ±5° of each axis and four diagonal sectors; horizontal sources have |ϕ| ≤5°.Source: §A.1 · p. 15

A.2 SCENE-LEVEL SPATIAL DESCRIPTION

  • For sound scenes, the model turns metadata-derived events into a coherent account of what happens, where, and when, using the overlap field as the authority for temporal relationships.The pipeline computes spatial and temporal facts deterministically from ground-truth metadata; the LLM performs only linguistic realization.Source: §A.2 · pp. 15–16
  • For speech scenes, utterances are replaced with placeholders during rephrasing, which must preserve their order and exact spatial meaning before transcripts are restored.Speakers receive deterministic referents derived from metadata, and verbatim transcripts are never exposed to the LLM.Source: §A.2 · pp. 15, 17
  • Scene descriptions track sound events and speakers by direction, motion, distance, and temporal overlap.The examples combine moving and stationary sounds with overlapping speech attributed to spatially located voices.Source: §A.2 · p. 18

A.3 SPATIAL QUESTION-ANSWER PAIRS

  • Three spatial QA categories are deterministically constructed, with three or four options per item; single-speaker identification questions use participant directions as options, while other questions remain open-ended.An LLM rephrases each question, its options, and its answer; sound-scene and speech-scene QA share templates.Source: §A.3 · pp. 18–19, Table 4 · p. 19

B.1.1 EVALUATION METRICS

The evaluation converts open-ended spatial captions and speech transcripts into structured event descriptions, then scores semantic matching, spatial attributes, and temporal consistency. Captions and transcripts differ only in how semantic correspondence is established.

B.1.1 EVALUATION METRICS

  • Generated descriptions are parsed into temporally ordered JSON events containing semantic content, direction, motion state, and potential motion trajectory.The structured representations support separate and joint evaluation of semantic, spatial, and temporal consistency.Source: §B.1.1 · p. 20
  • Predictions and references are matched one-to-one within each scene; captions use FENSE, while transcripts use WER to establish semantic correspondence.The same spatial-attribute and temporal-order scoring procedures then apply to both tasks.Source: §B.1.1 · pp. 20, 22
  • Recall measures the fraction of reference events with a semantic match, while motion-state accuracy is computed over matched events.Direction accuracy applies to matched static events with correctly predicted motion state; trajectory accuracy applies to matched moving events with correctly predicted motion state.Source: §B.1.1 · pp. 22–23
  • Temporal Order scores pairwise agreement between reference onset order and predicted narration order, allowing onset differences within δ as correct.Overall measures the fraction of reference events that are semantically matched, spatially correct, and temporally concordant.Source: §B.1.1 · pp. 22–23

B.1.2 RESULTS

Spatial audio-text alignment is stronger for coarse spatial cues than fine-grained attributes, and performance generally declines as scene complexity increases. The evaluation compares model predictions with oracle scores derived from ground-truth spatial descriptions.

B.1.2 RESULTS

  • On non-overlapping subsets, Recall > 70%, Motion State > 88%, Static Match > 50%, and Temporal Order > 96%; performance declines with overlap or moving sources.Table 5 reports Oracle | Model values on synthetic spatial captioning and speech-transcription datasets.Source: Table 5 · p. 23, §B.1.2 · p. 23
  • The model captures coarse-grained spatial cues, such as motion states, more effectively than fine-grained attributes such as motion trajectories.The oracle parser is reliable for semantic content, motion states, directions, and temporal order, but slightly less accurate for diverse, compositional trajectories.Source: §B.1.2 · p. 23
  • Overall scores are relatively low because every evaluated attribute must be correct simultaneously.The table's oracle results are upper bounds computed directly from ground-truth spatial descriptions.Source: §B.1.2 · p. 23, Table 5 · p. 23

B.2 ANALYSIS OF DIFFERENCES IN LOCALIZATION ACCURACY

The comparison tests whether integrating Spatial-Dasheng into MiDashengLM preserves localization capability by evaluating both systems on non-overlapping static sources. The integrated model improves the three-axis exact-match localization score.

B.2 ANALYSIS OF DIFFERENCES IN LOCALIZATION ACCURACY

  • 50.8% versus 44.7% three-axis exact-match F-score: MiDashengLM-Spatial improves over Spatial-Dasheng on non-overlapping static sources.Spatial-Dasheng frame-level coordinates are converted to textual directions for comparison with the integrated model's event-level textual directions.Source: Table 6 · p. 23, §B.2 · pp. 23–24
  • Both models perform better on the left-right axis than on the front-back and up-down axes.The authors suggest event-level prediction may consolidate spatial evidence over an event and be less sensitive to inconsistencies among independently estimated frame-level positions.Source: §B.2 · p. 24

B.3 SPATIAL AUDIO QUESTION-ANSWERING EVALUATION

MiDashengLM-Spatial performs above random guessing on all spatial question-answering categories, with lower overall accuracy in scenes containing overlapping sources. Spatial Counting is particularly difficult in those more complex scenes.

B.3 SPATIAL AUDIO QUESTION-ANSWERING EVALUATION

  • 80.6% on OV1 and 66.9% on OV2: MiDashengLM-Spatial exceeds random guessing in every category.Table 7 reports model and random-guess accuracy for seven fine-grained categories.Source: Table 7 · p. 24, §B.3 · p. 24
  • OV1 has no source overlap, while OV2 contains two overlapping sources and more sound events and scene-level context.The authors associate the lower OV2 accuracy with entangled concurrent-source cues and the need to represent more events jointly.Source: §B.3 · p. 24
  • Spatial Counting accuracy drops from 65.2% to 39.5%, the largest decline among the seven categories.Counting requires identifying and spatially distinguishing relevant events throughout the scene before aggregating them into one answer.Source: §B.3 · p. 24
Loading 2610.11156v1…