Source-linked AI summary
RMS-AQA: A Two-Stage Spatial Audio Question Answering Benchmark for Real-World Domestic Environments
Peihao Chen, Qing Wang, Lichun Fan, Yufeng Hao, Zhifeng Kong, Mengyao Zhu, Hengyi Hong, Hang Chen, Hang Su, Yujie Jian, Chao-Han Huck Yang, Shichao Hu, Jun Du, Jian Luan, Ke Li
TL;DR
Domestic assistants need to interpret what happened and where and when, but existing spatial audio QA benchmarks often do not explicitly link event grounding with later reasoning. RMS-AQA combines a two-stage benchmark with simulated and authentic FOA audio and a spatial plug-in; its evaluations identify concurrent sources, far-field acoustics, and sim-to-real mismatch as persistent challenges.
Problem
Existing SAQA benchmarks often separate event recognition from spatio-temporal reasoning or score only final answers, while authentic domestic multichannel audio is scarce.
Source: §1 · p. 1Method
RMS-AQA pairs FOA clips with two-stage questions, combining measured-RIR synthetic training data and authentic domestic recordings with a spatial plug-in for frozen audio-language backbones.
Source: §1 · pp. 1–2Results
Evaluations identify overlapping sources, far-field acoustics, and the sim-to-real gap as critical bottlenecks in domestic spatial audio reasoning.
Source: §5 · p. 4, §4.2 · p. 4Takeaways & Limitations
The authors identify bridging acoustic mismatch and improving fine-grained spatial localization in complex real-world scenarios as future research priorities.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Embodied assistants in domestic environments must infer what happened, where and when it occurred, and how to respond. To address this, we introduce RMS-AQA, a spatial audio question answering (SAQA) benchmark for real-world domestic environments. The benchmark features a two-stage question-answering (QA) format to comprehensively assess the ability of audio-language models (ALMs) to first ground audible sound events and subsequently perform complex spatio-temporal reasoning based on that grounding. To maximize acoustic realism, our dataset combines authentic real-world first-order Ambisonics (FOA) recordings with high-fidelity synthetic data generated using measured room impulse responses (RIRs). Furthermore, we provide a lightweight spatial plug-in that injects FOA-format data into frozen audio-language backbones. Experimental results reveal that the primary challenges stem from concurrent sources, far distance, and sim-to-real domain gap between RIR-synthesized and authentic recordings.
1. INTRODUCTION
Domestic audio assistants need to reason about grounded events across space and time, but existing benchmarks often separate event recognition from reasoning. RMS-AQA couples these stages and combines realistic simulated and recorded audio with a spatial plug-in for frozen audio-language models.
1. INTRODUCTION
- Existing SAQA benchmarks often isolate event recognition from spatio-temporal reasoning or score only the final answer, obscuring whether errors arise from grounding or inference.Domestic multichannel recordings are scarce, and simplified simulated RIRs may not generalize to actual homes.Source: §1 · p. 1
- RMS-AQA evaluates a two-stage chain in which Stage 1 grounds audible events and Stage 2 reasons about the scene conditioned on that grounding.The protocol is designed to prevent lucky guesses and make errors diagnosable as grounding or reasoning failures.Source: §1 · pp. 1–2
- The 177K-clip dataset uses measured real-room RIRs for scalable synthetic training scenes and authentic domestic recordings for validation and testing.This dual-source design supports evaluation of the sim-to-real domain gap.Source: §1 · pp. 1–2
- A lightweight spatial plug-in injects FOA cues into a frozen audio-language semantic backbone for multichannel adaptation.The design aims to add spatial information without discarding pretrained semantic knowledge.Source: §1 · pp. 1–2
2. DATASET AND BENCHMARK
RMS-AQA combines simulated and authentic domestic FOA audio, pairs clips with structured two-stage questions, and evaluates reasoning across six dimensions. Its primary metric credits Stage 2 only when both grounding and reasoning are correct.
2.1. Data Collection
- Recorded audio comes from 18 domestic rooms, while synthetic scenes use measured FOA RIRs collected across 1,396 rooms.Recorded validation and test sets cover 49 sound classes; the synthetic source pool is filtered to 324 domestic classes.Source: §2.1 · p. 2
- GPT-5.5 generates two-stage questions from structured metadata, Gemini 3.1 paraphrases them, and human reviewers verify validation QA pairs.Training uses simulated clips, validation mixes simulated and recorded clips, and testing uses recorded clips only; room assignments are separated to prevent leakage.Source: §2.1 · p. 2
2.2. Task Protocol
- Stage 1 grounds audible events, then Stage 2 receives that answer with the audio to reason across six balanced dimensions: SC, SL, TD, SR, TR, and AP.The staged design makes it possible to distinguish recognition errors from contextual reasoning errors.Source: §2.2 · p. 2
2.3. Evaluation Metric
- Grounded Stage-2 accuracy awards credit only when both the Stage-1 and Stage-2 answers are correct.Stage-1 and raw Stage-2 accuracy are also reported as complementary metrics.Source: §2.3 · pp. 2–3
3. BASELINE
The baseline adds a trainable FOA spatial branch to frozen semantic audio and language backbones, then fuses the aligned representations for two-stage QA. Training updates the spatial encoder, fusion projection, and LLM LoRA adapters; inference conditions Stage 2 on the predicted Stage-1 answer.
3.1. Spatial-Augmented Architecture
- The audio encoder and LLM remain frozen while a trainable spatial branch processes full FOA audio and supplies directional cues alongside semantic tokens from the W channel.The semantic and spatial paths are temporally aligned.Source: §3.1 · p. 3
- The spatial encoder processes four-channel log-Mel features and three-channel intensity vectors, using a time-preserving ResNet-18 and eight Conformer blocks to produce spatial tokens.The ResNet pools only along frequency before projection to 1024 dimensions.Source: §3.1 · p. 3
- A lightweight fusion module concatenates semantic and spatial tokens, projects them to the LLM dimension, and feeds the fused tokens into the LLM.Zero-initialized projection weights and bias make adaptation start from the semantic-only mapping.Source: §3.1 · p. 3
3.2. Training Details
- Training updates the spatial encoder, fusion projection, and LLM LoRA adapters, with Stage 2 conditioned on the reference Stage-1 answer.Both stages use autoregressive next-token supervision and a joint loss.Source: §3.2 · p. 3
- The baseline uses three training epochs with AdamW and rank-16 LoRA across all 28 LLM layers.Training uses a learning rate of 2 × 10^-4 and weight decay of 10^-2.Source: §3.2 · p. 3
- At inference, the predicted Stage-1 answer is inserted into the Stage-2 prompt, and the checkpoint with the highest validation AGS2 is selected.Stage 1 is decoded greedily for up to 48 new tokens.Source: §3.2 · p. 3
4. RESULTS AND ANALYSIS
Across validation, spatial augmentation improves grounded Stage-2 accuracy for all four adapted backbones, with adapted Qwen3-Omni achieving the strongest overall result. The analyses also reveal remaining challenges in grounding, spatial categories, scene complexity, and transfer from simulated to recorded audio.
4.1. Evaluated Systems
- The evaluation compares spatial-augmented AF3, AF-Next, MiDashengLM, and Qwen3-Omni against monaural zero-shot counterparts and closed-source zero-shot baselines.Qwen3.8-Omni-Flash is noted as a closed-source baseline with native FOA input.Source: §4.1 · p. 3
4.2. Results and Analysis
- 48.93% grounded Stage-2 accuracy makes adapted Qwen3-Omni the top system, +32.21% over its monaural zero-shot counterpart and +12.55% over Qwen3.8-Omni-Flash.Across the four adapted backbones, grounded Stage-2 gains over monaural zero-shot counterparts range from 23.32% to 35.28%.Source: §4.2 · p. 3, Table 2 · p. 4
- 52.60% (AF3) and 63.45% (Qwen3-Omni) raw Stage-2 accuracy under predicted context beat context-free reasoning by +8.48% and +5.45%, respectively.Oracle context raises accuracy to 60.05% (+15.93%) and 70.40% (+12.40%); the predicted-to-reference gap is about 7%.Source: Table 3 · p. 4, §4.2 · p. 4
- 31.40% on SL and 38.30% on SR are the strongest system’s lowest grounded category accuracies; AP reaches 63.45% raw Stage-2 but 48.93% grounded accuracy.Spatial augmentation yields its largest gains in SC and AP; the 14.52% AP gap separates raw Stage-2 from grounded accuracy.Source: §4.2 · p. 4
- 23.67% performance drop separates recorded from simulated scenes for adapted Qwen3-Omni; increasing concurrent sources and real far-field sources (≥4m) further reduce accuracy.The text attributes the real-recording decline to acoustic mismatch, complex reverberation, and weak direct sounds.Source: Figure 3 · p. 4, §4.2 · p. 4
5. CONCLUSION AND LIMITATIONS
RMS-AQA’s two-stage protocol separates perception failures from reasoning deficits, while its evaluations identify persistent acoustic bottlenecks in domestic environments.
5. CONCLUSION AND LIMITATIONS
- The two-stage evaluation protocol isolates perception failures from reasoning deficits and prevents score inflation from spurious guesses.It also provides diagnostic transparency about where errors arise.Source: §5 · p. 4
- A lightweight spatial plug-in injects multichannel cues into frozen audio-language backbones and drives significant performance improvements.Source: §5 · p. 4
- Overlapping sources, far-field acoustics, and the sim-to-real domain gap remain critical bottlenecks across simulated and authentic recordings.Source: §5 · p. 4
- Future work aims to bridge the acoustic mismatch and improve fine-grained spatial localization in complex real-world scenarios.Source: §5 · p. 4