Source-linked AI summary
Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources
Oh Hyun-Bin, Kazuki Shimada, Yuhta Takida, Kim Sung-Bin, Toshimitsu Uesaka, Takashi Shibuya, Kyeongyoon Lee, Tae-Hyun Oh, Yuki Mitsufuji
TL;DR
Audio-language models often treat clips as global event content, while localization models track source directions but offer limited semantic coverage. This paper introduces ST-AudioQA, a time-resolved FOA encoder, and an LLM-connected model for spatio-temporal audio QA. Experiments report stronger QA performance across perception, grounding, and compositional reasoning than static spatial and localization-oriented baselines.
Problem
Audio-language models often represent clips as global event content, while localization models track source directions over time but have limited semantic coverage for language reasoning.
Source: §1 · p. 1Method
The paper builds ST-AudioQA from FOA scenes with source metadata, trains a time-resolved encoder on event semantics and trajectories, and connects its tokens to an LLM.
Source: §1 · pp. 1–2, §3 · p. 3, §4.2 · p. 6Results
Takeaways & Limitations
The results suggest that spatio-temporal audio-language reasoning benefits from binding what sounds to where it is over time.
Source: §1 · p. 2The main benchmark uses controlled synthetic rendering and does not cover the full range of real acoustic conditions; broader real-world evaluation remains necessary.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Sound events are entities with semantic identities, locations, and trajectories, but current audio-language models usually reason about clips as global event content. Conversely, sound event localization models track source directions over time but offer limited semantic coverage for language reasoning. To address this gap, we introduce ST-AudioQA, a spatio-temporal audio QA dataset and benchmark built from first-order ambisonic (FOA) renderings of static and moving sound sources. Each scene provides source identity, activity, direction, distance, and motion metadata, enabling dense trajectory supervision and questions about what is sounding, where it is, how it moves, and how sources relate. We further propose ST-Audio Encoder, a time-resolved FOA audio encoder that learns event semantics together with source trajectories, and ST-AudioLM, which connects the audio tokens from the encoder to an LLM for spatio-temporal audio QA. Experiments show that this representation improves the semantic-localization tradeoff and yields stronger reasoning performance than static spatial and localization-oriented baselines.
1 Introduction
Audio-language models often represent sound as clip-level content, while localization models track spatial change but offer limited semantic coverage. The paper introduces a benchmark, encoder, and LLM-connected model for reasoning about sound identity and source trajectories together.
1 Introduction
- The benchmark addresses a gap between audio-language models with broad semantics but weak temporal localization and SELD models with limited language-level semantic coverage.Everyday sound events also have location, distance, and trajectories that can affect scene interpretation.Source: §1 · p. 1
- ST-Audio Encoder predicts AudioSet semantics alongside source activity, direction, and distance trajectories from time-resolved FOA audio.ST-AudioLM connects its audio tokens to an LLM for spatio-temporal question answering.Source: §1 · p. 2
- ST-AudioLM outperforms static-feature and localization-oriented baselines across single-source perception, two-source grounding, and compositional motion questions.The results point to the benefit of binding event identity to source location over time.Source: §1 · p. 2
- ST-AudioQA evaluates sound identity, time, spatial state, motion, and source relations for static and moving sources.It provides dense trajectory supervision and structured source metadata for training the encoder and audio LLM.Source: §1 · p. 2
2 Related Work
Prior work connects audio encoders to LLMs and develops spatial audio reasoning, while SELD models estimate source activity and direction over time. The paper distinguishes its work by targeting dynamic source-state reasoning with a finer-grained benchmark and an FOA encoder designed for semantic and trajectory tokens.
2 Related Work
- SELD models estimate source activity and direction over time, but constrained event taxonomies limit their suitability as broad-semantic LLM interfaces.Some extensions incorporate distance estimation and pretrained models.Source: §2 · p. 2
- Audio-language models typically encode global semantic content, while spatial audio-language benchmarks and models largely assume static sources.Dynamic source-state reasoning over time remains underexplored.Source: §2 · pp. 2–3
- ST-AudioQA targets finer-grained dynamic source-state reasoning than concurrent rendered-binaural QA, with an FOA encoder designed for semantic and trajectory tokens.The cited concurrent work uses a BAT-style encoder layer adaptation.Source: §2 · p. 3
3 ST-AudioQA: Spatio-Temporal Audio QA Benchmark
ST-AudioQA treats source trajectories, rather than clip-level positions, as the unit of spatio-temporal supervision. It renders FOA scenes with structured source states and generates question families that progress from single-source perception to compositional reasoning across trajectories.
3 ST-AudioQA: Spatio-Temporal Audio QA Benchmark
- ST-AudioQA formulates spatial audio QA as spatio-temporal grounding, tying each event to its location over time, motion, and direction and distance changes.The source trajectory, rather than a clip-level position, is the supervision unit.Source: §3 · p. 3
3.1 Spatio-Temporal FOA Audio Generation
- SoundSpaces 2.0 and Matterport3D meshes provide FOA room impulse responses for 10-second, 32 kHz clips with controllable trajectories and room-scale reverberation.Listeners and sources are sampled from navigable same-room candidates.Source: §3.1 · p. 3
- Each 10-second clip uses 40 spatial-state bins, with moving-source segments rendered through per-bin FOA responses and crossfaded.Trajectories pass audibility and RIR-stability checks; static sources use a constant position across bins.Source: §3.1 · p. 3
- AudioSet waveforms supply event content, while SoundSpaces supplies spatial propagation and trajectory annotations for one- or two-source scenes.Two-source scenes include static–static, static–dynamic, and dynamic–dynamic configurations.Source: §3.1 · p. 3
3.2 Spatio-Temporal QA Generation
- Question types progress from single-source perception, to two-source grounding, to compositional questions comparing grounded trajectories.The primitive families cover event semantics, movement status, time-anchored location, spatial change, and source grounding.Source: §3.2 · p. 4
4 Method
The method combines a time-resolved FOA encoder with an LLM interface and staged spatio-temporal QA training.
4.1 ST-Audio Encoder: Spatio-Temporal FOA Audio Encoder
- The encoder combines seven-channel FOA features with semantic and clip-level direction and distance tokens to learn event and static spatial summaries.A 3×3 convolutional frontend fuses the channels before Transformer patch embedding.Source: §4.1 · p. 5
- Frequency-averaged Transformer patches form temporal representations, interpolated to 40 bins per 10-second clip for activity, direction, and log-distance prediction.Temporal self-attention contextualizes the bins; static sources are represented as constant trajectories.Source: §4.1 · p. 5
- Three-stage encoder training progresses from semantic prediction to static localization and then dynamic trajectories while retaining semantic recognition through an active head and distillation.Static examples are replayed as constant trajectories during the dynamic stage.Source: §4.1 · p. 5
4.2 ST-AudioLM: Spatio-Temporal Audio Reasoning with LLM
- For QA, the frozen encoder supplies one semantic token and 40 ordered trajectory tokens; a two-layer MLP projects them into the LLM embedding space.The global direction and distance tokens supervise the encoder but are excluded from the LLM input.Source: §4.2 · p. 5, Figure 1 · p. 6
- The staged QA curriculum moves from single-source perception to two-source grounding and compositional relations across event, time, location, and motion.Training samples are cumulative across stages A, B, and C.Source: §4.2 · p. 6, Table 2 · p. 6
5 Experiments
Experiments evaluate whether ST-Audio Encoder combines event semantics with dynamic source tracking and whether its audio tokens support single-source, mixture, and compositional QA. ST-AudioLM improves on static-encoder QA baselines, while movement-conditioned spatial relations remain challenging.
5.1 Experimental Setup
- The encoder setup initializes trajectory training from a static FOA checkpoint and trains on 100K single-source moving clips; QA training freezes the encoder and base LLM.The audio-language setup trains only the MLP connector and LoRA adapters.Source: §5.1 · p. 6
- Baselines include static and dynamic encoders, zero-shot Qwen2-Audio, QA-trained BAT, and FOA encoder-to-LLM comparisons.The FOA static-encoder baseline shares ST-AudioLM’s 41-token interface but lacks trajectory-specific temporal attention, dynamic heads, and trajectory supervision.Source: §5.1 · pp. 6–7
- Encoder evaluation measures event mAP, direction and distance errors, and whole-path tracking; controlled QA families use answer accuracy.Traj. Acc@20° requires every active time bin to be localized within 20°.Source: §5.1 · p. 7
5.2 Experimental Results and Analyses
- ST-Audio Encoder achieves the best semantic mAP and distance error while remaining competitive with PSELDNets on dynamic localization.The localization-oriented baseline has the best DoA error and competitive trajectory accuracy, but substantially lower AudioSet mAP.Source: §5.2 · p. 7
- Zero-shot Qwen2-Audio performs poorly on distance and motion-sensitive spatial questions despite non-trivial event-verification performance.Type A evaluates single-source perception of event identity and spatial state over time.Source: §5.2 · p. 7
- ST-AudioLM improves on the strongest static FOA baseline in semantic recognition and spatial perception, with gains across single-source, mixture-grounding, and compositional QA.The clearest reported gains are on trajectory-sensitive questions and cross-source trajectory relations.Source: §5.2 · p. 8
- In two-source mixtures, ST-AudioLM further improves semantic recognition, source-conditioned localization, and spatial-change grounding over the strong static FOA baseline.The static baseline already performs strongly, indicating that the shared 41-token interface provides a useful basis for mixture QA.Source: §5.2 · p. 8
- ST-AudioLM achieves the best average accuracy on compositional questions, with its clearest gain on cross-source trajectory relations.Movement-conditioned spatial relations remain only modestly above chance for all models.Source: §5.2 · p. 8
6 Conclusion
The paper frames sound events as semantic entities with time-varying direction and distance, and presents a dataset, encoder, and audio-language model for reasoning about them. Experiments report improved QA across single-source perception, two-source grounding, and compositional source-time-space reasoning.
6 Conclusion
- The work introduces ST-AudioQA, ST-Audio Encoder, and ST-AudioLM to model dynamic sound sources as semantic entities with changing direction and distance.The encoder provides time-resolved FOA representations, and ST-AudioLM connects its tokens to an LLM.Source: §6 · p. 8
- ST-Audio Encoder preserves event semantics while achieving strong dynamic localization performance.Source: §6 · p. 8
- Connected to an LLM, its representations improve QA across single-source perception, two-source grounding, and compositional source-time-space reasoning.The conclusion highlights binding what is sounding to where it is over time.Source: §6 · p. 9
7 Limitations
The benchmark primarily uses controlled synthetic rendering and simplifies dynamic scenes, limiting how fully it represents real acoustic conditions and source motion.
7 Limitations
- Controlled synthetic rendering supplies precise trajectory labels and systematic QA generation but does not cover the full range of real acoustic conditions, microphone responses, source behaviors, noise, or outdoor environments.The real-world STARSS23-derived QA evaluation uses light adaptation, and broader real-world evaluation remains necessary.Source: §7 · p. 9
- Dynamic scenes use per-bin stationary FOA RIRs with crossfades and controlled 10-second trajectories, which may underrepresent continuous-motion acoustics.The main setting has one or two sources; denser mixtures, source ambiguity, entrances and exits, pauses, acceleration, and curved paths remain future directions.Source: §7 · p. 9
8 Ethical Considerations
The system may inherit bias from AudioSet labels and the simulated scene distribution; synthetic benchmark performance is not a substitute for real-world robustness.
8 Ethical Considerations
- The system may inherit bias from AudioSet labels and the simulated scene distribution.Speech-related and other audio labels may encode demographic or contextual bias, despite the system not using personally identifying information or aiming to infer speaker identity.Source: §8 · p. 9
- The authors will release generation scripts and metadata filters where possible, and caution that synthetic benchmark performance is not a substitute for real-world robustness.Source: §8 · p. 9