From Sounds to Scenes: A Benchmark for Evaluating Context-Aware Auditory Scene Understanding in Large Audio Language Models

1 Introduction

• CASU treats speech, events, and background as interacting layers whose relationships can confirm, contradict, or contextualize one another.Non-speech sounds may be irrelevant noise in one context and decisive logical constraints in another.Source: Figure 1 · p. 1, §1 · p. 2
• The benchmark uses a scalable semi-synthetic pipeline combining real-world scene recordings and events with semantically controlled synthetic speech.The pipeline enables precise control over cross-layer relationships for evaluation.Source: §1 · p. 2
• Four tasks probe scene-level reasoning: contextual reasoning, entity extraction, role inference, and counterfactual reasoning.The benchmark is designed to test reasoning about auditory scenes rather than reliance on isolated or incidental cues.Source: §1 · p. 2
• Models strong on single-layer perception can still fail at scene understanding, while removing speech, events, or background leads to collapse on CASU tasks.The results point to holistic integration of all auditory dimensions as necessary for scene understanding.Source: §1 · p. 2
• The paper presents CASU as a shift from isolated perception toward reasoning over structured interactions among acoustic layers.Its evaluation of state-of-the-art LALMs finds persistent limitations in multi-layer auditory understanding.Source: §1 · p. 2

2.1 Dataset Construction

• Natural recordings often lack temporal annotations and semantic relationships across acoustic layers, motivating a controlled, three-stage pipeline for evaluating context-aware reasoning.The stages are Script Generation, Layer Retrieval & Synthesis, and Time-Aligned Composition.Source: §2.1 · p. 2
• GPT-4o-generated structured JSON scripts specify background, speech, and acoustic-event layers, with descriptions and timestamps that encode narrative relationships.Human curation filters scripts for logical consistency and narrative realism.Source: §2.1 · pp. 2–3
• Speech is synthesized while background and event audio are retrieved, then aligned over the background base layer at scripted intervals and human-checked for balanced levels and intended causal logic.The speech and event layers are active only during their designated intervals.Source: §2.1 · pp. 3–4
• Sound retrieval ranks candidates using weighted semantic similarities between target and candidate descriptions and keyword tags, selecting the highest-scoring sample.The weighting parameter α balances semantic intent against keyword matching.Source: §2.1 · p. 3

2.2 Tasks and Questions

• Contextual Reasoning tests reconciling speech with scene cues, while Entity Extraction asks models to infer an entity that explains an observation or functional relationship.Background conditions or events may supplement or contradict spoken assertions.Source: §2.2 · pp. 4–5
• Role Inference tests social-role and relationship judgments from speech and scene context, while Counterfactual Reasoning tests how interpretations change under hypothetical acoustic evidence.The tasks target social inference and reasoning about dependencies between environmental cues and meaning.Source: §2.2 · p. 5
• An agent-based question-generation process uses scene scripts and descriptions to propose task-aligned questions, retains questions with critic-agent consensus, and applies human filtering.Human review checks logical validity, difficulty, and category suitability.Source: §2.2 · p. 5

3 Experiments

• The benchmark compares Qwen, Audio Flamingo, and other open-source LALMs alongside GPT-4o Audio and Gemini 2.0 Flash.The listed open-source systems include models trained mainly on speech data and models trained on diverse sound data.Source: §3 · p. 5
• Cascaded systems pass audio descriptions from a Qwen3-Omni or GPT-4o captioner to a GPT-4o or Qwen3-Instruct reasoner.The Qwen3-Omni captioner is described as having the same parameter count as Qwen3-Omni-Instruct but supporting only audio description.Source: §3 · p. 5
• Scene description is evaluated with BLEU-4 and BERTScore, layer recognition with judged event percentages and speech WER, and the four CASU tasks with multiple-choice accuracy.The CASU questions each have four answer options.Source: §3 · p. 5

4.1 Main Results

• Speech-centric models can reach 1 −WER ≥0.95 yet lag state-of-the-art models by 15% on scene understanding, while top models reach only 70–74% accuracy.The results describe accurate single-layer recognition as insufficient for resolving inter-layer reasoning.Source: §4.1 · pp. 5–6, Table 1 · p. 6
• Omni-modal models consistently outperform comparable cascaded pipelines, particularly on Counterfactual and Contextual Reasoning, where textual descriptions can discard subtle acoustic cues.The passage gives reverberation suggesting a large hall as an example of a cue that can be lost.Source: §4.1 · p. 6
• Entity Extraction relies more on non-speech evidence, while Role Inference is linguistically heavy but contextually grounded; Contextual and Counterfactual Reasoning are most demanding.Models with weak event perception or transcription collapse on the latter two tasks.Source: §4.1 · p. 6

4.2 Ablation Study on Audio Layers

• Removing speech reduces performance to near-random levels, while removing events and background also degrades results; adding events to speech raises Entity Extraction from 53.41% to 60.46%.Removing the acoustic scene particularly reduces Counterfactual Reasoning by 10.2% and Entity Extraction by 7.7%.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 Scene Difficulty on Speech

• Inter-person conversations degrade performance across all four tasks compared with monologues, with Role Inference accuracy dropping by approximately 12%.Source: Figure 4 · p. 7, §4.3 · p. 7

4.4 Influence of Scene Hints

• Scene hints improve Qwen2.5-Omni across all tasks, with the largest gain in Role Inference (+3.82%); GPT-4o Audio benefits less.The authors suggest stronger models may already infer scene context from audio.Source: Table 3 · p. 7, §4.4 · p. 7

4.5 Error Analysis

• The error analysis distinguishes perception or reasoning errors from answers based on question information or internal knowledge rather than audio cues.Source: Table 4 · p. 8, §4.5 · p. 8

Related Work

• Early audio benchmarks targeted speech, sound events, or music, while later suites unified datasets or expanded reasoning and instruction-tuned evaluation.Examples include OpenASQA, CompA, and MusicBench for specialized verticals, and AIR-Bench, AudioBench, MMAU, MMAU-Pro, SAKURA, and Dynamic-SUPERB for broader evaluation.Source: §5 · p. 8
• Existing LALMs have advanced toward unified omni-modal architectures, but are often trained and evaluated on loosely coupled streams that treat non-speech sounds as secondary.Source: §5 · p. 8
• Prior benchmarks rarely test whether models discern when background audio acts as noise or as a semantic anchor that changes speech interpretation.The paper frames CASU as formalizing scene-level interaction and reasoning.Source: §5 · p. 8

Conclusion

• CASU reframes auditory evaluation as holistic reasoning over interacting speech, event, and background layers rather than isolated perception.Source: §6 · p. 8
• The findings emphasize that models must discern the semantic utility of non-speech sounds instead of treating them as mere noise.Source: §6 · p. 8
• The authors present CASU as a stepping stone for future LALMs to bridge signal recognition and understanding real-world auditory scenes.Source: §6 · p. 8

• The benchmark may remain synthetic compared with fully human-labeled and curated datasets, despite its semi-synthetic pipeline and human-in-the-loop filtering.The pipeline is modular and configurable for adaptation to other domains and settings.Source: §7 · p. 9
• Audio scenes are under 30 seconds because current LALMs have limited audio-processing capacity, and some events may occur sooner after speech than in real-world scenes.The pipeline can generate longer audio if future LALMs can process it.Source: §7 · p. 9
• Current audio is limited to one-person monologues and two-person conversations, leaving N-person conversational dynamics for future work.The authors cite overlapping speech and current LALMs’ audio-processing limits as constraints.Source: §7 · p. 9