PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Large Audio Language Models Understand Audio Well? Speech, Scene and Events Understanding Benchmark for LALMs

Han Yin, Jung-Woo Choi

arXiv:2509.13148v3cs.SD

TL;DR

Existing benchmarks often separate speech, scene, and event understanding and do not account for changing energy levels between speech and background sound. SSEU-Bench evaluates these elements together, and the authors report that CoT-guided inference enhances joint audio understanding, while some LALMs show degraded performance on certain tasks.

  • Problem

    Existing benchmarks rarely evaluate speech, scene, and events jointly in one clip or test understanding across varying speech-to-background energy conditions.

    Source: §1 · p. 1

  • Method

    SSEU-Bench mixes foreground speech with background sound at varying SNRs for ASR, ASC, and AT, and uses sequential CoT reasoning for joint interpretation.

    Source: §1 · p. 1, §2.1 · p. 2

  • Results

    LALMs show promise on single-granularity tasks but remain significantly behind CLAP-based models, some degrade on joint tasks, and CoT can enhance joint performance.

    Source: §5 · p. 4

  • Takeaways & Limitations

    Unified LALM perception of speech, acoustic scenes, and sound events remains an open challenge.

    Source: §5 · p. 4

    Because real SNR distributions by scene are unavailable, the study uniformly assigns SNRs from –10 dB to 10 dB for each scene.

    Source: §3 · p. 3

Abstract

from arXiv · show

Recently, Large Audio Language Models (LALMs) have progressed rapidly, demonstrating their strong efficacy in universal audio understanding through cross-modal integration. To evaluate LALMs' audio understanding performance, researchers have proposed different benchmarks. However, key aspects for real-world interactions are underexplored in existing benchmarks, i.e., audio signals typically contain both speech and non-speech components, and energy levels of these components can vary significantly across different scenarios. Moreover, most benchmarks do not consider the joint understanding of speech, scene, and events within the same audio clip. In this work, we introduce SSEU-Bench, the first versatile audio understanding benchmark that explicitly accounts for energy differences between speech and non-speech audio, with both independent and joint understanding settings for speech, scene, and events. Furthermore, we demonstrate that some LALMs tend to underperform on certain tasks in a joint understanding setting. To address this issue, we introduce Chain-of-Thought, which effectively improves LALMs' joint audio understanding performance by decomposing complex tasks into simpler reasoning steps.

1. INTRODUCTION

Existing LALM benchmarks have broadened their task coverage, but often separate speech, scenes, and events and do not reflect varying speech-to-background energy. SSEU-Bench evaluates these elements together, with a Chain-of-Thought approach for joint interpretation.

1. INTRODUCTION

  • Prior benchmarks expanded from non-speech-focused OpenAQA and speech-augmented Open-ASQA toward broader audio-reasoning evaluations [7].Source: §1 · p. 1
  • Existing benchmarks rarely test speech and background events together across varying signal-to-noise ratios, or jointly analyze speech, scene, and events in one clip.The paper frames these as underexplored aspects of real-world human-machine interactions.Source: §1 · p. 1
  • SSEU-Bench mixes foreground speech and background events at different SNRs and evaluates ASR, ASC, and AT, while its CoT approach decomposes joint audio reasoning into sequential steps.The benchmark overview is shown in Fig. 1.Source: §1 · p. 1, Figure 1 · p. 2

2. SSEU-BENCH

SSEU-Bench mixes clean speech with background sound at specified SNRs and evaluates speech, scene, and sound-event understanding independently or jointly. Its joint-inference method guides models through staged reasoning before a final consistency check.

2.1. Audio Mixture Creation

  • The benchmark contains 21.72 hours of audio, including 2.71 hours of clean VCTK speech from 104 speakers and non-speech clips from DESED and MAESTRO-Real, across seven SNR conditions.After overlapping event categories were removed, 18 distinct sound-event classes remained.Source: §2.1 · p. 2

2.2. Audio Understanding Evaluation

  • ASR tests transcription robustness under varying levels of non-speech background interference.The model receives audio and a text prompt and must transcribe the spoken content.Source: §2.2 · p. 2
  • ASC estimates confidence for each of six target scenes by comparing the predicted scene’s text embedding with scene-class embeddings.The evaluation uses ChatGPT-Text-Embedding-3-Large3 and cosine similarity.Source: §2.2 · p. 2
  • AT evaluates fine-grained event understanding by scoring predicted non-speech events against 18 target event classes.The method uses a text embedding model to compute event-class confidence scores.Source: §2.2 · pp. 2–3
  • Joint understanding requires predictions for ASR, ASC, and AT, whereas independent understanding focuses on one task at a time.ASR transcribes speech; ASC identifies the acoustic scene; AT identifies non-speech events.Source: §2.2 · pp. 2–3

2.3. Chain-of-Thought-Guided Inference

  • The CoT-guided joint-inference method moves from energy and speech-onset analysis through transcription and scene/event candidates to cross-validation and correction.The final step selects one scene and events consistent with one another.Source: §2.3 · p. 3

3. EXPERIMENTAL SETUPS

The experimental setup evaluates four open-source LALMs alongside CLAP-based classification baselines, using task-specific metrics and uniformly assigned SNR conditions.

3. EXPERIMENTAL SETUPS

  • Four 7B-backbone open-source LALMs are evaluated, with means and standard deviations reported across three runs.The models are LTU-AS, Qwen2-Audio-Instruct, Kimi-Audio, and Step-Audio 2 Mini.Source: §3 · p. 3
  • CLAP, LAION-CLAP, and MGA-CLAP provide ASC and AT baselines, while WER, mACC, and mAP measure ASR, ASC, and AT, respectively.Ground-truth predictions are also used to compute upper-bound metrics for all three tasks.Source: §3 · p. 3
  • Because public real-scene SNR distributions are unavailable, the setup assigns SNRs uniformly from –10 dB to 10 dB for each scene.Source: §3 · p. 3

4. RESULTS AND DISCUSSIONS

SSEU-Bench results show that independent-task performance varies across models, while joint understanding can degrade task performance. CoT guidance improves joint understanding for the evaluated models, though Qwen2-Audio-Instruct was excluded from that CoT analysis.

  • 8.78% average WER is Kimi-Audio's lowest result; Step-Audio 2 Mini leads ASC at 38.2% average mACC and AT at 38.11% average mAP.Source: §4 · p. 3, Table 1 · p. 4
  • CLAP-based models outperform LALMs on scene and event understanding, while Step-Audio 2 Mini has the strongest LALM results on both tasks.Source: §4 · p. 3, Table 1 · p. 4
  • Joint understanding sharply reduces Qwen2-Audio-Instruct's performance; Kimi-Audio and Step-Audio 2 Mini tend to focus on some tasks at the expense of others.Source: §4 · p. 3, Table 2 · p. 4
  • Qwen2-Audio-Instruct is excluded from the CoT analysis because it frequently fails to produce expected outputs under CoT guidance.Source: §4 · p. 4
  • For Kimi-Audio, CoT keeps ASR relatively stable while improving ASC and AT under selected SNR conditions through scene-event correlation.Source: §4 · p. 4
  • CoT improves overall joint understanding; for Step-Audio 2 Mini, average AT mAP rises approximately 4% across all SNR conditions.Source: Figure 3 · p. 4, §4 · p. 4

5. CONCLUSIONS

The paper presents SSEU-Bench for speech and non-speech audio with energy differences, and reports that joint task performance can degrade for some LALMs. CoT guidance can enhance joint understanding, while unified perception remains an open challenge.

  • SSEU-Bench accounts for speech and non-speech audio energy differences to reflect realistic interactive scenarios.Source: §5 · p. 4
  • LALMs show promising single-granularity audio understanding but remain significantly behind CLAP-based models.Source: §5 · p. 4
  • Some LALMs degrade on certain tasks during joint understanding, while CoT-guided inference can enhance joint performance.Source: §5 · p. 4
Loading 2509.13148v3…