PaperlayerView exact revision on arXiv

Source-linked AI summary

When Capabilities Fail to Compose: Diagnosing the Compositionality Gap in Large Audio-Language Models

Chien-Feng Liu, Chih-Kai Yang, Bo-Han Feng, Yu-Hsuan Li Liang, Hung-yi Lee, Cheng-Fu Chou

arXiv:2609.36921v1cs.SD

TL;DR

Whether LALMs can reliably combine their strong individual audio capabilities remains underexplored. The paper evaluates composition by separating component recognition, cue-conditioned segment selection, and downstream execution. Across models and conditions, compositional performance degrades substantially relative to atomic performance, with the degradation varying by cue and model.

  • Problem

    Although LALMs show strong capabilities on individual audio tasks, evidence on how those capabilities change under controlled composition remains limited.

    Source: §1 · p. 1

  • Method

    The study evaluates two-segment inputs by separately measuring component capabilities, cue-conditioned target-segment selection, and compositional ASR or question answering across acoustic cues.

    Source: §1 · p. 1, §2.1 · p. 2

  • Results

    Compositional performance falls substantially below atomic performance across models, cue types, and downstream tasks, with degradation varying across conditions.

    Source: §4.1 · pp. 2–3

  • Takeaways & Limitations

    The gap does not arise at one common stage, so diagnosing it requires measuring component recognition, segment selection, and downstream execution separately.

    Source: §4.1 · p. 3

Abstract

from arXiv · show

Large audio-language models (LALMs) perform strongly on individual audio tasks, but whether these capabilities can be reliably composed remains underexplored. We conduct a controlled diagnostic study of capability composition in LALMs, requiring models to integrate audio-attribute recognition, cue-conditioned segment selection, and downstream ASR or question answering. We construct two-utterance inputs with distinct acoustic cues to evaluate composition across environmental sound, gender, and emotion cues, with ASR, Math QA, and Factual QA as downstream tasks. Across four open-source LALMs, compositional QA accuracy decreases in 39 of 40 model-task-cue settings, by an average of 26.7 percentage points. ASR exhibits a similarly consistent degradation, with WER increasing in 39 of 40 settings by an average of 28.5 percentage points, while the magnitude of degradation varies across models, cue types, and cue salience. We further probe these failures through output format, positional preference, and chain-of-thought (CoT) analyses. Our study reveals a systematic gap between possessing individual audio capabilities and reliably composing them.

1. INTRODUCTION

Although LALMs perform strongly on individual audio tasks, prior work had limited evidence on how measured capabilities change under controlled composition. This study evaluates that gap by separating component capabilities, cue-conditioned selection, and downstream execution.

1. INTRODUCTION

  • Prior work established compositionality gaps in LLMs and VLMs, but audio studies had limited analysis of how individual capabilities change under controlled composition.Existing work examined multi-hop audio reasoning, joint attribute reasoning, and audio-event composition; ART combined heterogeneous capabilities without explicitly comparing composite and constituent performance.Source: §1 · p. 1
  • Figure 1 illustrates a case where background-sound recognition, atomic math QA, and segment selection succeed individually but fail when composed to answer the cue-matched question.Table 1 is identified as summarizing the task decomposition, with T denoting the downstream task and i* the target segment index.Source: Figure 1 · p. 1, Table 1 · p. 2
  • The study finds substantial atomic-to-compositional degradation across models, cue types, and downstream tasks, and analyzes positional and formatting effects and explicit CoT prompting.Its contributions include a controlled evaluation framework and dataset for studying capability composition in LALMs.Source: §1 · p. 1
  • The evaluation decomposes compositional tasks into component capabilities, cue-conditioned segment selection, and downstream execution across three cue types and three downstream tasks.The cues are environmental sound, gender, and emotion; the downstream tasks are ASR, math QA, and factual QA.Source: §1 · p. 1

2. TASK FORMULATION AND DATA CONSTRUCTION

The task formulation uses two speech segments with distinct acoustic attributes and a query that identifies the target attribute before downstream processing. The dataset combines synthesized speech and question-answering examples with environmental-sound conditions, quality checks, and balanced target placement.

2.1. Task Formulation

  • Each sample concatenates two speech segments with distinct acoustic attributes, and a textual query specifies the target attribute while the other segment acts as a distractor.The model must identify the matching segment before performing ASR, math QA, or factual QA.Source: §2.1 · p. 2
  • Atomic tasks score both segments without attribute-conditioned selection, segment selection predicts the target index, and compositional tasks select the matching segment before executing the downstream task.This decomposition separates component perception, segment binding, and downstream execution.Source: §2.1 · p. 2

2.2. Dataset Construction

  • Speech was synthesized with CosyVoice 3 using CREMA-D reference audio, with six gender-balanced speakers covering angry, sad, and happy emotions.Factual QA questions came from TriviaQA, math QA from Spoken-MQA, and the same utterances were used for ASR with spoken text as reference.Source: §2.2 · p. 2
  • Environmental-sound cues used five ESC-50 classes mixed with speech at 0, 10, or 20 dB SNR, with two segments separated by 1.5 seconds of silence.Target attributes, positions, and distractor classes were balanced, with the target equally often in either segment position.Source: §2.2 · p. 2
  • Math and factual QA used four 600-sample subsets, totaling 4,800 two-segment samples across environmental-sound, gender, and emotion conditions.Samples were automatically checked for emotion consistency, speech quality, and transcription accuracy, then manually inspected for intelligibility, attribute consistency, and label correctness.Source: §2.2 · p. 2
  • Table 2 reports atomic recognition and segment-selection accuracy on math and factual QA subsets, with E0/E10/E20 denoting environmental-sound SNR conditions and Gen. and Emo. denoting gender and emotion.The caption identifies the table's scope and condition labels.Source: Table 2 · p. 2

3. EXPERIMENTAL SETUP

Four open-source LALMs were evaluated zero-shot with task-specific instructions, XML-style output formats, and greedy decoding. The study scores recognition and selection with accuracy, downstream QA with exact-match measures, and ASR with normalized WER, using two parsing approaches.

3. EXPERIMENTAL SETUP

  • The evaluation covered Qwen2.5-Omni-7B, MiniCPM-o 4.5, MiMo-Audio-7B, and Kimi-Audio-7B in zero-shot settings with task-specific instructions and greedy decoding.Models ran on an NVIDIA RTX A6000 using XML-style output formats.Source: §3 · p. 2
  • Atomic recognition and segment selection were scored by accuracy, while math QA used exact match, factual QA normalized exact match, and ASR normalized WER.Atomic segment predictions were scored independently rather than jointly at the sample level.Source: §3 · p. 2
  • Outputs were evaluated with tag-based parsing and GPT-4o-mini LLM-based parsing to assess sensitivity to output-format compliance.Under tag-based parsing, unparseable QA outputs count as incorrect and unparseable ASR outputs as empty hypotheses.Source: §3 · p. 2

4. RESULTS AND ANALYSIS

Across models, cues, and downstream tasks, composition degrades performance even when component capabilities are strong in isolation. The analyses show that degradation varies by cue, model, and stage, and that format and positional effects do not fully explain it.

4.1. Main Results

  • Compositional performance falls substantially below atomic performance across models, cue types, and downstream tasks, with degradation varying by selection behavior, cue salience, and output formatting.Compositional ASR WER can worsen as SNR rises even while atomic WER improves.Source: §4.1 · pp. 2–3, Table 3 · p. 3
  • Cue-conditioned selection does not follow recognition accuracy uniformly: on factual QA, Qwen2.5-Omni has 35.75% emotion-recognition accuracy versus Kimi-Audio’s 49.00%, but selection accuracy of 73.00% versus 61.83%.For environmental sounds, recognition and selection both decrease with increasing SNR in 15 of 16 model–task settings.Source: §4.1 · pp. 2–3

4.2. Format Following

  • Format compliance can deteriorate under composition, but parser sensitivity does not identify the source of score discrepancies and varies substantially by model.MiniCPM-o’s compliance falls from 92.71% on atomic tasks to 54.91% under composition; well-formed tags can still contain answers that strict matching misses.Source: §4.2 · pp. 3–4, Table 4 · p. 4

4.3. Positional Bias

  • All four models’ environmental-sound positional deviations from 50% increase monotonically from E0 to E20, while gender and emotion preferences vary by model.MiniCPM-o shifts from 41.00% first-segment selections at E0 to 29.08% at E20.Source: §4.3 · p. 4, Table 5 · p. 4

5. CAN EXPLICIT REASONING MITIGATE THE COMPOSITIONALITY GAP?

The study tests whether explicit CoT prompting improves compositional QA by spelling out the intended execution order. Its effects vary by task and model, so it does not consistently reduce the compositionality gap.

5. CAN EXPLICIT REASONING MITIGATE THE COMPOSITIONALITY GAP?

  • The CoT prompt makes the identify-cue-matched-segment, recover-question, and answer sequence explicit while retaining the baseline’s tagged final-answer format.Answer quality does not consistently track format compliance: MiniCPM-o’s tag-parsed math accuracy rises 41.67 points alongside a 68.50-point compliance increase, while LLM-parsed accuracy changes little.Source: §5 · p. 4
  • CoT prompting improves factual QA for all four models by 1.90 to 14.63 percentage points, but math QA accuracy decreases for two models and improves for two.Qwen2.5-Omni loses 11.17 percentage points in LLM-parsed math accuracy while format compliance remains unchanged.Source: §5 · p. 4
  • CoT does not consistently reduce the compositionality gap; its effects depend on downstream task and on whether gains arise from answer quality or format compliance.Kimi-Audio’s LLM-parsed math accuracy improves 6.10 points despite a 15.17-point compliance decline.Source: §5 · p. 4

6. CONCLUSION

The study finds a systematic gap between individual audio capabilities and their reliable composition. Failures vary across stages and conditions, so separate stage measurement is needed to diagnose them.

6. CONCLUSION

  • The compositionality gap can arise across atomic recognition, segment selection, and downstream execution, with the stage where degradation concentrates varying by cue type and model.Format compliance also degrades under composition but does not fully explain the observed performance gap.Source: §6 · p. 4
  • Cue-dependent positional tendencies show that selection behavior cannot be explained by recognition accuracy alone.The authors conclude that the gap may limit LALM reliability when complex audio tasks require cue-based content identification before downstream processing.Source: §6 · p. 4

Related papers

Loading 2609.36921v1…