Probing Large Audio-Language Models for Compositional Understanding of Sounding Actions

1 Introduction

• Human activities such as meal preparation emerge from temporally distributed, loosely coupled sound cues rather than any single event.The introduction contrasts this abstraction with the challenge of enabling machines to infer high-level actions from sound.Source: §1 · p. 1
• The benchmark probes compositional reasoning across exemplar typicality and distractor similarity using four complementary evaluation protocols.The protocols are closed-set recognition, open-ended prediction, event-grounded chain-of-thought, and hierarchical classification.Source: §1 · p. 2
• Current LALMs show a substantial gap between recognizing atomic sounds and inferring higher-level activities, especially for atypical examples and acoustically overlapping routines.The authors suggest this capability is distinct from what existing training paradigms and benchmarks capture.Source: §1 · p. 2
• The benchmark targets inference from egocentric acoustic event sequences to human actions and activities organized in a hierarchical taxonomy.It is paired with closed-set, open-ended, event-grounded, and hierarchical evaluation protocols.Source: §1 · p. 2

Audio-Language Models (ALMs)

• Audio-language models have progressed from contrastive audio-text learning to LALMs and structured reasoning, but generalization to inferring human activities from sound sequences remains unexamined.The paper positions its work as directly probing this gap.Source: §2 · p. 2

Benchmarks for Audio Understanding and Reasoning

• Prior audio-language benchmarks assess broad auditory perception and reasoning but include few activity examples and do not control typicality or distractor similarity.The benchmark is presented as complementary by isolating these two difficulty factors.Source: §2 · p. 2

Sound Event and Activity Recognition

• Sound-event datasets and tagging systems emphasize isolated event labels, while activity-grounded egocentric corpora have not been used to test acoustic-only compositional inference.Demirel et al. (2025) use audio and motion-sensor fusion for zero-shot activity classification, rather than probing reasoning over acoustic events alone.Source: §2 · pp. 2–3

3.1 Taxonomy of Sounding Actions

• The taxonomy organizes 99 activity nodes across up to five abstraction levels, with strict parent–child relations, after audio-identifiability screening.Activities are retained only when authors agree they are recognizable from sound in principle and sampled clips contain distinctive audible cues.Source: §3.1 · p. 3

3.2 Audio Exemplar Curation

• Exemplars are selected close to or far from each label’s CLAP prototype, while three near or distant distractors form a four-option item.Direct ancestor–descendant labels are excluded as distractors, and selection limits the influence of any one recording.Source: §3.2 · p. 4
• The benchmark uses single-label, 5–10-second segments and CLAP-based central or peripheral exemplars, yielding 1,938 clips across 85 labels.Fourteen taxonomy labels lacked sufficient extracted data; clips from HD-EPIC were not selected, so it contributed only to the taxonomy label space.Source: §3.2 · pp. 3–4

3.3 Benchmark Factorial Structure

• Crossing exemplar typicality with distractor similarity creates four difficulty conditions, from Central–Distant as easiest to Peripheral–Near as hardest.Peripheral–Near combines less typical examples with acoustically similar alternatives.Source: §3.3 · p. 4, Figure 2 · p. 4

4.1 Models

• Approximately 30 audio-language models are grouped into four categories, with thinking and no-thinking variants included where available.The groups are open-source generative, instruction-tuned, reasoning-augmented, and proprietary frontier models.Source: §4.1 · p. 5

4.2 Probe Family I — Flat Compositional Inference

• Flat inference protocols test forced-choice activity prediction, explicit event-grounded prediction, and free-form activity naming without a contrast set.The first two use a closed set; the open-ended protocol requests a noun phrase of up to five words.Source: §4.2 · p. 5

4.3 Probe Family II — Hierarchical Abstraction

• Hierarchical probes test coarse and fine activity prediction jointly in one pass or sequentially, with the second call conditioned on the first coarse prediction.The protocols assess inter-level consistency and whether separating coarse from fine reasoning improves inference.Source: §4.3 · p. 5

4.4 Evaluation Metrics

• Accuracy is the primary metric, with coarse, fine, and joint accuracies reported for hierarchical protocols and semantic equivalence accepted for open-ended predictions.Closed-set predictions are checked against the predefined vocabulary, tolerating minor formatting or typographical deviations.Source: §4.4 · pp. 5–6
• LLM-based judging produces near-identical results to fuzzy string matching on closed-set protocols and shows agreement with human annotators.For P3, judge–annotator agreement is κ = 0.745 versus a human–human baseline of κ = 0.714; closed-set agreement is 90.2% (κ = 0.803).Source: §4.4 · p. 6

5.1 Flat Compositional Inference

• Typicality sensitivity is largely independent of capability: Qwen3-Omni Captioner drops ∆≈−0.34, while Gemini-2.5-Flash-Lite is nearly invariant at ∆≈−0.02.Source: Figure 3 · p. 6, §5.1 · p. 6
• Closed-set P1/P2 rankings agree strongly (τ ≥0.63), but open-ended P3 rankings correlate weakly with closed-set protocols (τ ≤0.27).Source: §5.1 · pp. 6–7, Figure 4 · p. 7
• Joint difficulty compresses category differences: Frontier and Reasoning models reach 0.60 versus Open Source near 0.41 on C-Dist, converging near chance at P-Near.Frontier models partially recover at P-Dist after an initial drop at C-Near.Source: Figure 5 · p. 7, §5.1 · p. 7

Sequential decomposition improves over joint

• Event grounding (P1→P2) generally penalizes accuracy, whereas sequential taxonomy-aligned hierarchy improves 25/26 models on both easy and hard splits.Source: §5.1 · p. 7, Sequential decomposition improves over… · pp. 7–8, Figure 6 · p. 8
• Sequential errors are driven mainly by Call 1: worst models propagate initial mistakes, while top models chiefly benefit from fewer Call 1 errors.Source: Figure 7 · p. 8, Sequential decomposition improves over… · p. 8

5.3 Human Performance

• Humans outperform models on both P1 splits: average accuracy is 0.864 vs. 0.320 on easy and 0.580 vs. 0.275 on hard.Source: Table 2 · p. 9, §5.3 · p. 9
• Humans and models rank difficult activity classes only moderately similarly, with just 1/5 hardest-ranked classes overlapping on each split.Source: §5.3 · p. 9

• The work introduces a benchmark and factorial evaluation framework for compositional auditory understanding of human actions and activities.The framework probes a wide set of large audio-language models.Source: §6 · p. 9
• Under joint difficulty, model families converge near chance, and current LALMs do not reliably infer higher-level activities from atomic acoustic events solely from audio.The conclusion also reports that typicality robustness is largely decoupled from overall capability, event grounding hurts flat inference, and taxonomy-aligned hierarchical decomposition helps.Source: §6 · p. 9
• The factorial evaluation finds typicality robustness largely decoupled from overall capability, while taxonomy-aligned hierarchical decomposition helps and event grounding hurts flat inference.These patterns are reported across a wide set of LALMs.Source: §6 · p. 9
• The authors release data, model outputs, and code to support research on compositional auditory understanding.Source: §6 · p. 9

Limitations and Future Work

• The single-label, fixed-duration segment setting provides a clean foundation but leaves co-occurring, transitioning, and extended activities for future evaluation.Source: Limitations and Future Work · p. 10
• CLAP embeddings define exemplar typicality and distractor similarity, but these factors were not crossvalidated against an independent embedding model or knowledge-grounded CLAP refinement.Human evaluation showed convergent behavioral evidence for the resulting difficulty ordering.Source: Limitations and Future Work · p. 10
• Egocentric sourcing reflects available annotated corpora; broader cultural and linguistic contexts may be explored as datasets mature, and human judgment could further calibrate LLM-based open-ended evaluation.Source: Limitations and Future Work · p. 10
• The authors do not claim their findings generalize to other compositional auditory reasoning domains, including music, speech, environmental soundscapes, or nonegocentric recordings.Source: Limitations and Future Work · p. 10

A.1 Two-Round Audibility Filtering

• Two-pass filtering screens action audibility, advances only categories 3 and 4 to verification, then rates each surviving label using 1–2 minutes across 6–14 distinct source files; labels must survive both passes to enter the taxonomy.In verification, category 1 is discarded because the action is not reliably recoverable from audio, while categories 2–4 are retained.Source: §A.1 · p. 12

A.2 Prompt Bank

• The prompt bank spans direct-classification, instructional, and cloze/multiple-choice styles, with 15 closed-set, 5 hierarchical, and 10 open-ended templates.Source: §A.2 · p. 13

Direct classification

• Direct-classification prompts ask the model to identify the activity and return a single label from the available options.Source: Direct classification · p. 13

Instructional

• Instructional prompts vary the wording while directing the model to choose one option and return only a label.Source: Instructional · p. 13

Cloze/Multiple-choice

• Cloze and multiple-choice prompts frame activity identification as filling a blank or selecting one label from the options.Source: Cloze/Multiple-choice · p. 13

A.3 Full list of evaluated models

• The evaluated models are grouped as open-source generative, instruction-tuned, reasoning-augmented, and proprietary frontier audio-language models.Source: §A.3 · p. 13

A.4 LLM-as-Judge Evaluation

• Meta-Llama-3.1-8B-Instruct judges predictions, using lexical matching for closed-set protocols and semantic matching for the open-ended protocol.Source: §A.4 · p. 13
• For closed-set outputs, “Correct” requires lexical closeness to the ground-truth, allowing small spelling errors; vague, unrelated, or different activities are incorrect.Source: §A.4 · p. 13

Open-set judging prompt (P3)

• For open-ended outputs, “Correct” accepts a valid synonym, paraphrase, or closely matching description, but not a different, vague, or unrelated activity.Source: Open-set judging prompt (P3) · p. 13
• The LLM-as-Judge score is the proportion of items judged Correct for each model, benchmark split, and prompt template.For open-ended evaluation, raw free-form outputs are compared with the canonical taxonomy label.Source: Open-set judging prompt (P3) · p. 13

B Data and Reproducibility

• Source audio from Ego4D and Ego-Exo4D cannot be redistributed because of licensing terms.Researchers with licensed access can obtain that access through free registration.Source: B Data and Reproducibility · p. 14
• Released downstream artifacts include metadata, the taxonomy, prompts, distractors, predictions, LLM-judge outputs, and human validation results.These artifacts enable reported figures and tables to be recomputed without the original media.Source: B Data and Reproducibility · p. 14
• Deterministic benchmark construction code reproduces the benchmark and full evaluation pipeline for researchers with licensed source-dataset access.The code reconstructs the benchmark from released take ID and timestamp pairs.Source: B Data and Reproducibility · p. 14

C Dataset Licenses

• Ego4D and Ego-Exo4D are used under non-commercial research terms, while EPIC-KITCHENS uses CC BY 4.0 and HD-EPIC follows its parent corpus terms.No clips are redistributed; experiments are for non-commercial academic evaluation, and hosted benchmark assets carry equivalent non-commercial research terms.Source: C Dataset Licenses · p. 14

C.1 Use of AI Assistants

• LLM-based assistants were used only for manuscript writing tasks, not to produce results, modify evaluation code, or make scientific claims.The authors state that scientific content, experimental design, analysis, and conclusions are their own.Source: §C.1 · p. 14
• Figure 9 reports mean accuracy across four difficulty splits for P1, P2, and P3, with models sorted by mean P1 accuracy.The protocols are direct forced-choice, event-grounded two-stage, and open-ended LLM-judged, respectively.Source: Figure 9 · p. 15
• Figure 10 reports mean fine-grained accuracy across four difficulty splits for H1 and H2, with models sorted by mean H1 accuracy.H1 uses single-turn joint hierarchical abstraction; H2 uses multi-turn sequential hierarchical abstraction with an audio cue.Source: Figure 10 · p. 15