Source-linked AI summary
AudioICL-Bench: A Benchmark for Large Audio Language Model In-Context Learning
Jia-Hung Chen, Yi-Cheng Lin, Kai-Wei Chang, Ke-Han Lu, Hung-Yi Lee
TL;DR
Audio ICL results may reflect recognition of familiar tasks rather than learning new rules, a distinction that matters when audio problems depend on context-specific conventions. AudioICL-Bench tests episode-specific rule induction across nine tasks and two diagnostic axes; models succeed when perception is modest but struggle with temporal measurement and composing multiple rules.
Problem
Existing audio ICL results leave open whether demonstrations teach a new input-label mapping or merely cue a familiar task, despite practical audio settings where context-specific rules must be inferred from examples.
Source: §I · pp. 1–2Method
AudioICL-Bench evaluates five LALMs on nine tasks with episode-specific rules resampled per instance, organized by what must be learned from context and what must be perceived in audio.
Source: §I · p. 2Results
The best model averages 58.5% across six A2 tasks, no model exceeds 7.1% on A3 temporal tasks, and AudioOperatorHard reaches 10.0% at k=5.
Source: B. Where Does Audio Task Learning… · p. 6, D. Context-Side Complexity Interacts… · p. 6, §IV · p. 7Takeaways & Limitations
The two-axis taxonomy separates failures in extracting perceptual evidence from failures in learning or composing rules from demonstrations.
Source: §I · p. 2
Abstract
from arXiv · showhide
In-context learning (ICL) promises training-free adaptation for audio, where labeling every new condition is costly. Yet existing audio ICL studies largely measure Task Recognition, where demonstrations merely cue pre-trained capabilities, rather than Task Learning, where a genuinely new input-label mapping must be inferred from demonstrations alone. We introduce AudioICL-Bench, a diagnostic benchmark whose per-episode rules are resampled so that no correct answer is recoverable from prior knowledge. Its nine tasks are organized along two axes that separate what must be learned from demonstrations from what must be perceived in the signal, enabling failures to be attributed to either source. Across five Large Audio Language Models, the strongest models readily bind arbitrary sounds to new labels when perception is easy, but collapse on temporal measurement and composing multiple induced rules, revealing two primary capability boundaries: temporal perception and multi-rule composition.
I. INTRODUCTION
Prior audio ICL studies leave open whether demonstrations teach new input-label rules or merely cue pre-trained abilities. AudioICL-Bench is designed to isolate Task Learning and diagnose where it succeeds or fails.
I. INTRODUCTION
- Audio Task Learning requires inferring a new input-label mapping from demonstrations, while audio adds the challenge of perceiving patterns across multiple time scales –.Audio signals lack explicit unit boundaries, and useful patterns can range from sub-second timbral cues to multi-second event structures.Source: §I · pp. 1–2
- Existing audio ICL studies largely evaluate established tasks whose demonstrations help apply pre-trained mappings, leaving new-rule learning unisolated.The cited tasks include ASR, emotion and gender recognition, spoken commands, and language identification [12]–[18].Source: §I · p. 2
- AudioICL-Bench resamples episode-specific rules and organizes nine tasks along context-operation and audio-competency axes to isolate Task Learning.The rules make correct answers recoverable from demonstrations rather than pre-trained priors.Source: §I · p. 2
- Five LALMs bind arbitrary sounds when perception is modest, but fail at temporal measurement and at composing multiple induced rules.These results identify temporal perception and multi-rule composition as capability boundaries.Source: §I · p. 2
A. Design and Taxonomy
AudioICL-Bench uses controlled, mostly artificial tasks and two complementary axes to separate what models must learn from demonstrations from what they must perceive in audio.
A. Design and Taxonomy
- Artificial task designs let the benchmark vary context operations and audio competencies independently, unlike natural tasks where these factors tend to co-vary.The benchmark is primarily artificial, with one natural task; the distinction concerns task design, not recording quality [27]–[29].Source: A. Design and Taxonomy · pp. 2–3
- Episode-specific rules are resampled, making correct predictions dependent on demonstrations rather than pre-trained knowledge.Without demonstrations, performance remains near chance; acoustic signals are decoupled from pre-trained semantic associations.Source: A. Design and Taxonomy · pp. 2–3
- Axis-C distinguishes fast binding, induction, and structured composition: mapping evidence to labels, inferring a relation, and applying rules across multiple steps.These demands concern what must be learned or composed from demonstrations.Source: A. Design and Taxonomy · p. 3
- Axis-A distinguishes speech content, non-speech acoustic patterns, and temporal structure, including duration, onset/offset, and temporal-code perception.The axes are orthogonal and compositional, enabling diagnosis of which combination of context-side demand and audio-side competency accompanies failure.Source: A. Design and Taxonomy · p. 3
B. Artificial Tasks
The artificial tasks probe distinct combinations of rule learning and audio perception, from counting and duration estimation to arithmetic, acoustic remapping, and Morse decoding.
B. Artificial Tasks
- AudioCount and AudioCountHard test counting percussive events, while AudioLength and AudioLengthHard test duration estimation with or without inserted silence.AudioCountHard adds animal-vocalization distractors; AudioLengthHard requires measuring total active-audio duration.Source: B. Artificial Tasks · p. 3
- AudioOperator maps percussive sounds to arithmetic operations, AudioOperatorHard composes multiple induced mappings, and AudioRemap maps concatenated segments to anchor labels.MorseCode additionally requires learning letter-to-signal mappings and temporal conventions to decode novel sequences; single-character chance accuracy is 1/26≈3.8%.Source: B. Artificial Tasks · pp. 3–4
- Table II reports Qwen3-Omni-30B-A3B-Instruct results across demonstration-shot counts, while Table III compares models at k=5 under Specific prompting.Both tables report percentages with 95% bootstrap confidence intervals; Table III includes means over the six A2 and three A3 tasks.Source: Table II · p. 4, Table III · p. 4
C. Natural Task
The natural task tests whether models can distinguish normal from anomalous recordings within one machine type.
C. Natural Task
- Each AnomalyDetect episode uses recordings from a single machine type, with normal or anomalous demonstrations and a query from the same type.The task requires within-domain discrimination because both classes share the same source and spectral profile.Source: C. Natural Task · p. 4
- Models must detect subtle deviations from the demonstrated normal operating pattern.The task’s chance accuracy is 1/2=50%.Source: C. Natural Task · p. 4
- Unlike broad event-identity comparisons, AnomalyDetect tests fine-grained acoustic differences within a common source and operating context.This within-domain setup distinguishes it from the benchmark’s other A2 tasks.Source: C. Natural Task · p. 4
D. Prompting Strategy
The prompting strategy keeps the few-shot audio-label template fixed while varying task guidance, testing how instruction detail affects rule induction. Specific instructions avoid revealing episode-specific mappings, and General and Null prompts provide less textual guidance.
D. Prompting Strategy
- All configurations use a textual instruction field followed by k demonstration slots and one query slot, with each demonstration pairing audio with a text label.Multi-clip demonstrations can contain several tagged audio clips, while MorseCode prepends 26 letter-to-signal references.Source: D. Prompting Strategy · p. 4
- Specific prompting names the task-level objective for five tasks but uses the General instruction for AC, ACH, AL, and ALH to avoid revealing their target operations.For those four tasks, naming the operation would resolve the task without rule induction.Source: D. Prompting Strategy · p. 5
- General prompting gives every task the same instruction to identify hidden patterns and rules in the audio-label pairs and apply them to the query.This tests rule induction without task-specific priors.Source: D. Prompting Strategy · p. 5
- Null prompting leaves the instruction field empty, testing whether the audio-label demonstrations alone trigger in-context learning.Source: D. Prompting Strategy · p. 5
E. Evaluation Protocol
The evaluation uses deterministic decoding and task-specific parsing, then scores predictions by exact match. Zero-shot results provide context for prompt sensitivity, including chance-level interpretation for tasks with finite output sets.
E. Evaluation Protocol
- Models are decoded at temperature 0.0 with a maximum output length of 64 tokens, and predictions are scored by deterministic parsing followed by exact matching.Source: E. Evaluation Protocol · p. 5
- Task-specific extraction parses the last integer for integer-label tasks, normalizes anomaly keywords to a binary label, checks AudioRemap sequence length and uppercase strings, and normalizes MorseCode strings.Whitespace and punctuation are stripped for MorseCode matching.Source: E. Evaluation Protocol · p. 5
- Zero-shot performance is reported to contextualize prompt sensitivity, with finite-output tasks interpreted relative to their chance level.Full prompts, evaluation code, and the dataset are slated for release upon publication.Source: E. Evaluation Protocol · p. 5
III. RESULTS
The results section reports Qwen3 as the strongest overall model and presents shot-scaling behavior under Specific prompting across nine audio tasks. The figure plots exact-match accuracy against demonstration count for five LALMs.
III. RESULTS
- Qwen3 is identified as the strongest overall model among the five evaluated LALMs.Table II reports Qwen3 results across all three prompting configurations.Source: §III · p. 5
- Figure 2 plots exact-match accuracy against the number of demonstrations under Specific prompting across nine audio tasks.Each colored line represents one of five LALMs, shaded bands show 95% bootstrap confidence intervals, and dotted lines mark task-specific chance baselines.Source: Figure 2 · p. 5, §III · p. 5
- The reported shot-scaling curves compare model performance as the number of in-context demonstrations increases.Source: Figure 2 · p. 5
A. Validating the Evaluation Design
The evaluation design checks whether models can solve tasks without demonstrations and whether textual instructions affect learning. Qwen3 is near or below chance on nearly all tasks at k=0, while removing instructions degrades performance on tasks that otherwise show learning.
A. Validating the Evaluation Design
- At k=0, Qwen3 performs near or below chance on nearly all tasks, supporting the design goal that prior knowledge alone should not solve them.AnomalyDetect reaches 49.7% against a 50% chance baseline, and AudioOperator reaches 34.0% against an approximately 33% chance baseline.Source: A. Validating the Evaluation Design · p. 5
- AudioCount falls from 68.3% to 5.0% and AudioOperator from 98.7% to 73.7% under Null prompting, showing degradation when instructions are removed.Source: A. Validating the Evaluation Design · p. 6
- For AC, ACH, AL, and ALH, the Specific prompt is the General prompt by construction because naming each target operation would resolve the task directly.The restricted instruction therefore avoids leaking the rule being tested.Source: A. Validating the Evaluation Design · p. 6
B. Where Does Audio Task Learning Succeed?
Task Learning succeeds on non-speech pattern-recognition tasks but fails on temporal structure, and a single-character MorseCode ablation localizes that failure to temporal perception.
B. Where Does Audio Task Learning Succeed?
- Qwen3 averages 58.5% across six A2 tasks, including 98.7% on AudioOperator and 89.0% on AudioRemap.AudioCount reaches 68.3%, with performance rising from 0.3% at k=0 to 68.3% at k=5.Source: B. Where Does Audio Task Learning… · p. 6
C. Where Does Audio Task Learning Fail?
- No model exceeds a 7.1% A3 average, and the best single-task score is DeSTA’s 14.0% on AudioLengthHard.A3 averages remain near zero across model generations, unlike A2 averages, which rise from 15.6% for Qwen2 to 58.5% for Qwen3.Source: C. Where Does Audio Task Learning Fail? · p. 6
- With the 26-letter Morse dictionary supplied and only one character queried, Qwen3 reaches 11.5% accuracy while two other models remain near the 3.8% random-guess baseline.Removing the compositional demand still leaves poor performance, localizing the bottleneck to distinguishing Morse dots from dashes in the audio representation.Source: C. Where Does Audio Task Learning Fail? · p. 6, Table IV · p. 6
D. Context-Side Complexity Interacts With Audio Competency
Within tasks where audio perception is manageable, performance depends on context-side demands: single-rule induction and application can succeed, but composing multiple induced rules fails. The conclusion also identifies within-domain acoustic discrimination as a remaining difficulty.
D. Context-Side Complexity Interacts With Audio Competency
- Qwen3 reaches 89.0% on AudioRemap and 99.7% by k=3 on AudioOperator, but only 10.0% on AudioOperatorHard, where multiple induced rules must be composed.This contrast points to the joint demand of inducing and composing multiple rules, rather than composition or single-rule induction alone.Source: D. Context-Side Complexity Interacts… · p. 6
- AnomalyDetect remains near the 50% chance baseline despite its minimal context-side demand, with normal and anomalous recordings sharing a source and spectral profile.The authors suggest the acoustic difference between machine states is too subtle for current audio representations to capture from a few demonstrations.Source: D. Context-Side Complexity Interacts… · p. 6
IV. CONCLUSION
- The conclusion identifies temporal structure and compositional reasoning as capabilities current audio encoders do not provide.It argues that advancing audio Task Learning will require representations that explicitly capture temporal structure and support compositional reasoning.Source: §IV · pp. 6–7