Source-linked AI summary
AuditoryBench++: Can Language Models Understand Auditory Knowledge without Hearing?
Hyunjong Ok, Suho Yoo, Hyeonjun Kim, Jaeho Lee
TL;DR
Language models' auditory commonsense in text-only settings remains underexplored, despite its relevance to reasoning from textual descriptions. AuditoryBench++ evaluates this ability, and AIR-CoT adds auditory imagination during inference; the authors report that AIR-CoT outperforms off-the-shelf and auditory-knowledge-augmented models.
Problem
Method
Results
Takeaways & Limitations
The work provides a foundation for building language models that can imagine auditory information without direct audio input.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Even without directly hearing sounds, humans can effortlessly reason about auditory properties, such as pitch, loudness, or sound-source associations, drawing on auditory commonsense. In contrast, language models often lack this capability, limiting their effectiveness in multimodal interactions. As an initial step to address this gap, we present AuditoryBench++, a comprehensive benchmark for evaluating auditory knowledge and reasoning in text-only settings. The benchmark encompasses tasks that range from basic auditory comparisons to contextually grounded reasoning, enabling fine-grained analysis of how models process and integrate auditory concepts. In addition, we introduce AIR-CoT, a novel auditory imagination reasoning method that generates and integrates auditory information during inference through span detection with special tokens and knowledge injection. Extensive experiments with recent LLMs and Multimodal LLMs demonstrate that AIR-CoT generally outperforms both the off-the-shelf models and those augmented with auditory knowledge. The project page is available at https://auditorybenchpp.github.io.
1. INTRODUCTION
The paper addresses the limited study of auditory commonsense in text-only language models by introducing a benchmark and an auditory-imagination reasoning method. On AuditoryBench++, current LLMs and audio-language models without audio input perform near random on comparison tasks.
1. INTRODUCTION
- Auditory commonsense lets people infer sound-related details from text and communicate without describing every detail.Source: §1 · p. 1
- Current LLMs and LALMs without audio input perform close to random guessing on comparison tasks in AuditoryBench++.Source: §1 · p. 1
- AuditoryBench++ evaluates auditory knowledge across five tasks, including reasoning, extending earlier benchmarks limited to short-answer inference in two categories.Source: §1 · p. 1
- AIR-CoT uses explicit auditory imagination during inference, and the authors report superior performance to cascade approaches such as Imagine to Hear.The model emits an [imagine] token when auditory reasoning is needed, then integrates auditory information before continuing its reasoning.Source: §1 · p. 2
2. RELATED WORK
AuditoryBench++ differs from prior multimodal benchmarks by probing auditory reasoning from text alone. The paper also positions auditory imagination as a complementary reasoning modality to visual imagination.
2. RELATED WORK
- Unlike benchmarks that generally rely on perceptual inputs, AuditoryBench++ probes auditory commonsense from text without audio or image cues.This setting reflects cases where only textual descriptions are available.Source: §2 · p. 2
- Prior visual-imagination methods use visual thought processes as intermediate steps and have improved performance on multi-step reasoning tasks.Source: §2 · p. 2
- The paper explores auditory thoughts as a complementary modality to visual imagination for complex inference.Source: §2 · p. 2
3. AUDITORYBENCH++
AuditoryBench++ covers auditory comparisons, animal-sound recognition, and contextual reasoning, using a construction pipeline that combines existing resources, filtering, and verification.
Task Definition
- AuditoryBench++ comprises five tasks: pitch, duration, and loudness comparisons, animal-sound recognition, and auditory context reasoning.The tasks range from basic comparisons to contextually grounded reasoning.Source: §3 · p. 2
Construction Pipeline
- The benchmark uses a multi-stage construction pipeline combining existing resources, statistical estimation, systematic filtering, and human verification.Datasets receive a final human-filtering and refinement pass to check correctness and remove residual noise.Source: §3 · pp. 2–3
- Pitch comparisons use only the AuditoryBench wiki set, while animal recognition draws from its test and wiki sets with human filtering of choices and answers.Ambiguous animal examples, including fictional animals and mislabeled cases, are removed.Source: §3 · p. 2
- Duration and loudness comparisons are built from AudioTime segment annotations, with peak decibels used for loudness and statistically significant label-pair contrasts selected.The construction retains classes with at least 30 samples and uses IQR outlier removal and pairwise differences with p < 0.01.Source: §3 · pp. 2–3
- Context-reasoning questions adapt MMAU items into text-only problems using audio captions, GPT-4o rewrites, and human verification.Qwen2-Audio generates captions capturing sound sources, events, and acoustic properties before the questions are rewritten.Source: §3 · p. 3
4. AIR-COT
AIR-CoT equips language models to reason with auditory commonsense through two training stages: detecting spans that need auditory knowledge and injecting relevant auditory embeddings.
AIR-CoT
- AIR-CoT has two training stages: supervised span detection with special tokens, followed by training an imagination module to inject auditory knowledge into detected spans.Source: §4 · p. 3
Data preparation
- Training data is generated from an independent AuditoryBench pitch-comparison set, with [imagine] tokens marking spans that require auditory knowledge.Qwen2.5-32B generates reasoning-style outputs using few-shot examples and the data's context, span, and answer.Source: §4 · p. 3
Stage 1: Span detection via special token
- Stage 1 fine-tunes the model to emit special tokens around spans requiring auditory knowledge, with loss applied only to token generation.Downstream answer tokens are excluded from the loss to prioritize span detection without biasing final predictions.Source: §4 · p. 3
Stage 2: Knowledge injection via imagination
- At the [/imagine] token, Stage 2 pauses decoding, injects an audio-model embedding, and then continues the reasoning chain.CLAP produces the embedding, which a two-layer MLP adapts to the model's hidden size; Stage 2 training loss is computed only on answer tokens.Source: §4 · pp. 3–4
5. EXPERIMENTS
AIR-CoT improves on several auditory reasoning tasks, while duration and loudness remain challenging. The experiments compare it with language models, audio-language models, and methods that add auditory knowledge.
Baselines and metric
- Baselines include recent LLMs, audio-language models, auditory-knowledge methods, and supervised fine-tuning, with accuracy as the evaluation metric.The evaluated methods include AudioBERT, Imagine to Hear, and a standard supervised fine-tuning baseline on Qwen2.5 7B.Source: §5 · p. 4
Implementation details
- AIR-CoT uses Qwen2.5 7B in both stages, with a CLAP text encoder and 2-layer MLP for imagination and audio-embedding alignment.Stage 1 fine-tunes with special tokens; Stage 2 trains only the projector.Source: §5 · p. 4
- AudioBERT and Imagine to Hear use the same data for a fair comparison.Source: §5 · p. 4
Experimental results
- AIR-CoT substantially outperforms other methods on pitch comparison, animal sound recognition, and auditory context reasoning, but improvements remain limited for duration and loudness.The authors attribute the stronger results to semantic audio representations and pitch-related training data; duration and loudness depend on temporal and amplitude cues those representations do not capture well.Source: §5 · p. 4
6. CONCLUSIONS
The conclusion presents AuditoryBench++ as a text-only auditory-knowledge benchmark and AIR-CoT as an auditory-imagination method, with experiments showing AIR-CoT outperforms off-the-shelf and augmented models.
Conclusions
- AuditoryBench++ assesses auditory knowledge in text-only settings, while AIR-CoT equips LLMs with auditory imagination through span detection and knowledge injection.Source: §6 · p. 4
- Experiments show that AIR-CoT outperforms off-the-shelf models and models augmented with auditory knowledge.Source: §6 · p. 4
- The authors describe the work as a foundation for language models that imagine auditory information without direct audio input and support more natural multimodal reasoning.Source: §6 · p. 4