Source-linked AI summary
Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Model
Ziyang Ma, Zhuo Chen, Yuping Wang, Eng Siong Chng, Xie Chen
TL;DR
LALMs excel at audio perception but lack well-studied reasoning capabilities needed for complex auditory tasks. This paper is the first systematic exploration of CoT methods in LALMs across modalities and task types, finding improved performance overall while exposing hard-task limitations and a relationship between reasoning length and accuracy.
Problem
Reasoning capabilities in LALMs remain underexplored despite their importance for complex auditory understanding.
Method
The paper systematically evaluates representative CoT methods in LALMs across diverse auditory modalities, tasks, and difficulties.
Results
CoT methods improve LALM performance overall; Zero-Shot-CoT reaches 57.80% total accuracy versus 55.60% for the baseline, and Self-Consistency with Zero-Shot-CoT reaches 58.10%.
Takeaways & Limitations
Longer reasoning paths correlate positively with accuracy, indicating potential for scaling inference to improve instruction-following and reasoning in LALMs.
Takeaways & Limitations
Reasoning chains can inadvertently confuse current LALMs on harder questions, revealing limitations in their foundational capabilities.
Abstract
from arXiv · showhide
Large Audio-Language Models (LALMs) have demonstrated remarkable performance in tasks involving audio perception and understanding, such as speech recognition and audio captioning. However, their reasoning capabilities - critical for solving complex real-world problems - remain underexplored. In this work, we conduct the first exploration into integrating Chain-of-Thought (CoT) reasoning into LALMs to enhance their reasoning ability across auditory modalities. We evaluate representative CoT methods, analyzing their performance in both information extraction and reasoning tasks across sound, music, and speech domains. Our findings reveal that CoT methods significantly improve performance on easy and medium tasks but encounter challenges with hard tasks, where reasoning chains can confuse the model rather than improve accuracy. Additionally, we identify a positive correlation between reasoning path length and accuracy, demonstrating the potential of scaling inference for advanced instruction-following and reasoning. This study not only highlights the promise of CoT in enhancing LALM reasoning capabilities but also identifies key limitations and provides actionable directions for future research.
1. Introduction
LALMs perform well on audio perception and understanding, but their higher-level reasoning remains underexplored. This work investigates CoT reasoning for guiding LALMs through auditory reasoning tasks and finds benefits on easier tasks alongside challenges on harder ones.
- LALMs have demonstrated strong performance on audio perception and understanding tasks such as speech recognition and audio captioning.
- Existing LALM research primarily targets sensory-level transcription, classification, and summarization rather than multi-step auditory reasoning.
- CoT prompts models to generate explicit reasoning paths that guide step-by-step deduction toward predictions.
- The study systematically evaluates Manual-CoT, Zero-Shot-CoT, and Desp-CoT across auditory tasks.
- CoT methods improve easy and medium tasks but can confuse LALMs on hard tasks, while Self-Consistency and longer reasoning paths offer additional performance benefits.
2. Related Work
LALM research has largely emphasized perception and understanding, while reasoning remains comparatively neglected. Prior CoT work offers manual, automatic, hybrid, and multimodal approaches, but CoT had not yet been explored systematically for LALMs.
- LALMs combine an audio encoder, an LLM, and a projector to support general audio understanding.
- Most LALM research focuses on perception and understanding, leaving reasoning capabilities relatively underexplored.
- CoT elicits intermediate reasoning steps to improve complex-problem solving, but automatically generated chains can suffer from lower quality and hallucinations.
- Hybrid CoT methods combine manual and automatic chain construction to address weaknesses of either approach.
- Although CoT has been extended to multimodal reasoning in vision-language models, its methodologies had not been explored in LALMs.
3. Methods
The paper applies several Chain-of-Thought strategies to LALMs without changing their training paradigm, using handcrafted examples, a magic prompt, or an intermediate audio caption. It also evaluates Self-Consistency as a way to aggregate multiple reasoning paths.
- Manual-CoT: Manual-CoT uses few-shot examples containing audio, instructions, and reasoning chains to transfer a handcrafted reasoning structure to new inputs.The LALM uses the example set during inference to generate an output reasoning chain for each input audio and instruction.
- Zero-Shot-CoT: Zero-Shot-CoT prompts the LALM with a natural-language magic prompt instead of curated examples to generate a reasoning chain directly.The approach is described as efficient and scalable because it removes the need for task-specific handcrafted examples.
- Desp-CoT: Desp-CoT first generates a descriptive caption from the input audio, then concatenates that caption with the instruction to guide reasoning-chain generation.This creates a natural-language representation of the audio that serves as an anchor for the subsequent reasoning process.
- Self-Consistency: Self-Consistency samples multiple reasoning paths and selects the most consistent answer through marginalization and majority voting.The experiments use 5 sampling iterations followed by majority voting based on maximum probability.
4. Experiments
Experiments evaluate representative CoT methods on MMAU across sound, music, and speech, covering information extraction, reasoning, and difficulty levels. CoT generally improves performance, but gains weaken or reverse on hard questions, while longer reasoning and Self-Consistency show additional promise.
- Experimental Setup: MMAU evaluates LALMs across sound, music, and speech using information extraction and reasoning questions, with approximately 70% targeting reasoning.The benchmark contains multiple-choice questions spanning 27 skills.
- Main Results: 57.80% total accuracy makes Zero-Shot-CoT the strongest standalone method, ahead of the 55.60% normalized baseline, 57.00% Manual-CoT, and 56.30% Desp-CoT.The comparison covers both information extraction and reasoning tasks.
- Main Results: 53.40% reasoning versus 65.65% information extraction for Manual-CoT, and 54.39% versus 65.99% for Zero-Shot-CoT, shows lower reasoning accuracy.The reported differences indicate greater difficulty for reasoning tasks than information extraction tasks.
- Self-Consistency: 58.10% total accuracy is achieved by Zero-Shot-CoT with Self-Consistency, the highest reported result after marginalizing multiple sampled reasoning paths.The implementation uses five sampling iterations followed by majority voting based on maximum probability.
- Analysis: Accuracy generally rises with longer reasoning paths in sound, music, and total categories, with Zero-Shot-CoT averaging approximately 35 words and achieving the highest performance.Speech and Desp-CoT in sound are identified as apparent outliers.
- Analysis: CoT improves easy and medium questions but fails to improve, and can degrade, accuracy on hard information extraction and reasoning questions.The authors suggest that reasoning chains for challenging questions may confuse the model and identify a need for stronger foundational capabilities.
5. Conclusion & Future Work
The paper presents the first exploration of CoT reasoning in LALMs and systematically evaluates it across tasks, modalities, and difficulties. It finds benefits from CoT and Self-Consistency, a positive association between reasoning length and performance, and persistent challenges on hard questions that motivate dynamic reasoning and stronger foundations.
- This work provides the first exploration of CoT reasoning in LALMs across diverse tasks, modalities, and difficulty levels.
- CoT methods improve LALM performance, while Self-Consistency provides further performance improvement.
- Reasoning length is positively correlated with performance, indicating potential for scaling inference to improve instruction-following and reasoning capabilities.
- Hard questions remain challenging because reasoning chains can confuse the model, motivating dynamic reasoning strategies and stronger foundational capabilities.