Source-linked AI summary
Listen, Think, and Understand
Yuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky, James Glass
TL;DR
Existing audio models largely map sounds to predefined labels, while LLMs lack audio perception, motivating a model combining both capabilities. LTU integrates audio perception with language reasoning using the OpenAQA-5M dataset and a perception-to-understanding curriculum, achieving strong conventional-task performance and emerging audio reasoning.
Problem
Existing audio models generally classify sounds using predefined labels, whereas LLMs have reasoning ability but lack audio perception.
Method
LTU integrates an AST audio perception model with an open-source LLM, trains on unified audio question-answer tuples from OpenAQA-5M, and uses a perception-to-understanding curriculum.
Results
LTU outperforms CLAP across eight audio-classification benchmarks by 23.6% average relative improvement and exhibits audio reasoning abilities absent from existing audio models.
Takeaways & Limitations
LTU demonstrates a foundation audio reasoning model that handles conventional audio tasks alongside open-ended audio question answering.
Takeaways & Limitations
LTU focuses on general audio understanding and has limited ability to understand speech content, so it is not an automatic speech recognition model.
Abstract
from arXiv · showhide
The ability of artificial intelligence (AI) systems to perceive and comprehend audio signals is crucial for many applications. Although significant progress has been made in this area since the development of AudioSet, most existing models are designed to map audio inputs to pre-defined, discrete sound label sets. In contrast, humans possess the ability to not only classify sounds into general categories, but also to listen to the finer details of the sounds, explain the reason for the predictions, think about what the sound infers, and understand the scene and what action needs to be taken, if any. Such capabilities beyond perception are not yet present in existing audio models. On the other hand, modern large language models (LLMs) exhibit emerging reasoning ability but they lack audio perception capabilities. Therefore, we ask the question: can we build a model that has both audio perception and a reasoning ability? In this paper, we propose a new audio foundation model, called LTU (Listen, Think, and Understand). To train LTU, we created a new OpenAQA-5M dataset consisting of 1.9 million closed-ended and 3.7 million open-ended, diverse (audio, question, answer) tuples, and have used an autoregressive training framework with a perception-to-understanding curriculum. LTU demonstrates strong performance and generalization ability on conventional audio tasks such as classification and captioning. More importantly, it exhibits emerging audio reasoning and comprehension abilities that are absent in existing audio models. To the best of our knowledge, LTU is one of the first multimodal large language models that focus on general audio (rather than just speech) understanding.
1 INTRODUCTION
LTU addresses the gap between audio models that classify sounds with fixed labels and LLMs that reason without audio perception. It combines audio perception and language reasoning to support broader audio understanding.
- AudioSet-driven models improved audio tagging mAP from 31.4 to 47.3 but remained limited by discrete sound-label prediction.
- LTU integrates the AST audio perception model with the LLaMA language model and introduces the OpenAQA-5M audio question-answer dataset.OpenAQA-5M combines eight mainstream audio datasets and includes diverse open-ended pairs generated with Audio Instruction Generation.
- LTU outperforms CLAP across all eight audio-classification benchmarks by an average relative improvement of 23.6%.It does not require a predefined label set during inference.
- LTU answers open-ended audio questions with an 82.9% instruction-following and factual-correctness rate.The model also exhibits audio reasoning and comprehension abilities absent in existing audio models.
- The model targets general audio understanding rather than speech alone and is presented as an audio reasoning foundation model.
2 LTU MODEL ARCHITECTURE
LTU connects raw audio to a language model by encoding spectrogram patches into audio embeddings that are concatenated with text embeddings. Training preserves the base LLM while adapting audio-related components and predicting text autoregressively.
- Audio Encoder: The audio encoder converts each 10-second waveform into 32 projected audio embeddings that match LLaMA’s 4096-dimensional text-embedding size.AST processes a 1024 × 128 log-Mel spectrogram, producing 768-dimensional embeddings that are pooled, downsampled, and projected before concatenation with text embeddings.
- LLaMA Large Language Model: LTU uses LLaMA-7B with Vicuna instruction-following training as its language model.Vicuna is further trained on GPT-generated instruction-following prompts for reasoning and generation tasks.
- Low-rank Adapters: LoRA adapters modify frozen LLaMA parameters instead of fine-tuning the language model end to end.The design is intended to mitigate catastrophic forgetting and overfitting while improving training efficiency.
- Training Objective: LTU is trained by next-token prediction conditioned on previous text tokens and reference audio.
- Generation Setting: Generation uses temperature 0.1, Top K 500, Top P 0.95, and a repetition penalty of 1.1.The authors report that this plain setting generally works across tasks, while task-specific tuning may help but is less practical.
3 THE OPENAQA DATASET
OpenAQA-5M reformulates diverse audio tasks as question answering over public datasets, combining closed-ended supervision with large-scale open-ended instruction data. Its open-ended portion spans low-level perception to high-level audio understanding while retaining unanswerable examples.
- Dataset Construction: OpenAQA unifies nearly all audio tasks as (audio, question, answer) tuples, mapping task labels into a semantic space.The audio and question are model inputs, while the answer serves as the label.
- Dataset Construction: 845K unique public audio clips are used after relabeling existing datasets rather than collecting new audio.The source datasets include AudioSet, VGGSound, FSD50K, AudioCaps, FreeSound, Clotho v2, and Sound Bible.
- Dataset Composition: OpenAQA contains 1.9M closed-ended and 3.7M open-ended question-answer pairs.Both subsets are described as crucial to LTU training.
- Closed-ended Tasks: The closed-ended tasks cover classification, acoustic features, captioning, and temporal analysis.Their answers are generated with fixed-format or rule-based procedures, while question wording is diversified with GPT-3.5-Turbo.
- Open-ended QA Generation: The open-ended set addresses the difficulty of generating millions of diverse audio QA pairs by using audio meta-information as text input to GPT-3.5-Turbo.Audio Instruction Generation uses events, acoustic features, captions, and temporal information, while LTU ultimately trains on raw audio and generated QA pairs.
- Open-ended QA Generation: 3.7 million open-ended QA pairs span low-level event identification through high-level atmosphere understanding, with 6.5% intentionally unanswerable.The unanswerable examples are framed as training data for reducing hallucination and teaching the model when evidence is insufficient.
4 LTU TRAINING RECIPE
LTU uses a perception-to-understanding curriculum that first anchors the model to audio with objective closed-ended tasks, then introduces increasingly open-ended tasks. Removing this curriculum significantly reduces audio-task performance.
- Curriculum Design: Training begins by adapting the randomly initialized audio projection layer with closed-ended classification and acoustic-feature tasks while AST and LoRA remain frozen.
- Curriculum Design: Stages 2–4 progressively expand training from classification and acoustic features to all closed-ended tasks and finally to closed- and open-ended tasks.
- Rationale: The perception-to-understanding curriculum first teaches audio perception because early open-ended tasks encourage language-driven answers instead of conditioning on audio.Closed-ended tasks provide objective answers and stronger penalties for incorrect predictions, helping guide audio attention before open-ended reasoning.
- Implementation: Training uses batch size 256, linear learning-rate decay with warmup, a 108-token text cutoff, and four RTX A6000 GPUs for about three days.
5 EXPERIMENTS
Experiments show that LTU performs strongly on conventional audio tasks while supporting open-ended audio reasoning, explanation, scene understanding, and calibrated refusal. Ablations indicate that LoRA adapters and the perception-to-understanding curriculum are important for preserving language reasoning and audio-task performance.
- Audio Captioning: 17.0 and 11.9 SPICE scores on AudioCaps and Clotho, respectively, are comparable with state-of-the-art specialized models.LTU is trained on more diverse data, which may cause SPICE to underestimate semantically similar captions that use different vocabulary.
- Ablation Studies: Full fine-tuning slightly improves audio tasks but dramatically damages original language reasoning, whereas LoRA adapters preserve the pretrained language model.Removing LoRA adapters also causes noticeably lower performance.
- Ablation Studies: Audio-task performance significantly declines without the perception-to-understanding curriculum, while training on open-ended tasks also improves closed-ended performance.The curriculum trains perception before understanding, and open-ended training benefits closed-ended task performance.
- Audio Classification: LTU directly outputs text predictions, generalizes across closed-ended datasets and tasks, and requires no predefined label set during inference.The model also filters out non-prominent sounds automatically in its predictions.
- Open-Ended Audio Reasoning: LTU can answer follow-up questions about acoustic details and explain the reasons for its predictions, including correcting explanations after identified mistakes.Examples include describing bell-count, brightness, and pitch, and explaining why an atmosphere is considered dangerous.
- Open-Ended Audio Reasoning: LTU performs step-by-step reasoning, understands dangerous scenes, and connects sounds to actions or uses such as seeking shelter or dubbing a horror movie.These capabilities are presented as emerging audio reasoning and comprehension abilities absent from conventional audio models.
- Open-Ended Audio Reasoning: 69.3%→51.9% refusal rates show that removing unanswerable QA pairs increases hallucination risk on GPT-generated unanswerable questions.Close-ended performance also slightly drops from 50.3 to 50.0 after removal.
6 RELATED WORK
Related work combines language models with other modalities or speech, but the cited systems primarily emphasize vision, speech, or classification rather than general-audio understanding.
- Modern large language models demonstrate powerful reasoning and understanding abilities, motivating multimodal extensions.
- Most cited multimodal LLMs focus on vision, with LLaVA using instruction tuning and two-stage training as the closest concurrent visual work.
- Sound-domain systems such as WavPrompt and SpeechPrompt address speech understanding, while WavPrompt uses GPT-2 and a speech representation model for classification.
7 CONCLUSION
LTU combines audio perception with reasoning and understanding through diverse audio question-answering and a perception-to-understanding curriculum. It improves closed-ended performance while exhibiting emerging reasoning in open-ended tasks, but remains limited in speech-content understanding.
- Training: A perception-to-understanding curriculum and both closed-ended and open-ended audio question-answering tasks are key components of LTU training.Closed-ended tasks condition the model on audio initially, while open-ended tasks support advanced reasoning and understanding.
- Results: 23.6% average relative improvement on closed-ended tasks over CLAP accompanies emerging reasoning capability on open-ended tasks.The reported comparison covers closed-ended tasks, while the reasoning result concerns open-ended tasks.
- Limitation: LTU has limited ability to understand speech content and is not an automatic speech recognition model.
ETHICS STATEMENT
The paper uses publicly available audio data and notes potential benefits for hearing aids alongside possible security-related applications. Its broader literature discussion covers general audio classification and captioning research, including limitations of fixed label sets.
- Data use: The paper uses publicly available audio data and does not use private audio data for training.
- Potential benefits: The proposed audio language model may benefit individuals with disabilities through hearing-aid applications.
- Risks: The model may be repurposed for security-related tasks, so researchers, developers, and users should apply it responsibly and ethically.The paper states that its non-speech-recognition design comparatively mitigates associated risk.
- Related research: General audio research includes classification and captioning, while fixed-label models cannot generalize to unseen sound classes.Audio captioning generates free-text descriptions, addressing limitations of sound labels that are not informative enough for many applications.
B EXTENDED DISCUSSION OF LIMITATIONS
The discussion identifies limitations from model scale, temporal resolution, sparse temporal annotations, and residual hallucination, while examining encoder and alignment choices.
- Limitations: LLaMA-7B limits potential reasoning strength because larger language models may provide stronger reasoning and understanding abilities.The authors chose LLaMA-7B mainly because of computational-resource limits.
- Limitations: 2× temporal downsampling to 3.2Hz may limit fine-grained temporal reasoning, while higher temporal resolution could improve LTU.
- Limitations: Sparse temporal-level annotations may leave LTU weaker at localizing sound events than recognizing them.
- Limitations: Closed-ended and unanswerable training questions mitigate hallucination, but hallucination and bias may still occur rarely.
- Audio encoder: The CAV-MAE-pretrained AST improves AudioSet mAP over vanilla AST while shortening each 10-second clip’s patch sequence, improving Transformer efficiency.The reported values are 46.6 versus 45.9 mAP, with sequence lengths of 512 versus 1212.
- Audio-text alignment: Additional audio-text alignment does not significantly improve the same linear projection, while a larger Transformer alignment module worsens performance.The authors attribute this partly to trainable audio-encoder adaptation and the fact that audio embeddings serve as soft prompts.
G.2 CLOSED-ENDED TASK PERFORMANCE COMPARISON
LTU performs similarly to or better than Pengi on closed-ended classification while showing stronger open-ended answering and temporal-analysis capabilities, though temporal analysis remains weaker than classification and captioning.
- LTU and Pengi each win three of six closed-ended classification tasks; LTU averages 55.6 with LoRA and 56.7 when fully finetuned, versus Pengi’s 55.1.
- LTU dramatically outperforms Pengi on open-ended instruction following, although the comparison is not entirely fair because Pengi was not designed or trained for this task.
- LTU recognizes the order of two sound events with 70.9% accuracy on 550 ESC50 samples.
- LTU shows sound-event counting ability on an unseen task, with high correlation for many sound classes, but struggles for classes whose event appearances are difficult to define.
- Temporal analysis is weaker than classification and captioning, potentially because aggressive audio-token pooling harms fine-grained tasks and temporal-analysis training data are insufficient.
- Human evaluators rated LTU answers better than independently generated human answers, observing that LTU responses were usually more complete and detailed.
I.2 GPT-ASSISTED EVALUATION
GPT-assisted and probe evaluations examine LTU’s instruction following, acoustic-feature learning, and sensitivity to pitch changes. The results support semantic acoustic understanding, while captioning metrics can undervalue semantically correct outputs.
- Existing captioning metrics can assign 0 SPICE to semantically correct LTU outputs when they do not match keywords in the five ground truths.
- GPT-3.5-Turbo generated 10 acoustic descriptions for each sound class, emphasizing low-level features and paired concepts such as high-pitched and low-pitched.
- After lowering the pitch of 53 ambulance-siren evaluation audios, LTU’s pitch predictions aligned well with the actual pitch.
- The pitch probe indicates LTU learned pitch as an acoustic concept rather than merely associating high pitch with the ambulance-siren class.
K.1 SAMPLE ACOUSTIC FEATURES
The paper constructs diverse acoustic-feature descriptions and open-ended audio questions using GPT assistance, while training and evaluating LTU from multiple existing audio datasets. The generated OpenAQA data are notably diverse.
- The 3.7M open-ended OpenAQA subset contains highly diverse questions and answers, with over 95% appearing only once.
- GPT-3.5-Turbo generates complex open-ended questions and step-by-step answers from audio events, acoustic features, captions, and temporal information.
- Full fine-tuning produces lower Stage 3 training losses than LoRA, but only slightly improves audio-task performance and substantially reduces original language reasoning ability.
- The dataset combines relabeled public resources including AudioSet, VGGSound, FSD50K, AudioCaps, FreeSound, Clotho V2, and Sound Bible.
P.2.4 IN-DOMAIN EVALUATION
In-domain evaluation measures LTU on classification and captioning benchmarks whose training data overlap with LTU’s training data. The paper cautions that comparisons with dataset-specialized models are not exactly fair.
- Because LTU trains across multiple datasets and generates free-form outputs, its prediction search space is larger than conventional single-dataset models.
- Comparisons with dataset-specialized models are not exactly fair because LTU does not use dataset-specific training tricks.
- LTU is evaluated on VGGSound using top-1 accuracy, FSD50K and AudioSet using mAP, and AudioCaps and Clotho V2 using SPICE.