Source-linked AI summary
GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities
Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, Dinesh Manocha
TL;DR
Non-speech and non-verbal audio understanding remains less explored despite its importance for interaction with the world. GAMA combines multiple audio representations with an LLM, then uses CompA-R instruction tuning and soft audio-tag prompts for complex reasoning; it outperforms prior audio-language models across diverse tasks, while music understanding remains outside the evaluated scope.
Problem
Language-based understanding of non-speech sounds and non-verbal speech is less explored, although it is important for interacting with the world.
Method
GAMA integrates an LLM with Audio Q-Former and multi-layer AST representations, then instruction-tunes it with CompA-R and soft prompts containing audio event tags.
Results
GAMA outperforms prior audio-language models by significant margins on 16 datasets spanning four tasks, while CompA-R instruction tuning improves complex open-ended audio question answering and dense captioning.
Takeaways & Limitations
GAMA combines improved audio perception with complex reasoning and instruction following for open-ended audio question answering and dense captioning.
Takeaways & Limitations
The experiments do not extensively evaluate music understanding because GAMA was not trained on diverse, large-scale music datasets.
Abstract
from arXiv · showhide
Perceiving and understanding non-speech sounds and non-verbal speech is essential to making decisions that help us interact with our surroundings. In this paper, we propose GAMA, a novel General-purpose Large Audio-Language Model (LALM) with Advanced Audio Understanding and Complex Reasoning Abilities. We build GAMA by integrating an LLM with multiple types of audio representations, including features from a custom Audio Q-Former, a multi-layer aggregator that aggregates features from multiple layers of an audio encoder. We fine-tune GAMA on a large-scale audio-language dataset, which augments it with audio understanding capabilities. Next, we propose CompA-R (Instruction-Tuning for Complex Audio Reasoning), a synthetically generated instruction-tuning (IT) dataset with instructions that require the model to perform complex reasoning on the input audio. We instruction-tune GAMA with CompA-R to endow it with complex reasoning abilities, where we further add a soft prompt as input with high-level semantic evidence by leveraging event tags of the input audio. Finally, we also propose CompA-R-test, a human-labeled evaluation dataset for evaluating the capabilities of LALMs on open-ended audio question-answering that requires complex reasoning. Through automated and expert human evaluations, we show that GAMA outperforms all other LALMs in literature on diverse audio understanding tasks by margins of 1%-84%. Further, GAMA IT-ed on CompA-R proves to be superior in its complex reasoning and instruction following capabilities.
1 Introduction
GAMA addresses the limited exploration of language-based understanding for non-speech sounds and non-verbal speech by combining diverse audio representations with an LLM. CompA-R and CompA-R-test target complex audio reasoning and evaluate open-ended question answering.
- GAMA provides more detailed audio captions and answers questions requiring complex reasoning after diverse feature integration and CompA-R training.
- Understanding non-speech sounds and non-verbal speech through language remains less explored than spoken-language understanding, despite its importance for autonomous agents.
- GAMA integrates an LLM with Audio Q-Former and AST features, using an aggregation module to encode complementary semantic and surface-level audio information.The Audio Q-Former provides semantic generalization, while AST features capture surface-level audio properties.
- CompA-R is a synthetically generated instruction-tuning dataset designed to teach complex reasoning over audio events, their contexts, and related scene elements.Its instruction-response pairs are synthesized using audio metadata and human-written in-context examples.
- CompA-R-test is a human-labeled benchmark for open-ended audio question answering that requires complex reasoning.GAMA-IT shows significant improvements on this benchmark over literature baselines.
2 Related Work
Related work spans shared audio-language representations and instruction-tuned audio-language models. These approaches use language interfaces and instruction-response training to support audio tasks and reasoning.
- CLAP pioneered improving audio understanding through a language interface and achieved strong performance on retrieval and zero-shot classification tasks.
- Instruction tuning has improved multimodal alignment and reasoning capabilities, including logical, mathematical, and complex reasoning tasks.
3 Methodology
GAMA combines multiple audio representations with an LLM, then fine-tunes the model for audio understanding and complex audio reasoning. Its methodology includes the CompA-R dataset, soft prompting with event tags, and evaluation procedures spanning classification, captioning, and audio question answering.
- GAMA Architecture: GAMA couples an LLM with Audio Q-Former and AST features, using a multi-layer aggregator rather than only AST’s final layer.The aggregator integrates hidden features from multiple AST layers, while separate MLPs connect audio features to the word-embedding space as prefixes.
- Audio Q-Former: The Audio Q-Former is introduced to provide strong semantic generalization for arbitrary input audio through an alternative to CLAP-style audio-language encoders.Its features are passed through MLP layers before being added as prefixes to the text representation.
- Multi-Layer Aggregator: The multi-layer aggregator combines AST features from different depths because middle layers encode generic sounds and textures while deeper layers encode higher-level sound patterns.It uses two transformer-style blocks with self-attention, cross-attention, and feed-forward networks; the implementation uses AST layers j = 4 and k = 8.
- Soft Prompt: Soft prompting supplements instructions with audio event tags while allowing the model to use or partially use those tags adaptively despite inevitable classification errors.The tags are inserted through a fixed instruction template with a trainable soft prompt, rather than steering the model toward one fixed task.
- CompA-R: CompA-R targets complex reasoning about complete audio scenes, individual acoustic events, their relationships, and underlying contexts.It addresses failures of existing LALMs on questions requiring more than identifying or describing the audio, using a systematic multi-stage synthesis pipeline based on AudioSet-strong.
- CompA-R: CompA-R is synthesized by generating audio-video captions, combining them with event time slices, and prompting GPT-4 with human-written exemplars to produce instruction-response pairs.A subset of the generated data receives expert human verification for test-set construction.
- Training: GAMA is first fine-tuned on OpenAQA and four additional music datasets, then instruction-tuned on CompA-R as GAMA-IT using LoRA modules.The additional datasets are MusicCaps, MusicQA, NSynth, and Magna; only the LoRA modules of the LLM are trained during the instruction-tuning stage.
- Experimental Setup: Evaluation covers close-ended classification, captioning, open-ended AQA, complex open-ended AQA, and dense captioning, with metrics and zero-shot distinctions specified for the datasets.Classification evaluation distinguishes zero-shot, weak zero-shot, and seen datasets; audio is captioned and matched to labels using SentenceBERT for classification.
4 Results and Analysis
GAMA consistently outperforms the evaluated baselines across classification, captioning, open-ended audio question answering, and dense captioning, including complex reasoning tasks.
- Quantitative Results: GAMA outperforms baselines by 2%-67% on zero-shot classification, 1%-66% on weak zero-shot evaluation, and 1%-84% on in-domain evaluation.The evaluations cover VocalSound, TUT 2017, Beijing Opera, GTZAN, Medley-solos-DB, ESC-50, DCASE2017 Task 4, VGGSound, FSD50K, AudioSet, and NSynth.
- Quantitative Results: GAMA outperforms all baselines on open-ended and complex open-ended AQA and dense captioning.Table 2 evaluates these capabilities across multiple settings.
- Quantitative Results: 4%-50% on OpenAQA, 8%-58% on CompA-R-test, and 8%-30% on Dense Captioning are GAMA’s reported absolute improvements.Performance suffers most without the Audio Q-Former, while the Audio Q-Former is especially effective over CLAP in AQA.
- Qualitative Results: GAMA-IT produces responses that are more faithful, correct, and preferred by humans on CompA-R-test examples.The qualitative comparison covers GAMA-IT and other LALMs with audio chat or open-ended AQA capabilities.
5 Conclusion
The paper concludes that GAMA combines diverse audio representations with CompA-R instruction tuning to improve audio understanding, complex reasoning, and dense captioning, while identifying several scope and resource limitations.
- Conclusion: GAMA integrates multiple audio representations and outperforms prior audio-language models across 16 datasets spanning 4 tasks.The conclusion attributes the improved audio perception abilities to diverse knowledge from multiple audio representations.
- Conclusion: GAMA IT-ed on CompA-R outperforms baselines on complex open-ended AQA and dense captioning.CompA-R is synthesized for endowing an LALM with complex reasoning abilities.
- Limitations and Future Work: The experiments do not extensively evaluate or compare music understanding because GAMA was not trained on diverse, large-scale music datasets.The authors suggest that the architecture could support comprehensive music understanding if trained on such datasets.
- Limitations and Future Work: The study does not employ larger LLMs, such as 13B LLaMA versions, because of compute constraints.This is listed as a limitation of GAMA and the experimental setup.
- Limitations and Future Work: GAMA’s audio encoders have more parameters than those of the baselines, although they constitute only a fraction of the total LALM parameter count.The parameter-count difference is acknowledged as an experimental limitation.
A Additional Results
The appendix specifies prompts used to synthesize CompA-R, evaluate CompA-R responses, and generate dense audio captions.
- Additional Results: Figures 5 and 6 show the prompts for synthesizing CompA-R and evaluating model responses on CompA-R.Dense captioning uses the instruction: Write an audio caption describing the sound in detail.
C GAMA-IT vs GAMA and Evaluation Choices.
GAMA-IT is trained for complex reasoning after GAMA’s generic audio fine-tuning, but is not evaluated on general tasks because detailed descriptions are difficult to assess faithfully with retrieval metrics.
- Training and Evaluation: GAMA is first fine-tuned on OpenAQA and then instruction-tuned on CompA-R for complex reasoning.The instruction-tuned model is called GAMA-IT.
- Training and Evaluation: GAMA-IT is not evaluated on classification or vanilla captioning because it is aligned to generate detailed descriptions for complex reasoning.The authors report that retrieval-based evaluation methods do not faithfully assess such descriptions.
- Evaluation Choices: Sentence-BERT and CLAP can fail to retrieve the correct label even when a generated caption mentions the label as an audio event.The authors attribute this limitation to detailed descriptions and retrieval models focusing on high-level semantic meaning.
- Evaluation Choices: The soft prompt is used only during instruction tuning, not during fine-tuning, because high-level semantic cues may be unnecessary for generic audio tasks and inaccurate tags may add noise.The rationale is tied to generic classification and captioning objectives and imperfect acoustic-event classification.
D Additional Details: Human Study
The paper reports human-evaluation procedures for CompA-R-test, dense captioning, and OpenAQA, using trained raters and standardized scoring software.
- Recruitment: Three professionals with at least a Ph.D. evaluated dense captioning and CompA-R-test responses using headphones.They first analyzed the audio before judging response quality.
- CompA-R-test Evaluation: Each of the three annotators scored responses from 1-5, and the reported score was their average.Before evaluation, annotators received at least 10 example generations with corresponding scores.
- OpenAQA Evaluation: OpenAQA evaluation used 267 unique Amazon Mechanical Turk evaluators, with each generation scored by two evaluators.The same evaluation software was used for OpenAQA and CompA-R-test.
E Additional Details: Audio Q-Former
The appendix details Audio Q-Former training, its approximately 2.2M-pair audio-caption corpus, augmentation procedures, baselines, and manual verification of CompA-R-test.
- E.1 Audio Q-Former Training Details: Stage 1 Audio Q-Former pre-training used batch size 192, learning rates from 1e-4 to 1e-5, 5,000 warm-up steps, and 100 epochs on 8 A6000 GPUs.Stage 2 retained the settings but changed the batch size to 128.
- E.1 Audio Q-Former Training Details: Zero-shot audio-classification fine-tuning reused the Stage 1 corpus and objective while training AST, which was frozen during pre-training.The passage states that other settings remained the same apart from this fine-tuning change.
- E.2 Training Dataset Details: The Audio Q-Former training corpus contains approximately 2.2M audio-caption pairs and no speech-transcription pairs.Table 3 lists the individual datasets used to construct this corpus.
- F Baseline Details: AudioCLIP, CLAP, CompA-CLAP, and Pengi are described as baseline audio-language models with progressively broader audio-language or compositional capabilities.CLAP uses contrastive learning, while CompA-CLAP extends it for compositional reasoning.
- G.1 Annotation and Annotator Details: CompA-R was cleaned and CompA-R-test was verified by the paper authors, whose graduate training and research experience span speech, audio, and language processing.The authors state that this background supported their qualification to verify the data.
- G.1 Annotation and Annotator Details: At least three authors manually verified CompA-R-test instances with rationales, followed by cross-verification of all instances by the lead author.Verification was performed on local laptops without a purpose-built application.
H.1 GAMA Training Dataset Details
The training-dataset appendix identifies the datasets used for GAMA fine-tuning and instruction-tuning, including CompA-R's AudioSet-Strong origin.
- H.1 GAMA Training Dataset Details: Table 4 reports statistics for all datasets used to fine-tune and instruction-tune GAMA.
- H.1 GAMA Training Dataset Details: Table 5 reports statistics for CompA-R.
- H.1 GAMA Training Dataset Details: CompA-R is sourced entirely from the AudioSet-Strong dataset.
H.2 GAMA Evaluation Dataset Details
The evaluation-dataset appendix points to dataset statistics for GAMA's evaluation sets and identifies CompA-R-test as entirely sourced from AudioSet-Strong.
- H.2 GAMA Evaluation Dataset Details: Table 6 reports statistics for all datasets used to evaluate GAMA.
- H.2 GAMA Evaluation Dataset Details: Table 8 reports statistics for CompA-R-test.
- H.2 GAMA Evaluation Dataset Details: CompA-R-test is sourced entirely from the AudioSet-Strong dataset.
H.3 Other Details
The supplementary details describe GAMA’s implementation, datasets, evaluation setup, prompting procedures, and potential risks. They also provide examples of complex audio-reasoning instructions and model responses.
- Implementation: GAMA uses approximately 7B parameters, including LLaMA-2-7B, an Audio Q-Former, and trainable LoRA modules.The implementation also uses an AST audio encoder and four NVIDIA A6000 GPUs.
- Evaluation: Evaluation distinguishes zero-shot, weak zero-shot, and seen datasets, and uses dataset-specific metrics after captioning the audio.The supplementary tables list evaluation datasets, metrics, and dataset statistics.
- Potential Risks: The authors identify potential risks from biases inherited from the pretrained LLM and misuse of synthetic audio generated by the Audio Q-Former.These risks are presented as potential consequences of the model and its audio-generation backbone.
- Prompt Examples: The supplementary examples show prompts asking models to infer actions, skill levels, engine types, sound sources, musical forms, and scene atmosphere from audio.Examples include clock ticking with soft music, singing with beatboxing, and gunfire during a military parade.
- Evaluation Prompts: Caption evaluation uses GPT-4 as a judge, while CompA-R answers are evaluated with separate prompts.The supplementary figures document prompt instructions for caption augmentation and evaluation.