PaperlayerView exact revision on arXiv

Source-linked AI summary

DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment

Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Huck Yang, Sung-Feng Huang, Chih-Kai Yang, Chee-En Yu, Chun-Wei Chen, Wei-Chih Chen, Chien-yu Huang, Yi-Cheng Lin, Yu-Xiang Lin, Chi-An Fu, Chun-Yi Kuan, Wenze Ren, Xuanjun Chen, Wei-Ping Huang, En-Pei Hu, Tzu-Quan Lin, Yuan-Kuei Wu, Kuan-Po Huang, Hsiao-Ying Huang, Huang-Cheng Chou, Kai-Wei Chang, Cheng-Han Chiang, Boris Ginsburg, Yu-Chiang Frank Wang, Hung-yi Lee

arXiv:2507.02768v2eess.AScs.CLcs.SD

TL;DR

Existing LALMs can suffer catastrophic forgetting when audio-instruction data shifts from the backbone LLM’s native behavior. DeSTA has the backbone generate its own alignment targets and scales this approach into a diverse 5-million-sample corpus, yielding state-of-the-art or competitive results across audio-language benchmarks.

  • Problem

    Existing LALMs often suffer catastrophic forgetting because heterogeneous training targets can shift models away from the backbone LLM’s native behavior.

  • Method

    DeSTA uses structured audio descriptions and arbitrary prompts as inputs to the backbone LLM, whose generated responses become cross-modal alignment targets.

  • Results

    DeSTA2.5-Audio achieves state-of-the-art or competitive performance across Dynamic-SUPERB, MMAU, SAKURA, Speech-IFEval, and VoiceBench.

  • Takeaways & Limitations

    The results support self-generated alignment and carefully designed training-data distributions as practical considerations for robust, general-purpose LALMs.

Abstract

from arXiv · show

We introduce DeSTA2.5-Audio, a general-purpose Large Audio Language Model (LALM) designed for robust auditory perception and instruction-following. Recent LALMs augment Large Language Models (LLMs) with auditory capabilities by training on large-scale audio-instruction datasets. However, existing LALMs have often suffered from the catastrophic forgetting of the LLM's original abilities. Therefore, balancing knowledge retention and audio perception has become a critical challenge. To address this, we revisit the data construction pipeline and propose a self-generated cross-modal alignment strategy in which the backbone LLM generates its own training targets, named DeSTA. This approach aims at preserving the LLM's native language proficiency thereby enabling zero-shot generalization without task-specific tuning. We construct DeSTA-AQA5M, a large-scale, task-agnostic dataset containing 5 million training samples derived from 7,000 hours of audio spanning 50 diverse datasets, including speech, environmental sounds, and music. DeSTA2.5-Audio achieves state-of-the-art or competitive performance across a wide range of audio-language benchmarks, including Dynamic-SUPERB, MMAU, SAKURA, Speech-IFEval, and VoiceBench. Comprehensive comparative studies demonstrate that our self-generated strategy outperforms existing training strategies. Our findings underscore the importance of carefully designed data construction in LALM development and offer practical insights for building robust, general-purpose LALMs.

I. INTRODUCTION

The paper targets general-purpose LALMs that combine auditory perception with instruction-following while avoiding catastrophic forgetting. It proposes self-generated alignment and a diverse audio dataset, achieving strong benchmark performance with less training data.

  • General-purpose LALMs must process speech, non-verbal cues, background sounds, and music while following diverse commands.
  • Existing audio-instruction construction can cause catastrophic forgetting and weaken generalization to unseen tasks.
  • DeSTA has the backbone LLM generate training targets from structured audio descriptions and arbitrary prompts, preserving stylistic and semantic consistency.
  • DeSTA2.5-Audio achieves state-of-the-art or competitive results across multiple audio-language benchmarks, including Dynamic-SUPERB, MMAU, SAKURA, Speech-IFEval, and VoiceBench.
  • Comparative studies find self-generated targets outperform mismatched data sources, while LoRA can reduce complex-reasoning or out-of-domain performance despite alleviating distribution mismatch.
  • DeSTA-AQA5M contains 5 million audio-prompt-response triplets from 7,000 hours across 50 datasets covering speech, environmental sounds, and music.

II. RELATED WORKS

Related work addresses the audio-text modality gap through fusion-module fine-tuning and synthetic instruction data. DeSTA instead emphasizes targets matched to the backbone LLM, extending self-generated alignment beyond spoken-content-centric tasks.

  • General-purpose LALMs commonly fine-tune fusion modules with audio-text pairs, making training-data availability and construction central concerns.
  • Self-generated data matches training targets to a model’s inherent behavior and data distribution instead of relying on teachers or human annotations.
  • Multimodal self-generated approaches have often focused on spoken-content tasks such as ASR and spoken question answering.
  • DeSTA differs from heterogeneous-source instruction tuning by using a streamlined alignment strategy designed to avoid distributional mismatch.

III. METHODOLOGY

DeSTA constructs cross-modal alignment targets by having the backbone LLM respond to structured textual descriptions of audio under diverse prompts. The model then trains a lightweight modality adapter to map audio representations into the LLM’s semantic space while preserving the backbone’s output style.

  • Self-Generated Dataset Construction: DeSTA converts audio metadata into structured textual descriptions and pairs each description with a sampled instruction prompt.The initial pairs are represented as D_initial = {(x_audio, x_text)} before prompt-based target generation.
  • Self-Generated Dataset Construction: The backbone LLM generates each training response from the audio description and prompt, producing audio–description–prompt–response training examples.Prompts include descriptive, role-playing, and open-ended question types.
  • Model Training: The fusion model combines a frozen audio encoder and frozen instruction-tuned LLM with a trainable Q-Former modality adapter.The adapter is fine-tuned on triplets containing audio, prompts, and generated responses.
  • Model Training: Q-Former blocks extract features from multiple intermediate audio-encoder layers, aggregate them with learnable scalar weights, and project them into the LLM embedding dimension.The method optionally concatenates transcription-derived linguistic features with continuous audio features.
  • Model Training: The resulting audio embeddings and prompt embeddings are passed to the LLM for autoregressive next-token prediction.Training uses the standard next-token prediction loss on the generated targets.
  • Design Rationale: Self-generated targets remove stylistic mismatch with the backbone LLM, focusing optimization on cross-modal grounding rather than adapting to an external output distribution.The stated objectives are cross-modal grounding and stylistic adaptation.

A. Dataset

DeSTA-AQA5M is a large, diverse audio instruction-tuning corpus built from 50 public datasets and approximately 7,000 hours of speech, environmental sounds, and music. Multiple domain-specific prompts are used to generate about 5 million training triplets.

  • Corpus Composition: The corpus aggregates 50 public datasets selected for metadata covering paralinguistic, speaker, quality, environmental, and contextual audio information.The metadata spans attributes such as pitch, emotion, accent, gender, age, noise, reverberation, and environmental sounds.
  • Corpus Composition: Approximately 7,000 hours comprise 5,400 hours of speech, 1,000 hours of environmental sounds, and 500 hours of music.These domains provide the audio coverage for the dataset construction pipeline.
  • Prompt Construction: The instruction pool contains 4,000 speech prompts and 3,000 prompts for environmental sound and music categories, with upsampling used to balance domains.Each audio sample is paired with multiple prompts.
  • Dataset Scale: The resulting DeSTA-AQA5M dataset contains approximately 5 million audio–prompt–response triplets generated with low-temperature decoding.Responses were generated using temperature 0.05 and top-p 1.0.
  • Model Configuration: The model uses Llama3.1-8B-Instruct and Whisper-large-v3 with a six-layer, 64-query Q-Former adapter.Whisper intermediate layers 8, 16, 24, and 32 provide multi-scale acoustic features.

C. Comparison Study

The comparison study controls data and training conditions to isolate how target-generating LLMs, prompt diversity, and model configurations affect performance. Evaluation spans instruction-following, perception, reasoning, and spoken interaction across several audio-language benchmarks.

  • Comparison Design: A controlled 500,000-sample subset supports systematic comparisons of target-generation alternatives and model configurations.All comparison experiments use identical architectures and hyperparameter settings for ten epochs.
  • Target Generation: Dataset-level comparisons fix audio descriptions and prompts while varying the LLM that generates training targets.The compared generators include Llama3.1-8B-Instruct, Qwen2.5-7B-Instruct, Gemma3-12B-it, and Llama3.1-70B-Instruct.
  • Prompt Diversity: A caption-style dataset with one fixed prompt isolates the effect of prompt diversity.The fixed prompt is “What can you hear from the audio?”
  • Evaluation Coverage: The benchmark suite covers instruction-following, perceptual understanding, reasoning, and spoken interaction across distinct task types and domains.The evaluation includes Dynamic-SUPERB, MMAU, SAKURA, Speech-IFEval, and VoiceBench.
  • Evaluation Coverage: Speech-IFEval measures instruction-following rate independently of audio correctness and defines forgetting rate as the relative IFrate drop from the backbone LLM.A negative forgetting-rate score indicates reduced instruction-following ability relative to the text-only backbone.

B. Cascade Baselines

The study uses text-only cascade baselines and standardized inference prompts alongside representation-level CLAP-style evaluation. The latter extracts last-token LLM states and classifies audio by cosine similarity to candidate text labels.

  • Cascade Baselines: The ASR+LLM cascade transcribes audio with Whisper-large-v3 and feeds the transcript directly into Llama3.1-8B-Instruct.It preserves the LLM’s pretrained textual knowledge and reasoning but omits non-verbal acoustic information.
  • Inference Setup: The evaluation avoids advanced prompt engineering to prevent prompt-design effects from confounding comparisons.More advanced prompting techniques are left for future work.
  • Inference Setup: Standardized system prompts direct the model to focus on audio and instructions, with an additional no-explanation constraint for multiple-choice benchmarks.The cited multiple-choice evaluations include Dynamic-SUPERB Phase-1, MMAU, and SAKURA.
  • Benchmark Reporting: Table II reports results for Dynamic-SUPERB Phase-1, MMAU, SAKURA, and Speech-IFEval, with higher scores marked as better and cascade reference results marked by †.The table organizes the principal benchmark comparisons used in this evaluation.
  • Representation-Level Evaluation: CLAP-style evaluation extracts the last-token hidden state as an aggregated representation for either audio or text inputs.The backbone LLM serves as a universal encoder for variable-length inputs.
  • Representation-Level Evaluation: Audio is classified by selecting the candidate text label with the highest cosine similarity to its audio representation.The representation-level analysis targets gender, emotion, and animal sounds on SAKURA-Single.

VI. MAIN RESULTS

DeSTA2.5-Audio achieves leading or competitive performance across auditory perception, reasoning, knowledge, and instruction-following benchmarks, despite using substantially less training data than Qwen2-Audio-Instruct.

  • DeSTA2.5-Audio records the highest scores on Dynamic-SUPERB Phase-1 (69.53), MMAU (57.50), SAKURA-Multi (69.85), and Speech-IFEval (93.89).
  • Broader environmental-sound and music coverage improves performance over DeSTA2’s speech-focused training and extends the self-generated strategy across domains.
  • 7,000 hours of training data support competitive or superior results relative to Qwen2-Audio-Instruct’s 510,000 hours.The comparison highlights the efficiency of the training methodology.
  • DeSTA2.5-Audio performs strongly across speech content, semantics, paralinguistics, degraded speech, speaker identity, audio, and music tasks.
  • In SAKURA-Single, Qwen2-Audio-Instruct leads DeSTA2.5-Audio (81.20 vs. 76.65), whereas DeSTA2.5-Audio leads SAKURA-Multi with 69.85.
  • DeSTA2.5-Audio and DeSTA2 retain robustness on Speech-IFEval, with performance comparable to or surpassing text-based reference systems.

B. Results on Dynamic-SUPERB Phase-2

On Dynamic-SUPERB Phase-2, DeSTA2.5-Audio ranks first across domain-level comparisons while retaining broad generalization and uncertainty-aware behavior, though some specialized domains remain weaker than the cascade baseline.

  • DeSTA2.5-Audio ranks first with 14 domain wins and a 3.42 average relative score against the ASR+LLM baseline.The model does not use task-specific instruction tuning.
  • Negative relative scores occur in Quality Assessment, Speech, Voice, Hearing Disorder, and Phonetics, Phonology, and Prosody.The authors attribute these weaknesses to underrepresented audio attributes in the training corpus.
  • Other systems show more severe degradation, including LTU-AS in Phonetics, Phonology, and Prosody and SALMONN in Signal Characteristics Analysis.
  • DeSTA2.5-Audio often responds with uncertainty statements on unfamiliar tasks rather than hallucinated or misleading answers.The authors connect this behavior to trustworthiness and reliability in deployment scenarios.
  • VoiceBench evaluates spoken instructions and generated responses, emphasizing general knowledge and interactive behavior through task-specific metrics.
  • On VoiceBench, DeSTA2.5-Audio scores 74.52, exceeding VITA-1.5 (64.53) and Qwen2-Audio-Instruct (55.80), but remains below ASR+LLM’s 77.48.

D. Representation-Level Analysis

Representation-level analyses show that DeSTA2.5-Audio progressively develops audio-text alignment, with integration concentrated in deeper LLM layers; controlled comparisons favor self-generated targets and diverse prompts.

  • Similarity-based classification demonstrates robust cross-modal grounding and discriminative audio features comparable to instruction-based evaluation.
  • Gender alignment approaches convergence by the first epoch, while emotion and animal-sound concepts improve steadily during training.
  • Alignment accuracy remains near random in shallow layers but surges around layer 22, indicating deferred audio-text integration in deeper semantic layers.
  • Self-generated targets consistently outperform external-model targets and provide greater training stability under identical audio-description pairs.
  • Cross-model target generation can destabilize training: Llama3.1 trained on Qwen2.5 data degenerates after epoch four, while Gemma3 data causes premature plateauing.
  • Self-generated settings achieve high instruction-following and similarity-based scores, whereas external-model settings degrade in both metrics.The authors attribute the downstream difference to distributional discrepancy when the training target is the only changed variable.
  • Diverse prompts improve generalization within the self-generated framework without task-specific instruction pairs.

B. With LoRA Adapter

The LoRA study finds that self-generated targets make LoRA adapters largely unnecessary, whereas externally generated targets preserve some in-domain audio performance but degrade broader capabilities. Overall, reducing training-target distribution mismatch matters more for generalization than architectural modification alone.

  • B. With LoRA Adapter: Under self-generation, adding LoRA layers yields similar or only slightly better performance, indicating that a lightweight modality adapter is sufficient for cross-modal alignment.The reported comparison is from setup C1.
  • B. With LoRA Adapter: Qwen2.5-generated data matches self-generation on Dynamic-SUPERB, MMAU, and SAKURA-Single but significantly degrades SAKURA-Multi and Speech-IFEval.The latter benchmarks require additional text knowledge and instruction-following ability.
  • B. With LoRA Adapter: LoRA adapters can mitigate distribution mismatch for in-domain tasks but fail to preserve broader capabilities on out-of-domain benchmarks.The authors interpret LoRA as encoding mismatch between external targets and the backbone’s native behavior in these scenarios.
  • B. With LoRA Adapter: Reducing discrepancy between training data and the backbone’s native distribution is more critical for preserving generalization than architectural modifications alone.This conclusion is drawn from the comparison of self-generated and externally generated targets.
  • B. With LoRA Adapter: The text-description bridge remains limited because not all acoustic nuances can be effectively captured through textual representations.The authors identify audio-to-audio supervision and preference learning from human feedback as future directions.

APPENDIX

The appendix records the authors’ collective contributions across method design, data collection, evaluation, writing, and manuscript refinement. It also identifies leadership, technical guidance, evaluation, and discussion roles among contributors.

  • APPENDIX: All authors contributed meaningfully to method design, data collection, evaluation, writing, and refinement of the paper.The appendix states that authors participated in multiple aspects of the project.
  • APPENDIX: Ke-Han Lu proposed the initial idea, led the overall project direction, conducted most experiments, and primarily drafted the manuscript.The appendix describes these as his primary contributions.
  • APPENDIX: Other contributors provided technical expertise, research guidance, evaluation of the model and baselines, and discussion that refined the experiments and manuscript.The appendix attributes these roles across the named collaborators.
Loading 2507.02768v2…