PaperlayerView exact revision on arXiv

Source-linked AI summary

Alignment Is All You Need: Instruction-Free Training for General Audio-Language Models

Xuanru Zhou, Yiwen Shao, Jiahong Li, Dong Yu

arXiv:2608.18132v1cs.CLcs.SDeess.AS

TL;DR

This paper asks whether adapting an LLM to audio requires extensive task-specific post-training or whether alignment alone can transfer its capabilities. It freezes the audio encoder and LLM, training only a lightweight projector with instruction-free, automatically generated responses, and matches or surpasses heavily post-trained LALMs across four benchmarks using substantially less data.

  • Problem

    It remains unclear whether LLM reasoning and instruction-following capabilities can be transferred to a new modality through alignment alone rather than extensive task-specific supervision.

  • Method

    The method freezes the audio encoder and LLM, training only a lightweight projector with instruction-free audio-response pairs automatically expanded from captions.

  • Results

    Across MMAU, MMAR, MMSU, and MMAU-Pro, the approach matches or surpasses heavily post-trained LALMs using substantially less data, reaching 80.8 / 77.4 on MMAU Sound.

  • Takeaways & Limitations

    Competitive audio-language models can emerge from alignment alone, reducing multimodal construction to lightweight projector training within the evaluated setting.

  • Takeaways & Limitations

    The empirical study is limited to the audio domain and 7B scale, leaving validation on larger models and other modalities for future work.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) are typically built through a multi-stage pipeline consisting of cross-modal alignment, supervised fine-tuning (SFT), and preference optimization. This pipeline assumes that adapting an LLM to a new modality requires extensive task-specific supervision. However, pretrained LLMs already possess strong reasoning and instruction-following abilities. As LLMs evolve rapidly, an important question remains: can we efficiently transfer these capabilities to a new modality with minimal intervention, and is alignment alone sufficient for building a multimodal model? We introduce an Instruction-Free Alignment-Only large audio-language model (LALM) that keeps both the audio encoder and the LLM fully frozen, learning only a lightweight projector. Borrowing insights from AzeroS [1], we train on (audio, response) pairs from Self-Generated Data Construction, where an LLM expands captions into free-form responses without explicit task instructions. Across MMAU, MMAR, MMSU, and MMAU-Pro, our approach matches or surpasses heavily post-trained baselines using substantially less data. By keeping the LLM frozen, our model preserves its native instruction-following competence and can port seamlessly across model generations. Our results suggest that competitive MLLM can emerge from alignment alone, reducing multimodal extension to a lightweight projector-training problem that generalizes across modalities and adapts rapidly to each new LLM release.

1 Introduction

The paper argues that competitive audio-language models can be built through alignment alone, without updating the audio encoder or LLM. Its instruction-free, projector-only approach preserves the LLM’s instruction-following ability while reducing multimodal training to a lightweight alignment problem.

  • Motivation: Post-training can specialize an LLM toward task-specific supervision, eroding its universal decoding behavior and requiring expensive reruns for each new LLM generation.This motivates testing whether cross-modal alignment alone is sufficient.
  • Method: Self-Generated Data Construction feeds paired captions into the frozen LLM without instructions, using its responses as training targets instead of manually collected human responses.The caption acts as a semantic surrogate for audio, and projector training matches the LLM’s caption-conditioned response distribution.
  • Method: The proposed model freezes both the audio encoder and LLM, training only a lightweight projector without task instructions.The projector aligns audio representations directly to the LLM’s latent semantic space.
  • Findings: Instruction-free alignment-only training matches or surpasses heavily post-trained LALMs on MMAU, MMAR, MMSU, and MMAU-Pro using substantially less data.The approach generalizes across these evaluation benchmarks despite omitting supervised fine-tuning and preference optimization.
  • Findings: Keeping the LLM frozen preserves its pretrained instruction-following capability, while the automatic pipeline avoids manually curated QA pairs, task templates, and instruction taxonomies.Encoder and LLM ablations indicate that performance depends jointly on encoder representations and LLM capability rather than data scaling alone.

2 Related Work

Prior MLLM and LALM development has centered on modality alignment followed by task-specific instruction tuning, while newer methods seek to exploit pretrained LLM generalization and audio representations learned beyond fixed label spaces.

  • Multimodal alignment and instruction tuning: MLLMs commonly follow two stages: modality alignment and Task-Specific Instruction Tuning (TSIT), with RLHF and DPO used to align responses with human intent.This paradigm has also been adopted in the audio domain for building Large Audio-Language Models (LALMs).
  • Instruction-free and description-based alignment: Recent methods reduce reliance on TSIT by exploiting pretrained LLM generalization through cross-modal consistency, continuation-based alignment, and description-based training.AudioChatLLaMA enforces consistent responses across speech and text, BLSP uses continuation objectives, and DeSTA2 and DeSTA2.5 incorporate richer acoustic information through descriptions.
  • Audio representation learning: Audio representation learning comprises task-specific supervision, self-supervised learning, and other paradigms for developing general-purpose models across diverse audio understanding tasks.Task-specific learning supports event, speech, and speaker recognition but is limited by predefined label spaces, whereas self-supervised learning leverages large-scale unlabeled data to improve generalization.

3 Methodology

The method adapts a frozen audio encoder and frozen LLM by training only a projector, using instruction-free audio–response alignment data. This preserves pretrained instruction-following while seeking global alignment beyond the finite instruction sets used in SFT.

  • Alignment-only architecture: The model freezes the audio encoder and LLM, training only a projector that maps audio representations into the LLM’s input embedding space.The projector connects already learned modality-specific representations and language capabilities without retraining either component.
  • Alignment-only architecture: The LALM comprises a frozen encoder E, frozen autoregressive LLM L, and learnable position-wise projector Pθ producing an audio prefix.The LLM autoregressively generates the response from this continuous prefix, with no task instruction prepended.
  • Self-generated data construction: Training uses captions expanded into free-form responses, avoiding manual task selection, templates, and fixed task categories characteristic of standard SFT.Caption sources include ground-truth paired text and synthetic captions generated by Qwen3-Omni-Captioner [68].
  • Instruction-free training: The LLM receives only h = Pθ(E(x)) and is trained on response tokens from D_align, while the encoder and LLM remain frozen.The alignment-phase LLM must be identical to the expander used to construct the responses.
  • Alignment objective and limits: Instruction-free alignment targets complete alignment, stronger than partial alignment, by training the projector to support arbitrary test-time instructions rather than only instructions in T_train.The achievable ceiling still depends on encoder-retained mutual information, the frozen LLM’s response space, and alignment-data coverage.

4 Experiments

Experiments evaluate the alignment-only framework across four audio-understanding benchmarks and vary the audio encoder, LLM generation/alignment pairing, data scale, and caption source. Results show strong benchmark performance, encoder-dependent strengths, cross-generational portability, and nonuniform benefits from scaling or synthetic captions.

  • Evaluation Benchmarks: The evaluation spans MMAU, MMAR, MMSU, and MMAU-Pro, covering broad audio reasoning, mixed speech-sound-music understanding, speech-specific phenomena, and challenging expert-annotated skills.MMAU has 27 tasks and 10,000 clips; MMAR has 1,000 QA triplets; MMSU has 47 speech-focused tasks; MMAU-Pro evaluates three task types on instances under one minute.
  • Main Results: 80.8 / 77.4 on MMAU Sound lets AudioSet-Zipformer exceed Audio-Flamingo 3’s 79.6 / 75.8 despite training on 46× less data.Both compared instantiations use the frozen Qwen2.5-7B-Instruct backbone, identical projector recipe, and CaptionStew 400K with 10% added speech; they differ only in audio encoder.
  • Encoder Ablation: AudioSet-Zipformer leads MMAU Avg. at 68.2, MMAR at 54.3, and MMAU-Pro Avg. at 52.8, while Whisper-large-v2 performs best on speech-leaning MMAU Speech at 60.4 and MMSU at 50.6.Encoder strengths align with pretraining: AudioSet supervision favors sound and music, whereas ASR pretraining favors speech; joint audio-language encoders trail AudioSet-Zipformer by 7–13 points on MMAU Avg.
  • LLM Portability: 61.2 vs. 60.6 on MMAU Avg. shows Qwen2.5-7B-Instruct and Qwen3-8B remain in the same performance band under matched generation and alignment LLMs.The matched results are 48.4 vs. 49.8 on MMAR and 47.3 vs. 48.3 on MMAU-Pro Avg., supporting projector-only transfer across LLM generations.
  • Data Scaling: Data scaling produces task- and encoder-dependent trends: MMAU remains flat from CaptionStew 400K to 4M, while MMAR rises for three of four encoders.The supplied results characterize scaling as nonuniform rather than a shared monotonic curve across benchmarks.
  • Caption Sources: 12.6 points is the largest ground-truth-caption advantage on MMAU, whereas Qwen3-Omni AuT’s synthetic captions match ground truth on MMAU and MMAU-Pro and exceed it by +1.4 on MMAR and +4.6 on MMSU.Ground-truth captions generally lead for AudioSet-Zipformer, Whisper-large-v2, and Qwen2.5-Omni, but the advantage collapses for Qwen3-Omni AuT.

5 Limitation and Conclusion

The paper introduces an instruction-free, alignment-only recipe for LALMs that freezes the audio encoder and LLM while training only a lightweight projector, achieving competitive performance without extensive post-training. It concludes that alignment alone can simplify multimodal LLM construction, while acknowledging that the empirical study is limited to audio.

  • Conclusion: The method freezes the audio encoder and LLM, training only a lightweight projector for instruction-free LALM alignment.This recipe achieves competitive performance without extensive post-training.
  • Conclusion: Extensive ablations support the claim that competitive multimodal LLMs can emerge from alignment alone.This reduces multimodal LLM construction to a lightweight projector-training task.
  • Limitation: The empirical study is limited to the audio domain.The paper explicitly identifies this as a limitation of its evaluation.

A Detailed Analysis of SFT Failure Modes

The paper identifies two SFT failure modes that hinder general-purpose audio-language interfaces: instruction-induced feature collapse and modality neglect. SFT can suppress audio information irrelevant to a task while allowing the LLM to bypass auditory evidence through linguistic shortcuts.

  • A Detailed Analysis of SFT Failure Modes: SFT introduces instruction-induced feature collapse and modality neglect, hindering the development of a general-purpose audio-language interface.These are presented as two phenomena arising despite SFT’s effectiveness for optimizing specific tasks.
  • Instruction-Induced Feature Collapse: Task-conditioned gradients teach the projector to suppress audio features irrelevant to the target loss, including emotional prosody, background acoustics, and speaker identity.For automatic speech recognition, these dimensions do not contribute to transcript loss reduction, causing an irreversible collapse of information.
  • Modality Neglect (The Shortcut Problem): When instructions and responses correlate strongly, the LLM can model p(y | i) directly, bypassing the audio prefix hθ(x) and neglecting modality grounding.The resulting model follows task templates and linguistic patterns rather than the actual auditory evidence.

B The Information-Theoretical Mechanism of Global Alignment · C Zipformer Model · D Source Datasets for CaptionStew

The paper explains global alignment through information-theoretic distribution matching with a frozen LLM, then details the Zipformer-M encoder and CaptionStew source data used for alignment-only training. Freezing preserves a stable semantic reference and general reasoning priors while limiting adaptation to a lightweight projector.

  • B The Information-Theoretical Mechanism of Global Alignment: Standard SFT compresses intermediate representations toward task-specific labels, whereas high-entropy targets preserve richer information from the input.The paper frames this contrast through the Information Bottleneck principle.
  • B The Information-Theoretical Mechanism of Global Alignment: Global alignment matches audio and caption response distributions, anchoring the audio prefix to the LLM’s structured semantic manifold without task-specific guidance.A frozen LLM supplies the stable reference frame for this distributional matching.
  • B The Information-Theoretical Mechanism of Global Alignment: At test time, instructions act as directional perturbations on the aligned manifold, guiding the frozen LLM toward outputs through its pretrained reasoning paths.The framework requires no further adaptation after global alignment.
  • B The Information-Theoretical Mechanism of Global Alignment: Freezing the LLM provides a static semantic anchor, simplifies projector optimization, and avoids task-specific warping of its high-dimensional manifold.The paper presents this as an optimization-stability and parameter-efficiency choice.
  • B The Information-Theoretical Mechanism of Global Alignment: Freezing also preserves universal reasoning priors and prevents catastrophic forgetting when alignment data is much smaller than the original LLM training sets.The resulting model remains a universal decoder rather than becoming task-specialized.
  • C Zipformer Model: Zipformer-M uses six Transformer stages with progressively decreasing and increasing frame rates to capture fine- and coarse-grained temporal patterns efficiently on long sequences.Its U-Net-inspired architecture is selected for memory efficiency and strong performance across audio tasks.
  • C Zipformer Model: The encoder uses the 2,2,3,4,3,2 block configuration, fuses outputs at 25 Hz, and produces 768-dimensional frame-level embeddings.BiasNorm, Swoosh activations, and ScaledAdam compatibility are described as enhancements for stability and convergence.
  • C Zipformer Model: Zipformer-M is pretrained on AudioSet with multi-tag classification over 527 classes, and its resulting embeddings are used for alignment-only training.A linear classifier is added after the encoder during pretraining.

E Dataset Statistics

Table 6 catalogs the corpora used in the work, separating instruction-free projector-training data, an added speech mixture, and QA data for targeted SFT. Instruction-Free data pairs each clip with a caption or caption surrogate, while the other blocks serve distinct coverage and fine-tuning roles.

  • Dataset organization: Table 6 lists every corpus used in the work.The datasets are organized into Instruction-Free, speech corpora, and QA blocks.
  • Instruction-Free data: Instruction-Free data pairs each clip with a caption, transcript, or paralinguistic annotation as a caption surrogate for projector training.These pairs are trained under the instruction-free recipe.
  • Speech coverage: The speech corpora block appends a 10% speech mixture to each CaptionStew subset to increase speech coverage.This mixture is separate from the QA data used for targeted SFT.
  • Targeted SFT: The QA block is used only for targeted SFT in Sec. 4.6.5.It is not part of the instruction-free projector-training data.

F Configuration for Response Generation · G Message Templates and Decoding

Response generation expands captions into free-form targets using a prompted Qwen expander, with truncation and stochastic decoding controls. Training remains instruction-free, while evaluation uses task-specific message templates and beam-search decoding.

  • F Configuration for Response Generation: The expander g uses Qwen2.5-7B-Instruct or Qwen3-8B to transform each caption c into a free-form response r = g(c).The generated response serves as the training target.
  • F Configuration for Response Generation: Captions exceeding 7,500 tokenized tokens are tail-truncated, preserving space for responses within the 8,192-token context window.The caption is inserted into the user turn unmodified after truncation.
  • F Configuration for Response Generation: Generation samples with temperature 0.6, top-p 0.9, top-k 20, and max_new_tokens = 512; budget-exhausted outputs receive a <|truncated|> suffix for filtering.Filtered outputs are excluded before entering Dalign.
  • F Configuration for Response Generation: Serving uses vLLM on four NVIDIA A100 (40 GB) GPUs in bf16 with tensorparallel size 4, GPU-memory utilization 0.9, seed 1234, and batches of max_num_seqs = 32.CPU producers apply chat templates in parallel and feed one vLLM engine.
  • G Message Templates and Decoding: Alignment-Only training is instruction-free: the user turn contains only the audio token and the assistant turn contains the targeted response, without a system prompt.Optional speech-QA SFT prepends the question to the user turn while retaining the same two-turn structure.
  • G Message Templates and Decoding: Evaluation uses a multiple-choice template for closed-set items and a concise open-ended template for instruction-following and open-ended MMAU-Pro items.Both variants add a short system prompt and leave the assistant turn empty.
  • G Message Templates and Decoding: The four benchmarks covered by evaluation templates are MMAU, MMAR, MMSU, and MMAU-Pro, including its closed-QA and open-ended splits.Closed-set items force answer-only output, whereas open-ended items request concision.
  • G Message Templates and Decoding: Evaluation decodes with beam search using num_beams = 4 and max_tokens = 256, with per-batch max_duration = 500 audio seconds.Inference runs on one NVIDIA A100 (40 GB) in bf16 using the projector’s EMA training weights.

H Projector Downsampling Rate

Projector downsampling controls audio-token density, with the strongest results generally occurring when the input frequency falls within 6.25–12.5 Hz. Sweeps show that excessive density sharply harms performance, while sparser settings can improve instruction-following for some encoders.

  • Downsampling principle: The default selects downsampling r to place audio-token frequency in the 6.25–12.5 Hz band, despite native encoder rates ranging from 12.5 to 50 Hz.The projector downsamples encoder outputs along time before passing tokens to the LLM.
  • Whisper-large-v2: At r = 4, Whisper-large-v2 reaches MMAU 66.30 and MMSU 50.61, the section’s strongest result.This corresponds to 12.5 Hz audio-token frequency.
  • Whisper-large-v2: At 25 Hz, Whisper-large-v2’s MMAU collapses to 46.10, with speech dropping 29 points, music 20, and sound 12.These modality changes are 60.36 to 31.53 for speech, 62.87 to 42.81 for music, and 75.68 to 63.96 for sound.
  • Joint-pretrained encoders: Qwen2.5-Omni peaks at r = 2 with MMAU 59.0, while Qwen3-Omni’s AuT encoder peaks at r = 2 with MMAU 61.2.These optima correspond to 12.5 Hz and 6.25 Hz, respectively.
  • Joint-pretrained encoders: For Qwen3-Omni, r = 1 sacrifices 2 MMAU points but raises MMAU-Pro instruction-following from 67.74 to 77.43 and improves the speech subset by 1.8 points.The higher-density 12.5 Hz setting trades general MMAU for instruction-following performance.
  • Joint-pretrained encoders: AudioSet-Zipformer performs best at r = 8, corresponding to 6.25 Hz and four MMAU points above r = 4.Its native encoder frame rate is 50 Hz.

I Scaling Behavior (Cont’d) · J Effect of Caption Source (Cont’d)

Scaling alignment data largely saturates closed-set MMAU performance, while MMAU-Pro Open-ended and MMAR continue improving with scale. The caption-source analysis provides a per-subset comparison across encoders, caption sources, and benchmarks.

  • I Scaling Behavior (Cont’d): Alignment data scales from 576.8K to 4.18M samples while keeping Qwen2.5-7B-Instruct and the projector configuration fixed.All samples come from CaptionStew with 10% speech mixed in.
  • I Scaling Behavior (Cont’d): MMAU average changes by at most 2 points across the full 10× scale-up for every encoder, indicating headline saturation.Zipformer changes from 68.20 to 67.30, Whisper from 66.30 to 64.30, Qwen2.5-Omni from 59.00 to 59.60, and Qwen3-Omni from 61.20 to 61.40.
  • I Scaling Behavior (Cont’d): MMAU-Pro Open-ended rises from 50.84 to 67.52 on Zipformer and from 49.46 to 55.15 on Qwen2.5-Omni as data scales.MMAR also gains across three encoders: Zipformer +3.8, Qwen3-Omni +4.6, and Whisper +2.6.
  • I Scaling Behavior (Cont’d): MMAU Speech degrades with more data for Whisper, Qwen2.5-Omni, and Qwen3-Omni, while Zipformer holds.The reported changes are Whisper 60.36 to 52.55, Qwen2.5-Omni 53.31 to 49.75, and Qwen3-Omni 54.95 to 52.55; the fixed 10% speech mixture causes dilution as caption data grows.
  • I Scaling Behavior (Cont’d): The scaling results support an alignment-ceiling reading: richer-output tasks keep gaining, whereas MMAU’s closed-set ceiling is set by the encoder and LLM.MMAU-Pro Open-ended and MMAR benefit from scale, while closed-set MMAU remains saturated.
  • J Effect of Caption Source (Cont’d): Table 9 presents a per-subset breakdown of each encoder’s performance across different caption sources and benchmarks.This continues the caption-source analysis from Sec. 4.6.4 of the main text.

K Samples

The K Samples section presents three audio-caption-response examples illustrating how free-form responses elaborate diverse audio descriptions. The examples cover music, an engine with indistinct speech, and a woman’s voice amid clattering and crinkling sounds.

  • Rock ballad example: A rock-ballad caption describing a passionate male vocalist, medium tempo, bass, drums, and guitar receives a response elaborating vocal performance and instrumentation.The response suggests considering the vocalist’s range, emotion, and delivery across choruses and verses.
  • Vehicle engine example: A caption of a vehicle engine with a man’s indistinct voice receives a response proposing settings such as a parking lot or driveway.The response preserves the absence of discernible speech while expanding possible situational interpretations.
  • Background-noise example: A caption featuring a woman’s voice with clattering and crinkling background noise receives a response that interprets the sounds as potentially occurring during household activities.The response associates clattering with dishes or utensils and crinkling with paper or thin plastic.
Loading 2608.18132v1…