PaperlayerView exact revision on arXiv

Source-linked AI summary

Alignment Is All You Need: Instruction-Free Training for General Audio-Language Models

Xuanru Zhou, Yiwen Shao, Jiahong Li, Dong Yu

arXiv:2608.18132v1cs.CLcs.SDeess.AS

TL;DR

Whether alignment alone can transfer pretrained LLM capabilities to audio without extensive task-specific supervision remains an open question. We freeze the audio encoder and LLM, train only a projector using instruction-free audio-response pairs, and match or surpass heavily post-trained LALMs across four benchmarks with substantially less data.

  • Problem

    Whether alignment alone can build competitive LALMs, given current pipelines’ reliance on later SFT and preference optimization, remains an open question.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    The approach freezes the audio encoder and LLM, training only a lightweight projector with instruction-free audio-response pairs generated by expanding captions into free-form responses.

    Source: §1 · p. 2

  • Results

    Across MMAU, MMAR, MMSU, and MMAU-Pro, the model matches or surpasses heavily post-trained LALMs with substantially less data, scoring 52.8 on MMAU-Pro Avg., highest among listed open-source LALMs.

    Source: §1 · p. 2, §4.5 · p. 7

  • Takeaways & Limitations

    The findings suggest multimodal LLM construction can be simplified to lightweight projector training within the demonstrated audio setting.

    Source: §5 · p. 9

    Evidence is limited to audio models at 7B scale, and alignment cannot recover missing encoder information or add capabilities absent from the base model.

    Source: §5 · p. 9

Abstract

from arXiv · show

Multimodal large language models (MLLMs) are typically built through a multi-stage pipeline consisting of cross-modal alignment, supervised fine-tuning (SFT), and preference optimization. This pipeline assumes that adapting an LLM to a new modality requires extensive task-specific supervision. However, pretrained LLMs already possess strong reasoning and instruction-following abilities. As LLMs evolve rapidly, an important question remains: can we efficiently transfer these capabilities to a new modality with minimal intervention, and is alignment alone sufficient for building a multimodal model? We introduce an Instruction-Free Alignment-Only large audio-language model (LALM) that keeps both the audio encoder and the LLM fully frozen, learning only a lightweight projector. Borrowing insights from AzeroS [1], we train on (audio, response) pairs from Self-Generated Data Construction, where an LLM expands captions into free-form responses without explicit task instructions. Across MMAU, MMAR, MMSU, and MMAU-Pro, our approach matches or surpasses heavily post-trained baselines using substantially less data. By keeping the LLM frozen, our model preserves its native instruction-following competence and can port seamlessly across model generations. Our results suggest that competitive MLLM can emerge from alignment alone, reducing multimodal extension to a lightweight projector-training problem that generalizes across modalities and adapts rapidly to each new LLM release.

1 Introduction

The paper argues that competitive general-audio language modeling can emerge from instruction-free alignment alone, with the audio encoder and LLM frozen and only a projector trained. It reports matching or surpassing heavily post-trained baselines with less data while preserving instruction-following ability.

1 Introduction

  • The paper motivates alignment-only training by noting that task-specific post-training can erode universal decoding behavior and must be rerun for each new LLM generation.In current practice, supervised fine-tuning and preference optimization are credited with most performance gains, while alignment is treated mainly as initialization.Source: §1 · pp. 1–2
  • The approach freezes the audio encoder and LLM, trains only a lightweight projector, and uses caption-conditioned free-form responses without instructions or human-curated supervision [1].Captions serve as semantic surrogates for audio; the same frozen LLM generates responses and receives the projected audio during training.Source: §1 · p. 2, Figure 1 · p. 2
  • Instruction-free alignment-only training matches or surpasses heavily post-trained LALMs on MMAU, MMAR, MMSU, and MMAU-Pro with substantially less data.The paper presents this as a central finding and contribution.Source: §1 · pp. 2–3
  • Keeping the LLM frozen preserves pretrained instruction-following ability, yielding state-of-the-art results among open-source LALMs on MMAU-Pro’s instruction-following split.The recipe also transfers across LLM generations under a matched-generator condition.Source: §1 · p. 2
  • Performance is jointly bounded by encoder informativeness and LLM capability: more alignment data mainly helps open-ended reasoning, while closed-set recognition saturates when encoder limits dominate.The saturation point depends on data coverage and diversity, and benchmark-leading encoders track the capabilities emphasized during pretraining.Source: §1 · pp. 2–3
  • Dense synthetic captions are not essential per sample; simple open-source captions suffice when aggregated across diverse datasets.This finding supports the approach’s reliance on automatically constructed training data rather than manually curated QA pairs or instruction templates.Source: §1 · pp. 2–3

2 Related Work

Prior audio-language models commonly combine modality alignment with task-specific instruction tuning, while instruction-free approaches seek to use pretrained LLMs’ generalization instead. Audio representation learning spans supervised, self-supervised, and audio-language pretraining, with this work evaluating alignment-only training across five frozen encoders.

2.1 Multimodal Alignment and Instruction Tuning

  • Audio MLLMs commonly pair modality alignment with task-specific instruction tuning, but prompt-distribution overfitting can produce template-driven responses, including ASR-style transcriptions for non-speech audio.RLHF and DPO are preference-alignment approaches; audio systems include Qwen-Audio, Qwen2-Audio, SALMONN, and WavLLM.Source: §2.1 · p. 3
  • Recent approaches leverage pretrained LLM generalization through cross-modal response consistency, continuation objectives, or synthetic description targets, while Shao et al. [1] formalizes instruction-free speech training.AudioChatLLaMA aligns speech and text responses, BLSP uses continuation-based alignment, and DeSTA2/DeSTA2.5 combine transcripts with paralinguistic metadata to generate targets.Source: §2.1 · p. 3

2.2 Audio Representation Learning

  • Audio representation learning includes task-specific supervised, self-supervised, and audio-language pretraining paradigms, and this work evaluates alignment-only training across five frozen encoders from these paradigms.The passage notes limitations in predefined label spaces, domain specialization, task coverage, and caption supervision, alongside efforts to expand audio coverage and improve caption quality.Source: §2.2 · p. 3

3 Methodology

The method freezes the audio encoder and LLM while training a projector to connect audio representations to the LLM, using instruction-free responses generated from captions. Its theoretical framing treats broad generalization as a goal of global alignment, with realization depending on alignment-data coverage.

3.1 Motivation and Problem Setup

  • The model freezes the audio encoder and LLM, training only a projector that maps audio features into the LLM’s embedding space to elicit responses without task instructions.The projector is the system’s only trainable component.Source: §3.1 · p. 4

3.2 Self-Generated Data Construction

  • Caption-based data construction uses dataset text or synthetic captions as audio surrogates, then expands each caption into a fluent, free-form response without manual task templates.Synthetic captions come from Qwen3-Omni-Captioner [68].Source: §3.2 · p. 4

3.3 Instruction-Free Alignment-Only Training

  • Training feeds only the audio prefix, supervises response tokens from Dalign, and applies causal language-modeling loss with gradients updating only the projector; the expander and alignment LLM must be identical.The encoder and LLM remain frozen, preserving pretrained priors and enabling rapid convergence.Source: §3.3 · pp. 4–5

3.4 Theoretical Analysis

  • The approach assumes modern LLMs can perform nearly any task when given a faithful textual representation, making global input alignment—not new reasoning paths—the adaptation goal.The paper characterizes modern LLMs as universal decoders [6].Source: §3.4 · p. 5

3.4.1 Alignment Completeness

  • Complete alignment requires audio and caption prefixes to elicit identical response distributions without instructions; the objective is intended to generalize beyond SFT’s training instructions to any test-time instruction.The formal definition equates the two prefixes’ output distributions under an instruction, while complete alignment uses no instruction.Source: §3.4.1 · p. 5

3.4.2 Information-Capability Bound

  • Realizing the frozen LLM’s capability depends on alignment-data coverage: audio diversity must span encoder features, and response richness must span the LLM’s semantic behaviors.The projector reduces the information-side slack, while freezing the LLM preserves its response-space breadth.Source: §3.4.2 · p. 5

4 Experiments

The experiments evaluate a frozen-encoder, frozen-LLM audio-language model trained through a lightweight projector across four audio-understanding benchmarks. The final comparison and ablations show how performance varies with encoder, LLM matching, data scale, caption source, and targeted speech tuning.

4.1 Model Architecture; 4.2 Dataset; 4.3 Training Configuration

  • The setup freezes the audio encoder and LLM and trains only a lightweight projector, using CaptionStew subsets augmented with 10% speech data.For Whisper-large-v2 + Qwen2.5-7B, the projector has 31.2M trainable parameters; the reported 400K-subset run uses 600k steps and takes about 46 hours on an 8-GPU A100 node.Source: §4.3 · p. 6

4.4 Evaluation Benchmarks

  • The four benchmarks span broad audio reasoning, mixed-media reasoning, speech understanding, and challenging long-form or instruction-following tasks.MMAU has 10,000 QA pairs across 27 tasks; MMAR has 1,000 QA triplets across four reasoning layers; MMSU has 5,000 pairs across 47 speech tasks; and MMAU-Pro includes 5,305 instances across 49 skills, with evaluation restricted to clips under one minute.Source: §4.4 · pp. 6–7

4.5 Main Results

  • AudioSet-Zipformer leads open-source LALMs on MMAU-Pro Avg. at 52.8, while the Whisper variant reaches 60.4 / 60.1 on MMAU Speech and 50.6 on MMSU.AudioSet-Zipformer also scores 80.8 / 77.4 on MMAU Sound, ahead of Audio-Flamingo 3’s 79.6 / 75.8 despite using 46× fewer samples and 34× less audio; it is on par with GPT-4o-Audio on MMAU-Pro Avg. (52.8 vs. 52.5). It trails Audio-Flamingo 3 and Qwen2.5-Omni on MMAU test Avg., while Whisper reaches 55.4 on MMAU-Pro open-ended response and the models top open-source LALMs on MMAU-Pro IF (62.9 / 72.6 vs. 61.3).Source: Table 1 · p. 7, §4.5 · p. 7
  • Speech and MMAR remain weaknesses against stronger systems: the Whisper variant trails ALARM and Qwen2.5-Omni by 10–17 points on speech subsets, and AudioSet-Zipformer scores 54.3 on MMAR.AudioSet-Zipformer is within 1.1 points of Gemini 2.5 Flash on MMAU test Avg. (66.3 vs. 67.4) and exceeds GPT-4o-Audio (60.8) by 5.5, but trails Gemini by 14.1 and Audio-Flamingo 3 by 4.2 on MMAR.Source: §4.5 · p. 7, Table 1 · p. 7

4.6.1 Effect of Audio Encoder

  • With the LLM, training data, and projector fixed, AudioSet-Zipformer leads on MMAU Avg. (68.2), MMAR (54.3), and MMAU-Pro Avg. (52.8).Whisper-large-v2 is strongest on speech-leaning subsets, while the joint audio-language encoders trail AudioSet-Zipformer by 7–13 points on MMAU Avg.; benchmark winners track the axis emphasized during encoder pretraining.Source: §4.6.1 · pp. 7–8, Table 2 · p. 8

4.6.2 Effect of LLM Backbone

  • Matched response generation and alignment yield similar results with Qwen2.5-7B-Instruct and Qwen3-8B, but mismatching them lowers Qwen3-8B’s MMAU Avg. by 5.9 points and MMAU-Pro Avg. by 7.8.Under matched settings, MMAU Avg. is 61.2 vs. 60.6, MMAR is 48.4 vs. 49.8, and MMAU-Pro Avg. is 47.3 vs. 48.3; the results support porting the recipe to newer LLMs through projector retraining when the generator matches the alignment LLM.Source: §4.6.2 · p. 8, Table 3 · p. 8

4.6.3 Scaling Behavior

  • Scaling CaptionStew from 400K to 4M samples leaves MMAU flat across encoders, raises MMAR for three encoders, and produces no shared trend on MMSU or MMAU-Pro Avg.The reported interpretation is that additional alignment data primarily benefits richer open-ended reasoning, while closed-set recognition is constrained by encoder representations and gains diminish after minimal alignment data.Source: §4.6.3 · p. 8, Figure 2 · p. 9

4.6.4 Effect of Caption Source

  • Caption-source effects depend on the encoder: synthetic captions match ground truth for Qwen3-Omni AuT on MMAU and MMAU-Pro and exceed it by 1.4 on MMAR and 4.6 on MMSU.Ground-truth captions retain a small consistent advantage for AudioSet-Zipformer, Whisper-large-v2, and Qwen2.5-Omni, including a 12.6-point MMAU gap for Qwen2.5-Omni.Source: §4.6.4 · p. 9, Figure 2 · p. 9

5 Limitation and Conclusion

The paper presents an alignment-only LALM recipe that freezes the audio encoder and LLM while training a lightweight projector, achieving competitive performance without extensive post-training. Its evidence is limited to audio and 7B models, and performance remains constrained by the base model and the semantic focus of caption/response supervision.

5 Limitation and Conclusion

  • Freezing the audio encoder and LLM while training only a lightweight projector yields competitive LALM performance without extensive post-training.The authors conclude that alignment alone can simplify multimodal LLM construction to a lightweight projector-training task.Source: §5 · p. 9
  • The empirical study covers only the audio domain and 7B scale, leaving larger models and other modalities for future validation.The authors identify both broader modality coverage and scaling to larger models as open validation needs.Source: §5 · p. 9
  • Alignment cannot overcome the base model’s performance ceiling by recovering missing encoder information or creating new LLM capabilities.Source: §5 · p. 9
  • Caption/response supervision primarily captures semantic information; richer super-semantic signals may require specialized QA supervision or stronger representations.Source: §5 · p. 9

A Detailed Analysis of SFT Failure Modes

SFT can optimize specific audio tasks but may hinder a general-purpose audio-language interface through instruction-induced feature collapse and modality neglect.

Instruction-Induced Feature Collapse

  • Task-conditioned SFT makes the projector suppress audio features irrelevant to the target task, such as prosody, background acoustics, and speaker identity.For example, ASR training [46] rewards transcript-relevant information, so the model may lose features needed to analyze a speaker’s mood at test time.Source: A Detailed Analysis of SFT Failure Modes · p. 16
  • This feature pruning is irreversible: once SFT removes dimensions that do not help its training task, the model cannot recover them for unseen tasks.The passage illustrates this with mood analysis after training that filters out speaker-related cues.Source: A Detailed Analysis of SFT Failure Modes · p. 16

Modality Neglect (The Shortcut Problem)

  • When instructions and target responses are highly correlated, a high-capacity LLM may model p(y | i) [89] and bypass the audio prefix hθ(x).This shortcut prioritizes task templates and linguistic patterns over grounding generation in auditory evidence.Source: A Detailed Analysis of SFT Failure Modes · p. 16

B The Information-Theoretical Mechanism of Global Alignment · C Zipformer Model · D Source Datasets for CaptionStew

High-entropy responses support broad audio-information preservation and alignment to a stable frozen-LLM semantic manifold. The sections also describe the Zipformer-M encoder and CaptionStew sources used in experiments.

B The Information-Theoretical Mechanism of Global Alignment

  • High-entropy LLM-expanded responses make alignment preserve richer acoustic information than narrow task labels, because the projector must maximize I(hθ(x); x) under the Information Bottleneck account [91, 92].Source: B The Information-Theoretical… · p. 16
  • Matching audio prefixes to faithful-caption coordinates on the frozen LLM’s semantic manifold creates a stable reference that lets test-time instructions steer pretrained reasoning without adaptation.Source: B The Information-Theoretical… · pp. 16–17
  • Freezing the LLM avoids alignment-corpus-driven manifold warping and catastrophic forgetting, preserving universal reasoning priors while enabling competitive performance with minimal computation and rapid convergence.Source: B The Information-Theoretical… · p. 17

C Zipformer Model

  • Zipformer-M [73] uses six U-Net-inspired Transformer stages at 50, 25, 12.5, 6.25, 12.5, and 25 Hz, with residual and upsampling connections to capture multi-resolution temporal patterns.Source: C Zipformer Model · p. 17
  • Its 2,2,3,4,3,2 stage blocks fuse at 25 Hz into 768-dimensional embeddings; BiasNorm, Swoosh, and ScaledAdam compatibility support stable long-sequence training.Source: C Zipformer Model · p. 17
  • The encoder is pretrained on AudioSet with 527-class multi-tag classification, and its embeddings feed alignment-only training.Source: C Zipformer Model · p. 17

D Source Datasets for CaptionStew

E Dataset Statistics

The dataset statistics distinguish instruction-free alignment data, added speech corpora, and QA data reserved for targeted SFT.

E Dataset Statistics

  • Instruction-Free pairs clips with captions or transcript/paralinguistic surrogates; a 10% speech mixture augments each CaptionStew subset, while QA data is used only for targeted SFT in Sec. 4.6.5.Table 6 lists every corpus used in the work.Source: E Dataset Statistics · p. 17

F Configuration for Response Generation · G Message Templates and Decoding

The appendices specify how captions are expanded into free-form training targets and define the training, evaluation, and decoding configurations used for the audio-language model. Together, they distinguish instruction-free alignment training from optional speech-QA SFT and from the two evaluation prompt styles.

F Configuration for Response Generation

  • The response-generation stage expands each caption c into a free-form response r = g(c), which serves as the training target.Examples of audio, captions, and generated responses are provided in Appendix K.Source: F Configuration for Response Generation · p. 17
  • Qwen2.5-7B-Instruct or Qwen3-8B acts as the expander, prompted as a listener with the unmodified caption as the user turn.Captions longer than 7,500 tokens are tail-truncated to leave room for the response within the 8,192-token context window.Source: F Configuration for Response Generation · p. 18
  • Response generation samples with temperature 0.6, top-p 0.9, top-k 20, and max_new_tokens = 512.Outputs that reach the token budget receive a <|truncated|> suffix and can be filtered before entering Dalign.Source: F Configuration for Response Generation · p. 19
  • Generation runs in bf16 under vLLM on four NVIDIA A100 (40 GB) GPUs with tensor parallelism size 4 and GPU-memory utilization 0.9.CPU producers apply the chat template in parallel; one vLLM engine batches up to max_num_seqs = 32 prompts, using seed 1234.Source: F Configuration for Response Generation · p. 19

G Message Templates and Decoding

  • Alignment-Only training uses an audio-token user turn and a targeted-response assistant turn, without a system prompt; optional speech-QA SFT prepends the question to the user turn.Both training variants retain the same two-turn shape.Source: G Message Templates and Decoding · p. 19
  • Evaluation uses a multiple-choice template for MMAU, MMAR, MMSU, and MMAU-Pro closed-QA, and a concise open-ended template for instruction-following and MMAU-Pro open-ended splits.Both templates add a short system prompt and leave the assistant turn empty; the multiple-choice prompt requests only the answer, while the other asks for a concise response.Source: G Message Templates and Decoding · p. 19
  • Evaluation decodes with beam search using num_beams = 4, max_tokens = 256, and per-batch max_duration = 500 audio seconds.Inference uses bf16 on one NVIDIA A100 (40 GB) and the projector’s EMA training weights.Source: G Message Templates and Decoding · p. 20

H Projector Downsampling Rate

The sweep shows that audio-token frequency, rather than downsampling factor itself, governs performance: every encoder’s best setting falls within 6.25–12.5 Hz, with benchmark-dependent preferences inside that band.

H Projector Downsampling Rate

  • Every encoder performs best at 6.25–12.5 Hz audio-token frequency; within that band, speech and instruction-following favor denser tokens, while sound and music favor sparser tokens.Rows outside the band degrade, with excessive density causing sharper losses than density below the band.Source: H Projector Downsampling Rate · p. 20
  • At r = 4 (12.5 Hz), Whisper-large-v2 reaches MMAU 66.30 and MMSU 50.61; doubling density to 25 Hz drops MMAU to 46.10.At 25 Hz, speech drops 29 points (60.36 to 31.53), music 20 (62.87 to 42.81), and sound 12 (75.68 to 63.96).Source: H Projector Downsampling Rate · p. 20
  • Qwen2.5-Omni peaks at r = 2 (12.5 Hz, MMAU 59.0), while Qwen3-Omni AuT peaks at r = 2 (6.25 Hz, MMAU 61.2).For Qwen3-Omni, r = 1 trades 2 MMAU points for MMAU-Pro instruction-following of 77.43 versus 67.74 and a 1.8-point speech-subset gain.Source: H Projector Downsampling Rate · p. 20
  • AudioSet-Zipformer performs best at r = 8, corresponding to 6.25 Hz from its 50 Hz native rate, and scores four MMAU points above r = 4.Source: H Projector Downsampling Rate · p. 20

I Scaling Behavior (Cont’d) · J Effect of Caption Source (Cont’d)

Scaling alignment data leaves MMAU largely saturated while MMAU-Pro Open-ended and MMAR continue to improve; the caption-source section reports a per-subset, per-encoder breakdown across sources.

I Scaling Behavior (Cont’d)

  • The scaling comparison uses CaptionStew at 576.8K samples (1.6Kh), 1.18M (3.7Kh), and 4.18M (14.2Kh), with 10% speech mixed in.Qwen2.5-7B-Instruct and the projector configuration stay fixed; Table 8 reports full scaling results across encoders and data sizes.Source: I Scaling Behavior (Cont’d) · p. 20, Table 8 · p. 21
  • MMAU average drifted by at most 2 points over 10× scaling: Zipformer 68.20 to 67.30, Whisper 66.30 to 64.30, Qwen2.5-Omni 59.00 to 59.60, and Qwen3-Omni 61.20 to 61.40.Sound and Music subscores show similar saturation; Speech moves more and is treated separately.Source: I Scaling Behavior (Cont’d) · p. 20, Table 8 · p. 21
  • MMAU-Pro Open-ended rises on Zipformer (50.84 to 67.52) and Qwen2.5-Omni (49.46 to 55.15), while MMAR gains across three encoders: Zipformer +3.8, Qwen3-Omni +4.6, and Whisper +2.6.Both benchmarks continue gaining after MMAU saturates, rewarding generative answers or event-sequence reasoning.Source: Table 8 · p. 21, I Scaling Behavior (Cont’d) · p. 21
  • MMAU Speech fell with scale for Whisper (60.36 to 52.55), Qwen2.5-Omni (53.31 to 49.75), and Qwen3-Omni (54.95 to 52.55), while Zipformer held.The 10% speech mixture stays fixed in proportion, so absolute speech samples grow; the paper interprets the decline as dilution relative to the expanding caption-style corpus, not under-coverage.Source: I Scaling Behavior (Cont’d) · p. 21, Table 8 · p. 21
  • The alignment-ceiling interpretation is that data scales richer-output audio understanding, while the closed-set MMAU ceiling depends on the encoder and LLM rather than alignment data.The richer-output benchmarks identified are MMAU-Pro Open-ended and MMAR.Source: I Scaling Behavior (Cont’d) · p. 21

J Effect of Caption Source (Cont’d)

K Samples

The samples illustrate how free-form responses expand short audio captions into detailed descriptions, creative suggestions, and plausible contextual interpretations.

K Samples

  • The rock-ballad response turns the caption’s musical details into suggestions about vocals, instrumentation, structure, lyrics, production, and emotional themes.It frames these as elements to consider when further developing or describing the song.Source: K Samples · p. 22
  • The vehicle-audio response proposes parking-lot, driveway, and service-area scenarios, while noting that the exact setting cannot be determined from the clip.The caption describes an engine and a man’s voice but says no speech content is discernible.Source: K Samples · p. 23
  • The response to the woman’s voice and background clattering and crinkling elaborates possible household sounds and settings while retaining the caption’s neutral atmosphere.It presents these contexts as possibilities, such as kitchen activity or household chores.Source: K Samples · p. 23

Related papers

Loading 2608.18132v1…