PaperlayerView exact revision on arXiv

Source-linked AI summary

Symbiotic Architecture for Post-Hoc Audio Extension of Frozen Language Models

Yotaro Kubo, Qi Sun, Yujin Tang

arXiv:2609.30784v1cs.SDcs.CLeess.AS

TL;DR

Conventional audio-language-model adaptation fine-tunes text-pretrained LLMs, risking catastrophic forgetting while audio prefilling scales with backbone size. This paper introduces an injector that writes audio-conditioned representations into a frozen LLM’s KV cache. It outperforms a frozen encoder-only baseline, approaches a fine-tuned reference on audio tasks, and preserves text-only performance by construction.

  • Problem

    Conventional ALMs fine-tune text-pretrained LLMs for audio, while backbone-based prefilling scales with LLM size and fine-tuning can degrade earlier capabilities.

    Source: §1 · p. 1

  • Method

    A narrower injector writes audio-conditioned, layer-wise KV-cache representations into the LLM without updating its weights.

    Source: §1 · p. 1

  • Results

    The symbiotic model substantially outperforms the frozen encoder-only baseline, especially on non-ASR tasks, approaches the fine-tuned reference, and preserves frozen-LLM text-task scores exactly.

    Source: §6 · p. 4, §5.3 · p. 4

  • Takeaways & Limitations

    The architecture decouples audio-injection cost from backbone width and preserves the backbone’s original text-only performance by construction.

    Source: §6 · p. 4

    Experiments used a single small backbone, so the scalability advantage remains to be verified with larger backbones.

    Source: §6 · p. 4

Abstract

from arXiv · show

This paper proposes an architecture for equipping large language models (LLMs) with audio-understanding capabilities without fine-tuning their weights. The proposed symbiotic architecture employs an injector module that writes audio-conditioned vectors directly into the target LLM's short-term memory, i.e., the key-value (KV) cache, enabling the LLM to behave as an audio language model (ALM). The architectural advantages are twofold. First, it improves the scalability of ALMs: because the proposed method bypasses the LLM during audio injection, the injection cost is governed by the injector width rather than the backbone width, and can therefore scale more slowly than the cost of full-backbone prefilling. Second, since the training scheme does not update the LLM weights, the original capabilities of the LLM are preserved without the risk of degradation from fine-tuning. The effectiveness of the proposed method is evaluated on both audio-understanding tasks (automatic speech recognition, audio question answering, and acoustic scene classification) and text-only tasks. We confirm that, while activating fewer parameters during audio prefilling, our architecture outperforms the conventional method with a frozen LLM and approaches the performance of a fine-tuned ALM, all while preserving the backbone LLM's original text-only task performance by construction.

1. INTRODUCTION

Conventional audio language models fine-tune a text-pretrained LLM, but audio prefilling scales with the backbone and fine-tuning risks catastrophic forgetting. The proposed symbiotic architecture injects audio-conditioned vectors into the LLM’s KV cache without updating its weights.

1. INTRODUCTION

  • ALMs combine an input encoder, adapter, and backbone LLM, which is typically fine-tuned to handle audio embeddings absent from its text pretraining.They target tasks including ASR, audio question answering, and acoustic scene classification.Source: §1 · p. 1
  • Audio prefilling in conventional ALMs is performed by the backbone LLM, so its cost scales with backbone size and long audio sequences make that cost substantial.The paper identifies decoupling prefilling from the backbone as important as LLMs grow.Source: §1 · p. 1
  • Audio-instruction fine-tuning can degrade capabilities acquired earlier, and the multimodal data mixtures commonly used to counter this make post-training more complicated and expensive.The paper calls this degradation catastrophic forgetting.Source: §1 · p. 1
  • The symbiotic architecture uses an injector to write audio representations into the LLM’s KV cache, optimizing this input-dependent, layer-wise context instead of updating backbone weights.Delegating prefilling to a narrower injector decouples its cost from backbone size and avoids degradation from audio fine-tuning.Source: §1 · p. 1

2. RELATED WORK

Related work explores modality-specific routing, audio subsampling, and lightweight or post-hoc adaptation. The paper distinguishes its injector-based approach by shifting audio prefilling away from the backbone LLM.

2. RELATED WORK

  • MoE models activate only a subset of parameters, and MoST routes tokens to modality-specific experts, but its backbone MoE framework leaves prefilling cost dominated by the backbone LLM.The proposed architecture shares the use of different modules by modality, with routing predetermined by input modality.Source: §2 · pp. 1–2
  • Figure 1 contrasts the conventional monolithic architecture with a symbiotic design accommodating a large decoder and a fast, small injector.In the monolithic design, the backbone transforms audio embeddings into the KV cache even after subsampling.Source: Figure 1 · p. 2
  • SSR-Connector and SALMONN use adaptive audio subsampling, techniques the proposed method could combine with to further improve prefilling efficiency.SSR-Connector aligns segments to text tokens; SALMONN uses Q-Former for fixed-length segment representations.Source: §2 · p. 2
  • SLM adapts audio tasks with a lightweight adapter while leaving LLM parameters unchanged, whereas Gemma 4 maps audio directly to LLM inputs through an affine transformation.The paper compares its method with SLM and reports limitations of SLM on generic non-speech audio tasks.Source: §2 · p. 2
  • LLaMA-Adapter also leaves LLM parameters unchanged by optimizing an image-conditioned KV prefix, but modifies the computation graph unlike the proposed approach.The paper aims to reuse existing backbone-LLM software infrastructure.Source: §2 · p. 2

3. SYMBIOTIC ARCHITECTURE

The symbiotic architecture transfers audio prefilling from the backbone LLM to an injector that writes directly to the LLM’s KV cache. This makes injection cost independent of backbone width while leaving the LLM weights untouched.

3. SYMBIOTIC ARCHITECTURE

  • In the monolithic architecture, the backbone LLM transforms audio embeddings into the KV cache, so audio prefilling remains computationally expensive even with subsampling.The audio sequence reaches response generation through the backbone’s KV cache.Source: §3 · p. 2
  • The symbiotic injector transforms audio embeddings into the KV cache, allowing a narrower injector whose injection cost does not scale with LLM width.This permits use of arbitrarily large LLMs without a proportional increase in injection cost.Source: §3 · p. 2
  • The injector acts as a context controller that introduces arbitrary vectors into the KV cache while keeping LLM weights untouched, adapting a text-trained LLM into an ALM without fine-tuning.The paper presents this as addressing both prefilling scalability and catastrophic forgetting.Source: §3 · p. 2

4. INJECTOR MODULE

The injector maps audio representations to layer-wise KV vectors through CNN and adapter blocks, with scale matching and randomized RoPE addressing compatibility and sequence-length robustness.

4. INJECTOR MODULE

  • The injector has the same number of layers as the backbone LLM because it must produce a KV cache for every LLM layer.It is a stacked CNN built from LConv modules.Source: §4 · p. 2, §4.1 · p. 2

4.1. CNN Block

  • The CNN block uses LConv modules with depthwise convolution between two pointwise dense layers, focusing on local audio context complementary to the backbone Transformer’s global context.The experiments use 2,048-dimensional inputs and a kernel size of 15; self-attention alternatives were unstable in preliminary experiments.Source: §4.1 · p. 2

4.2. Adapter Block

  • The adapter block subsamples CNN outputs by fixed-stride max pooling and maps them into the backbone’s key and value spaces with separate affine transformations.The fixed stride is 8, and RMSNorm is applied to key and value vectors for stabilization.Source: §4.2 · p. 2

4.3. KV Scale Matching

  • KV scale matching adjusts injector key and value magnitudes to match the backbone’s internal distributions using RMSNorm scales and additional head-wise rescaling.For Qwen3, key scales use corresponding backbone RMSNorm parameters, while value scales are initialized from calibration sentences designed to emulate audio-task instruction-response pairs.Source: §4.3 · p. 3

4.4. Noisy RoPE Training

  • Noisy RoPE randomizes position offsets and time scales during training to improve robustness to input lengths unseen during training.Conventional RoPE is recovered with τ = 0 and κ = 1; experiments use τMAX = 256, κMIN = 0.75, and κMAX = 1.5.Source: §4.4 · p. 3

5. EXPERIMENTS

Experiments compare Symbiotic with frozen-backbone and fine-tuned baselines across audio and text tasks. Symbiotic reduces audio-task errors relative to Encoder-only, approaches Monolithic while activating fewer prefilling parameters, and preserves the frozen LLM’s text-task scores; additional tests examine prefilling speed and long-utterance robustness.

5.1. Model Configuration

  • The experiments used WavLM-large with LoRA and Qwen3-0.6B; the baselines were Encoder-only with the same encoder and backbone, and Monolithic with LoRA on the backbone.The conventional models stack eight 1,024-dimensional audio representations before mapping them into the 1,024-dimensional LLM embedding space.Source: §5.1 · p. 3

5.2. Training Configuration

  • Training data covered ASR, AQA, and ASC; Clotho-AQA examples were retained when at least two annotators agreed, using the majority answer.LibriSpeech was speed-perturbed with 40% probability, and utterances longer than 25 seconds after perturbation were excluded; Table 2 lists the training datasets.Source: §5.2 · p. 3, Table 2 · p. 4

5.3. Experimental Results and Discussion

  • Symbiotic substantially reduced error rates versus Encoder-only, especially on non-ASR tasks, across ASR, AQA, and ASC evaluations.Table 1 reports WER for LibriSpeech and classification error rates for Clotho-AQA and CochlScene; it also identifies parameters involved in audio and text prefilling.Source: §5.3 · p. 4, Table 1 · p. 4
  • Symbiotic closely approached Monolithic while reducing parameters activated during audio prefilling from 1,108M to 552M.The remaining CochlScene gap may stem from the CNN-centric injector design.Source: §5.3 · p. 4
  • 157.1 seconds versus 199.8 seconds: Symbiotic prefilling was faster than Monolithic on LibriSpeech dev-other with an H100 and batch size 4.Although the injector requires more FLOPs in theory, the authors attribute the shorter wall-clock time to fewer kernel invocations and cache-friendly parameter size.Source: §5.3 · p. 4
  • 120.68% to 10.49%: noisy RoPE reduced long-utterance WER, and KV scale matching further reduced it to 4.70%.The long-utterance set comprised 86 utterances longer than 25 seconds, excluded from training; the authors report gains in robustness to unseen sequence lengths.Source: Table 4 · p. 4, §5.3 · p. 4
  • Because its backbone stays frozen, Symbiotic’s text-only scores match Frozen LLM by construction, while audio-instruction fine-tuning caused severe catastrophic forgetting.The text evaluation used WikiText-2 and HellaSwag for next-token prediction and GSM8K for reasoning.Source: §5.3 · p. 4, Table 3 · p. 4

6. CONCLUSIONS

The symbiotic architecture connects an audio encoder to a frozen backbone LLM through its KV cache, decoupling audio-injection cost from backbone width. In compact experiments it approached a fine-tuned reference on audio tasks while preserving text-only performance, but larger-backbone scalability remains unverified.

6. CONCLUSIONS

  • In the compact regime, the architecture approached a fine-tuned reference on audio tasks while activating fewer parameters during audio prefilling and preserving text-only performance exactly.Among frozen-backbone methods, it also substantially outperformed the encoder-only SLM baseline, especially on non-speech audio tasks.Source: §6 · p. 4
  • The architecture couples an audio encoder with a backbone LLM through its KV cache, decoupling audio-injection cost from backbone width without updating the LLM.Keeping the backbone unchanged also eliminates catastrophic forgetting by construction.Source: §6 · p. 4
  • Experiments used a single small backbone, so the scalability advantage with larger backbones remains to be verified.Source: §6 · p. 4
Loading 2609.30784v1…