PaperlayerView exact revision on arXiv

Source-linked AI summary

In-Context Adapter Generation for Compositional Generalization in Large Audio Language Models

Yiwei Guo, Siqi Huang, Bohan Li, Jing Peng, Hankun Wang, Zhihan Li, Kai Yu

arXiv:2610.34413v1eess.AScs.AIcs.SD

TL;DR

Whether LALMs can systematically recombine familiar capabilities and task schemas into unseen compositions remains underexplored. The paper introduces T2L-ICAG, which generates instruction-conditioned LoRA adapters from token-level task information, and reports consistent gains over static LoRA and T2L-emb across two LALM backbones.

  • Problem

    Whether LALMs can transfer familiar capabilities and task schemas to novel compositions without direct training examples remains unclear.

    Source: Introduction · p. 1

  • Method

    T2L-ICAG extends Text-to-LoRA to LALMs, using a token-level Transformer hypernetwork to generate instruction-conditioned LoRA adapters.

    Source: Introduction · pp. 1–2

  • Results

    Across multiple compositional task schemas, T2L-ICAG consistently outperforms static LoRA and T2L-emb, with clear gains on structurally demanding tasks.

    Source: Conclusion · p. 7

  • Takeaways & Limitations

    The results suggest that instruction-conditioned adapter generation is a promising direction for improving compositional generalization in LALMs.

    Source: Conclusion · p. 7

    Robust generalization to unseen task compositions remains an open challenge.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

Large audio language models (LALMs) have achieved strong performance on standard speech and audio understanding tasks, yet their ability to systematically generalize to novel task compositions remains underexplored. Taking a meta-learning view, we extend Text-to-LoRA (T2L) to LALMs, generating instruction-conditioned LoRA adapters with a Transformer hypernetwork rather than learning a single fixed adapter shared across instructions. To exploit fine-grained task information, we introduce in-context adapter generation (ICAG), a token-level hypernetwork architecture; we call the resulting method T2L-ICAG. We evaluate this approach on two representative LALMs using SIFT-50M for supervised fine-tuning and construct a controlled compositional evaluation suite covering five task classes. Across two LALM backbones, T2L-ICAG consistently improves over static LoRA and also the global embedding-based T2L baseline. Coverage and capacity analyses further show that these gains are larger on structurally out-of-domain tasks and cannot be explained by parameter count alone. These results suggest that generating task-specific adapters from instructions is a promising direction for improving compositional generalization of LALMs.

Introduction

The paper investigates whether LALMs can recombine familiar acoustic capabilities and task schemas into unseen compositions, a question that remains underexplored in audio. It extends instruction-conditioned LoRA generation to LALMs and introduces token-level ICAG; across two backbones, the approach improves over static LoRA and global embedding-based T2L.

  • The study asks whether LALMs can systematically recombine familiar capabilities and task schemas into novel compositions without direct training examples of those compositions.Source: Introduction · p. 1
  • Audio composition requires interpreting novel instruction structures and binding requested functionalities to the appropriate acoustic evidence and inputs, while exhaustive composition training is costly.Source: Introduction · p. 1
  • The paper extends Text-to-LoRA to LALMs, generating instruction-conditioned LoRA weights with a hypernetwork instead of training a static adapter.Source: Introduction · p. 1
  • T2L-ICAG uses a bidirectional Transformer hypernetwork that attends to instruction tokens, preserving fine-grained task information for generating adapters for unseen compositions.Source: Introduction · p. 2
  • Experiments use Kimi-Audio and Qwen2.5-Omni with SIFT-50M, and T2L-ICAG outperforms the global embedding-based T2L-emb baseline.Source: Introduction · p. 2
  • Across two LALM backbones, T2L-ICAG outperforms static LoRA and T2L-emb, with larger gains on structurally novel tasks that cannot be explained by training-data overlap or parameter count alone.Source: Introduction · p. 2

Background

The background situates the work in audio-language modeling, compositional generalization, and hypernetwork-based meta-learning. It notes that audio LALMs are capable multi-task models, while systematic generalization and hypernetwork adaptation remain underexplored in this setting.

Large Audio Language Models

  • LALMs combine audio encoders and decoder-only language models to support audio and speech tasks such as recognition, translation, captioning, and reasoning.Source: Large Audio Language Models · p. 2

Compositional Generalization

  • Compositional generalization is transfer to novel structures built from familiar components; it remains underexplored in audio LALMs, and complex audio processing alone does not establish it.Source: Compositional Generalization · p. 2

Meta-Learning and LoRA Hypernetworks

  • Hypernetworks generate model parameters for meta-learning, including LoRA weights, but meta-learning and hypernetwork adaptation remain largely underexplored for LALMs.Source: Meta-Learning and LoRA Hypernetworks · p. 2

Method

The method replaces directly optimized LoRA updates with adapters generated by an instruction-conditioned hypernetwork. Its ICAG architecture uses instruction tokens and layer-module queries in a shared Transformer context to produce fine-grained LoRA weights.

Formulation of LoRA Hypernetworks

  • Instead of directly optimizing adapter updates, the hypernetwork generates them and is trained using the same supervised fine-tuning criterion.Training gradients from cross-entropy loss flow from the adapter weights back to the hypernetwork parameters.Source: Formulation of LoRA Hypernetworks · pp. 2–3
  • Each layer-module pair receives a generated LoRA adapter, with the hypernetwork taking the instruction and pair indices as inputs.The full adapter collection is generated across target layers and modules.Source: Formulation of LoRA Hypernetworks · pp. 2–3

Hypernetwork Architecture

  • T2L-ICAG replaces T2L’s global sentence-embedding MLP with a Transformer hypernetwork for generating instruction-specific adapters.The paper describes the global embedding-based T2L approach as suboptimal for LALMs.Source: Hypernetwork Architecture · p. 3
  • ICAG combines instruction-token embeddings with learnable layer-module query tokens in a bidirectional Transformer encoder.The encoder’s layer-module prefix states are passed through module-specific linear heads to generate the LoRA matrices.Source: Hypernetwork Architecture · p. 3
  • The backbone and its input embedding matrix remain frozen; only the hypernetwork is updated, and adapter queries attend directly to instruction tokens.This formulation also removes the need for a pretrained sentence-embedding model, reducing computational cost relative to Charakorn et al. (2025).Source: Hypernetwork Architecture · p. 3

Data and Tasks

The evaluation combines SIFT-50M training with composed speech-understanding tasks and audits overlap between evaluation tasks and training data. The audit finds substantially more out-of-domain tasks under exact-form than primitive-level coverage.

Data and Tasks

  • Training uses about 4.5 million English, closed-ended SIFT-50M samples from the acoustic-level, comparison, and content-level partitions.SIFT-50M contains 50 million multilingual speech-text pairs with natural-language instructions.Source: Data and Tasks · p. 3

Compositional Tasks for Evaluation

  • Evaluation uses EvalSIFT’s acoustic-level, comparison, and content-level partitions, with 500 samples in each partition.The composed tasks also use standard test sets for atomic tasks and are formed from those tasks’ source datasets.Source: Compositional Tasks for Evaluation · pp. 3–4
  • The compositional benchmark covers atomic, comparative, selective-execution, sequential-execution, and multi-function tasks built from speech-understanding components.The benchmark combines familiar acoustic functionalities and task schemas into new task forms.Source: Compositional Tasks for Evaluation · p. 4

Train-Test Coverage

  • The coverage audit matches evaluation tasks to SIFT samples by primitive functionality and task schema, then separately checks exact task-form overlap.Tasks with more than 1,000 matches are in-domain, 1–1,000 are sparse, and zero matches are OOD.Source: Train-Test Coverage · p. 4
  • 18 of 29 tasks are exact-form OOD, compared with 7 OOD at the primitive level; all nine selective tasks are exact-form OOD.Most sequential and multi-function tasks overlap with SIFT only at the primitive level.Source: Train-Test Coverage · p. 4, Table 1 · p. 4

Experimental Results

T2L-ICAG improves compositional performance over static LoRA, with stronger gains on novel task forms and consistent average gains across backbones and ranks. Coverage, transfer, and architecture analyses associate these improvements with instruction-conditioned adaptation rather than parameter count alone.

Model, Training and Evaluation

  • Rank-4 adapters are applied to query and value projections; ASR evaluation uses WER and JSON-schema IFR, while selective ASR also separates selection accuracy from transcription quality.The Kimi-Audio hypernetwork has about 30M parameters, compared with about 1.3M LoRA parameters; relative performance score normalizes across metric scales.Source: Model, Training and Evaluation · pp. 4–5

Overall Performance Across Task Schemas

  • T2L-ICAG achieves the best average Kimi-Audio performance; versus rank-4 LoRA, it gains 7.2 RPS points on sequential tasks and 13.6 on multi-function tasks.On in-domain tasks, its performance is similar to multi-task LoRA, with occasional mild improvements.Source: Overall Performance Across Task Schemas · p. 5, Table 2 · p. 6

Generalization Beyond Training Coverage

  • T2L-ICAG leads in all three exact-form coverage regimes and gains 9 points over rank-4 LoRA on exact-form OOD tasks.It also maintains high RPS across primitive-level in-domain, sparse, and OOD partitions; parameter-matched rank-80 LoRA remains below it across all six partitions.Source: Table 3 · p. 6, Generalization Beyond Training Coverage · p. 6

Compositional Transfer of Atomic Functionalities

  • T2L-ICAG stays closer than static LoRA to atomic-implied performance on harder multi-input and selection-style gender and language tasks, though gaps remain.Expected performance is estimated from atomic recognition, and its values depend on task construction, label distribution, and possible shortcuts.Source: Figure 3 · p. 6, Compositional Transfer of Atomic… · pp. 6–7

Varying Hypernetwork Architectures

  • On Kimi-Audio, T2L-emb’s average compositional RPS is 90.99, below rank-4 LoRA’s 92.02 and T2L-ICAG’s 98.35.T2L-emb is competitive on in-domain EvalSIFT tasks, but its generalization is weaker on structurally complex, sparsely covered, and OOD tasks; ICAG preserves token-level task structure.Source: Varying Hypernetwork Architectures · p. 7

Varying LoRA Ranks

  • Across ranks 2, 4, and 8, T2L-ICAG improves average RPS over static LoRA on both backbones: Kimi-Audio gains 8.8%, 6.1%, and 7.9% RPS points, and Qwen2.5-Omni gains 3.2%, 4.7%, and 6.2%.The averages aggregate more than 30 task metrics; group-level exceptions occur, and increasing static LoRA rank beyond 8 does not improve compositional generalization.Source: Figure 4 · p. 7, Varying LoRA Ranks · p. 7

Conclusion

T2L-ICAG improves compositional transfer across multiple task schemas, especially structurally demanding tasks. The authors nevertheless describe robust generalization to unseen compositions as an open challenge.

Conclusion

  • T2L-ICAG consistently outperforms static LoRA and T2L-emb across multiple compositional task schemas, with clear gains on structurally demanding tasks.Overlap and atomic-functionality analyses suggest the gains reflect better composition of known primitives rather than simple memorization or more parameters.Source: Conclusion · p. 7
  • Robust generalization to unseen task compositions remains an open challenge.Source: Conclusion · p. 7

Appendix

The appendix supplies additional dataset, evaluation, experimental, and analysis details.

Appendix

  • The appendix reports dataset construction, evaluation protocols, experiment setups, and additional results on LoRA ranks, translation, mismatched instructions, newer LALMs, singular values, and judge consistency.Source: Appendix · p. 8

Task Definitions and Evaluation Metrics

The evaluation defines comparative, selective-execution, multi-input, and multi-task speech tasks, with metrics designed to distinguish task performance from selection and formatting errors.

Task Definitions and Evaluation Metrics

  • Comparative tasks use paired speech inputs to compare speakers on acoustic attributes, including pitch and duration, with pairs filtered by attribute differences.Emotion-verification negatives exclude highly similar emotion categories; examples include joy and surprise.Source: Task Definitions and Evaluation Metrics · p. 8
  • Selective execution tasks require first selecting a target speaker using one acoustic attribute, then performing a second task on that speaker.The target-gender SQA case uses CN-College-Listen dialogue and asks about the selected speaker's activities and content.Source: Task Definitions and Evaluation Metrics · p. 8
  • For target-X ASR, SAcc measures whether the inferred speaker matches the specified target, while mWER measures transcription quality against the better-matching reference.Together, the metrics separate target selection from transcription quality.Source: Task Definitions and Evaluation Metrics · p. 8
  • Multi-input and multi-task scenarios combine ASR, gender recognition, or language identification; ASR outputs use JSON, whose compliance is measured by IFR.Lenient WER normalizes JSON tokens and keys; multi-task WER is evaluated on instruction-following samples, while language and gender use string matching or an LLM judge.Source: Task Definitions and Evaluation Metrics · p. 8
  • The target-conditioned transcript metric is distinct from cpWER because this setting produces one transcript under an attribute-conditioned selection criterion.Source: Task Definitions and Evaluation Metrics · p. 8

Task Primitives and Data Audits

The audit represents tasks through acoustic primitives and task schemas, then estimates how well SIFT training samples cover evaluation tasks at primitive and exact-form levels.

Task Primitives and Data Audits

  • Because SIFT lacks task-level annotations, the authors use customized audits and LLM annotations to label training samples with primitives and schemas.Qwen3-32B performs the annotation; the sets are described as sufficient for the SIFT subspace, not the entire speech and audio understanding space.Source: Task Primitives and Data Audits · p. 9
  • The audit defines acoustic primitives as basic audio abilities and task schemas as task structures independent of those abilities.The listed schemas include single- or multi-input, single- or multi-task, and comparison; the primitive set includes speech, acoustic, and semantic abilities.Source: Task Primitives and Data Audits · p. 9
  • Training coverage is counted at primitive level, matching primitive and schema sets, and exact-form level, matching the exact ordering of functionalities.Exact-form audits use heuristic string templates followed by iterative retrieval refinement with LLM agents.Source: Task Primitives and Data Audits · p. 9
  • Figure 5 plots training-sample multiplicity for 29 evaluation tasks on a shared logarithmic scale, marking zero-coverage tasks and a 1,000-sample sparse-versus-in-domain threshold.The tasks are grouped into five task groups.Source: Figure 5 · p. 9, Task Primitives and Data Audits · p. 9

Model Details

The supplemental experiments show that larger static LoRA ranks do not account for compositional gains, while T2L-ICAG also transfers to translation. Additional analyses examine instruction matching, model behavior, adapter spectra, and evaluation protocols.

Model Details

  • Kimi-Audio and Qwen2.5-Omni Thinker are evaluated with understanding-focused configurations, one-epoch AdamW training, and deterministic greedy decoding.The audio-generation layers are omitted, and Qwen2.5-Omni Thinker uses the same configurations as Kimi-Audio; training uses learning rate 5 × 10−5, batch size 16, and four A800 GPUs.Source: Model Details · p. 9, Training Details · p. 9, Inference Settings · p. 9

Translation Tasks

  • 30.3 BLEU-4 for T2L-ICAG exceeds 23.9 with +LoRA and 23.5 with +T2L-emb; its average COMET is 0.775 versus 0.721, 0.718, and 0.732, respectively.The evaluation covers seven CoVoST2 X-to-English directions; Figure 6 reports BLEU-4 and COMET by source language.Source: Translation Tasks · pp. 11–12, Figure 6 · p. 12

Case Study: Using Mismatched Instructions

  • Every mismatched hypernetwork instruction lowers gender-verification performance, with larger deviations from the task instruction associated with greater decreases.The LLM instruction remains the task-inherent instruction while the hypernetwork instruction varies.Source: Case Study: Using Mismatched… · p. 12, Table 8 · p. 15

Consistency across LLM Judges

LLM Judge Prompts

  • The evaluation uses a shared compositional-task judge template, with synonym-aware rules for language identification and distinct rating formats for spoken question answering.EvalSIFT uses a binary correctness prompt; SLUE SQA uses a 0–5 rating, while AudioBench multiple-choice evaluation uses a binary 0–1 rating.Source: LLM Judge Prompts · p. 13, Figure 10 · p. 16, Figure 11 · p. 17

Larger LoRA Ranks Are Not Enough

  • Under distribution shift, static LoRA peaks at r = 4 on primitive-level and exact-form OOD tasks; higher ranks lack consistent gains and can substantially reduce RPS, while rank-4 T2L-ICAG beats the best static LoRA in all six overlap partitions.Source: Larger LoRA Ranks Are Not Enough · p. 11

Empirical Observations on Singular Values

  • T2L-ICAG adapters generally have larger, flatter singular-value spectra and greater layer-wise dispersion than static LoRA across visualized tasks and target modules.The authors treat this as an empirical pattern, not a causal explanation of performance gains, and leave the mechanism for future work.Source: Empirical Observations on Singular… · pp. 12–13, Figure 8 · p. 15

Performance of State-of-the-Art LALMs

  • On Qwen3.5-Omni-Plus, Gender Verification reaches 57.73% versus 97.60% atomic gender recognition; accuracy is 16.07% on same-gender pairs and 99.39% on different-gender pairs.The model performs strongly on most tasks, but its unauditable training data make it a reference for attainable benchmark performance rather than controlled evidence of compositional generalization.Source: Performance of State-of-the-Art LALMs · p. 12, Table 7 · p. 14, Figure 9 · p. 15

Related papers

Loading 2610.34413v1…