PaperlayerView exact revision on arXiv

Source-linked AI summary

AzeroS: Extending LLM to Speech with Self-Generated Instruction-Free Tuning

Yiwen Shao, Wei Liu, Jiahong Li, Tianzi Wang, Kun Wei, Meng Yu, Dong Yu

arXiv:2601.06086v1cs.CLcs.SDeess.AS

TL;DR

Speech–LLMs often rely on manually curated task-specific instruction data, which can be costly and generalize poorly to unseen tasks. This paper uses a frozen LLM to generate instruction-free supervision from textual speech representations, and reports state-of-the-art performance across semantic and paralinguistic benchmarks.

  • Problem

    Speech–LLMs often rely on manually curated task-specific instruction data, while task-focused alignment can limit generalization to unseen tasks.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    SIFT uses a frozen LLM to generate supervision from oracle textual representations without explicit instructions, while AZeroS trains projection modules with the LLM and audio encoders frozen.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    AZeroS achieves state-of-the-art performance across semantic and paralinguistic benchmarks, including VoiceBench and AIR-Bench Speech evaluations.

    Source: §1 · p. 2, §6 · p. 14

  • Takeaways & Limitations

    SIFT avoids collecting task-specific instruction–response pairs and is reported to improve training efficiency and scalability.

    Source: §6 · p. 14

    SIFT’s optimality depends on the frozen LLM intrinsically surfacing the information contained in the oracle input.

    Source: §5.7 · p. 12

Abstract

from arXiv · show

Extending large language models (LLMs) to the speech domain has recently gained significant attention. A typical approach connects a pretrained LLM with an audio encoder through a projection module and trains the resulting model on large-scale, task-specific instruction-tuning datasets. However, curating such instruction-tuning data for specific requirements is time-consuming, and models trained in this manner often generalize poorly to unseen tasks. In this work, we first formulate that the strongest generalization of a speech-LLM is achieved when it is trained with Self-Generated Instruction-Free Tuning (SIFT), in which supervision signals are generated by a frozen LLM using textual representations of speech as input. Our proposed SIFT paradigm eliminates the need for collecting task-specific question-answer pairs and yields the theoretically best generalization to unseen tasks. Building upon this paradigm, we introduce AZeroS (Auden Zero-instruction-tuned Speech-LLM), which is trained on speech-text pairs derived from publicly available corpora, including approximately 25,000 hours of speech with ASR transcripts and 3,000 hours of speech with paralinguistic labels. Built upon Qwen2.5-7B-Instruct, the model updates only two lightweight projection modules (23.8 million parameters each), while keeping both the LLM and audio encoders frozen. Despite the minimal training cost and modest data scale, AZeroS achieves state-of-the-art performance on both semantic and paralinguistic benchmarks, including VoiceBench, AIR-Bench Foundation (Speech), and AIR-Bench Chat (Speech).

1 Introduction

AZeroS uses frozen-LLM-generated supervision and lightweight projection modules to extend speech–LLMs without task-specific instruction tuning. The paper reports state-of-the-art results on semantic and paralinguistic benchmarks.

  • Cascaded speech–LLM pipelines can propagate ASR errors and lose prosody or emotion, motivating end-to-end speech processing.Source: §1 · p. 2
  • Task-specific instruction data is often manually curated at scale, motivating an approach that leverages a frozen LLM’s generalization capacity.Source: §1 · p. 2
  • SIFT generates supervision from oracle textual representations of speech using a frozen LLM, without explicit instructions.Source: §1 · p. 2
  • AZeroS updates only projection modules while keeping the LLM and audio encoders frozen.Source: §1 · p. 2
  • AZeroS achieves state-of-the-art performance on semantic and paralinguistic benchmarks, including VoiceBench and AIR-Bench (Speech).Source: §1 · p. 2

2 Related Work

Prior speech–LLMs often depend on task-specific instruction data, while instruction-free alignment approaches have sought broader generalization. Existing approaches differ in whether they cover paralinguistic information and whether they rely on external ASR.

  • Task-specific instruction-tuned systems can become biased toward ASR-style responses and remain limited to their predefined task sets.Source: §2 · p. 3
  • AudioChatLLaMA and BLSP support zero-shot speech-task generalization through speech–text alignment, but primarily target semantic information.Source: §2 · p. 3
  • SpeechEmotionLlama adds emotion-conditioned response alignment, while Desta2 uses transcripts and paralinguistic metadata for descriptive targets.Source: §2 · p. 3
  • Desta2.5 expands self-generated descriptions with many prompts, but the Desta series still relies on external ASR and cannot natively support speech interaction.Source: §2 · p. 3

3 Methodology

The methodology motivates instruction-free audio–text alignment, then uses a frozen LLM to generate supervision from oracle text and trains an audio mapping to reproduce it. SIFT’s proposed generalization advantage depends on targets that capture the oracle text’s information without instructions.

3.1 Motivation: Unlocking the Universal Decoder

  • Instruction tuning can make encoders discard features irrelevant to a specified task and encourage responses grounded in prompts rather than audio.Source: §3.1 · p. 4

3.2 Problem Setup

  • The framework maps audio x through hθ into a frozen LLM’s input space, with a frozen audio encoder and a trainable projector composing hθ.Source: §3.2 · p. 4

3.3 Self-Generated Data Construction

  • The oracle text should capture diverse audio details, and the frozen LLM should elicit them in its response without an instruction.Source: §3.3 · p. 5
  • SIFT is the optimal branch when the LLM self-elicits the oracle content; otherwise, the framework uses prompted SIT as a fallback.Source: §3.3 · p. 5
  • The comparison frames AudioChatLlama as transcript-limited and DESTA-2/2.5 as relying on rigid instructions and auxiliary ASR, constraints SIFT aims to avoid.Source: Table 1 · p. 5

3.4 SIFT: Self-Generated Instruction-Free Tuning

  • SIFT minimizes prediction error on LLM-generated targets with no instruction, directing training toward audio-to-LLM modality alignment.Source: §3.4 · p. 5

3.5 Why Instruction-Free Generalizes Best

  • The paper argues that instruction-free targets reflecting the oracle text’s full content require general rather than task-dependent alignment, supporting unseen instructions at test time.Source: §3.5 · p. 6

4 AZeroS: System Overview

AZeroS is built to test SIFT, coupling an encoder–projector–LLM architecture with a paired data-generation and training process.

4.1 Model Architecture

  • AZeroS uses Qwen2.5-7B-Instruct with TTA and Auden-Voice encoders, each paired with a 23.8M projector that reduces features from 25 Hz to 6.25 Hz.TTA supplies semantic representations, while Auden-Voice captures paralinguistic cues.Source: §4.1 · p. 6

4.2 Training Paradigm

  • The system pairs self-generated target construction with training: the frozen LLM receives only the textual representation, generating response y paired with speech x.The LLM and audio encoders remain frozen during training; only the lightweight projector is updated.Source: §4.2 · pp. 6–7
  • Training proceeds in two stages: transcript-based semantic alignment with TTA first, then transcript-plus-metadata alignment by updating only the paralinguistic projector.Source: §4.2 · p. 7

5 Experimental Setup and Results

Experiments test semantic-only and joint semantic–paralinguistic alignment, then compare AZeroS with other speech–LLMs. AZeroS offers a strong balance across semantic and paralinguistic benchmarks, while ablations and a case study probe its training choices and self-elicit assumption.

5.1 Data Source

  • Data-SP provides 3,548 hours with transcripts and paralinguistic annotations, while Data-S provides 22,383 hours with ASR transcripts only.Both data sets are described as publicly available.Source: §5.1 · p. 7

5.2 Evaluation Benchmarks

  • VoiceBench evaluates semantic understanding across spoken-instruction tasks, while AIR-Bench-Speech emphasizes paralinguistic cues alongside semantic components.The AIR-Bench Foundation subset uses multiple-choice questions and Chat uses open-ended question–answer pairs.Source: §5.2 · pp. 7–8

5.3 Stage 1: Semantic Alignment

  • On VoiceBench, TSITs scores 22.26, SITs 57.89, and SIFTs 68.70, leading the authors to select instruction-free SIFTs for Stage 1.The reported scores are overall scores; the study describes SIFTs as having the most generalizable semantic alignment.Source: §5.3 · pp. 8–9, Table 3 · p. 9

5.4 Stage 2: Joint Semantic and Paralinguistic Alignment

  • 73.63 versus 68.70 overall on VoiceBench followed training on Data-S rather than Data-SP; joint alignment improves AIR-Bench, and mixing SIFTs with SIFTsp yields the final model.The authors report slight VoiceBench degradation after Stage 2 before combining the two training configurations.Source: §5.4 · pp. 9–10, Table 4 · p. 10

5.5 Comparison to Other Systems

  • 72.91 on VoiceBench places AZeroS near the text-only (77.52) and cascaded (76.05) bounds, while it substantially outperforms other end-to-end models on AIR-Bench.The comparison reports the strongest overall balance between semantic and paralinguistic understanding.Source: Table 5 · p. 11, §5.5 · p. 11

5.6 Ablation Study

  • Two-stage training slightly outperforms one-stage training; the dual encoder beats single Whisper, while the study attributes the primary gains to SIFT.The one-stage setup merges all training data for joint training.Source: §5.6 · pp. 11–12, Table 6 · p. 12

5.7 Case Study: When Self-Elicit Fails (Dialog-LLM)

  • When a system prompt suppresses self-eliction, SIFT targets become information-sparse and fine-grained performance degrades; mixing SITssp with SIFT recovers performance.The ssp alignment nevertheless imprints a conversational, concise response style, unlike the more analytical and detailed sp behavior.Source: §5.7 · pp. 12–13

6 Conclusion

The conclusion presents AZEROS as an efficient speech–LLM built with SIFT and reports strong semantic and paralinguistic results from a small training footprint. It attributes gains primarily to tuning rather than architectural complexity and identifies vision–language models and unified instruction datasets as future directions.

6 Conclusion

  • SIFT uses a frozen LLM to generate supervision from textualized speech representations, avoiding large-scale manually curated instruction–response pairs.The authors characterize this approach as improving efficiency and scalability.Source: §6 · p. 14
  • AZEROS achieves state-of-the-art performance across semantic and paralinguistic benchmarks, approaches text-only semantic upper bounds, and surpasses prior cascaded and end-to-end systems in paralinguistic understanding.The reported benchmarks include VoiceBench, AIR-Bench Foundation (Speech), and AIR-Bench Chat (Speech).Source: §6 · p. 14
  • AZEROS’s performance gains stem primarily from the self-generated tuning paradigm rather than architectural complexity, with consistent improvements across ablation studies.The authors suggest SIFT may offer a general and extensible principle for aligning multimodal encoders with LLMs.Source: §6 · p. 14
  • Future work will test self-generated supervision for vision–language models and investigate integrating it with conventional task-specific instruction tuning.The authors say these directions may further validate the approach’s generality and inform scalable, instruction-efficient multimodal learning.Source: §6 · p. 14

A Data Example

The examples illustrate outputs for different input and instruction setups, including metadata-aware descriptions, transcript repetition, and conversational responses. They show how the response can preserve transcript content while also using speaker attributes or elaborating conversationally.

A Data Example

  • In the SIFTs repetition example, the English output repeats the utterance, while the Chinese example repeats the transcript’s meaning.The examples are labeled as self-generated instruction tuning for semantic alignment.Source: A Data Example · p. 19
  • In one English-language example, the response offers general advice about unexpected interruptions; in another, a Chinese-language response describes Wuyishan and lists attractions a visitor could consider.Source: A Data Example · p. 19
  • The metadata-aware description examples state speaker attributes alongside transcript content, including a male, happy voice and a young female speaker.The examples pair English and Chinese speech inputs with descriptions of speaker characteristics and utterance content.Source: A Data Example · p. 20
Loading 2601.06086v1…