Source-linked AI summary
Teaching LLMs to Hear Who Spoke What: Metadata-Supervised Pretraining for Encoder-Free Speech-LLMs
Mohan Shi, Ruchao Fan, Sunit Sivasankaran, Keqi Deng, Jinyu Li
TL;DR
Speech encoders can prioritize linguistic content over acoustic cues, while systematic pretraining for encoder-free Speech-LLMs remains underexplored. The paper introduces metadata-supervised pretraining with speaker-aware utterance composition and span masking. Under matched training-data conditions, its encoder-free model outperforms a randomly initialized encoder-based counterpart and is competitive with models pretrained on substantially larger corpora.
Problem
Speech encoders may discard fine-grained acoustic cues, while systematic pretraining strategies for encoder-free Speech-LLMs remain underexplored.
Source: §1 · p. 1Method
Metadata-supervised pretraining uses speech attributes such as speaker identity and emotion, with speaker-aware utterance composition and random span masking.
Source: §1 · pp. 1–2Results
Under matched training-data conditions, the encoder-free model outperforms its randomly initialized encoder-based counterpart and is competitive with models using substantially larger encoder-pretraining corpora.
Source: §1 · pp. 1–2Takeaways & Limitations
Abstract
from arXiv · showhide
Encoder-based speech large language models (Speech-LLMs) commonly employ pretrained speech encoders that prioritize linguistic content but may discard fine-grained acoustic cues essential for speaker discrimination and paralinguistic understanding. Encoder-free Speech-LLMs instead map Mel-spectrogram features directly into the LLM input space through lightweight embedding layers, enabling the LLM to learn from low-level acoustic features. However, systematic pretraining strategies for encoder-free Speech-LLMs remain underexplored, limiting their ability to compensate for the absence of large-scale pretrained speech encoders. We propose metadata-supervised pretraining (MSP), which leverages speech attributes such as speaker identity and emotion to develop speaker-discriminative and paralinguistic capabilities. We further introduce speaker-aware utterance composition (SAUC) to strengthen speaker discrimination and apply random span masking to regularize pretraining. We primarily evaluate our approach on joint ASR and speaker diarization in multi-speaker conversations, complemented by experiments on paralinguistic speech-understanding tasks. Under matched training-data conditions, our encoder-free model outperforms its randomly initialized encoder-based counterpart. With limited metadata-annotated data, it is competitive with models using speech encoders pretrained on substantially larger corpora, outperforming them in several settings. These results demonstrate the potential of encoder-free architectures for building native multimodal LLMs that acquire diverse speech capabilities.
1. INTRODUCTION
Speech encoders often prioritize linguistic content and may lose acoustic cues needed for speaker identity and other paralinguistic attributes; richer representations can require added complexity or dedicated encoder training. The paper proposes metadata-supervised pretraining for encoder-free Speech-LLMs and evaluates its speech capabilities.
1. INTRODUCTION
- Encoder-based Speech-LLMs may discard fine-grained acoustic cues for speaker identity and emotion because their encoders prioritize linguistic content.Prior approaches use dual encoders or auxiliary objectives, but can increase architectural or training complexity and require dedicated encoder training.Source: §1 · p. 1
- Encoder-free models expose low-level audio features to the LLM, but prior work had not established an explicit pretraining strategy for speaker-discriminative and paralinguistic representations.Existing evaluations also make these abilities difficult to isolate from instruction-following and reasoning effects.Source: §1 · p. 1
- The paper introduces metadata-supervised pretraining (MSP), using attributes such as speaker identity and emotion, plus speaker-aware utterance composition (SAUC) and random span masking.Evaluation centers on joint ASR and diarization, with complementary paralinguistic tasks.Source: §1 · pp. 1–2
- Under matched training-data conditions, the encoder-free model outperforms its randomly initialized encoder-based counterpart and is competitive with models using substantially larger encoder-pretraining corpora.It outperforms those larger-corpus models in several settings.Source: §1 · pp. 1–2
2. METHOD
The method pretrains an encoder-free Speech-LLM on speech metadata and transcription, then fine-tunes it for downstream tasks. Metadata targets, composed multi-speaker inputs, and span masking provide the main training design choices.
2. METHOD
- MSP pretrains the encoder-free model before separate fine-tuning for joint ASR and speaker diarization or paralinguistic understanding.The approach retains the encoder-free backbone and uses task-specific targets for downstream fine-tuning.Source: §2 · p. 2
2.1. Encoder-free Speech-LLM
- The frontend converts 80-dimensional log-Mel features through lightweight convolutional downsampling and linear projection into the LLM embedding space, without speech encoder blocks.The frozen LLM is adapted with LoRA, and speech embeddings are concatenated with prompt embeddings.Source: §2.1 · p. 2
2.2. Metadata-supervised Pretraining
- MSP predicts paralinguistic attributes alongside transcription because transcription supervision alone does not explicitly encourage learning speaker identity, emotion, or other such attributes.The encoder-free model receives the Mel spectrogram directly, exposing low-level acoustic detail.Source: §2.2 · p. 2
2.2.1. Meta labels
- Each utterance’s metadata record includes gender, emotion, language, speaker identity, and transcription, with paralinguistic fields preceding lexical content.Speaker identifiers are global and namespaced by source corpus.Source: §2.2.1 · p. 2
2.2.2. Speaker-aware Utterance Composition
- SAUC concatenates utterances on a shared timeline, mixes them into one audio stream, and orders corresponding metadata targets by onset to create long-form multi-speaker examples.Junctions use overlaps or silence gaps; overlaps are limited to different speakers, with at most two active speakers.Source: §2.2.2 · p. 2
2.2.3. Input Masking
- During pretraining, span masking replaces selected log-Mel frames with per-utterance, per-channel means plus Gaussian noise.Masking is applied only during pretraining, at the frontend’s 12.5 Hz temporal resolution.Source: §2.2.3 · p. 2
3. EXPERIMENTAL SETTINGS
The experiments use large-scale speech data, a fixed pretraining setup, and separate evaluations for joint ASR and diarization and for paralinguistic understanding. Comparisons include matched-training baselines and previously reported models.
3.1.1. Pretraining Data
- The metadata corpus totals approximately 10k hours, and MSP mixes metadata examples with transcription-only ASR examples at a 3:1 ratio.An additional scaling variant adds one epoch of ASR initialization on one million hours of internal data and expands metadata data with Emilia-en.Source: §3.1.1 · p. 3
3.1.2. Pretraining Setup
- Pretraining runs for 80k steps on 16 H200 GPUs, updating the convolutional frontend, modality projector, and LoRA parameters.The setup uses 80 ms of speech per token and input masking with p = 0.04 and L = 4, corresponding to an expected masked fraction of approximately 15%.Source: §3.1.2 · p. 3
3.2.1. Joint ASR and Speaker Diarization
- Joint ASR and diarization are evaluated on AMI IHM-Mix and CH109 Full using WDER and cpWER, with a fixed prompt to reduce instruction-following effects.Fine-tuning uses approximately 10k hours of multi-speaker speech and only the LLM prediction loss for the encoder-free model.Source: §3.2.1 · p. 3, Table 1 · p. 3
3.2.2. Paralinguistic Speech Understanding
- Paralinguistic evaluation covers IEMOCAP emotion recognition, VoxCeleb gender and age prediction, and WavLLM speaker verification, using accuracy except for age MAE.Evaluation samples are excluded from pretraining and fine-tuning.Source: §3.2.2 · p. 3
3.3. Baselines
- Baselines include a randomly initialized encoder-based model trained with the same setup, an encoder-free model without MSP, and previously reported results.The related-work results are from [15].Source: §3.3 · p. 3
4. EXPERIMENTAL RESULTS
Across joint ASR and diarization, metadata supervision and SAUC improved results, while masking and target design affected outcomes. MSP also improved paralinguistic tasks, and encoder-free models compared favorably with encoder-based counterparts under matched training-data conditions.
4.1. Joint ASR and Speaker Diarization
- Under matched-data conditions, encoder-free models beat randomly initialized encoder-based models on all four metrics; input masking also beat reproduced JEDIS-LLM on both AMI metrics.Without the auxiliary speaker-decoder loss, JEDIS-LLM had higher WDER than the encoder-free baseline on AMI and CH109.Source: §4.1 · p. 3
4.2. Ablation of Pretraining Objectives and SAUC
- Utterance-level MSP improves all four metrics over pure ASR; simulated overlaps and silence gaps further improve all four after concatenation alone proves insufficient.With overlaps and gaps, WDER falls from 6.05 to 5.38 on AMI and from 3.18 to 2.77 on CH109; composition is on-the-fly.Source: §4.2 · p. 4
4.3. Metadata Target Design
- Moving transcription before speaker identity degrades all four metrics, while adding timestamp supervision increases cpWER on both sets without improving WDER.The authors suggest preceding text may encourage lexical shortcuts and precise timestamps may divert capacity from other targets.Source: §4.3 · p. 4
4.4. Input Span Masking
- At p = 0.04 (about 15% masking), WDER and cpWER are lowest on both sets; aggressive masking degrades both AMI metrics and CH109 cpWER.The comparison fixes span length at L = 4, or 320 ms.Source: Table 4 · p. 4, §4.4 · p. 4
4.5. Paralinguistic Speech Understanding
- MSP improves all four paralinguistic tasks over the reproduced encoder-free baseline; it beats encoder-based MSP on emotion recognition and speaker verification.It matches encoder-based gender accuracy and yields slightly higher age MAE; ASR initialization further improves all four tasks.Source: Table 5 · p. 4, §4.5 · p. 4
5. CONCLUSION
The conclusion presents metadata-supervised pretraining for encoder-free Speech-LLMs, combining rich metadata targets, SAUC, and moderate input masking. Experiments show competitive performance across joint ASR and diarization and paralinguistic tasks under matched-data conditions.
5. CONCLUSION
- MSP uses rich metadata targets to promote speaker discrimination and paralinguistic understanding beyond transcription.SAUC composes long-form multi-speaker inputs with random overlaps and silence gaps, while moderate input masking regularizes pretraining.Source: §5 · p. 4
- Under matched training-data conditions, the encoder-free model outperforms its randomly initialized encoder-based counterpart overall, with fewer parameters after removing the speech encoder.Scaling ASR and metadata pretraining provides further task-dependent gains.Source: §5 · p. 4
- The results demonstrate the potential of encoder-free architectures to acquire diverse speech capabilities through metadata-supervised pretraining.Source: §5 · p. 4
6. GENERATIVE AI USE DISCLOSURE
The authors disclose using Claude Opus 5 for language editing and proofreading.
6. GENERATIVE AI USE DISCLOSURE
- Claude Opus 5 was used for language editing, including proofreading and improving manuscript clarity and readability.Source: §6 · p. 5
- The authors state that they produced and verified the technical content, experimental design, results, and conclusions.Source: §6 · p. 5
- After generative-AI use, the authors reviewed and edited the manuscript and take full responsibility for its content.Source: §6 · p. 5