Source-linked AI summary
Speech Block Influence: Component-Specific Layer Scoring for Pruning Speech LLMs
Siyu Yao, Du Q. Huynh, Lian Xu, Mark Reynolds
TL;DR
Speech LLM pruning scores developed for text LLMs do not transfer effectively to models with encoder-adapter components and multimodal sequences. The paper introduces component-specific SBI scores and finds improved pruning robustness across three speech LLMs, with text-only calibration providing a cheaper alternative for estimating decoder-layer importance.
Problem
Layer-pruning scores developed for text LLMs transfer poorly to speech LLMs, which add encoder and adapter components and process both audio and text embeddings.
Source: §1 · p. 1Method
SBI scores encoder-layer removal at the adapter output and decoder-layer input-output similarity over text tokens only.
Source: §1 · p. 1, §3.1 · p. 2, §3.2 · p. 2Results
Takeaways & Limitations
Abstract
from arXiv · showhide
Speech LLMs are costly to deploy in resource-constrained settings. Layer pruning can cut this cost, but existing scoring metrics transfer poorly to speech LLMs: they assume a decoder-only architecture with homogeneous token sequences, whereas speech LLMs add encoder and adapter components and process multimodal sequences. We propose Speech Block Influence (SBI), the first layer-importance scoring framework designed for speech LLM pruning that consists of two component-specific scores: SBI-Enc measures the effect of encoder-layer removal at the adapter's output to better reflect downstream impact; SBI-Dec measures layer-wise input-output similarity over text-token positions only to avoid audio-token dominance. Across three speech LLMs, SBI improves pruning robustness, with stronger encoder performance at higher pruning rates and more reliable decoder layer selection by scoring text tokens rather than the audio-dominated full sequence. We further find that text-only calibration yields decoder rankings highly correlated with those from speech-text calibration, suggesting a cheaper alternative to measure decoder layer importance.
1. INTRODUCTION
Speech LLMs are costly to deploy, and pruning scores designed for text LLMs do not transfer effectively to their additional components and multimodal sequences. The paper introduces component-specific Speech Block Influence scores and reports that text-only calibration offers a cheaper way to estimate decoder-layer importance.
1. INTRODUCTION
- Speech LLMs strain resource-constrained deployment because encoder and decoder layers increase memory footprint and forward-pass latency.The architecture also includes an adapter between the speech encoder and decoder-only transformer.Source: §1 · p. 1
- Text-LLM pruning scores fit speech LLMs poorly because speech models add encoder and adapter components and combine audio embeddings with text.The paper reports that ShortGPT’s Block Influence (BI) does not transfer effectively to speech LLM pruning.Source: §1 · p. 1
- SBI scores encoder layers at the adapter output and decoder layers over text tokens only, improving pruning robustness over BI.The framework is presented as the first layer-importance scoring framework designed for speech LLM pruning.Source: §1 · p. 1
- Text-only calibration rankings closely correlate with speech-text calibration rankings, suggesting a cheaper way to estimate decoder-layer importance.The paper presents this as a separate contribution from its component-specific scores.Source: §1 · p. 1
2. RELATED WORK
Prior work establishes layer redundancy and pruning approaches in text LLMs, while speech-LLM pruning studies have covered narrower tasks. This paper proposes speech-specific scoring and evaluates a broader task range.
2. RELATED WORK
- ShortGPT’s BI scores a layer by how much it changes hidden states, while layer pruning removes entire transformer blocks.Prior pruning approaches also remove individual weights or structured units such as attention heads and MLP channels.Source: §2 · p. 1
- Gromov et al. use angular distance between contiguous layer blocks and report deeper layers as more redundant in text LLMs.Their results indicate that deeper layers can be pruned with minimal loss.Source: §2 · p. 1
- SBI-Enc compares adapter outputs with and without an encoder layer; SBI-Dec compares decoder-layer input and output text tokens.The figure contrasts these scores with BI computed over audio tokens or all tokens.Source: Figure 1 · p. 2
- Earlier speech-LLM pruning studies evaluated ASR and AST, whereas this study also evaluates broader speech understanding with multiple-choice questions.The paper also distinguishes its work by proposing speech-specific layer scoring and studying natively trained speech LLMs.Source: §2 · pp. 1–2
3. THE SBI FRAMEWORK
SBI scores encoder and decoder layers separately, measuring encoder-layer effects at the adapter output and decoder-layer similarity over text tokens. The framework also specifies calibration and pruning procedures.
3. THE SBI FRAMEWORK
- SBI separates encoder and decoder scoring: it measures encoder-layer effects at the adapter output and decoder-layer input-output similarity over text tokens.The following subsections specify calibration data and how layers are removed.Source: §3 · p. 2
3.1. Encoder layer scoring at the adapter’s output
- A low SBI-Enc score indicates that removing an encoder layer leaves the adapter output nearly unchanged, marking the layer as redundant.The method scores at the adapter output because internal encoder changes may have little downstream effect after adapter projection and downsampling.Source: §3.1 · p. 2
3.2. Decoder layer scoring with text tokens
- SBI-Dec applies the original BI formulation only to text-token positions, avoiding scores dominated by audio tokens.Audio tokens outnumber text tokens by 6× to 22× in the calibration set, and audio-dominated scoring can cause collapse after removing as few as one layer in some models.Source: §3.2 · p. 2
3.3. Calibration set
- The speech-text calibration set uses 100 low-perplexity examples drawn from four audio corpora; transcribed counterparts test text-only calibration with content held constant.The speech clips are sampled from Common Voice, IEMOCAP, VoxCeleb, and Gaokao; the text-only comparison changes modality, not content.Source: §3.3 · p. 2
3.4. Pruning procedure
- The procedure scores the unpruned model once, removes its k lowest-scoring layers, prunes encoder and decoder separately, and applies no healing.SBI-Enc requires LEnc +1 calibration-set passes; SBI-Dec requires one pass.Source: §3.4 · p. 2
4. EXPERIMENTAL SETUP
The setup evaluates three speech LLMs and compares component-specific pruning scores against BI-based baselines across ASR and multiple-choice tasks.
4. EXPERIMENTAL SETUP
- Audio Flamingo 3 uses a Qwen2.5-7B decoder (28 layers), and Voxtral Mini 3B uses a Ministral-3B decoder (30 layers); all three models use 32-layer encoders initialized from Whisper-large-v3.Source: §4 · p. 3
- Evaluation covers ASR on Common Voice and multiple-choice answering on MMSU and VoiceBench’s OBQA subset.WER measures ASR performance, while MCQ performance is measured by accuracy.Source: §4 · p. 3
- Encoder pruning compares SBI-Enc with BI, while decoder pruning compares SBI-Dec with BIA∪T; each component is pruned independently by up to 10 layers.Source: §4 · p. 3
5. RESULTS AND DISCUSSION
Across three speech LLMs, the results show component-specific differences in pruning robustness and close agreement between decoder rankings from text-only and speech-text calibration.
5. RESULTS AND DISCUSSION
- SBI-Enc and BI share seven to nine of their top-10 encoder layers, while SBI-Enc orders differing selections toward shallower layers.In decoder rankings, audio-token dominance makes BIA∪T closely follow BIA, whereas SBI-Dec selects deeper layers.Source: §5 · p. 3
5.1. Encoder pruning
- SBI-Enc gives the lowest Common Voice WER for all three models and better preserves Qwen2-Audio and Audio Flamingo 3 performance, with a minor difference for Voxtral.For Qwen2-Audio at 31% encoder pruning, WER is 40% under BI versus 25% under SBI-Enc; Voxtral’s top-10 sets share nine layers.Source: §5.1 · p. 3
- At 31% encoder-layer pruning, SBI-Enc matches or outperforms BI in every model; below 20%, their curves generally overlap and cross, except for Audio Flamingo 3 on OBQA, indicating neither criterion consistently prevails.Source: §5.1 · p. 3
5.2. Decoder pruning
- SBI-Dec is the better decoder-pruning criterion on almost every task, delaying MCQ collapse relative to BIA∪T across models.Audio Flamingo 3’s ASR performance is an exception; Fig. 3 compares the pruning criteria and deepest-k.Source: Figure 3 · p. 4, §5.2 · p. 4
- Deepest-k is not equivalent to SBI-Dec: it raises WER faster in all three models and collapses Voxtral after pruning two to three layers.Speech LLMs generally degrade substantially before 20% of decoder layers are removed.Source: §5.2 · p. 4
5.3. Text-only calibration
- Text-only and speech-text calibration produce almost identical decoder rankings, with Spearman’s ρ ≥0.93 for every model.The table compares SBI-Dec and BItext rankings; the lowest-scoring layer sets are identical in eight of twelve settings and differ by at most one layer.Source: Table 1 · p. 4, §5.3 · p. 4
6. CONCLUSION AND FUTURE WORK
The conclusion presents SBI as a component-specific framework for speech LLM pruning, reports improved robustness and cheaper decoder calibration, and identifies future directions.
6. CONCLUSION AND FUTURE WORK
- SBI scores encoder layers at the adapter output and decoder layers over text tokens only.Source: §6 · p. 4
- Across three speech LLMs, SBI improves pruning robustness over BI in most settings, and text-only calibration offers a cheaper way to estimate decoder layer importance.Source: §6 · p. 4
- Future work may test lightweight fine-tuning with SBI and joint encoder-decoder pruning; the reported results use no healing and prune components independently.Source: §6 · p. 4
7. COMPLIANCE WITH ETHICAL STANDARDS
The study reports using publicly released speech corpora and no newly collected human-subject data.