PaperlayerView exact revision on arXiv

Source-linked AI summary

Pruning for Efficiency, Paying in Fairness: Demographic Disparities in Pruned Speech-LLMs

Ganesh Pavan Kartikeya Bharadwaj Kolluri, Michael Kampouridis, Ravi Shekhar

arXiv:2609.38106v1cs.SDcs.CL

TL;DR

Because aggregate WER can hide existing demographic disparities, the paper systematically tests encoder pruning across model scales and demographic groups. It finds disparities across scales, with concealment at the largest scale, while Common Voice gaps persist without clear amplification.

  • Problem

    Aggregate WER is commonly used to validate pruned models, while ASR systems already perform unevenly across demographic groups, leaving pruning’s effect on that unevenness an open question.

    Source: §1 · p. 1

  • Method

    The study systematically prunes Whisper encoders at three scales in the SLAM-ASR pipeline, retraining the projector at each pruning depth.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    13.5 to 24.5 percentage points: the Black–Asian WER gap grew on Whisper Large after eight removed layers; across scales gaps persisted, while Common Voice gaps did not clearly amplify.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Aggregate WER alone can conceal subgroup degradation, so pruning effects should be assessed by demographic group.

    Source: §1 · pp. 1–2

    Evidence is limited to one Whisper/Qwen2.5-3B setup, top-down layer pruning, and a single random seed; the main analysis uses English Fair-Speech.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Speech-LLMs are expensive to run, making compression important for real-world deployment. However, compressed models are usually selected using aggregate word error rate (WER), which can hide how pruning affects different demographic groups. In this work, we systematically study the effect of audio encoder pruning on SLAM-ASR for different demographic groups. Using the Fair-Speech and Common Voice datasets, we found that the pruning does not affect all demographic groups equally; the gap between best- and worst-performing groups increases in fold. These disparities appear across all three encoder scales, but only the largest model initially hides them behind aggregate WER. LoRA adaptation improves WER for every group, but benefits groups already performing well more strongly and widens for certain groups. On Common Voice English, Danish, and Dutch, accent gaps persist but do not clearly widen, showing that the fairness effects of pruning vary across datasets and must be measured directly. Our findings suggest that for pruned models, deployment decisions should include per-group WER, with the worst-performing group's error rate as an explicit criterion.

1 Introduction

Speech-LLMs are costly to deploy, and aggregate WER can conceal demographic disparities when models are pruned. This study evaluates pruning and LoRA across encoder scales and datasets, finding that effects vary across groups and settings.

1 Introduction

  • Speech-LLMs pair a pretrained speech encoder with a large language model through a lightweight projector, making these systems computationally expensive.Compression is a routine step between training and deployment.Source: §1 · p. 1
  • After eight removed layers, the Black–Asian WER gap grows from 13.5 to 24.5 pp, while the first prune improves aggregate WER.On Whisper Large, Black speakers’ WER rises significantly even as aggregate WER falls from 21.6 to 21.1%.Source: Figure 1 · p. 1, §1 · p. 2
  • The study evaluates SLAM-ASR with three Whisper encoder scales, top-down pruning, and projector retraining at each depth.Fair-Speech is the primary corpus, with Common Voice English, Dutch, and Danish added for cross-lingual study.Source: §1 · pp. 1–2
  • Pruning disparities appear at all three encoder scales, but aggregate WER conceals them only for the largest model.On the small and medium models, the first prune raises aggregate WER by 10.8 and 2.3 pp; Common Voice accent gaps persist without clear amplification.Source: §1 · p. 2
  • LoRA improves WER for every group but widens the Black-to-Asian ratio at every matched pruning depth.Recovery is stronger for groups the model already performs well on.Source: §1 · p. 2
  • Aggregate WER alone cannot validate a pruned speech model as fair; deployment decisions should include per-group WER and the worst-performing group.The work presents a systematic study of demographic disparity across pruning depth, model scale, and LoRA adaptation.Source: §1 · p. 2

2 Related work

Prior ASR research documents demographic performance gaps and introduces demographic datasets for measuring them. Work on compression reports varied fairness effects, while speech-LLM pruning and adaptation leave subgroup outcomes insufficiently studied.

2.1 Bias in ASR

  • ASR disparities have been reported across race, dialect, gender, age, accent, and language background, including in large pretrained models.Common Voice captures optional accent, gender, and age information; Fair-Speech covers age, gender, ethnicity, socio-economic status, and geographic variation.Source: §2.1 · p. 2

2.2 Model Compression, Scale, and Bias

  • Compression studies report that bias effects vary by method, domain, demographic axis, and model scale.Computer-vision work found pruning and quantization can amplify bias, while speech embedding probes found different effects for distillation and row pruning.Source: §2.2 · pp. 2–3
  • Prior speech-LLM studies examined response content or benchmarked WER disparities across architectures, but did not test pruning’s effect on transcription disparity.A prior Whisper-pruning study reported aggregate WER only; whether LoRA recovery is shared evenly across groups was also unexamined.Source: §2.2 · p. 3

3 Experimental Setup

The experiments vary Whisper encoder depth in a fixed SLAM-ASR pipeline, use Common Voice and Fair-Speech for training and evaluation, and measure group-level WER and disparity.

3.1 System and Pruning

  • Three Whisper scales (12, 24, and 32 layers) are pruned top-down in two-layer steps, with the projector retrained at every depth to form deployable systems.The Qwen2.5-3B decoder is held fixed so encoder capacity is the varied component.Source: §3.1 · p. 3

3.2 Datasets

  • Common Voice 22 trains the projector and LoRA adapters, using English as the primary system and separate Danish and Dutch systems for cross-lingual analyses.Training uses official train/test splits with disjoint speakers; English, Danish, and Dutch training subsets contain 100, 4.2, and 54 hours, respectively.Source: §3.2 · p. 4
  • Fair-Speech is evaluation-only and supplies ethnicity, socio-economic status, gender, age, and first-language labels; Common Voice 22 supports evaluation by accent, gender, and age.Common Voice group-level analyses use only test utterances with the relevant optional self-reported attribute.Source: §3.2 · p. 4

3.3 Implementation Details

  • The projector concatenates five frames for temporal downsampling, then applies two linear layers, ReLU, dropout, and LayerNorm; LoRA adapts LLM attention projections.The projector is reinitialized and trained from scratch at each pruning depth; Danish uses a lower LoRA rank after higher-rank preliminary runs overfit.Source: §3.3 · p. 4

3.4 Evaluation Details

  • By L-8, the Black–Asian WER gap nearly doubles from 13.5 to 24.5 pp, while Black speakers have the highest WER at every depth.At L-2, aggregate WER improves while Black speakers degrade.Source: Table 3 · p. 5
  • Disparity is measured with worst-group WER, the best-to-worst absolute gap, and a ratio; paired bootstrap testing uses 2,000 resamples.Claims are limited to depths with aggregate WER at or below 40%, and groups are analyzed only with at least 200 utterances and 30 minutes of audio.Source: §3.4 · p. 5

4 Results and Discussions

Across Fair-Speech, pruning’s disparity effects vary by demographic axis and encoder scale; LoRA lowers WER but can widen racial disparities, while Common Voice gaps persist without consistent amplification.

4.1 Pruning and demographic disparity (RQ1)

  • Ethnicity and gender gaps widen, socioeconomic disparity persists, and the age gap narrows by leveling down.The gender gap grows from 3.3 to 11.1 pp; the age gap narrows because the oldest group degrades slowest.Source: Figure 2 · p. 6, §4.1 · p. 6
  • The Black–Asian WER gap grows from 13.5 pp to 24.5 pp by L-8, while the L-2 aggregate improves even as Black speakers worsen.At L-2, aggregate WER falls from 21.6% to 21.1%, while Black speakers’ error rate rises by 0.9 pp.Source: §4.1 · pp. 5–6

4.2 The role of encoder scale (RQ2)

  • At the first prune, aggregate WER rises by 10.8 pp on small and 2.3 pp on medium, but improves by 0.5 pp on large-v2.Black speakers have the highest WER at every scale and depth; beyond the first prune, the absolute Black–Asian gap widens through L-8.Source: §4.2 · pp. 6–7, Figure 3 · p. 7

4.3 The effect of LoRA adaptation (RQ3)

  • LoRA lowers aggregate WER, including from 37.6% to 33.0% at eight removed layers, but widens the Black-to-Asian ratio at every depth.At ten removed layers, it recovers 8.0 pp for Asian speakers and 3.3 pp for Black speakers.Source: Figure 4 · p. 7, §4.3 · p. 8

4.4 Beyond Fair-Speech: Common Voice English, Dutch, and Danish

  • On Common Voice English and Dutch, disparities persist without systematic widening; English aggregate WER worsens from the first prune.In English, the India/South Asia-to-US ratio changes from 1.42 at L-0 to 1.39 at L-8; Dutch accent and gender ratios remain broadly stable.Source: §4.4 · pp. 8–9
  • Common Voice Danish supports coverage reporting rather than group-effect conclusions because annotations are sparse and aggregate WER is already 35.5% unpruned.Most groups, including all female speakers, fall below the analysability threshold; the 40% usability threshold is crossed after four removed layers.Source: Table 8 · p. 8, §4.4 · p. 9
  • Across corpora, a baseline performance gap does not predict pruning’s effect; it must be evaluated separately by corpus, demographic axis, and model scale.Fair-Speech’s concealed amplification did not consistently transfer to Common Voice English and Dutch.Source: §4.4 · pp. 8–9

5 Conclusion

On Fair-Speech, pruning widens demographic disparities, and aggregate WER can conceal group-level degradation. LoRA also favors better-performing groups, while cross-lingual gaps persist without clear amplification; the findings motivate worst-group-aware evaluation and compression.

5 Conclusion

  • On Fair-Speech, pruning two Whisper Large layers raises the Black-to-Asian WER ratio from 1.99 to 2.15 despite slightly improving aggregate WER.Source: §5 · p. 9
  • The absolute gap widens from 13.5 to 24.5 pp; pruning also increases the low-SES vs affluent gap from 5.9 to 7.2 points.Source: §5 · p. 9
  • LoRA improves aggregate WER at every depth but widens the Black-to-Asian ratio by benefiting better-performing groups more; Common Voice gaps persist without clear amplification.Sparse annotations and high WER prevent reliable group-level conclusions for Danish.Source: §5 · p. 9
  • The findings motivate fairness-aware compression objectives that prioritize worst-group rather than average performance.Source: §5 · p. 9

Limitations

The study evaluates one encoder, one LLM backbone, and one compression method, with a single random seed. Its dataset scope, demographic coverage, and mitigation analysis are also limited.

Limitations

  • The experiments cover only Whisper, Qwen2.5-3B, and top-down layer pruning, with one random seed (42).Source: Limitations · p. 9
  • The main analysis uses English Fair-Speech; cross-lingual evidence is thin, and Danish models were too degraded for group comparison.Source: Limitations · p. 9
  • Demographic axes are analyzed separately despite shared speakers, leaving intersectional effects unexamined; the study evaluates no mitigation techniques.Source: Limitations · p. 9

A Appendix: Training Details

Training settings vary by language to accommodate corpus size and convergence, while other optimization settings remain constant across encoder scales and pruning depths.

A Appendix: Training Details

B Appendix: Relative-to-baseline WER on Fair-Speech

The appendix reports Fair-Speech WER relative to each group’s or model’s unpruned baseline, including tables for large-model ethnic groups and cross-scale comparisons.

B Appendix: Relative-to-baseline WER on Fair-Speech

  • At L-2, aggregate relative WER is 0.98 while Black-speaker relative WER is 1.03, the highest of any group.Source: Table 10 · p. 12
  • Only large-v2 has a pruning depth where aggregate WER improves while Black-speaker WER degrades.Source: Table 11 · p. 12
  • Table 12 reports Fair-Speech ethnicity results for Whisper medium, with utterance counts per group and italicized values beyond aggregate WER 40%.Source: Table 12 · p. 12

C Appendix: Per-group WER at small and medium scale

The appendix reports Black speakers’ WER ranking across encoder scales and depths, and documents Fair-Speech coverage and additional demographic axes.

C Appendix: Per-group WER at small and medium scale

D Appendix: Fair-Speech coverage, SES, gender, and age

E Appendix: Per-group WER under LoRA adaptation

The appendix presents per-group WER for LoRA-adapted Whisper large on Fair-Speech and explains how uncertainty is reported.

E Appendix: Per-group WER under LoRA adaptation

  • Table 16 reports per-group WER under LoRA adaptation for Fair-Speech with Whisper large, complementing Figure 4 and the base-model results in Table 3.Source: E Appendix: Per-group WER under LoRA… · p. 12
  • LoRA-run per-utterance outputs were not retained, so the reported SD is estimated from stored resampling intervals.It reflects the precision of group WER rather than per-utterance spread.Source: E Appendix: Per-group WER under LoRA… · pp. 12–13
  • Table 13 gives Fair-Speech ethnicity results for Whisper small, with italic values beyond the usable range where aggregate WER exceeds 40%.Source: Table 13 · p. 13
  • Table 14 reports utterance and audio-minute coverage for Fair-Speech demographic groups, each exceeding the 200-utterance and 30-minute thresholds.Source: Table 14 · p. 13

F Appendix: Common Voice English, Dutch, and Danish

Across the corpora, demographic disparities persist, but amplification hidden by an improving aggregate appears only for Fair-Speech with the largest encoder.

  • On Common Voice English, India/South Asia trails US and England at every depth, while the relative gap shifts from 1.42 at L-0 to 1.39 at L-8.Source: F Appendix: Common Voice English,… · p. 13
  • On Fair-Speech, the male–female WER gap widens from 3.3 pp at L-0 to 11.1 pp at L-8; female speakers improve slightly after the first prune.Source: Figure 5 · p. 13
  • On Fair-Speech, affluent speakers retain a WER advantage, but the Low-to-Affluent ratio falls from 1.36 to 1.24 as all groups degrade.Source: Figure 6 · p. 13, Table 15 · p. 14
  • On Dutch, Belgian speakers trail Netherlands speakers at every scale and usable depth, while the ratio stays between 1.1 and 1.3; the gender ratio remains near 1.3.The accent groups shown meet the analysability threshold, and each figure panel ends where its model leaves the usable range.Source: F Appendix: Common Voice English,… · p. 13, Figure 8 · p. 14
  • Danish aggregate WER is 35.5% unpruned and crosses the 40% usability threshold during pruning; sparse annotations preclude group-level conclusions.Source: F Appendix: Common Voice English,… · pp. 13–14
  • On Fair-Speech, the oldest group degrades by 9.9 pp by L-8, while the 31–45 group degrades by 19.8 pp and becomes worst-performing by L-6.The worst–best gap narrows by leveling down, not by improving the group that started worst.Source: Figure 7 · p. 14
  • Across the three corpora, disparities persist, but amplification hidden under an improving aggregate appears only on Fair-Speech with the largest encoder.Source: F Appendix: Common Voice English,… · p. 14

Related papers

Loading 2609.38106v1…