Source-linked AI summary
Adaptive Fisher-Whitened Cross-Covariance for Low-Resource Speech Recognition
Asmee Mishra, Mengjie Qian, Brechtje Post, Kate Knill
TL;DR
Low-resource ASR adaptation is difficult when languages have little labeled speech and are underrepresented in pre-training, while conventional PEFT does not explicitly use downstream task information to define adaptation subspaces. This paper applies FCCA to Whisper and Qwen3-ASR and adds cross-layer sharing and adaptive-rank extensions. AR-FCCA most consistently improves on standard FCCA across both architectures while retaining the same trainable-parameter budget.
Problem
Many low-resource languages are underrepresented in speech-model pre-training and have little labeled speech, while conventional PEFT does not explicitly use downstream task information to define adaptation subspaces.
Source: §1 · p. 1Method
The paper applies FCCA to Whisper and Qwen3-ASR and introduces AC-FCCA for cross-layer sharing and AR-FCCA for rank allocation across projection matrices under a fixed parameter budget.
Source: §1 · p. 1Results
Takeaways & Limitations
Task-informed subspace construction can support low-resource speech adaptation, and adaptive rank allocation improves parameter efficiency without increasing the trainable budget.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Adapting multilingual speech foundation models to low-resource languages remains difficult, especially for languages that are poorly represented during pre-training. While parameter-efficient fine-tuning (PEFT) reduces the cost of adapting large models, conventional approaches such as LoRA rely on generic low-rank parameterizations and do not explicitly use downstream task information to define the adaptation subspace. To investigate whether task-informed PEFT can better support low-resource ASR, we apply Fisher-Whitened Cross-Covariance Analysis (FCCA) to Whisper and Qwen3-ASR, and introduce two complementary extensions: Asymmetric-Coupled FCCA (AC-FCCA), which exploits structured cross-layer sharing, and Adaptive-Rank FCCA (AR-FCCA), which reallocates adaptation capacity across projection matrices under a fixed parameter budget. Under controlled multilingual experiments, we evaluate these approaches on languages that are poorly represented or unsupported during pre-training alongside well-represented languages. Standard FCCA is competitive with, and usually outperforms, trainable-parameter-budget-matched LoRA. AR-FCCA provides the most consistent improvement over standard FCCA across both model architectures, with statistically significant gains in several evaluation settings, while retaining the same number of trainable parameters. These results show that task-informed subspace construction can be effective for low-resource speech adaptation, and that adaptive rank allocation provides a robust way to improve parameter efficiency without increasing model capacity.
1. INTRODUCTION
Speech foundation models can recognize many languages, yet low-resource and endangered languages often have little labeled speech and remain underrepresented in pre-training. This work tests task-informed PEFT for multilingual low-resource ASR, extending FCCA with cross-layer sharing and adaptive rank allocation; experiments across Whisper and Qwen3-ASR find AR-FCCA consistently improves on standard FCCA.
1. INTRODUCTION
- Underrepresented and endangered languages may have only a few hours of labeled speech, making efficient adaptation of multilingual speech models an important research problem.Speech technologies can support practical applications, linguistic research, and language documentation.Source: §1 · p. 1
- PEFT adapts a small subset of parameters, and task-informed methods use downstream data to identify adaptation directions; FCCA derives paired subspaces from Fisher-whitened input–error statistics.Despite promising results for large language models, FCCA's effectiveness for low-resource speech adaptation remains unexplored.Source: §1 · p. 1
- The study tests whether task-informed PEFT transfers to low-resource ASR by adapting FCCA to speech foundation models.It proposes AC-FCCA for structured cross-layer sharing and AR-FCCA for reallocating adaptation capacity across projection matrices under a fixed parameter budget.Source: §1 · p. 1
- AR-FCCA consistently improves standard FCCA across multilingual benchmarks on Whisper and Qwen3-ASR while remaining competitive with strong PEFT baselines.The work also reports statistically significant improvements over standard FCCA in several settings.Source: §1 · p. 1
2. PROPOSED METHODS
The methods use FCCA to construct task-informed adaptation subspaces and propose two extensions: AC-FCCA shares aligned structure across adjacent layers, while AR-FCCA allocates ranks across matrices under a fixed trainable-core budget.
2. PROPOSED METHODS
- The proposed extensions address distinct adaptation choices: whether subspaces should be shared across layers and how adaptation capacity should be distributed across matrices.Both build on FCCA's task-informed low-rank adaptation framework.Source: §2 · p. 2
2.1. Fisher-Whitened Cross-Covariance Adaptation (FCCA)
- FCCA trains only the r^2 core parameters between fixed task-informed bases, which are derived from Fisher-whitened input–error cross-covariance statistics.Unlike LoRA-XS, which uses singular directions from pretrained weights, FCCA derives its bases from downstream task statistics.Source: §2.1 · p. 2
2.2. Asymmetric-Coupled FCCA (AC-FCCA)
- AC-FCCA quantifies alignment between orthonormal bases, with larger scores indicating more shared subspace structure, and optimizes shared and private bases by alternating block-coordinate updates.After convergence, the shared and private bases are concatenated and the opposite-side basis is recovered as in AC-FCCA-H.Source: §2.2 · p. 2
- AC-FCCA shares the better-aligned side of adjacent-layer subspaces while keeping the opposite side layer-specific, reflecting projection-dependent alignment asymmetry.Its shared–private variant splits the coupled rank-128 subspace into 64 shared and 64 private directions.Source: §2.2 · p. 2
2.3. Adaptive-Rank FCCA (AR-FCCA)
- AR-FCCA assigns matrix-specific ranks according to retained Fisher-whitened spectral energy per added trainable parameter while preserving the total core budget.At the selected ranks, it applies the standard FCCA construction; the experiments use a uniform FCCA baseline rank of 128.Source: §2.3 · p. 2
3. EXPERIMENTAL SETUP
Experiments use FLEURS language data and compare conventional fine-tuning and PEFT baselines with FCCA variants. Whisper is tested on five languages, while Qwen3-ASR provides cross-model evaluation on two unsupported languages.
3. EXPERIMENTAL SETUP
- The evaluation uses Asturian, Sorani Kurdish, and Kyrgyz as primary low-resource targets, with Mandarin and Persian as better-represented controls.Adaptation uses only 7.5–10.5 hours of target-language FLEURS training data; results are reported on held-out test sets.Source: §3 · p. 3
- Whisper medium and Qwen3-ASR-1.7B provide different architectures; Qwen3-ASR is evaluated on Asturian and Sorani Kurdish, which it does not support.Whisper medium is an encoder–decoder model with cross-attention, while Qwen3-ASR combines an audio encoder with an autoregressive LLM-style decoder.Source: §3 · p. 3
- The comparisons include pretrained inference, full fine-tuning, LoRA, standard FCCA, AC-FCCA, and AR-FCCA.LoRA is the PEFT baseline; FCCA and its variants are compared against it alongside vanilla and full fine-tuning.Source: §3 · p. 3
- FCCA methods adapt attention projections across 288 Whisper matrices and 208 Qwen3-ASR matrices, using 256 calibration examples and base rank 128.The methods train 0.61% of Whisper-medium parameters and 0.20% of Qwen3-ASR parameters; calibration is a one-off cost.Source: §3 · p. 3
- Evaluation reports CER for Mandarin and WER for other languages after language-specific normalization and post-processing.Decoding is deterministic greedy generation without an external language model, with a repetition guard to suppress hallucinations.Source: §3 · p. 3
4. RESULTS AND DISCUSSION
On Whisper, FCCA-based methods offer parameter-matched gains over LoRA, with AC-FCCA improving FCCA on most languages and AR-FCCA giving the most consistent gains. On Qwen3-ASR, AR-FCCA also improves FCCA, while AC-FCCA variants increase WER; adaptive ranks retain greater spectral utility than uniform ranks throughout training.
4.1. Whisper Baseline Performance
- Vanilla Whisper performs poorly on unsupported languages, while LoRA-8 trails LoRA-128 by 2–8% WER; FFT is strongest among conventional approaches across most languages.Vanilla Whisper has 8.88% CER on Mandarin, leaving limited room for improvement with fewer than 10 hours of training data.Source: §4.1 · p. 3, Table 2 · p. 4
4.2. Task-informed PEFT - FCCA variants
- FCCA trains 16 times fewer parameters than LoRA-128, with WER 0.36–7.14 percentage points higher, but beats parameter-matched LoRA-8 on three Whisper languages and nearly matches it on others.Source: §4.2 · p. 3, Table 2 · p. 4
- AC-FCCA beats FCCA on four of five Whisper languages; AC-FCCA-H significantly improves Sorani WER from 44.97% to 44.35% and Mandarin CER from 10.46% to 10.19%.Both AC-FCCA constructions also achieved higher Fisher-weighted gradient capture than standard FCCA on a disjoint probe set.Source: §4.2 · p. 3, Table 2 · p. 4
- Across Whisper and Qwen3-ASR, AR-FCCA most consistently improves FCCA, reducing Whisper WER by 0.32–1.13 points and reaching 18.56% Asturian and 42.98% Sorani WER on Qwen3-ASR.Whisper gains are significant on Sorani Kurdish and Mandarin; Qwen3-ASR gains are significant on both evaluated languages.Source: §4.2 · pp. 3–4, Table 2 · p. 4, Table 3 · p. 4
4.3. Generalization to Qwen3-ASR
- On Qwen3-ASR, FCCA beats parameter-matched LoRA-6 by 0.17% Asturian WER and 0.77% Sorani WER but trails LoRA-128; AC-FCCA variants increase WER.Source: §4.3 · p. 4
4.4. Analysis of Adaptive Rank Allocation
- Whisper concentrates adaptive rank in decoder output projections; across both models and all languages, initial AR-FCCA allocations retain more task-relevant spectral utility than uniform ranks at every checkpoint.Source: §4.4 · p. 4
5. CONCLUSION
Across Whisper and Qwen3-ASR, AR-FCCA most consistently improved on standard FCCA at the same trainable-parameter budget, with significant gains in several settings. Standard FCCA was highly parameter-efficient, while AC-FCCA added only limited gains.
5. CONCLUSION
- AR-FCCA consistently improved over standard FCCA across both models and multiple languages at the same trainable budget, with statistically significant gains in several settings.It reallocates the fixed budget across projection matrices.Source: §5 · p. 4
- Standard FCCA provided a highly parameter-efficient alternative to conventional PEFT with a matched parameter budget.Source: §5 · p. 4
- AC-FCCA variants delivered only limited additional gains over standard FCCA.Source: §5 · p. 4
- AR-FCCA assigned markedly different ranks across layers and projections, retaining higher FCCA spectral utility than uniform rank allocation throughout training.Source: §5 · p. 4
- Systematic parameter-matched LoRA comparisons such as LoRA-XS remain unaddressed and are left for future work.Source: §5 · p. 4