Source-linked AI summary
CAFE: Confining Feature Distillation to Language-Routed Adapters for Compact Multilingual Automatic Speech Recognition
Tianchi Liu, Nicholas Moratelli, Khurram Azeem Hashmi, Mohammadreza Zolfaghari, Rishabh Jain, Louis Lu, Changdae Park, Amril Nazir
TL;DR
In compact multilingual ASR, it is not well understood whether feature distillation adds to logit distillation when languages share an encoder. The paper analyzes the gradients of common feature losses and confines CKA to adapters that English bypasses. CAFE improves over the Baseline mainly in English, with a smaller multilingual gain.
Problem
Whether feature distillation adds to logit distillation in multilingual ASR with a shared encoder is not well understood, despite the need to serve English and lower-resource languages together.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The findings support confining feature distillation to parameters English bypasses rather than resolving the language trade-off through the loss.
Source: §1 · p. 2, §8 · p. 8Each configuration is trained once; the authors say their checks provide only partial evidence and do not fully replace a multi-seed study.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Feature distillation is usually expected to add to logit distillation. We find that this expectation does not hold for two common feature losses in compact multilingual automatic speech recognition (ASR), and analyze why through their gradients. The gradient of a projection-based loss is orthogonal to the task gradient, while that of Centered Kernel Alignment (CKA), a similarity-based loss, is weakly aligned with it. CKA lowers the word error rate (WER) on non-English languages, but at the expense of data-rich English. Three loss-level remedies (one trained, two probed) all fall short, and the CKA gradient overlaps with the subspace English uses in the shared encoder by 97.5% to 99.5% (about 81% by chance). Guided by this diagnosis, we propose CAFE (Confined-Adapter Feature distillation), which confines the CKA gradient to adapters that English bypasses, addressing the trade-off in the parameters rather than in the loss. On the Open ASR Leaderboard, CAFE lowers the average WER from 8.55% to 8.02%, beats an adapter-only capacity control at 8.51%, and makes English more stable across checkpoints. At 254M parameters, it trails the 0.6B teacher by 2 WER points and is competitive with much larger models across 48 test sets in 24 languages. We use 564k hours of training data and state-of-the-art models to keep the findings relevant to practice.
1 Introduction
The paper asks whether feature distillation adds value beyond logit distillation in compact multilingual ASR, where languages share an encoder. It diagnoses why common feature losses can trade English performance for non-English gains and introduces CAFE to confine that trade-off to adapters.
1 Introduction
- Multilingual ASR must serve data-rich English and lower-resource languages with one model, yet the gap between them persists.The paper frames this as a challenge for multilingual systems.Source: §1 · p. 1
- Neither of two common feature losses adds to logit distillation: the projection-based loss has almost no effect, while CKA trades English performance for non-English gains.The projection gradient is orthogonal to the task gradient, while the CKA gradient acts on directions English uses.Source: §1 · p. 2
- The multilingual CKA gradient overlaps English’s encoder subspace by 97.5–99.5%, versus about 81% by chance, explaining why masking English fails and other loss-level remedies fall short.The paper argues against two other loss-level remedies.Source: §1 · p. 2
- CAFE confines the CKA gradient to adapters that English bypasses, moving the trade-off from the loss to the parameters.This is the proposed response to the shared-encoder conflict.Source: §1 · p. 2
- CAFE improves mainly in English, beats an adapter-only Capacity control, halves English WER variation across checkpoints, and trails the 0.6B teacher by 2 WER points at 254M parameters.The paper reports the smaller multilingual gain alongside the English gains.Source: §1 · p. 2
2 Related Work
Prior work uses feature matching and CKA for model compression, and language-specific adapters for multilingual transfer. This paper distinguishes its setting and uses adapters to constrain the distillation gradient rather than to add transfer capacity.
2 Related Work
- Feature distillation matches activations or relations among examples, while another line improves the logit term; this work finds two common feature losses do not help in its setting.The cited feature-matching approaches include Romero et al. (2015), Zagoruyko and Komodakis (2017), Park et al. (2019), and Tian et al. (2020).Source: §2 · p. 2
- Prior CKA and feature-distillation work reports positive results, but this paper limits its different findings to compact, pruning-initialized multilingual ASR trained with strong logit distillation on English-dominated data.The paper reports that VkD-style projection has almost no effect and CKA helps lower-resource languages relative to English.Source: §2 · p. 2
- Earlier multilingual distillation adds language-specific gated experts for per-language accuracy, whereas these adapters keep the feature-distillation gradient away from the shared encoder.The comparison is with Ferraz et al. (2024).Source: §2 · p. 2
- Unlike adapter work that uses language-specific modules for transfer, this paper uses them to restrict where a distillation gradient acts; gradient surgery is a poor fit because the gradients are orthogonal, not opposed.The gradient-surgery discussion cites Yu et al. (2020) and Huang et al. (2025).Source: §2 · pp. 2–3
3 Preliminaries
The experiments distill a pruned 254M student from a 0.6B teacher and evaluate English and multilingual WER on the Open ASR Leaderboard. The objective combines supervised and logit losses with optional feature losses, including projection-based matching and CKA.
3 Preliminaries
- A 254M student initialized by structured pruning is distilled from Parakeet-TDT-0.6B-v3; evaluation uses Open ASR Leaderboard EN-7 and ML-9 WER with averages over the last seven checkpoints.EN-7 and ML-9 summarize seven English and nine multilingual test sets, respectively.Source: §3 · p. 3
- The training objective combines supervised transducer and logit-distillation losses with an optional feature-distillation loss; the Baseline sets the feature-loss weight to zero.The logit term is a temperature-scaled KL divergence between student and teacher label and duration distributions, summed over both heads.Source: §3 · p. 3
- VkD projects student features into the teacher dimension and matches normalized frame-wise features, while linear CKA compares feature similarity structure without requiring a projection.The w/o norm VkD variant omits LayerNorm; CKA loss is averaged over the distilled layers.Source: §3 · p. 3
4 Analysis: When and Why Feature Distillation Helps or Hurts
In compact multilingual ASR, projection-based feature distillation stalls while CKA trains but trades English performance for non-English gains. Gradient analysis and three loss-level remedies indicate that the trade-off is difficult to isolate within the shared encoder.
4.1 Which feature loss actually trains the student?
- Feature gradients are about 10^3 times smaller than the supervised gradient; VkD is task-orthogonal, while CKA has weak positive alignment with cosine about 0.09.The much larger orthogonal part of CKA’s gradient adds representation structure that helps some languages and hurts others.Source: Table 1 · p. 3, §4.1 · p. 3
- Projection losses stop decreasing after warmup, whereas CKA falls to about a quarter of its initial value; the projection loss also stays flat in the full recipe.Figure 2 shows feature-loss trajectories and step time for feature-only runs.Source: §4.1 · p. 3, Figure 2 · p. 4
- The projection loss is not stalled by limited student capacity: 640 dimensions capture about 99% of teacher-feature energy, while projection costs about 230× more per step than without projection.A CKA step is about 2.8× faster than a projection-loss step.Source: §4.1 · p. 4
4.2 CKA helps non-English languages and hurts English
- CKA achieves the best multilingual WER at 6.83% on ML-9 but the worst English WER at 11.25% on EN-7.The English degradation appears late in training, while English WER follows the Baseline early on.Source: §4.2 · p. 4, Table 2 · p. 4
4.3 Why the trade-off is hard to fix at the loss level
- Masking English from CKA leaves English WER at 11.21% versus 11.25% and raises multilingual WER from 6.83% to 7.18%; loss-gap gating also fails to identify the harm.The loss gap is positive on 96%–98% of utterances and is not larger on English.Source: §4.3 · p. 5
- The multilingual CKA gradient overlaps English’s activation subspace by 97.5%–99.5%, versus about 81% by chance, so projecting it out would remove most of CKA.The overlap is stable during training; multilingual gains and English harm use the same encoder directions.Source: §4.3 · p. 5
- Because languages share one representation and CKA statistics come mostly from multilingual data, the proposed loss-level remedies do not separate the gains from the harm.The authors conclude that the non-English side needs its own parameters.Source: §4.3 · pp. 4–5
5 Proposed CAFE: Confined-Adapter Feature Distillation
CAFE routes non-English utterances through adapters and confines feature distillation to those adapters. This keeps the CKA gradient out of the shared encoder that English uses.
5 Proposed CAFE: Confined-Adapter Feature Distillation
- CAFE applies the feature loss through adapters used by non-English utterances, so its gradient does not reach the shared encoder.The adapters are shared across non-English languages; English bypasses them.Source: §5 · pp. 5–6
- The adapters begin with zero updates, leaving the model identical to the Baseline at initialization.They are the only language-dependent parameters.Source: §5 · pp. 5–6
- A stop-gradient confines CKA updates to the adapters, while supervised and logit losses continue updating the shared encoder.The CKA term matches adapted non-English features to teacher features, using the same fixed feature-loss weight without a late-training schedule.Source: §5 · p. 6
6 Experimental Setup
The experiments train a pruned multilingual ASR student with shared data, optimization, and schedule across systems. Evaluation spans 48 held-out test sets in 24 languages, with controls designed to isolate adapter and feature-loss effects.
6 Experimental Setup
- All systems use a 24-layer, 640-dimensional student trained for 350k steps with AdamW and Noam annealing; feature distillation targets layers 3, 7, 11, 15, 19, and 23.The teacher has 1024 dimensions, and CAFE adapters have rank 64.Source: §6 · p. 6
- Training uses about 116.9 million utterances, or 564k hours, across 25 languages and 10 corpora.English accounts for 14% of sampled training frames despite being the largest language by hours.Source: §6 · p. 6
- Evaluation covers 48 held-out test sets, 296k utterances, and 24 languages, including the Open ASR Leaderboard and multilingual Common Voice and MLS sets.The Open ASR Leaderboard includes EN-7 English short-form sets and ML-9 sets from FLEURS and MLS.Source: §6 · p. 6
- Checkpoints are saved every 6k steps, and checkpoint averaging smooths consecutive-checkpoint fluctuations at no extra training cost.The reported ranking holds for averaging windows of 5, 7, and 10 checkpoints.Source: §6 · p. 6
- The four-system 2 × 2 comparison uses CAFE versus Capacity to isolate confined CKA and Capacity versus Baseline to isolate adapters.CAFE without stop-gradient tests the effect of removing that constraint.Source: §6 · p. 6
7 Results
CAFE improves on the Baseline and an adapter-only Capacity control, while reducing English checkpoint variation; its adapter overhead is small, and it is competitive with larger systems across multiple test suites.
Main results
- EN-7 WER falls from 9.85% to 9.03% and ML-9 from 7.24% to 7.00% with CAFE, which beats Capacity on both measures.The main comparison reports checkpoint-averaged WER; both English gains are significant, and adapters alone yield only a small English gain.Source: §7 · p. 6, Table 3 · p. 7
Replication on 8 languages
- EN-7 WER improves from 10.11% to 8.71% and ML-9 from 6.62% to 6.53% in the matched 8-language replication.The repeated pattern with less data indicates that the effect is not specific to the 25-language setup.Source: §7 · p. 7
Adapter overhead
- Adapters lower throughput by about 2.0%–3.8% while leaving latency and memory almost unchanged.Efficiency figures use the single-GPU protocol; Table 4 compares CAFE with the same model without adapters.Source: §7 · p. 7, Table 4 · p. 7
Stability and stop-gradient
- English WER standard deviation is 0.32 for CAFE versus 0.82 for the Baseline, and removing stop-gradient raises EN-7 WER to 10.44%.Without stop-gradient, ML-9 remains 6.95%, matching CAFE, while English standard deviation reaches 0.69.Source: §7 · p. 7
Comparison with external systems
- Among compact systems, CAFE is most accurate on the multilingual leaderboard sets, MLS-8, and Common Voice v25, while OWSM v4 small is slightly better on English.On MLS-8, CAFE's WER is 6.58%, below Whisper-medium at 7.76%; larger models win on some suites.Source: §7 · p. 7
- Qwen3-ASR-0.6B has no 24-language Common Voice mean because some languages are outside its supported set.The external comparison spans 48 test sets after removing four duplicated MLS languages.Source: §7 · p. 7
CAFE (Ours)
CAFE routes adapters around English and confines CKA to them; training curves show steadier English WER, while the teacher gap varies by test suite.
CAFE (Ours)
- CAFE matches the teacher on MLS-8 but trails it on English short-form, with WER of 9.03% versus 6.83%.The authors partly attribute the English gap to the student seeing less English data and omit corpora with potential legal concerns to support release.Source: CAFE (Ours) · p. 8
- Adapters carry the multilingual gain and must be skipped for English: applying them to every utterance raises EN-7 WER to 15.95%, while never applying them raises ML-9 to 8.04%.A 0.19%-parameter language gate handles unknown languages at a cost of 0.05 points in average WER.Source: CAFE (Ours) · p. 8
- In the second half of training, Baseline English WER spikes above 14%, while CAFE stays lower, varies less, and keeps improving.The curves are decoded on test sets for analysis only and are not used for model selection.Source: CAFE (Ours) · p. 8, Figure 4 · p. 8
- English bypasses the adapters and receives no CKA gradient, yet CAFE improves over Capacity from 9.67% to 9.03% through shared parameters.The authors attribute this gain to less late-training interference on the shared encoder.Source: CAFE (Ours) · p. 8
8 Conclusion
Feature distillation does not help uniformly in compact multilingual ASR: gradient analysis explains the trade-off, and CAFE confines CKA to adapters English bypasses. CAFE improves mainly English performance, also helps multilingual WER, beats an adapter-only control, and reduces English checkpoint variation.
8 Conclusion
- In compact multilingual ASR, projection-based feature-distillation gradients are orthogonal to the task, whereas CKA gradients are weakly aligned, lowering non-English WER but raising English WER.Source: §8 · p. 8
- CAFE confines CKA gradients to language-routed adapters that English bypasses, moving the remedy from the loss to the parameters.Loss masking fails because the CKA gradient overlaps almost entirely with the subspace English uses; diagnostics also argue against gating or projecting it.Source: §8 · p. 8
- CAFE improves mainly English performance with a smaller multilingual gain, beats an adapter-only control, and more than halves English WER variation across checkpoints.These outcomes support the conclusion that feature distillation's effects depend on where its gradient is applied.Source: §8 · p. 8
Limitations
The findings are limited to students initialized by structured pruning; whether they hold with random initialization remains open. Each configuration was trained once, and the authors say the checks only partly address the absence of a multi-seed study. CAFE also needs an English/non-English decision, supplied by a gate in deployment.
Limitations
- The findings apply to students initialized by structured pruning; whether the same pattern holds with random initialization is left for future work.The authors describe pruning as standard at this data scale and limit their claims to that setting.Source: Limitations · p. 9
- Each configuration was trained once, and the authors say their checks provide only partial evidence rather than replacing a multi-seed study.The 8-language pair uses a different seed and language set, while checkpoint-level tests capture some within-run variation.Source: Limitations · p. 9
- CAFE needs to know whether an utterance is English; the main results use test-set language labels, while deployment uses a released language gate.The gate adds 0.19% parameters and raises average WER from 8.02% to 8.07%, still below the Baseline's 8.55%.Source: Limitations · p. 9
Ethical Statement
The study uses public research datasets under their licenses, reports language and dialect results, and notes that performance outside its test sets is not guaranteed. Its efficiency protocol measures size, throughput, latency, and peak GPU memory under controlled conditions.
Ethical Statement
- The study uses public research datasets under their licenses, collects no new human-subject data, and conducts no annotation study.It avoids speaker re-identification and follows official train/test splits, with additional Common Voice overlap removal.Source: Ethical Statement · p. 9
- The authors report dialectal English and per-language results, while noting that performance outside the test sets is not guaranteed.They intend the system for accessibility, on-device, and offline transcription, and advise consent and safeguards against misuse.Source: Ethical Statement · p. 9
- Training uses 48 H100 GPUs and 564k audio hours, while the resulting 254M-parameter student is described as fast and low-memory for inference.The paper characterizes training as a significant one-time energy cost and inference as the dominant recurring deployment cost.Source: Ethical Statement · p. 9
A Efficiency measurement protocol
- Inference cost is measured by model size, RTFx, P50 latency, and peak GPU memory, with measurements made on one GPU under fixed conditions.RTFx is reported at batch sizes 1 and 32; P50 latency uses batch size 1, and peak memory uses batch size 32.Source: A Efficiency measurement protocol · p. 13
- Efficiency timings use MLS-8 clips capped at 30 seconds, decoded to WAV in advance, so measurement includes model computation rather than I/O.The same 4,000-utterance set is used across models and GPUs.Source: A Efficiency measurement protocol · p. 13
B Per-dataset results for external systems
The appendix contextualizes external-system comparisons, training and test data, checkpoint-window robustness, deployment gating, and significance tests. It finds the main CAFE pattern persists across checkpoint windows and reports that gating preserves gains with a small average-WER increase.
B Per-dataset results for external systems
- Table 6 compares systems to show that the student and teacher are competitive for their size, not to rank systems, since some external models cover far more languages, including over 1600 for omniASR.All systems, including the teacher, use the same decoding and scoring code and official Open ASR Leaderboard text normalizer. The teacher’s SPGISpeech WER is 3.98%, close to Whisper-medium’s 2.92%.Source: B Per-dataset results for external… · p. 13, Table 6 · p. 14
C Training and evaluation data
- Leaving out Fisher and YouTube-Commons mainly affects spontaneous, accented English with named entities, while all systems use the same training and test data.The authors connect the omissions to larger teacher-student gaps on SPGISpeech, Earnings, and AMI.Source: C Training and evaluation data · p. 15
D Robustness to the checkpoint-averaging window
- CAFE stays ahead of the Baseline across 5-, 7-, and 10-checkpoint averaging windows, with average WER between 8.02% and 8.06%.The Baseline's English WER rises from 9.85% to 9.99% at the widest window, while multilingual WER is stable for both.Source: D Robustness to the… · p. 17
E Language gate for deployment
- The language gate reaches 98.9% accuracy, and gated average WER rises only from 8.02% to 8.07%, remaining ahead of the Baseline.Applying adapters to every utterance raises EN-7 from 9.03% to 15.95%; applying them to none raises ML-9 from 7.00% to 8.04%.Source: E Language gate for deployment · p. 18
F Significance testing
- Significance tests support CAFE's larger English gain and smaller multilingual gain; against Capacity, checkpoint-level Wilcoxon tests give p=0.047 on EN-7 and p=0.031 on ML-9.The authors caution that seven checkpoint pairs limit the sign test's resolution.Source: F Significance testing · p. 18, Table 13 · p. 18