Source-linked AI summary
Disentangling Linguistic and Paralinguistic Information with Routed Sparse Autoencoders
Beimnet Bekele Guta, Xiaoyu Yang, Guangzhi Sun, Philip C. Woodland
TL;DR
Speech encoders carry linguistic and paralinguistic information in shared, entangled representations, while sparse feature discovery alone does not guarantee predefined factorisation. The paper combines a TopK sparse autoencoder with route-specific supervision and cross-factor adversaries; probes and route interventions show factor-specific separation across encoders and corpora.
Problem
Speech encoders place linguistic and paralinguistic properties in a shared entangled representation, and sparse feature discovery does not itself route codes into predefined factor-specific subspaces.
Source: §1 · p. 1Method
The framework combines a TopK sparse autoencoder with unit-level linguistic and paralinguistic routing, route-specific task supervision, and cross-factor adversarial objectives.
Source: §1 · p. 1Results
Independent probes show factor-specific retention and suppression across SPEAR and WavLM, while route interventions transfer the swapped factor and preserve information carried by the unchanged route.
Source: §5 · p. 4, §4.2 · p. 4Takeaways & Limitations
The route organisation is observed with both SPEAR XLarge and WavLM Large and is retained under cross-corpus evaluation on MSP-Podcast.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Self-supervised speech encoders contain linguistic and paralinguistic information in a shared, entangled representation space. We combine a TopK sparse autoencoder with route-specific supervision and cross-factor adversaries. Across frozen SPEAR and WavLM encoders, independent probes show factor-specific retention and suppression: linguistic information remains stronger in the linguistic route, while paralinguistic factors, including speaker identity, emotion, and prosody, are retained in the paralinguistic route and substantially reduced in the linguistic route. The route organisation learned on LibriSpeech persists on MSP-Podcast without representation-side retraining. Feature-space route interventions further transfer the swapped factor while largely preserving the information carried by the unchanged route. These results show consistent route-selective separation across encoders, corpora, independent probes, and representation-level interventions.
1. INTRODUCTION
Self-supervised speech encoders support varied tasks, but linguistic and paralinguistic properties remain entangled in shared representations. This work addresses the gap between discovering sparse features and explicitly routing them into factor-specific subspaces.
1. INTRODUCTION
- Speech representations support phone, speech recognition, speaker, emotion, and prosody tasks, yet the same activation can carry multiple kinds of information.This entanglement makes selective interpretation, removal, or modification of individual properties difficult.Source: §1 · p. 1
- Sparse autoencoders map dense representations to wider sparse codes, but sparsity alone does not ensure units align with factors selected in advance.Dictionaries can depend on model width and training choices, and reconstruction need not recover a canonical factorisation.Source: §1 · p. 1
- Prior speech and audio SAE studies primarily analyse discovered features rather than route sparse codes into task-specific subspaces under an explicit disentanglement objective.This is the gap the paper addresses.Source: §1 · p. 1
- The paper combines a TopK sparse autoencoder with unit-level routing, route-specific task supervision, and cross-factor adversaries to organise linguistic and paralinguistic routes.Phone recognition supervises the linguistic route; speaker identification, emotion recognition, and prosody prediction supervise the paralinguistic route.Source: §1 · p. 1
- The framework is evaluated on frozen SPEAR and WavLM representations across LibriSpeech and MSP-Podcast using independent probes for linguistic and paralinguistic information.The evaluation also includes reconstruction-only, fixed-routing, and quota-freeze SAEs, Eta-WavLM, adversarial ablations, frozen cross-corpus transfer, and controlled route interventions.Source: §1 · p. 1
2. METHODOLOGY
A frozen speech encoder feeds a TopK sparse autoencoder whose units are divided into linguistic and paralinguistic routes. Route-specific supervision and cross-factor adversaries train the routes to retain intended information while reducing information associated with the other factor.
2. METHODOLOGY
- A frozen speech encoder feeds a TopK SAE, and its active units are partitioned into linguistic and paralinguistic routes while reconstruction and task-supervision losses train the SAE, masks, and task heads.The speech encoder remains frozen.Source: §2 · pp. 1–2, Figure 1 · p. 2
2.1. Sparse Autoencoder
- The SAE projects frame-level encoder representations into an overcomplete sparse code, with TopK selecting active elements for reconstruction.The dictionary width K exceeds the input dimension D; TopK fixes the active-unit count without shrinkage on retained activations, while AuxK uses inactive units to predict reconstruction residuals and encourage their reuse.Source: §2.1 · p. 2
2.2. Task-Specific Routing
- Learnable complementary masks assign each latent unit to either the linguistic or paralinguistic route, which are trained to retain intended information and reduce recoverability of the other factor.Learned masks are global across frames and utterances; fixed routing predetermines membership and budgets, whereas quota-freeze learns assignments before freezing them and setting separate route quotas.Source: §2.2 · p. 2
2.3. Task Supervision and Adversarial Training
- Phone recognition supervises the linguistic route, while speaker, emotion, and prosody tasks supervise the paralinguistic route; cross-factor adversaries target the opposite route.A gradient reversal layer is the identity in the forward pass and multiplies the backward gradient by −γ; complementary masks ensure the complete sparse code remains available for reconstruction.Source: §2.3 · pp. 2–3
3. EXPERIMENTAL SETUP
Experiments use frozen SPEAR XLarge and WavLM Large encoders, LibriSpeech and MSP-Podcast, and fixed or learned route configurations. Independent post-hoc probes and controlled recipient–donor interventions assess information retention, suppression, transfer, and preservation.
3.1. Data, Encoders, and Configurations
- The study uses LibriSpeech train-clean-100 and MSP-Podcast, evaluating phone recognition on both and speaker, emotion, and prosody tasks on MSP-Podcast.Frozen SPEAR XLarge and WavLM Large representations are layer-normalised and uniformly averaged; the upstream encoders remain frozen during SAE training.Source: §3.1 · p. 3
- All SAEs use dictionary width K = 5120, TopK k = 256, AuxK budget 64, and AuxK weight 1/32; fixed routing assigns 4096/1024 units with active budgets 240/16.Quota-freeze assignments are learned until step 4,000, then mean activity over 20 sampled training batches determines post-freeze budgets.Source: §3.1 · p. 3
3.2. Evaluation Protocol
- Independent probes are trained post hoc on frozen representations, separately from the SAE, masks, and training adversaries, and use PER, SID accuracy, emotion UAR, prosody correlations, and reconstruction MSE.For cross-corpus evaluation, LibriSpeech-trained models are tested on an MSP-Podcast subset while the SAE, decoder, routing masks, and quotas stay frozen; only MSP-specific phone and speaker probes are trained.Source: §3.2 · p. 3
- Controlled interventions swap a donor’s paralinguistic route into a recipient or swap the donor’s linguistic route, then use independent phone and speaker evaluators on decoded features.The protocol uses 250 held-out recipient–donor pairs with different speakers and at most 10% duration difference; donor routes are resampled to match recipient frame counts.Source: §3.2 · p. 3
4. RESULTS
Across encoders and corpora, the routes show factor-specific retention and suppression; controlled swaps support distinct functional roles, and adversarial objectives provide complementary suppression.
4.1. Linguistic and Paralinguistic Separation
- Across SPEAR and WavLM, phone information remains stronger in zL, while speaker, emotion, and prosody are more recoverable from zP on MSP-Podcast.Reconstruction-only preserves both information types without producing the selected factorisation; fixed routing more strongly suppresses paralinguistic information in zL, while quota-freeze more strongly suppresses linguistic information in zP and offers a more balanced reconstruction and selectivity trade-off.Source: §4.1 · p. 3, Table 1 · p. 4
- Without fine-tuning, LibriSpeech-trained SPEAR routing retains the opposing route pattern on MSP-Podcast under both fixed and quota-freeze configurations.Fixed routing yields 43.92/99.72% PER and 12.82/82.71% SID accuracy; quota-freeze yields 44.56/90.38% PER and 3.00/82.29% SID accuracy.Source: §4.1 · p. 4
4.2. Controlled Route Interventions
- Swapping zP preserves recipient phone information while transferring donor speaker identity, whereas swapping zL transfers donor phone information while preserving recipient speaker identity.The reported swaps support functionally distinct routes under both routing configurations.Source: §4.2 · p. 4, Table 2 · p. 4
4.3. Ablations
- Phone and speaker adversaries provide complementary suppression by reducing phonetic recoverability from zP and speaker recoverability from zL.Without adversaries, desired phone and speaker information is retained, but cross-factor information remains recoverable.Source: §4.3 · p. 4, Table 3 · p. 4
5. CONCLUSION
This work investigates disentangling pretrained speech representations with sparse autoencoders. Under suitable inductive biases, the routes show factor-specific retention and suppression across encoders and cross-corpus evaluation, while swapping them reveals distinct functional roles.
5. CONCLUSION
- With suitable inductive biases, a sparse autoencoder can produce strong route-selective separation in pretrained speech representations.Route-specific task supervision, cross-factor adversaries, and a stable routing boundary organize latent units into linguistic and paralinguistic routes.Source: §5 · p. 4
- Independent probes show that each route retains its intended information while substantially reducing cross-factor recoverability.This pattern is observed with both SPEAR XLarge and WavLM Large and retained under cross-corpus evaluation on MSP-Podcast.Source: §5 · p. 4
- Route swapping shows that the learned linguistic and paralinguistic routes have distinct functional roles.Source: §5 · p. 4