When Do Self-Supervised Representations Help Speech Restoration?

• Prior work uses SSL representations either as training guidance or as conditioning features, while SSL feature cleansing first recovers clean SSL features from degraded speech.Cleansing has used language models or fine-tuned SSL models; the language-model approach is described as more computationally demanding.Source: §1 · p. 1
• Under severe degradation, cleansing degraded SSL features toward clean ones can misalign with perceptual quality, challenging the assumption that minimizing feature distance improves restoration.The collapse is observed across multiple SSL encoders and perceptual-quality proxies.Source: §1 · p. 1
• ERI injects intermediate restoration-backbone features into the SSL through a lightweight conditioning network, supplying partially restored evidence without a separate pre-restoration model.The intermediate features are intended to mitigate perceptual correlation collapse.Source: §1 · p. 1
• ERI achieves competitive performance on the VoiceFixer benchmark.Source: §1 · p. 1

2.1. Some utterances lose their identity in the SSL feature space at low SNRs, while others remain firmly anchored to their clean counterparts.

• In WavLM feature space, some degraded utterances converge toward noise embeddings despite audible speech, while others at the same SNR remain near their clean anchors.Some samples enter the noise-occupied region around 0 dB, but the transition varies by utterance.Source: §2.1 · p. 2

2.2. Lower SSL loss does not always translate into higher perceptual quality, especially when speech degradation is severe.

• The analysis compares clean–degraded SSL cosine distance with quality predictions across four degradation severities, using multiple SSL encoders and neural and model-free quality measures.The degradation pipeline varies noise, bandwidth limitation, clipping, reverberation, and packet loss.Source: §2.2 · p. 2
• Beyond a critical degradation severity, SSL feature distance no longer rank-orders perceptual quality well, so minimizing SSL loss does not always improve perceived quality.Source: §2.2 · pp. 2–3
• The collapse appears across four SSL representations and persists beyond bootstrap uncertainty, while LSD shows the same trend despite not perfectly tracking subjective quality.Source: §2.2 · p. 3

3. EARLY RESTORATION INJECTION

• ERI reshapes SEMamba++ bottleneck features, projects and pools them to match WavLM, then concatenates them with WavLM features as a condition.The conditioning feature is projected back to WavLM's channel dimension before injection.Source: §3 · p. 3
• Cleansed layer-24 WavLM features are fused into SEMamba++ with an AdaLN-based module, using SEMamba++ layer 2 features and fusion at layer 3.WavLM is fine-tuned with LoRA.Source: §3 · p. 3
• Stop-gradient blocks SSL loss from updating SEMamba++ and blocks spectrogram-level SR loss from reaching WavLM, keeping the models focused on their respective objectives.Without the stop-gradient before SSL condition fusion, SSL loss was not effectively minimized and performance degraded.Source: §3 · p. 3

3.1. Training objective

• The end-to-end generator objective adds an L2-normalized MSE loss on layer-24 SSL features to the LSGAN-based SR generator loss.The total generator loss is Ltotal = LG + LSSL.Source: §3.1 · p. 3

4.1. Training

• Training uses VCTK-0.92 and 300 hours of LibriTTS-R speech, with noise, reverberation, bandwidth limitation, and clipping simulated; full ERI training runs for an additional 300k steps.Training SNR ranges from -5 to 20 dB, and bandwidth cutoffs range from 2-7 kHz.Source: §4.1 · p. 3

4.2. Evaluation

• System comparisons use the official VoiceFixer test set covering noise, reverberation, bandwidth limitation, and clipping, against GAN, generative, SSL-based, and regressive baselines.The listed baselines are VoiceFixer, SEMamba++, AnyEnhance, SIDON, and TF-GridNet.Source: §4.2 · p. 4
• Evaluation measures perceptual quality with SCOREQ, UTMOS, and OVRL, signal fidelity with PESQ, intelligibility with ESTOI, and phoneme similarity with LPS.Within-model comparisons use a simulated set built from the training section's test split.Source: §4.2 · pp. 3–4

VoiceFixer benchmark

• ERI attains the best PESQ, ESTOI, and LPS, and second-best UTMOS, among the evaluated VoiceFixer test-set baselines.Its real-time factor rises from 0.015 for SEMamba++ to 0.020 for ERI on a single RTX 4090 GPU, despite added parameters.Source: §5 · p. 4

ERI effectiveness

• ERI improves over the baseline with a parameter gap below 1M, remains effective with a 300k-step Stage 1 backbone, and retains its strengths with HuBERT.The baseline comparison increases r and α and attaches LoRA to both FFNs to compensate for ERI’s parameter increase.Source: §5 · p. 4, Table 2 · p. 4

Perceptual gain ratio

• The perceptual gain ratio substantially favors ERI, indicating that SSL-loss minimization translates more effectively into perceptual quality gain with ERI.The ratio is the SCOREQ increment per 10^-3 reduction in SSL loss, anchored after initial training stabilization.Source: Figure 4 · p. 4, §5 · p. 4

Design study

• The design study finds that using two ERI layers performs better than one, removing stop-gradient degrades perceptual quality, and AdaLN adds no significant gain over concatenation.The study used 300k Stage 1 training steps; the authors attribute ERI’s benefit to intermediate features rather than increased capacity.Source: §5 · p. 4, Table 3 · p. 4

Conclusion

• Under severe degradation, SSL feature cleansing does not fully align with the restoration objective, a trend observed across multiple neural quality proxies.The proxy agreement suggests the trend is not an artifact of a single metric.Source: §6 · p. 4
• Early Restoration Injection conditions SSL feature cleansing on partially restored intermediate features from the restoration model.The method adds simple operations, and experiments attribute its gain to the injected features rather than added capacity or a more expressive fusion mechanism.Source: §6 · p. 4
• ERI improves over the unconditioned baseline at matched parameter count and performs competitively against multiple baseline models.Source: §6 · p. 4