Source-linked AI summary
Domain-Incremental Learning for Generative Speech Enhancement
Manjunath Mulimani, Annamaria Mesaros, Minje Kim, Jesper Rindom Jensen
TL;DR
Generative speech enhancement must adapt across changing acoustic domains, but fine-tuning can cause forgetting and language-model-based generative DIL remains underexplored. This paper develops a GPT-based enhancement model and incrementally adapts it with domain-specific LoRA; it reports gains over baselines while identifying adapter selection as a limitation.
Problem
Generative speech enhancement faces sequential domain shifts, while DIL for LM-based generative models and broader dataset-driven shifts remains underexplored.
Source: §1 · p. 1Method
The paper trains a GPT-based generative enhancement model on discrete speech tokens, then adapts it sequentially with domain-specific LoRA while sharing a frozen base model.
Source: §1 · pp. 1–2, §2.3 · p. 2Results
DIL-GenSE consistently outperforms baseline methods in the domain-aware setting and improves over fine-tuning in the domain-agnostic setting.
Source: §4 · p. 4, §3.3 · p. 4Takeaways & Limitations
Domain-specific adapters support incremental enhancement with 1.18M parameters each, about 1.1% of the base model's parameters.
Source: §4 · p. 4Domain-agnostic performance depends on accurate adapter selection, and imperfect selection can underperform the domain-aware setup.
Source: §3.3 · p. 4
Abstract
from arXiv · showhide
We propose a domain-incremental learning framework for generative speech enhancement (SE) that learns from a sequence of datasets or domains recorded under diverse acoustic conditions. Fine-tuning a pretrained model on continuously evolving domains leads to catastrophic forgetting of previously acquired knowledge, while zero-shot generalization often fails to adequately adapt to unseen domains. To address these challenges, we first develop a novel language model-based generative SE model that we then use as a pretrained backbone and incrementally adapt it to acoustically mismatched domains using lightweight domain-specific Low-Rank Adaptation. The proposed framework enables the model to acquire enhancement capabilities for new domains while preserving performance on previously learned domains. Evaluated on four heterogeneous speech datasets, our approach effectively adapts to new domains without forgetting previously learned domains.
1. INTRODUCTION
Speech enhancement models face changing acoustic domains, where zero-shot performance may degrade and fine-tuning can cause forgetting. This work develops a domain-incremental framework for generative speech enhancement and evaluates it across heterogeneous datasets.
1. INTRODUCTION
- Generative enhancement uses semantic representations to produce natural-sounding speech and suppress noise, and language-model approaches have shown stronger unseen-domain generalization than other generative approaches.The paper develops a GPT-based model trained from scratch with discrete semantic and global speech tokens from the Spark-TTS BiCodec tokenizer.Source: §1 · p. 1
- Fine-tuning on a new domain can degrade performance on earlier domains, while zero-shot performance may fall under substantial shifts; storing all data for retraining can be impractical.Practical barriers include privacy concerns, storage requirements, and computational constraints.Source: §1 · p. 1
- Domain-incremental learning handles multiple domains arriving over time and evaluates performance across all domains observed so far, unlike two-domain source-to-target adaptation.For speech enhancement, it must preserve high-quality speech generation across earlier domains while adapting to new acoustic conditions.Source: §1 · p. 1
- LM-based generative speech enhancement remains largely unexplored in domain-incremental learning, especially under shifts spanning different speech datasets rather than new noise types alone.Earlier DIL-SE studies target discriminative models, while this work addresses autoregressive token-generation enhancement.Source: §1 · p. 1
- The framework adapts a pretrained generative enhancement model to sequentially arriving domains using domain-specific LoRA adapters while preserving shared knowledge in the frozen base model.It is evaluated on four heterogeneous speech enhancement datasets with shifts in speech, speakers, recording conditions, environments, and noise.Source: §1 · p. 2
2. METHOD
The method defines sequential domain learning, builds an autoregressive token-generation enhancement model, and incrementally adapts it with domain-specific LoRA while keeping its base weights frozen.
2.1. DIL tasks setup and notations
- Each domain supplies noisy-clean speech pairs; after pretraining on D0, the model encounters additional datasets sequentially, without access to earlier-domain data during current training, and is evaluated on all seen domains.Domains may differ acoustically, though some overlap is unavoidable.Source: §2.1 · p. 2
2.2. Base model for generative speech enhancement
- The base model predicts clean-speech semantic and global tokens autoregressively from noisy-speech acoustic embeddings, then decodes the generated tokens into an enhanced waveform.The conditioning embeddings come from WavLM-Large; clean speech is tokenized with Spark-TTS BiCodec.Source: §2.2 · p. 2
2.3. DIL for generative speech enhancement
- For each domain, LoRA adds trainable low-rank matrices to the shared frozen base weights; inference uses the known domain's adapter or selects the minimum-entropy adapter when domain identity is unavailable.In the domain-agnostic setting, the input is evaluated through the available adapters and the lowest predictive-entropy output is selected.Source: §2.3 · pp. 2–3
3. EVALUATION AND RESULTS
Across five datasets, DIL-GenSE adapts to sequential domains while retaining performance on earlier ones, outperforming generalization and fine-tuning baselines. Evaluation also compares domain-agnostic adapter selection and GenSE on EARS, with that comparison limited to the fairest available domain.
3.1. Datasets, training setup, and baselines
- The benchmark uses D0 for base training and learns DNS, EARS, LibriTTS-R, then VoiceBank incrementally, evaluating each step on domains seen so far.The five datasets span differing speech, recording, acoustic, and noise conditions; baselines include zero-shot generalization, fine-tuning, and joint fine-tuning.Source: §3.1 · p. 3
3.2. Implementation details and evaluation metrics
- Evaluation reports average SpeechBertScore across seen domains and average forgetting; implementation inserts rank-8 LoRA into GPT-2 attention and feed-forward layers.The model is trained incrementally for 10 epochs per domain; DNSMOS is also reported and used for comparison with GenSE.Source: §3.2 · p. 3
3.3. Results
- D1’s shared noise recordings with pretraining support stronger zero-shot generalization, while fine-tuning loses prior-domain performance after adapting to mismatched LibriTTS-R and VoiceBank domains.The authors attribute the EARS decline after LibriTTS-R fine-tuning to the larger distribution mismatch between those domains.Source: §3.3 · p. 4, Figure 2 · p. 4
- DIL-GenSE adapts to highly mismatched domains while preserving earlier-domain performance, outperforming GRL, FT, and joint FT in the domain-aware evaluation.It uses domain-specific LoRA adapters with a frozen base model; joint FT uses data from previously encountered domains.Source: §3.3 · pp. 3–4, Table 1 · p. 3
- DIL-GenSE’s domain-agnostic performance depends on selecting the appropriate LoRA adapter; imperfect selection can underperform the domain-aware setup, though it still beats FT.The authors identify adapter selection, rather than LoRA adaptation itself, as the primary limitation and suggest more reliable selection could improve results.Source: §3.3 · p. 4, Table 2 · p. 4
- On EARS, domain-aware DIL-GenSE outperforms GenSE across all evaluation metrics, despite using one decoder and lightweight adapters rather than two LM decoders.The comparison is restricted to EARS because GenSE’s pretraining data overlap with the other domains makes them unfair comparisons.Source: §3.3 · p. 4, Table 3 · p. 4
4. CONCLUSION
DIL-GenSE incrementally adapts generative speech enhancement with lightweight domain-specific LoRA adapters while mitigating catastrophic forgetting. It outperforms baselines in domain-aware evaluation, though domain-agnostic performance depends on accurate adapter selection.
4. CONCLUSION
- DIL-GenSE consistently outperforms baseline methods in the domain-aware setting while using only 1.18M-parameter LoRA adapters, about 1.1% of the base model’s size.The framework incrementally learns new domains while mitigating catastrophic forgetting.Source: §4 · p. 4
- Each domain-specific LoRA adapter contains 1.18M parameters, about 1.1% of the base model’s parameters.Source: §4 · p. 4
- DIL-GenSE also improves over fine-tuning in the domain-agnostic setting, but performance remains dependent on accurate adapter selection.Source: §4 · p. 4
- Improving domain-agnostic adaptation is identified as an important direction for future research.Source: §4 · p. 4