PaperlayerView exact revision on arXiv

Source-linked AI summary

GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling

Jixun Yao, Hexin Liu, Chen Chen, Yuchen Hu, EngSiong Chng, Lei Xie

arXiv:2502.02942v1eess.AScs.SD

TL;DR

Speech enhancement methods often overlook semantic information, while noisy input complicates semantic and acoustic modeling. GenSE uses discrete tokens and hierarchical language-model generation; it reports improved enhancement quality and generalization over baselines, with inference speed remaining a bottleneck.

  • Problem

    Speech enhancement approaches often overlook semantic information, while noise complicates modeling both semantic and acoustic aspects of speech.

    Source: §1 · pp. 1–2

  • Method

    GenSE treats enhancement as conditional language modeling with a single-quantizer acoustic codec, a noisy-to-semantic stage, and a semantic-to-speech stage using token-chain prompting.

    Source: §1 · p. 2, §3.3.1 · p. 6, §3.3.2 · p. 6

  • Results

    GenSE improves enhancement quality and generalization over baseline systems; its OVL score is 3.43 without reverb and 3.19 with reverb.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    The two-stage design addresses noisy semantic tokens before acoustic generation, while token-chain prompting uses noisy acoustic tokens to support speaker-timbre consistency.

    Source: §1 · p. 2

    Autoregressive generation slows inference and limits GenSE’s real-time applicability.

    Source: §A.8 · p. 21

Abstract

from arXiv · show

Semantic information refers to the meaning conveyed through words, phrases, and contextual relationships within a given linguistic structure. Humans can leverage semantic information, such as familiar linguistic patterns and contextual cues, to reconstruct incomplete or masked speech signals in noisy environments. However, existing speech enhancement (SE) approaches often overlook the rich semantic information embedded in speech, which is crucial for improving intelligibility, speaker consistency, and overall quality of enhanced speech signals. To enrich the SE model with semantic information, we employ language models as an efficient semantic learner and propose a comprehensive framework tailored for language model-based speech enhancement, called \textit{GenSE}. Specifically, we approach SE as a conditional language modeling task rather than a continuous signal regression problem defined in existing works. This is achieved by tokenizing speech signals into semantic tokens using a pre-trained self-supervised model and into acoustic tokens using a custom-designed single-quantizer neural codec model. To improve the stability of language model predictions, we propose a hierarchical modeling method that decouples the generation of clean semantic tokens and clean acoustic tokens into two distinct stages. Moreover, we introduce a token chain prompting mechanism during the acoustic token generation stage to ensure timbre consistency throughout the speech enhancement process. Experimental results on benchmark datasets demonstrate that our proposed approach outperforms state-of-the-art SE systems in terms of speech quality and generalization capability.

1 INTRODUCTION

The introduction argues that speech enhancement can benefit from speech semantics, but conventional approaches often neglect them and noisy inputs complicate their use. GenSE casts enhancement as conditional language modeling, combining discrete speech tokens, hierarchical generation, and token-chain prompting.

Motivation

  • Conventional speech enhancement often overlooks semantic information that could support intelligibility, speaker consistency, and speech quality, especially in complex noise or unseen domains.Noisy speech also makes semantic-token modeling difficult, which can affect subsequent acoustic-token generation.Source: §1 · pp. 1–2
  • Applying clean-speech generation directly to enhancement is difficult because noise distorts semantic tokens and can degrade acoustic-token generation.Source: §1 · p. 2
  • Residual multi-quantizer codecs produce long acoustic-token sequences that increase language-model prediction costs and require additional prediction designs.Source: §1 · p. 2

Contributions

  • GenSE combines a reorganized single-quantizer codec, separate noisy-to-semantic and semantic-to-speech stages, and token-chain prompting for timbre consistency.The prompt concatenates noisy and clean semantic tokens with noisy acoustic tokens for the S2S module.Source: §1 · p. 2

2 RELATED WORK

Related work spans generative enhancement, speech tokenization, and neural audio codecs. These lines of work motivate discrete representations and language-model prediction while leaving challenges in speech quality, similarity, and prediction complexity.

2.1 GENERATIVE SPEECH ENHANCEMENT

  • Generative enhancement includes GAN, VAE, flow-based, and diffusion models; discrete-token approaches include codec-latent enhancement and LM-based SELM, which remains below clean speech in quality and similarity.Genhancer argues that codec tokens provide an efficient latent domain for speech modeling.Source: §2.1 · pp. 2–3

2.2 SPEECH TOKENIZATION

  • Speech language models use semantic tokens for higher-level phonetic and linguistic content and acoustic tokens for detailed waveform information and reconstruction.Source: §2.2 · p. 3

2.3 NEURAL AUDIO CODEC

  • Typical residual-quantization codecs use eight 1024-entry codebooks, but their many tokens complicate LM prediction; single-quantizer alternatives can sacrifice reconstruction quality or require a reference encoder.Source: §2.3 · p. 3

3 GENSE

GenSE uses a codec, a pretrained speech representation model, and two decoder-only language models in a two-stage enhancement pipeline. Its codec compresses speech into acoustic tokens, while the stages first produce clean semantic tokens and then generate clean speech.

3.1 OVERALL ARCHITECTURE

  • GenSE combines SimCodec, pretrained XLSR semantic representations, and two decoder-only language models in a noisy-to-semantic then semantic-to-speech pipeline.The S2S stage uses semantic tokens and noisy acoustic tokens while preserving speaker characteristics.Source: §3.1 · pp. 3–4, Figure 1 · p. 4

3.2.1 SIMCODEC OVERVIEW

  • SimCodec uses a single quantizer with a larger codebook to provide acoustic-token targets for S2S with fewer tokens.A two-stage training and codebook-reorganization process addresses low codebook usage and slow convergence associated with a large codebook.Source: §3.2.1 · p. 4

3.2.2 EXPLORATION OF CODEBOOK SIZE AND QUANTIZE SPACE

  • Reducing eight quantizers to one initially made codec training difficult to converge, so the authors explored larger codebooks from 2^8 to 2^13 and evaluated usage and reconstruction quality.Source: §3.2.2 · pp. 4–5, Figure 2 · p. 5

3.2.3 CODEBOOK REORGANIZATION

  • SimCodec first trains two smaller codebooks using group quantization, then selects frequently used entries and pairwise concatenates them into a new codebook of size N * K.The second stage adapts the projection layer while reusing encoder and decoder parameters, allowing quick convergence in a similar quantized space.Source: §3.2.3 · pp. 5–6, Figure 3 · p. 5

3.3 HIERARCHICAL MODELING FRAMEWORK

  • The hierarchical framework separates noisy-speech denoising into an N2S front-end and clean-speech generation into an S2S back-end to reduce prediction difficulty and improve stability.Source: §3.3 · p. 6, §3.3.2 · p. 7

3.3.1 NOISY-TO-SEMANTIC TRANSFORMATION

  • N2S extracts semantic tokens with a pretrained self-supervised model and trains a language model to predict clean semantic tokens autoregressively using noisy semantic tokens.XLSR produces frame-level representations every 20 ms and discretizes them into cluster indices; the clean and noisy semantic tokens are denoted ˆs and ¯s.Source: §3.3.1 · p. 6

3.3.2 SEMANTIC-TO-SPEECH GENERATION

  • S2S predicts clean acoustic tokens from prior clean acoustic tokens, noisy and clean semantic tokens, and noisy acoustic tokens, then SimCodec decodes them into enhanced speech.The token-chain prompt uses noisy acoustic tokens to retain original-speaker characteristics and support timbre consistency.Source: §3.3.2 · pp. 6–7

4 EXPERIMENTS AND RESULTS

The experiments compare GenSE with speech-enhancement baselines, test its generalization and subjective ratings, evaluate SimCodec, and ablate framework components.

4.1 EXPERIMENTAL SETUP

  • Training uses varied clean speech, WHAM! and DEMAND noise, and simulated reverberation; DNS Challenge and CHiME-4 provide evaluation sets.The study compares seven named baselines and uses objective and subjective evaluation metrics.Source: §4.1 · pp. 7–8

4.2 EXPERIMENTAL RESULTS ON SPEECH ENHANCEMENT

  • 3.43 without reverb and 3.19 with reverb OVL; GenSE also achieves the highest reported SECS scores, 0.67 without reverb and 0.65 with reverb.The authors report that GenSE outperforms baseline systems on objective enhancement metrics.Source: §4.2 · p. 8
  • On CHiME-4, GenSE achieves the best quality and speaker similarity and lowest WER among baselines, with less degradation under shifted noise.Table 2 reports enhancement comparisons on the CHiME-4 dataset.Source: Table 2 · p. 8, §4.2 · p. 8
  • GenSE receives higher NMOS and SMOS than every baseline, with higher medians and subjective-score distributions described as superior.Figure 4 shows violin plots for speech naturalness and speaker similarity; white points mark medians.Source: §4.2 · pp. 8–9, Figure 4 · p. 9

4.3 EXPERIMENTAL RESULTS ON SIMCODEC

  • 1.73 PESQ, 0.17 STOI, 1.09 MCD, and 1.31 UTMOS: SimCodec outperforms DAC at similar bandwidth and uses the fewest tokens.Table 3 compares reconstruction quality across codec models and identifies quantizer count and tokens per second.Source: §4.3 · p. 9, Table 3 · p. 9

4.4 ABLATION AND ANALYZE

  • Removing token-chain prompting lowers SECS, while replacing hierarchical modeling with a single LM lowers DNSMOS.The ablations assess prompting without noisy semantic and acoustic tokens, and direct acoustic-token prediction with one LM.Source: §4.4 · p. 10, Table 4 · p. 10

5 CONCLUSION

The conclusion presents GenSE as an LM-based generative speech-enhancement system built around SimCodec and hierarchical modeling.

5 CONCLUSION

  • SimCodec uses a reorganization process and a single quantizer to improve codebook usage under a larger codebook size.The conclusion says it delivers high-quality reconstruction with fewer tokens, reducing downstream-task complexity and difficulty.Source: §5 · p. 10
  • GenSE uses a two-stage process: an N2S front-end predicts clean semantic tokens, then an S2S back-end synthesizes clean speech.A token-chain prompting mechanism is designed to maintain speaker consistency.Source: §5 · p. 10
  • The reported experiments show that GenSE outperforms state-of-the-art speech-enhancement systems in subjective and objective evaluations.The conclusion says ablations and codec comparisons further confirm the effectiveness of its components.Source: §5 · p. 10

A APPENDIX

The appendix lists supplemental design, training, configuration, comparison, visualization, and limitation materials.

A APPENDIX

  • The supplement discusses GenSE design questions, SimCodec training objectives, and SimCodec and LM configurations.These topics are listed as Appendix A.1–A.3.Source: A APPENDIX · p. 15
  • It provides baseline-system details, codec ablations, quantization-strategy comparisons, and a visual enhanced-sample case.These topics are listed as Appendix A.4–A.7.Source: A APPENDIX · p. 15
  • The supplement discusses limitations and future work.This topic is listed as Appendix A.8.Source: A APPENDIX · p. 15

A.1 ADDITIONAL DISCUSSIONS ON THE DESIGN OF GENSE FRAMEWORK

The additional discussions probe semantic-extractor flexibility, perceptual evaluation, and N2S autoregressive modeling. They report similar results across two SSL extractors, favorable ABX judgments for GenSE, and a speed–metric trade-off for non-autoregressive modeling.

A.1 ADDITIONAL DISCUSSIONS ON THE DESIGN OF GENSE FRAMEWORK

  • Replacing XLSR with WavLM produced similar GenSE results, supporting compatibility with different SSL semantic extractors.XLSR was selected for its multilingual speech representations.Source: §A.1 · p. 15, Table 5 · p. 16
  • Intrusive metrics such as PESQ and STOI may not precisely reflect generative speech quality because realistic outputs can differ from their clean references.The authors also use ABX testing to assess perceived closeness to clean speech.Source: §A.1 · p. 16
  • GenSE was judged significantly closer to clean speech than baseline systems, and ABX participants consistently preferred it; Figure 5 defines “No Preference” as enhanced speech perceived as realistic as clean speech.The test assesses how distinguishable enhanced speech is from clean samples.Source: Figure 5 · p. 16, §A.1 · p. 16
  • Non-autoregressive N2S inference is significantly faster, but all reported metrics degrade relative to autoregressive modeling.The authors attribute the autoregressive model’s more precise and stable transformation to the need to preserve dependencies and sequence consistency for downstream S2S generation.Source: Table 6 · p. 17, §A.1 · p. 17

A.2 TRAINING OBJECTIVES OF SIMCODEC

This section describes SimCodec’s training objectives and compares it with WavTokenizer. SimCodec performs favorably in the reported codec comparison, while Figure 6 relates reconstruction quality to compression and token rate.

A.2 TRAINING OBJECTIVES OF SIMCODEC

  • SimCodec training combines multiscale mel-spectrogram L1/L2 reconstruction, quantizer commitment, generator/discriminator adversarial losses, and discriminator feature matching.The final objectives are a weighted sum.Source: §A.2 · p. 17
  • 0.65 kbps SimCodec outperforms WavTokenizer at 0.5 kbps by a large margin and achieves similar performance to WavTokenizer at 0.9 kbps, with 10 additional tokens per second.The comparison used models trained on LibriSpeech; SimCodec also supports a codebook size of 8192 versus WavTokenizer’s 4096.Source: Table 7 · p. 18, §A.2 · p. 18
  • Figure 6 plots PESQ against Kbps, with circle size representing tokens per second, for comparing codec quality and compression.The figure compares SimCodec with state-of-the-art acoustic codec models.Source: Figure 6 · p. 18, §A.2 · p. 18

A.3 MODEL CONFIGURATION

The section gives training configurations for SimCodec and the language models.

A.3 MODEL CONFIGURATION

  • SimCodec trains on 960 hours of 16-kHz LibriSpeech for 50k first-stage and 10k second-stage steps; language models train for 1 million steps on 8 A100 GPUs.SimCodec training uses 2 A100 GPUs and batch size 128; language-model training uses batch size 256, AdamW at 1e-4, 5k warmup steps, and an inverse-square-root learning-rate schedule.Source: §A.3 · p. 18

A.4 BASELINE SYSTEMS

The section outlines the baseline systems used for comparison, spanning spectral, diffusion, stochastic-regeneration, and language-model approaches.

A.4 BASELINE SYSTEMS

  • FullSubNet (Hao et al., 2021) fuses full- and sub-band features; Inter-Subnet (Chen et al., 2023) adds subband interaction with global spectral information.The listed global information includes cross-band dependencies and global spectral patterns.Source: §A.4 · p. 19
  • CDiffuSE (Lu et al., 2022) conditions diffusion on noisy speech, while SGMSE (Welker et al., 2022) starts reverse diffusion from noisy speech mixed with Gaussian noise.SGMSE uses 30 diffusion steps to generate high-quality clean speech estimates.Source: §A.4 · p. 19
  • StoRM (Lemercier et al., 2023) guides diffusion with a predictive estimate; SELM (Wang et al., 2024c) uses discrete tokens and language models; DOSE uses condition augmentation.StoRM’s predictive model removes vocalizing and breathing artifacts; DOSE is diffusion-based.Source: §A.4 · p. 19

A.5 ABLATION FOR DIFFERENT CODEC MODEL IN SE

GenSE’s codec comparisons show that SimCodec performs better than DAC alternatives, while reducing bandwidth trades some enhancement performance for faster inference. SimCodec’s token reduction and reorganization support a more efficient prediction process while maintaining competitive reconstruction quality.

A.5 ABLATION FOR DIFFERENT CODEC MODEL IN SE

  • Replacing SimCodec with single-quantizer DAC significantly reduces DNSMOS and speaker similarity, and increasing DAC quantizers still does not surpass SimCodec.The comparison considers hybrid autoregressive/non-autoregressive and parallel prediction for codecs with multiple quantizers.Source: Table 8 · p. 19, §A.5 · p. 19
  • The 50Hz, 0.65kbps GenSE version uses half as many acoustic and prefix tokens and achieves over 20% speedup, but lower bandwidth degrades performance, particularly DNSMOS.Despite the degradation, it still outperforms most baseline systems.Source: §A.5 · pp. 19–20, Table 9 · p. 20
  • SimCodec reduces LM prediction token counts while maintaining competitive reconstruction quality; its reorganization and larger codebook further improve reconstruction quality and compression efficiency.The reduced token count simplifies prediction and contributes to a more efficient and stable enhancement process.Source: §A.5 · p. 20

A.6 COMPARISON RESULTS FOR USING DIFFERENT QUANTIZATION STRATEGIES IN SIMCODEC.

The reorganization quantization strategy outperforms CVQ on PESQ, STOI, and MCD, with slightly lower UTMOS. A severe-noise visualization favors hierarchical modeling, while the authors identify data coverage and autoregressive inference speed as limitations.

A.6 COMPARISON RESULTS FOR USING DIFFERENT QUANTIZATION STRATEGIES IN SIMCODEC.

  • Reorganization outperforms CVQ in PESQ, STOI, and MCD, with only a slight degradation in UTMOS; FSQ has lower reconstruction quality.The authors attribute FSQ’s challenges to smaller latent dimensions, high gradient variance, and less accurate early-training approximation; group FSQ needs several quantizers.Source: Table 10 · p. 20, §A.6 · p. 20

A.8 LIMITATION AND FUTURE WORKS

  • The authors say GenSE could benefit from larger training datasets and more model parameters, with broader data coverage aimed at improving generalization to unseen conditions.The target coverage includes diverse noise types, speaker characteristics, and acoustic environments.Source: §A.8 · pp. 20–21

A.7 VISUALIZE OF THE ENHANCED SPECTROGRAM FROM SEVERELY DEGRADED SAMPLES

  • For a severely noisy signal, GenSE’s spectrogram shows clear harmonic structures and speech formants, while variants without hierarchical modeling show more distortion and residual noise.The authors interpret the visual comparison as evidence of better noise reduction and signal reconstruction with hierarchical modeling in this challenging condition.Source: §A.7 · p. 20, Figure 7 · p. 21

A.8 LIMITATION AND FUTURE WORKS

  • Autoregressive generation slows GenSE and limits real-time applicability; the authors identify prediction-pattern changes, quantization, and acceleration as directions needed for practical deployment.They propose alternating semantic and acoustic token prediction as a possible route to real-time performance.Source: §A.8 · p. 21
Loading 2502.02942v1…