PaperlayerView exact revision on arXiv

Source-linked AI summary

TIMUSE: Text-Initialized Masked Semantic Token Modeling for Universal Speech Enhancement

Yiwei Guo, Hao Qiu, Mohan Ye, Zhijie Deng, Zehua Chen, Kai Yu

arXiv:2610.34814v1eess.AS

TL;DR

Discrete-token speech enhancement can produce plausible speech while changing linguistic content or speaker characteristics. TIMUSE pairs text-initialized masked semantic restoration with flow-matching acoustic reconstruction. It reports a stronger quality–fidelity balance than existing discrete systems, with text-DLM initialization selectively improving content preservation under severe degradations.

  • Problem

    Discrete generative USE can produce perceptually plausible speech while altering linguistic content or speaker characteristics, motivating restoration that balances quality and fidelity.

    Source: §1 · p. 1

  • Method

    TIMUSE predicts clean semantic tokens with a text-initialized masked generator, then reconstructs speech through flow matching conditioned on those tokens and the degraded acoustics.

    Source: §1 · p. 1, §2.1 · p. 2, §2.4 · p. 2

  • Results

    TIMUSE offers a strong overall quality–fidelity balance among discrete systems, and text-DLM initialization selectively improves content preservation, particularly under severe content-degrading conditions.

    Source: §4 · p. 4, §3.4 · p. 4

  • Takeaways & Limitations

    The results support using text-pretrained priors for USE and indicate potential for DLMs in future universal speech signal processing.

    Source: §4 · p. 4

Abstract

from arXiv · show

Discrete-token-based universal speech enhancement (USE) enables strong sequence modeling but often suffers from a quality--fidelity trade-off, where perceptually plausible outputs may alter linguistic content or speaker characteristics. We present TIMUSE, a two-stage USE framework combining text-initialized masked semantic restoration with acoustic reconstruction. A pretrained diffusion language model (DLM) is adapted with LoRA to recover clean semantic tokens using masked generative modeling, while a conditional flow matching model reconstructs clean 24\,kHz speech from the recovered semantic tokens and original degraded speech. Both system-level and controlled comparisons show that TIMUSE achieves a better balance between perceptual quality and fidelity than existing discrete USE models. Ablations further show that text DLM initialization brings a targeted gain under severe degradations.

1. INTRODUCTION

Universal speech enhancement must restore clean speech across diverse degradations, but discrete generative systems can produce perceptually plausible outputs that alter linguistic content or speaker traits. TIMUSE uses text-initialized masked semantic restoration followed by acoustic reconstruction, and reports a better quality–fidelity balance than existing discrete systems.

1. INTRODUCTION

  • Severe degradations can remove linguistic evidence, while discrete generative systems may alter linguistic content or speaker characteristics despite producing perceptually plausible speech.USE targets recovery across noise, reverberation, bandwidth limitation, clipping, and mixtures.Source: §1 · p. 1
  • Prior systems use acoustic tokens or semantic-token restoration followed by acoustic generation, but the quality–fidelity trade-off remains.Acoustic-token systems jointly model linguistic and acoustic information; Metis-SE and SenSE use semantic tokens in different generation pipelines.Source: §1 · p. 1
  • Bidirectional text diffusion language models offer a linguistic prior for inferring missing content from the available degraded utterance.The paper motivates transferring a pretrained text DLM to semantic-token restoration.Source: §1 · p. 1
  • TIMUSE combines text-initialized masked semantic restoration with flow-matching acoustic reconstruction, ranking first among discrete systems on five of six URGENT 2026 metrics.Under matched training, it outperforms all compared baselines across metric–benchmark combinations, with particularly large content-accuracy gains.Source: §1 · p. 1
  • The authors report that semantic tokens and masked generative modeling provide gains, while text-DLM initialization further improves content preservation, especially under severe degradations.They present this initialization as, to their knowledge, the first demonstration that a pretrained text DLM benefits USE.Source: §1 · p. 1

2. TIMUSE: TEXT-INITIALIZED MASKED SEMANTIC TOKEN MODELING

TIMUSE first predicts clean semantic tokens from degraded-speech features, then reconstructs speech using those tokens alongside the original degraded acoustics. Its method combines learned multi-layer speech conditioning, text-initialized masked token prediction, and conditional flow matching.

2.1. Problem Formulation and Overview

  • TIMUSE separates restoration into clean semantic-token prediction from degraded-speech features and acoustic reconstruction that retains the original 24 kHz degraded waveform as a reference.The conditioning encoder operates on speech resampled to 16 kHz; the acoustic stage produces the final 24 kHz waveform.Source: §2.1 · p. 2

2.2. Degraded-Speech Conditioning

  • The degraded-speech conditioner mixes hidden states from all layers of a frozen Whisper large-v3 encoder, then normalizes and projects them to the restoration model's hidden dimension.The encoder processes speech resampled to 16 kHz; learned layer weights allow access to different acoustic and linguistic abstraction levels.Source: §2.2 · p. 2

2.3. Text-Initialized Masked Semantic Token Modeling

  • The restoration target is a clean S3 semantic-token sequence extracted from the clean speech, using one codebook of 4096 entries at 50 tokens per second.The masked generator is initialized from Dream-v0-Instruct-7B, with a new S3 embedding table and output head; the pretrained backbone is adapted through LoRA.Source: §2.3 · p. 2
  • Training corrupts target tokens through a diffusion-style masking process that combines random token masking and contiguous spans, then predicts clean tokens from the corrupted sequence, speech features, and instruction.The cross-entropy objective is averaged over masked positions; fully masked targets are included in training to match the inference starting state.Source: §2.3 · p. 2

2.4. Acoustic Reconstruction with Conditional Flow Matching

  • The acoustic stage conditions a flow-matching model on the original degraded-speech Mel spectrogram and predicted clean semantics, with token-only classifier-free guidance.The null-token guidance branch removes the semantic condition but retains the acoustic observation; a frozen BigVGAN vocoder converts the reconstructed Mel spectrogram into 24 kHz speech.Source: §2.4 · p. 2

3. EXPERIMENTS

TIMUSE is evaluated on two diverse USE benchmarks through system-level comparisons, matched-training comparisons, and ablations. The experiments show strong benchmark performance and distinguish the contributions of semantic targets, masked modeling, and text-DLM initialization.

3.1.1. Training Data

  • Training uses multilingual clean speech, noise, and room responses, with language sampling fixed at 6:1:2 for English, Mandarin, and other MLS languages.The two-stage online degradation policy shifts from balanced everyday degradations toward more severe communication and compound distortions.Source: §3.1.1 · p. 3, §3.1.2 · p. 3

3.1.2. Training and Inference Recipe

  • Training runs for 50k updates in Stage 1 and 100k in Stage 2; inference uses two-step masked decoding followed by eight-step CFM sampling.The reported average real-time factor is 0.104 on an RTX 5090 GPU.Source: §3.1.2 · p. 3

3.1.3. Evaluation Data and Metrics

  • The evaluation covers AATC 2025 and URGENT 2026, using perceptual-quality metrics alongside linguistic-content and speaker-identity fidelity metrics.Because AATC lacks clean transcripts, it uses pseudo-CAcc derived from ASR transcriptions of clean speech.Source: §3.1.3 · p. 3

3.2. System-Level Comparison

  • TIMUSE ranks first among discrete systems on five of six URGENT 2026 metrics and leads all evaluated systems on BAK, OVRL, and NISQA.Its SIG score is slightly below those of UniSE and Metis-SE, while it offers the strongest overall balance among discrete systems.Source: Table 1 · p. 3, §3.2 · p. 3

3.3. Controlled Experiments

  • TIMUSE ranks first in all twelve metric–benchmark combinations, with CAcc gains of 7.46 percentage points on AATC 2025 and 5.80 points on URGENT 2026.The matched training data and recipe make these gains more directly reflective of model design.Source: Table 2 · p. 4, §3.3 · p. 4

3.4. Ablation and Analysis

  • Text initialization’s CER gain generally grows with severity for noise, downsampling, clipping, and quantization, and is positive at high packet-loss rates, though several mild conditions are indistinguishable.The severity-controlled comparison uses the same corrupted utterances for TIMUSE and a parameter-matched randomly initialized semantic-token MGM.Source: Figure 2 · p. 4, §3.4 · p. 4
  • Under controlled ablations, semantic tokens and masked generative modeling drive the overall gains, while text-DLM initialization specifically strengthens content preservation.The comparisons hold token type and parameter budget fixed when assessing modeling family, and use matched semantic-token MGM variants to assess initialization.Source: §3.4 · p. 4

4. CONCLUSION

TIMUSE combines text-initialized masked semantic-token prediction with flow-matching acoustic reconstruction. It performs strongly across the reported benchmarks, with ablations indicating that text-DLM initialization particularly benefits content preservation under severe degradations.

4. CONCLUSION

  • TIMUSE combines text-initialized masked semantic-token prediction with flow-matching-based acoustic reconstruction.The conclusion describes this as a two-stage USE framework.Source: §4 · p. 4
  • TIMUSE offers a strong overall balance among discrete systems on URGENT 2026 and leads all metrics on AATC 2025 and URGENT 2026 under matched training.Source: §4 · p. 4
  • Ablations attribute overall gains to semantic token targets and masked modeling, while text-DLM initialization selectively improves content preservation, particularly under severe content-degrading conditions.Source: §4 · p. 4

5. COMPLIANCE WITH ETHICAL STANDARDS

The study used publicly available speech data and required no additional ethical approval.

5. COMPLIANCE WITH ETHICAL STANDARDS

  • The study used publicly available speech data and required no additional ethical approval.Source: §5 · p. 5

Related papers

Loading 2610.34814v1…