PaperlayerView exact revision on arXiv

Source-linked AI summary

XingChen-UniFE: A Unified Generative Speech Front-End via Complementary Acoustic-Semantic Modeling

Kelan Kuang, Qihan Hu, Jian Kang, Jie Li, Yongxiang Li, Yi Liu

arXiv:2610.29164v1eess.AScs.SD

TL;DR

Real-world speech degradation and task-specific systems motivate a unified approach to restoration and target speaker extraction. XingChen-UniFE fuses acoustic and semantic representations in a LALM; the authors report strong results across restoration tasks and validate the modeling and fusion strategy.

  • Problem

    Speech restoration systems often use task-specific architectures, while severe degradation requires acoustic, linguistic, and speaker information beyond signal reconstruction alone.

    Source: §1 · p. 1

  • Method

    XingChen-UniFE uses a dual-stream encoder for complementary acoustic and semantic representations, joined by bidirectional cross-attention to condition a LALM.

    Source: §1 · pp. 1–2

  • Results

    XingChen-UniFE achieves the best results on all denoising metrics and leads in PESQ and SBS for TSE on Libri2Mix, with best PESQ, SBS, and LPS under TSE&Denoise.

    Source: §4.1 · p. 4

  • Takeaways & Limitations

    The reported results validate the proposed acoustic-semantic modeling and fusion strategy for discrete speech generation in GSR and TSE.

    Source: §5 · p. 4

Abstract

from arXiv · show

This paper presents XingChen-UniFE, a unified large audio language model (LALM) for general speech restoration (GSR) and target speaker extraction (TSE). The proposed model employs complementary semantic and acoustic representations through a dual-stream encoder, where a self-supervised learning model provides hierarchical semantic features and a learnable acoustic branch captures fine-grained structures from complex spectra. A bidirectional cross-attention module further facilitates information exchange between the two representations. The fused representation conditions the LALM for discrete speech token generation with task and speaker information. Experiments demonstrate strong performance on both GSR and TSE and robust generalization to unseen degradation conditions.

1. INTRODUCTION

Real-world degradation and the use of separate systems for restoration and speaker extraction motivate a unified approach. XingChen-UniFE combines acoustic and semantic representations, with explicit cross-attention, to support diverse tasks in one LALM.

1. INTRODUCTION

  • Real-world noise, reverberation, transmission distortion, and interfering speakers degrade speech, while separate restoration and extraction systems increase training and deployment costs.Source: §1 · p. 1
  • Discriminative restoration systems often use task-specific architectures and objectives, while severe degradation calls for modeling acoustic detail, linguistic content, and speaker characteristics.Source: §1 · p. 1
  • XingChen-UniFE proposes a unified LALM for speech restoration and target speaker extraction, conditioning discrete speech-token generation on fused linguistic and acoustic cues.Source: §1 · p. 1
  • Its dual-stream encoder combines SSL representations with complex spectral features, and bidirectional cross-attention explicitly models interactions between acoustic and semantic representations.Source: §1 · pp. 1–2

2. PROPOSED METHOD

XingChen-UniFE conditions a large audio language model on complementary acoustic and semantic speech representations to generate speech tokens for restoration and target-speaker extraction. Its encoder fuses parallel feature streams, while the generation pipeline incorporates task and speaker conditions.

2.1. Overall Architecture

  • The model generates speech tokens conditioned on degraded speech, a task token, and enrollment speech for TSE.Its encoder supplies the LALM with a conditional representation; generated tokens are rearranged and decoded by the codec.Source: §2.1 · p. 2

2.2. Complementary Representation Learning

  • A complementary encoder uses parallel semantic and acoustic branches followed by bidirectional feature fusion.The fused representation provides the LALM's conditional input.Source: §2.2 · p. 2, §2.1 · p. 2

2.2.1. Semantic and Acoustic Feature Encoding

  • The semantic branch averages hidden representations across SSL layers, while the acoustic branch processes complex STFT features through convolutional and Conformer blocks.The acoustic branch's convolutional blocks reduce frequency dimension while preserving temporal resolution.Source: §2.2.1 · p. 2

2.2.2. Bidirectional Feature Fusion

  • Two fusion layers exchange information bidirectionally between semantic and acoustic representations before concatenating and projecting their outputs.Each layer alternates Query and Key/Value roles, with residual connections, Layer Normalization, and a shared FFN.Source: §2.2.2 · p. 2

2.3. Conditional Token Generation

  • During training, target speech is codec-encoded into acoustic and semantic tokens, interleaved, and rearranged into a delayed autoregressive sequence.The LALM predicts tokens conditioned on the fused input, enrollment representation for TSE, task token, and preceding generated tokens; the codec decoder reconstructs speech.Source: §2.3 · pp. 2–3

3. EXPERIMENTAL SETUP

The model combines a decoder-only LALM with a frozen WavLM semantic branch, trainable acoustic branch, and H-Codec. Training uses on-the-fly simulated degradation and a large, multi-source speech, noise, and interference pool.

3.1. Implementation Details

  • The decoder-only Transformer has 28 layers and approximately 502 M trainable parameters, with WavLM and H-Codec frozen.It is trained for 400k steps with batch size 64 using AdamW and a Noam schedule with 4k-step warm-up.Source: §3.1 · p. 3

3.2. Training Data

  • Training degradation is simulated on the fly with random combinations of noise, reverberation, interfering speech, clipping, packet loss, and bandwidth limitation.Audio is resampled to 16 kHz and segmented into 3–15-second clips.Source: §3.2 · p. 3
  • The training pool contains approximately 10k hours of target speech, 405k hours of noise, and 4640 hours of interfering speech from multiple datasets.Target utterances from Emilia and LibriHeavy with DNSMOS scores below 3.5 are discarded.Source: §3.2 · p. 3

4. RESULTS

XingChen-UniFE is evaluated against generative multi-task baselines on GSR and TSE, with reported results across restoration tasks and unseen degradations. Ablations examine the contributions of its semantic and acoustic branches, cross-attention, and phase information.

4.1. Comparison Experiment

  • On Libri2Mix, XingChen-UniFE leads all denoising metrics, PESQ/SBS for TSE, and PESQ/SBS/LPS for TSE&Denoise; GSR LPS/WER margins widen on Mixed.Source: §4.1 · p. 4, Table 2 · p. 4
  • For unseen codec distortion, XingChen-UniFE leads SBS/WER; for wind noise, it leads LPS/SBS/WER.Source: §4.1 · p. 4, Table 3 · p. 4

4.2. Ablation Study

  • The semantic branch contributes more to SBS/LPS/WER, while the acoustic branch mainly affects PESQ/STOI; these metrics reflect linguistic consistency versus perceptual quality and intelligibility.Source: §4.2 · p. 4, Table 4 · p. 4
  • Cat Fusion and w/o C.A. degrade performance; w/ RI outperforms w/ Mag, particularly under mixed degradations, highlighting phase information.Source: §4.2 · p. 4, Figure 3 · p. 4

5. CONCLUSION

XingChen-UniFE is a discrete speech generation model for GSR and TSE. Its dual-stream encoder and bidirectional cross-attention integrate acoustic and semantic representations, and the reported results validate this modeling and fusion strategy.

  • XingChen-UniFE is a discrete speech generation model for GSR and TSE.Source: §5 · p. 4
  • A dual-stream encoder and bidirectional cross-attention integrate complementary acoustic and semantic representations.Source: §5 · p. 4
  • The results demonstrate the effectiveness of the model and validate its acoustic-semantic modeling and fusion strategy.Source: §5 · p. 4

Related papers

Loading 2610.29164v1…