Source-linked AI summary
SpeechX: Neural Codec Language Model as a Versatile Speech Transformer
Xiaofei Wang, Manthan Thakker, Zhuo Chen, Naoyuki Kanda, Sefik Emre Eskimez, Sanyuan Chen, Min Tang, Shujie Liu, Jinyu Li, Takuya Yoshioka
TL;DR
Existing audio-text speech models leave gaps in transforming noisy speech and in using reference transcriptions for enhancement. SpeechX unifies generation and transformation with a task-prompted neural codec language model; it performs across TTS, editing, and enhancement tasks, though codec accuracy constrains its performance.
Problem
Comprehensive audio-text speech enhancement models that leverage reference transcriptions are lacking, and prior approaches typically require separate expert models for different tasks.
Source: §I · p. 1Method
SpeechX extends neural codec language modeling with task-dependent prompts and multi-task training to generate acoustic tokens from textual and acoustic inputs.
Source: §I · p. 2, A. Overview · p. 3, D. Model training · p. 5Results
SpeechX performs comparably or better than expert models across its tasks; it has superior WER but lower DNSMOS and PESQ than expert models for noise suppression and target speaker extraction.
Source: §I · p. 2, 1) Result overview · p. 7Takeaways & Limitations
SpeechX unifies clean- and noisy-speech generation and transformation, including transcript-informed enhancement and editing that retains background noise.
Source: §I · p. 2, §VI · p. 10SpeechX's performance is constrained by neural codec accuracy; codec processing lowered PESQ from 4.64 to 2.69 for clean speech.
Source: 6) Limitation of current neural codec… · p. 9
Abstract
from arXiv · showhide
Recent advancements in generative speech models based on audio-text prompts have enabled remarkable innovations like high-quality zero-shot text-to-speech. However, existing models still face limitations in handling diverse audio-text speech generation tasks involving transforming input speech and processing audio captured in adverse acoustic conditions. This paper introduces SpeechX, a versatile speech generation model capable of zero-shot TTS and various speech transformation tasks, dealing with both clean and noisy signals. SpeechX combines neural codec language modeling with multi-task learning using task-dependent prompting, enabling unified and extensible modeling and providing a consistent way for leveraging textual input in speech enhancement and transformation tasks. Experimental results show SpeechX's efficacy in various tasks, including zero-shot TTS, noise suppression, target speaker extraction, speech removal, and speech editing with or without background noise, achieving comparable or superior performance to specialized models across tasks. See https://aka.ms/speechx for demo samples.
I. INTRODUCTION
Existing audio-text speech models support zero-shot TTS and some transformations, but their coverage of noisy-speech tasks and transcription-informed enhancement remains limited. SpeechX uses task-dependent tokens in a neural codec language model to unify speech generation and transformation across clean and noisy settings.
I. INTRODUCTION
- Existing generative models have limited coverage of speech transformation: editing models handle clean signals only, [21], and target speaker extraction remains unaddressed by them –.A denoising model also requires noisy speech to be surrounded by clean segments.Source: §I · p. 1
- Traditional denoising and target speaker extraction rely on distinct regression-based expert models, while comprehensive audio-text enhancement using reference transcriptions is lacking [26]–.The few transcription-informed studies cited focus on particular enhancement tasks,.Source: §I · p. 1
- The proposed unified models are motivated by the need for robustness to acoustic distortions and flexible architectures that can accommodate additional speech-generation tasks.The introduction frames these as properties needed for use in acoustically challenging environments and future task extensions.Source: §I · pp. 1–2
- SpeechX extends neural codec language modeling with task-dependent tokens in multi-task learning to perform zero-shot TTS, enhancement, speech removal, and clean or noisy speech editing.Its inputs include text and acoustic tokens, with text optional for some tasks.Source: §I · p. 2, Figure 1 · p. 2
- SpeechX reports comparable or superior performance to expert models across its tasks and adds background-sound-preserving editing and transcript use for enhancement.It also proposes integrating non-speech enhancement tasks, such as zero-shot TTS, to leverage reference transcriptions for intelligibility.Source: §I · p. 2
II. RELATED WORK
Autoregressive Transformers provide a language-modeling basis for speech generation, including zero-shot speaker adaptation. Earlier multi-task generative speech models still leave noisy-speech processing and conventional enhancement tasks insufficiently covered; SpeechX extends VALL-E to unify generation and transformation for clean and noisy speech.
A. Autoregressive generative models
- VALL-E [16] and SPEAR-TTS [17] model text and acoustic tokens as one stream, enabling zero-shot speaker adaptation from clips as short as three seconds without a separate duration model.These properties motivate the paper's choice of autoregressive Transformers for SpeechX.Source: A. Autoregressive generative models · p. 2
B. Multi-task generative speech models
- Voicebox supports many tasks through masked speech prediction but remains limited to clean-speech generation, unlike SpeechX's noisy-speech and enhancement coverage.SpeechX extends VALL-E with task-dependent prompts and handles noise suppression, target speaker extraction, and editing that retains background noise.Source: B. Multi-task generative speech models · p. 3
III. METHOD
SpeechX uses a neural codec language model conditioned on textual and acoustic prompts to generate desired speech codes. Its autoregressive and non-autoregressive Transformers support multiple codec layers, while task prompting enables varied input-output tasks without changing the architecture.
A. Overview
- SpeechX conditions codec-code generation on phoneme-based textual and audio-based acoustic prompts, then decodes the generated codes into a waveform.Task tokens are added to the acoustic prompt to specify the task or desired output.Source: A. Overview · p. 3
B. Neural codec language model
- SpeechX uses EnCodec's eight code layers, with the AR Transformer generating layer one and the NAR Transformer generating layers two through eight.Each EnCodec layer produces discrete codes from 1024 entries at 75 Hz.Source: B. Neural codec language model · p. 3, A. Overview · p. 3
- The AR model predicts the first-layer output codes from the prompts and preceding first-layer codes.The model is optimized by minimizing the negative log-likelihood of the first-layer codes.Source: B. Neural codec language model · p. 3
- Unlike VALL-E [16], SpeechX conditions its AR model on a separate acoustic prompt, so zero-shot TTS inference does not require transcribing the enrollment audio.SpeechX's simple end-to-end architecture also avoids separate speaker-embedding or duration models beyond the neural codec model.Source: B. Neural codec language model · p. 3, A. Overview · p. 3
- A shared NAR model generates each higher codec layer using the prompts and codes from all preceding layers.The preceding-layer acoustic tokens are embedded and summed for the shared model.Source: B. Neural codec language model · pp. 3–4
C. Task-based prompting
SpeechX uses task-specific tokens and prompt arrangements to specify the desired output across speech generation, enhancement, extraction, removal, and editing tasks. The prompting scheme can be extended with new tasks without changing the underlying architecture.
C. Task-based prompting
- Noise suppression and speech removal use task tokens to produce clean speech or the noise signal, respectively, from noisy audio; text prompts are optional for both.Noise suppression uses <ns>, while speech removal uses <sr>.Source: C. Task-based prompting · p. 4
- Target speaker extraction combines an enrollment clip with mixed speech and uses a task token to produce the target speaker's clean speech; text input is optional.The enrollment audio is assumed to be three seconds long.Source: C. Task-based prompting · p. 4
- Zero-shot TTS uses enrollment audio as the acoustic prompt and generates speech codes based on the input text.The synthesized speech is intended to reflect the input text and resemble the enrollment speaker's characteristics.Source: C. Task-based prompting · p. 4
- Clean speech editing marks the segment to replace with <soe>, <mask>, and <eoe>, while preserving the unreplaced speech and speaker characteristics.In clean editing, <mask> can let the model adapt output length; practical workflows can locate edit boundaries from an ASR transcript that a user edits.Source: C. Task-based prompting · pp. 4–5
- Noisy speech editing supplies the full noisy input and targets changed speech content while retaining background noise, including noise in the edited segment.The formulation is more challenging than clean editing because the model must distinguish speech from noise.Source: C. Task-based prompting · p. 4
- New tasks can be added by integrating their prompting schemes and training from an existing checkpoint, initializing only new task-token embeddings and leaving the architecture unchanged.Task-based prompting lets the model specify the desired output during inference.Source: C. Task-based prompting · pp. 4–5
D. Model training
SpeechX first trains on zero-shot TTS, then continues with multi-task training. Training samples tasks equally and presents text prompts half the time for noise suppression, speech removal, and target speaker extraction.
D. Model training
- Tasks are sampled with equal probability at each model update to avoid unduly favoring any one task.The text prompt is included with 50% probability for noise suppression, speech removal, and target speaker extraction.Source: D. Model training · p. 5
- SpeechX is initialized by training for zero-shot TTS before continuing with all tasks in multi-task learning.The paper reports that this two-stage strategy substantially enhances performance across all tasks.Source: D. Model training · p. 5
IV. EVALUATION SETUPS
Evaluations use datasets derived from LibriSpeech test-clean, with test samples lasting four to ten seconds. The setups cover TTS, enhancement, speaker extraction, speech editing, and speech removal under specified prompt and corruption conditions.
IV. EVALUATION SETUPS
- All evaluation datasets derive from LibriSpeech test-clean, and selected test samples last four to ten seconds.These choices were made to keep experiments manageable and consistent across tasks.Source: §IV · p. 5
A. Evaluation data
- TTS uses a three-second clip from another utterance by the same speaker; noise suppression mixes test speech with MUSAN noise at 0–20 dB SNR and tests textless and text-guided prompts.The noise-suppression task aims to recover uncorrupted speech.Source: A. Evaluation data · p. 5
- Target speaker extraction mixes speech with another speaker at 0–20 dB SIR and uses a three-second same-speaker enrollment clip; speech removal uses the noise-suppression dataset and only textless prompts.Extraction is evaluated with both textless and text-guided settings.Source: A. Evaluation data · p. 5
- Clean editing replaces 10–50% of an utterance and uses correct edit boundaries; noisy editing adds MUSAN noise at 0–20 dB SNR and targets changed speech while preserving noise.For noisy editing, the task target also preserves speaker characteristics and the unreplaced input portion.Source: A. Evaluation data · p. 5
B. Metrics
The paper evaluates generated speech with task-specific objective metrics covering transcription fidelity, speaker similarity, perceived quality, and speech-removal accuracy.
B. Metrics
- WER measures fidelity to the provided transcription and is used across tasks except speech removal.The experiments use NeMo’s stt en conformer transducer large ASR model, selected for its observed stability and robustness to noise and processing artifacts.Source: B. Metrics · pp. 5–6
- SIM measures speaker-characteristic coherence using cosine similarity between embeddings of generated and desired speech.Speaker embeddings come from NeMo’s TitaNet-Large; the metric is used in zero-shot TTS and clean and noisy speech editing.Source: B. Metrics · p. 6
- DNSMOS OVRL predicts perceived quality for noise suppression and target speaker extraction; extraction uses a personalized DNSMOS model..Source: B. Metrics · p. 6
- PESQ evaluates noise suppression and target speaker extraction against clean references, assessing fidelity to the original clean data.Unlike DNSMOS, PESQ is intrusive and requires clean reference signals.Source: B. Metrics · p. 6
- MCD measures speech-removal accuracy by comparing estimated noise with ground-truth noise audio..Source: B. Metrics · p. 6
V. EXPERIMENTS
Training uses simulated task-specific speech mixtures and edits, with pseudo-transcribed semantic prompts, and compares random with VALL-E initialization.
A. Training data
- SpeechX training data uses LibriLight’s 60,000 hours of untranscribed English reading speech from over 7,000 speakers.Training mini-batches are formed from randomly selected tasks at each iteration.Source: A. Training data · p. 6
- Simulated training examples cover noise suppression, speech removal, target speaker extraction, and clean and noisy speech editing.Noise suppression and speech removal use DNS-noise mixtures at −5 to 20 dB SNR; extraction uses speaker mixtures at −5 to 20 dB SIR. Editing replaces 10%–70% of an utterance, with noise added for noisy editing.Source: A. Training data · p. 6
- Because LibriLight lacks reference transcripts, the authors derive phoneme-sequence semantic prompts using pseudo-labels from a Kaldi model.The Kaldi model was trained on 960-hour Librispeech data with 3x speed perturbation.Source: A. Training data · p. 6
B. Model and training configurations
- SpeechX AR and NAR share a 12-layer, 16-head Transformer with 1024-dimensional embeddings and 4096-dimensional feed-forward layers.Random initialization trains for 800k iterations; VALL-E initialization uses a 400k-iteration zero-shot TTS model and then 400k further iterations, with different warm-up lengths.Source: B. Model and training configurations · p. 6
C. Baseline expert models
SpeechX is compared with task-specific experts, and reported results show benefits from VALL-E initialization alongside competitive performance across generation and transformation tasks.
C. Baseline expert models
- The comparison uses VALL-E for zero-shot TTS, DCCRN for noise suppression, VoiceFilter for target speaker extraction, and A3T for speech editing.DCCRN was trained on Microsoft’s internal dataset and fine-tuned with an ASR objective; VoiceFilter uses a bidirectional LSTM configuration.Source: C. Baseline expert models · pp. 6–7
1) Result overview:
- Across tasks, SpeechX beat experts on enhancement WER and clean editing, improved speech-removal MCD, and slightly edged VALL-E on TTS WER with comparable SIM.Table II presents the cross-task comparison; the reported TTS speaker-similarity score was comparable to VALL-E.Source: Table II · p. 7, 1) Result overview · p. 7
- VALL-E checkpoint initialization benefited SpeechX across all tasks, especially in WER..Source: 1) Result overview · p. 7
2) Speech editing for clean and noisy speech:
SpeechX edits noisy speech while retaining speaker identity and background noise; textual prompts improve intelligibility in noise suppression and target speaker extraction.
2) Speech editing for clean and noisy speech:
- 13.95% WER and 0.65 SIM: SpeechX edited noisy speech while retaining speaker identity and background noise, versus A3T’s 32.17% WER.The paper reports that noisy editing is harder than clean editing and that SpeechX retained background noise in the examples.Source: 2) Speech editing for clean and noisy… · p. 7
- Table IV scopes multi-task-training comparisons: random-initialized models train for 800k iterations, while VALL-E-initialized models use 400k TTS then 400k multitask iterations..Source: Table IV · p. 8
3) Effectiveness of text input in noise suppression and target speaker extraction:
- Figure 2 compares pre- and post-edit noisy-signal mel spectrograms; the examples show foreground speech changes while background noise remains.The pre-edit signal is EnCodec-compressed and decompressed without intermediate processing.Source: 2) Speech editing for clean and noisy… · p. 7, Figure 2 · p. 8
- The authors call for a metric to quantitatively evaluate noise-cloning capability..Source: 3) Effectiveness of text input in… · p. 8
- Omitting textual input increased WER in both noise suppression and target speaker extraction, while DNSMOS and PESQ degraded modestly.For target speaker extraction, the DNSMOS impact was significant; acoustic-only WER remained comparable to expert-model baselines.Source: 3) Effectiveness of text input in… · p. 8
4) Effect of multi-task training:
The experiments compare multi-task training with single-task experts and test how adding different task combinations affects performance. Multi-task training improves zero-shot TTS WER while supporting additional tasks, with trade-offs across editing conditions.
4) Effect of multi-task training:
- Single-task expert models were initialized from a VALL-E checkpoint trained for zero-shot TTS, then further updated with task-specific data.The VALL-E checkpoint had 400k training iterations, followed by 400k iterations of single-task training.Source: 4) Effect of multi-task training · p. 8
- Multi-task training improved zero-shot TTS WER and enabled additional tasks; noise-suppression and target-extraction experts had worse WER, while editing experts were slightly better.SIM, DNSMOS, and MCD were almost the same across the compared models.Source: 4) Effect of multi-task training · p. 8, Table V · p. 9
- Adding speech editing improved zero-shot TTS WER; adding noise suppression and removal hurt clean editing but helped noisy editing, while target extraction did not impair suppression or removal.Source: 4) Effect of multi-task training · p. 9
5) Phoneme vs. Byte Pair Encoding (BPE):
The section compares phoneme and BPE prompts, examines performance constraints from the neural codec, and summarizes SpeechX’s range of speech tasks.
5) Phoneme vs. Byte Pair Encoding (BPE):
- Phoneme-based SpeechX performed significantly better, especially in WER, on zero-shot TTS and clean speech editing; BPE was marginally better for noise suppression and target speaker extraction.The authors speculate that ASR-derived erroneous training transcriptions contributed to BPE’s WER degradation in text-based generation tasks.Source: 5) Phoneme vs. Byte Pair Encoding (BPE) · p. 9
6) Limitation of current neural codec model:
- PESQ fell from 4.64 to 2.69 for clean speech input after EnCodec processing.The authors note that the large PESQ drop may partly reflect a mismatch between PESQ and EnCodec’s training objective.Source: Table VII · p. 9, 6) Limitation of current neural codec… · pp. 9–10
VI. CONCLUSION
- SpeechX unifies zero-shot TTS, speech editing, noise suppression, speech removal, and target speaker extraction, including editing speech with background noise.Source: §VI · p. 10