PaperlayerView exact revision on arXiv

Source-linked AI summary

Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generation

Maohao Shen, Shun Zhang, Jilong Wu, Zhiping Xiu, Ehab AlBadawy, Yiting Lu, Mike Seltzer, Qing He

arXiv:2410.20336v1cs.CLcs.AIcs.SDeess.AS

TL;DR

Speech generation from text-dominant LLMs remains under-explored. The paper introduces TTS-Llama and builds MoLE-Llama through late-fusion PEFT with text and speech experts. MoLE-Llama performs comparably to TTS-Llama on TTS while demonstrating strong text QA performance.

  • Problem

    Enabling pretrained text-based LLMs to generate speech remains an under-explored research area.

    Source: §I · p. 1, A. Text-to-Speech · p. 1

  • Method

    The paper fine-tunes a text LLM for TTS, then uses late fusion and mixture-of-LoRA experts to combine speech and text capabilities.

    Source: §I · p. 1, §IV · p. 2

  • Results

    MoLE-Llama performs comparably to TTS-Llama on TTS and shows strong text QA performance across three benchmarks, with comparable MMLU performance to SPIRIT-LM.

    Source: B. Results · p. 4

  • Takeaways & Limitations

    The results demonstrate a PEFT-based late-fusion approach for adding speech generation while retaining text QA capability.

    Source: §I · p. 1, B. Results · p. 4

Abstract

from arXiv · show

Large language models (LLMs) have revolutionized natural language processing (NLP) with impressive performance across various text-based tasks. However, the extension of text-dominant LLMs to with speech generation tasks remains under-explored. In this work, we introduce a text-to-speech (TTS) system powered by a fine-tuned Llama model, named TTS-Llama, that achieves state-of-the-art speech synthesis performance. Building on TTS-Llama, we further propose MoLE-Llama, a text-and-speech multimodal LLM developed through purely late-fusion parameter-efficient fine-tuning (PEFT) and a mixture-of-expert architecture. Extensive empirical results demonstrate MoLE-Llama's competitive performance on both text-only question-answering (QA) and TTS tasks, mitigating catastrophic forgetting issue in either modality. Finally, we further explore MoLE-Llama in text-in-speech-out QA tasks, demonstrating its great potential as a multimodal dialog system capable of speech generation.

I. INTRODUCTION

Speech generation remains an under-explored extension of text-dominant LLMs, while many speech language models focus on TTS and are trained from scratch. This work uses PEFT and mixture-of-experts to build TTS and text-speech models, including one explored for text-in-speech-out QA.

I. INTRODUCTION

  • Many recent TTS language-model approaches focus solely on TTS and train a speech language model from scratch, demanding computation and speech data.Source: §I · p. 1
  • The paper targets the under-explored challenge of enabling pretrained text-based LLMs to generate speech.Most existing efforts instead extend text LLMs for speech input and understanding.Source: §I · p. 1
  • The authors use PEFT rather than full pretraining or fine-tuning to add speech generation to text-based LLMs.Source: §I · p. 1
  • MoLE-Llama combines text-only QA and TTS through late fusion and is also explored for text-in-speech-out QA.The authors describe it as the first text-speech multimodal LLM achieved through purely late-fusion PEFT.Source: §I · p. 1

II. RELATED WORK

Prior TTS work spans continuous-signal synthesis and language modeling over discrete speech tokens. The section identifies adapting pretrained text LLMs for speech generation as an under-explored direction.

A. Text-to-Speech

  • Recent speech language models use discrete tokens, including separate high-level semantic and low-level acoustic tokens, for speech synthesis.Earlier systems commonly generated mel spectrograms for a vocoder to convert into waveforms.Source: A. Text-to-Speech · p. 1
  • Most cited methods train speech language models from scratch, leaving speech generation from pretrained text LLMs under-explored.Source: A. Text-to-Speech · p. 1
  • TTS-Llama’s pipeline uses a fine-tuned Llama model to extract semantic information, then conditions an acoustic model to extract features for synthesis.Source: Figure 1 · p. 2
  • A prior attempt to integrate a pretrained text LLM with a speech LM did not achieve promising speech-generation performance through fine-tuning alone.Source: A. Text-to-Speech · p. 2

B. Text-Speech Multimodal Language Model

Prior speech language models cover multiple speech tasks but lack the conversation or reasoning capabilities of state-of-the-art text LLMs, while some integration approaches require expensive full-scale training. TTS-Llama instead generates semantic tokens from text, then acoustic tokens for waveform synthesis.

B. Text-Speech Multimodal Language Model

  • Existing speech language models support multiple speech tasks, but lack state-of-the-art text LLMs’ conversation and reasoning capabilities.Other approaches add speech dialog or continuation, but require expensive full-scale pretraining or fine-tuning; text-capability loss is also a concern.Source: B. Text-Speech Multimodal Language Model · p. 2

III. TTS-LLAMA: A TTS SYSTEM POWERED BY FINE-TUNED TEXT LLAMA MODEL

  • TTS-Llama generates high-level semantic tokens from raw text, then an acoustic language model maps them to acoustic features for waveform synthesis.The semantic and acoustic targets are formed by tokenizers that project information into a quantized latent space.Source: §III · p. 2
  • The semantic-token generator adapts pretrained Llama3-8B-Instruct with LoRA, training input embeddings, the prediction head, and injected adapters.Source: §III · p. 2
  • A semantic tokenizer extracts 4,096 discrete tokens, while a convolutional autoencoder with residual vector quantization extracts acoustic tokens.Source: §III · p. 2
  • A lightweight autoregressive acoustic transformer generates multi-codebook tokens from semantic tokens, and a reused tokenizer decoder converts them to waveform.The acoustic LM has fewer than 0.2B parameters.Source: §III · p. 2

IV. MOLE-LLAMA: A TEXT-SPEECH MULTIMODAL LLM VIA MIXTURE-OF-LORA EXPERTS

TTS-Llama’s speech-generation fine-tuning is associated with catastrophic forgetting on text QA. MoLE-Llama addresses this with separate text and speech LoRA experts, unified through late fusion.

IV. MOLE-LLAMA: A TEXT-SPEECH MULTIMODAL LLM VIA MIXTURE-OF-LORA EXPERTS

  • TTS-Llama’s MMLU evaluation shows catastrophic forgetting in text QA, motivating a late-fusion approach.Source: §IV · p. 2
  • MoLE-Llama uses separately trained text and speech experts, then merges them into a unified multimodal LLM.Source: §IV · pp. 2–3
  • The three-stage procedure adds speech with TTS fine-tuning, preserves text capability with text instruction tuning, then unifies experts with mixture-of-LoRA.The figure notes that a speech QA expert can also be added during Stage 2.Source: Figure 2 · p. 3

A. Three-stage Training Procedure

MoLE-Llama is trained in three stages to combine speech and text capabilities. Separate TTS and text experts are unified by a trainable router while other parameters remain frozen.

a) Stage-1: TTS Expert Fine-tuning

b) Stage-2: Text Expert Fine-tuning

  • Stage 2 aims to restore text capabilities by freezing Stage 1’s re-trained input embedding and prediction head and fine-tuning the LoRA adapter on text QA.Source: A. Three-stage Training Procedure · p. 3

c) Stage-3: Mixture-of-LoRA Experts

  • Stage 3 combines the TTS and text experts in one model, training only a router to select the modality-aware expert while keeping other parameters frozen.The system is trained end-to-end on text QA and TTS data.Source: A. Three-stage Training Procedure · p. 3

B. One Step Towards Speech QA

The paper explores text-in-speech-out speech QA as a step toward responding to requests in both text and speech. It also reports zero-shot TTS MOS scores for two baselines.

  • The authors explore text-in-speech-out speech QA as a step toward responding to requests in both text and speech modalities.Source: B. One Step Towards Speech QA · p. 3
  • Voice Craft scored 2.85±0.13 for human likeness and 3.17±0.11 for audio quality; Your TTS scored 1.92±0.11 and 2.68±0.11, respectively.Source: B. One Step Towards Speech QA · p. 3, Table I · p. 3
  • The speech-QA exploration uses chain-of-modality instruct-tuning, then fine-tunes a speech-QA expert to generate text tokens followed by semantic tokens.A final stage integrates the text-QA and speech-QA experts into one multimodal LLM.Source: B. One Step Towards Speech QA · p. 3

A. Settings

The experiments use speech, text-QA, and speech-QA training data, with evaluations spanning TTS, text normalization, and text QA. The setup includes parameter-efficient fine-tuning and specified training and evaluation conditions.

a) Datasets

  • The speech training mixture totals 50K hours paired with transcripts, with about 60K internally curated text-normalization sentences added.Text QA uses 2M in-house QA pairs; speech QA uses about 1M samples with text instructions and text-and-speech responses, after filtering coding- and math-heavy QA.Source: A. Settings · pp. 3–4

b) Model Parameters

  • MoLE-Llama uses 1.4B trainable parameters in Stage 1, 0.3B in Stage 2, and a 30M-parameter router in Stage 3.The speech vocabulary has 4096 semantic tokens; LoRA adapters use rank 128 and alpha 64.Source: A. Settings · p. 4

c) Training Details

  • Training uses supervised fine-tuning with Llama Instruct prompts, AdamW, and a cosine learning-rate schedule, on 8 H100 GPUs for up to 20K steps.The maximum sequence length is 2048 and batch size is 256.Source: A. Settings · p. 4

d) Evaluation Metrics

  • TTS is evaluated on zero-shot synthesis and text normalization, using MOS ratings for human likeness and audio quality with 95% confidence intervals.The zero-shot test uses 50 samples; the text-normalization test uses 25 samples containing at least two special characters.Source: A. Settings · p. 4

e) Baseline Methods

  • Text QA accuracy is measured on MMLU, GPQA, and ARC Challenge; TTS comparisons include YourTTS and VoiceCraft, while text QA is compared mainly with Llama3-8B-Instruct.MMLU results for SPIRIT-LM are included for reference.Source: A. Settings · p. 4

B. Results

TTS-Llama outperforms the reported TTS baselines on zero-shot and text-normalization tasks, while MoLE-Llama with text QA performs comparably on TTS. MoLE-Llama also mitigates the text-QA forgetting observed in TTS-Llama and performs strongly across three benchmarks.

a) TTS Task

  • TTS-Llama outperforms all baselines on zero-shot and text-normalization TTS, and achieves competitive text-normalization results without a frontend module.MoLE-Llama (TTS+text QA) performs comparably to TTS-Llama on TTS while offering additional text-QA capability.Source: B. Results · p. 4

b) Text QA Task

  • MoLE-Llama mitigates TTS-Llama’s catastrophic forgetting on text QA, with both versions performing strongly across three benchmarks.MoLE-Llama achieves performance comparable to SPIRIT-LM on MMLU using late-fusion PEFT; SPIRIT-LM uses 1–2 orders of magnitude more training data and full fine-tuning.Source: B. Results · p. 4

VI. CONCLUDING REMARKS

The work presents TTS-Llama, which uses PEFT to enable speech generation from a text-based LLM, and MoLE-Llama, which combines text-QA and TTS without catastrophic forgetting. The conclusion identifies late fusion’s potential and notes two areas for extension.

  • MoLE-Llama combines text-QA and TTS capabilities using a mixture of LoRA experts without catastrophic forgetting in either modality.Source: §VI · p. 4
  • TTS-Llama demonstrates the effectiveness of enabling TTS generation by applying only PEFT fine-tuning to a text-based LLM.Source: §VI · p. 4
  • The authors identify late fusion as a promising approach for enabling text-speech multimodal LLMs.Source: §VI · p. 4
  • Future work could close the gap between the text-QA expert and the original text-only LLM, and improve speech-output QA beyond the chain-of-modality technique.Source: §VI · p. 4
Loading 2410.20336v1…