Source-linked AI summary
Spirit LM: Interleaved Spoken and Written Language Model
Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Mary Williamson, Gabriel Synnaeve, Juan Pino, Benoit Sagot, Emmanuel Dupoux
TL;DR
SPIRIT LM addresses the challenge of combining text LLM capabilities with expressive speech generation. It continuously pretrains LLaMA 2 on text, speech, and interleaved speech-text tokens, with HuBERT-based BASE and pitch-and-style-based EXPRESSIVE versions. The models generate speech and text cross-modally, support few-shot tasks, and preserve sentiment within and across modalities, while remaining limited by English-only evaluation, 7B scale, reduced text performance, and unresolved safety work.
Problem
Existing speech models and text-to-speech pipelines limit unified cross-modal generation and expressive speech modeling.
Method
SPIRIT LM continuously pretrains LLaMA 2 on text, speech, and word-level interleaved speech-text tokens, adding pitch and style tokens in its EXPRESSIVE version.
Results
SPIRIT LM generates speech and text cross-modally, learns few-shot tasks across modalities, and preserves sentiment within and across modalities.
Takeaways & Limitations
A single interleaved language model can provide multimodal semantic abilities together with expressive speech generation.
Takeaways & Limitations
Evaluation is restricted to English and 7B models, text performance trails the initial LLaMA 2 model, and further safety and alignment work is needed.
Abstract
from arXiv · showhide
We introduce Spirit LM, a foundation multimodal language model that freely mixes text and speech. Our model is based on a 7B pretrained text language model that we extend to the speech modality by continuously training it on text and speech units. Speech and text sequences are concatenated as a single stream of tokens, and trained with a word-level interleaving method using a small automatically-curated speech-text parallel corpus. Spirit LM comes in two versions: a Base version that uses speech phonetic units (HuBERT) and an Expressive version that models expressivity using pitch and style units in addition to the phonetic units. For both versions, the text is encoded with subword BPE tokens. The resulting model displays both the semantic abilities of text models and the expressive abilities of speech models. Additionally, we demonstrate that Spirit LM can learn new tasks in a few-shot fashion across modalities (i.e. ASR, TTS, Speech Classification). We make available model weights and inference code.
1 Introduction
SPIRIT LM combines text-language-model capabilities with speech generation by training a single model on text, speech, and interleaved speech-text sequences. It supports cross-modal generation, few-shot learning, and expressive speech, while requiring safety precautions because it can generate harmful content.
- Motivation: SPIRIT LM combines the generative abilities of text LLMs with the expressive capacities of speech-language models.The model is designed to understand and generate language in either speech or text modality.
- Capabilities: SPIRIT LM extends few-shot learning to text, speech, and cross-modal tasks such as ASR, TTS, and speech classification.The authors evaluate comprehension in both modalities and cross-modal prompting.
- Expressivity: SPIRIT LM EXPRESSIVE adds pitch and style information to phonetic speech representations and introduces the STSP sentiment-preservation benchmark.The benchmark evaluates sentiment preservation within and across speech and text modalities.
- Approach: The model is trained on text-only, speech-only, and interleaved speech-text sequences using a shared token stream.Speech and text are encoded into tokens and processed with next-token prediction.
- Safety: SPIRIT LM can generate harmful content, so user-facing applications require red-teaming and safety instruction-tuning.The paper specifically recommends safety work for applications built with the model.
2 Related Work
Prior speech language models often focus on speech-only data or specific tasks, while text-to-speech pipelines constrain expressive speech outside the text model. SPIRIT LM instead uses shared speech-text representations and targets cross-modal generation and expressive modeling.
- Speech representations: Textless NLP derives discrete speech tokens that can capture linguistic content and prosody for language modeling from raw audio.SPIRIT LM uses phonetic HuBERT tokens, optionally combined with pitch and style tokens, alongside text BPE units.
- Related systems: Existing systems include speech-only models, acoustic-token TTS systems, and cross-modal models for ASR, TTS, and speech continuation.The related work spans GSLM, AudioLM, Vall-E, SpeechGPT, Spectron, and related approaches.
- Positioning: SPIRIT LM differs by supporting cross-modal generation, expressive speech and text generation, and zero-shot and in-context learning with larger models.The authors position these capabilities as distinguishing their work from prior speech/text continuation systems.
3 SPIRIT LM Training Recipe
SPIRIT LM continuously pretrains LLaMA 2 on text, speech, and aligned speech-text data represented as interleaved tokens. The BASE model uses HuBERT speech units, while EXPRESSIVE adds pitch and style tokens to model speech expressivity.
- Training data: SPIRIT LM continuously pretrains LLaMA 2 on text-only, speech-only, and aligned speech-text datasets.The model is trained with a mixture of modality-specific and aligned sequences.
- Speech representations: SPIRIT LM BASE uses HuBERT phonetic tokens, whereas SPIRIT LM EXPRESSIVE concatenates HuBERT, pitch, and style tokens.The expressive version sorts these token types by timestamps and deduplicates HuBERT and pitch tokens.
- Interleaving: Aligned speech and text are interleaved at word boundaries so the model can learn correspondence between modalities.Speech and text spans are sampled randomly at each training step, with modality tokens marking each span.
- Expressivity: Expressive speech modeling supplements phonetic units with pitch and style features to capture and generate speech emotion without speaker identity.Pitch is quantized from F0, while style features are normalized to remove speaker information.
- Training data: The training corpus includes 300B text tokens, 460K hours of speech, and 110K hours of aligned speech-text data.Aligned data are automatically curated and may contain alignment errors.
4 Speech and Text Understanding
SPIRIT LM combines speech and text modeling in one interleaved-token language model, supporting semantic understanding, cross-modal generation, and few-shot learning across modalities. Its performance shows benefits from interleaving and pretrained knowledge, while expressive speech units impose a moderate understanding cost.
- Speech- and Text-only Tasks: SPIRIT LM BASE competes with baselines on speech WUGGY, BLIMP, and StoryCloze while preserving competitive text performance.It outperforms baselines by a large margin on StoryCloze, which requires more advanced speech semantic abilities.
- Ablation Experiments: Interleaving training improves speech understanding and generation compared with speech-only or parallel-data alternatives.Speech-only fine-tuning performs more than 6 points lower on spoken StoryCloze, while parallel-data variants are more than 10 points lower on cross-modal Topic-StoryCloze.
- Expressivity comes with a moderate modeling cost: SPIRIT LM EXPRESSIVE performs lower than BASE on lexical, grammatical, and semantic tasks because expressive units lengthen sequences and increase modeling cost.The expressive version adds pitch and style units to HuBERT tokens, increasing sequence complexity for the same raw speech input.
- Cross-Modal Performance: SPIRIT LM supports cross-modal StoryCloze, with S→T performance about 5 points above speech-only performance while T→S remains on par with speech-only performance.The results indicate stronger text generation than speech generation when prompted with speech.
- Pretrained Knowledge is Essential for Few-Shot Learning: Pretrained knowledge enables rapid few-shot transfer: after 25k steps, SPIRIT LM BASE exceeds 75% intent-classification accuracy, while a randomly initialized model remains below 20%.The comparison supports the importance of initializing from pretrained LLAMA 2 for speech-modality few-shot learning.
5 Expressivity Modeling
This section evaluates Spirit LM’s expressive speech modeling using pitch and style tokens, sentiment-preservation tasks, and zero- and few-shot settings across modalities.
- Expressivity Modeling: Spirit LM models expressivity with pitch and style tokens alongside phonetic speech tokens.The evaluation covers tokenization quality and sentiment preservation.
- Sentiment Preservation: The sentiment-preservation benchmark tests whether generated text or speech retains the positive, negative, or neutral sentiment of a written or spoken prompt.Prompts and continuations can remain within one modality or cross between speech and text.
- Sentiment-Rich Prompts: The benchmark uses expressive speech from EXPRESSO-READ and EMOV, plus transcribed expressive dialogue from EXPRESSO-ASR.The datasets provide speech and text prompts with aligned expressive or emotional content.
- Evaluation Settings: Evaluation measures sentiment consistency with fine-tuned text and speech classifiers on generated continuations.Generation parameters are tuned on development sets, and test prompts are used for zero-shot and few-shot evaluation.
- Results: In zero-shot evaluation, SPIRIT LM EXPRESSIVE surpasses BASE in every direction except T→T, where they perform comparably.Against the cascade baseline, EXPRESSIVE outperforms it in all directions.
- Results: Few-shot prompting improves every direction except S→S, while sentiment preservation is stronger within the same modality than across modalities.S→T is the lowest-scoring direction, and high prompt scores indicate room to improve expressive preservation.
6 Responsible AI in Speech and Text
This section evaluates added toxicity in Spirit LM’s text and speech generation. It finds comparable ETOX results with a cascade baseline but higher MUTOX added toxicity, especially for speech-to-speech generation.
- Scope: The safety evaluation focuses on added toxicity, defined as toxicity in generated content exceeding that of the source prompt.The study addresses verbal toxicity and leaves non-verbal toxic content for future work.
- Evaluation: HOLISTICBIAS prompts are used to generate text-to-text and speech-to-speech outputs, with toxicity measured using MUTOX, ETOX, and ASR-ETOX.Speech outputs are transcribed with ASR before ETOX evaluation.
- Metrics: For ETOX and ASR-ETOX, added toxicity means more toxic words in the output than in the prompt; for MUTOX, it requires a score increase above 0.7.These criteria compare toxicity in the generated output with toxicity in the input prompt.
- Results: SPIRIT LM and the ASR + LLAMA 2 + MMS-TTS cascade have comparable ETOX results, but SPIRIT LM has higher MUTOX added toxicity, especially in S→S.The authors suggest this may relate to more toxic content in the speech training data and leave mitigation to future work.
- Demographic Analysis: Added toxicity is highest for gender and sex and sexual orientation, and lowest among ability and nationality, with little distributional difference across modalities or metrics.The analysis covers 13 demographic axes in HOLISTICBIAS.
7 Limitations and Broader Impacts
Spirit LM combines multimodal semantic abilities with expressive speech generation, while retaining important safety, training, evaluation, and scaling limitations. Its broader deployment therefore requires further architectural, safety, and alignment work.
- Harmful applications: SPIRIT LM inherits risks including harmful applications, bias, toxic generations, and speaker impersonation or inconsistent speaker robustness.The paper discusses watermarking, RLHF, and dataset diversity as mitigation strategies.
- Future work: Combining style and pitch tokens with phonetic tokens enables expressive speech generation alongside multimodal semantic abilities.The authors identify further architectural and training improvements as opportunities to advance speech generation.
- Scope limitations: The evaluation is restricted to English, so quality and safety in non-English languages remain unassessed.The authors call for further investigation beyond English.
- Scope limitations: The experiments use only 7B models, leaving the effects of scaling beyond 7B unresolved.The authors suggest that larger-scale experiments could improve performance.
- Training and alignment: Multimodal training currently reduces text performance relative to the initial LLAMA 2 model, and the foundational models require additional safety and alignment work.The authors identify training refinement and alignment with user expectations as open needs.
8 Conclusion
Spirit LM is a LLAMA 2-based model that generates speech and text cross-modally. By alternating speech and text during training, it can change modalities while generating content, though future work targets capability, transparency, and safety improvements.
- Conclusion: SPIRIT LM is based on LLAMA 2 and generates both speech and text in a cross-modal manner.The model was evaluated using speech and text metrics.
- Conclusion: Alternating speech and text in the input sequence enables fluid generation that changes from one modality to another.The conclusion describes this capability as arising from the training procedure.
- Future work: The authors plan further improvements in model capability, transparency, and safety.
A LM Training Optimization
The training and prompting setup extends LLAMA 2 with speech and modality tokens, then uses few-shot prompts and task-specific markers to control generation. The appendix also notes a limited effect from changing word-span length.
- Training: The 7B LLAMA 2 model is continuously pretrained after adding randomly initialized speech and modality token embeddings.Training uses a 4k sequence length, batch size 4 per GPU, 64 A100 GPUs, and 200K steps.
- ASR prompting: ASR prompting uses special start and end flags because omitting them can cause hallucinated output after transcription.
- Few-shot prompting: Ten-shot prompting produces the best reported performance for ASR and TTS prompting setups.ASR and TTS examples use modality-specific markers and speech start or stop tokens.
- ASR prompting: For ASR, the [ASR] token replaces [SPEECH] in models trained with parallel ASR data to trigger transcription prediction.
- Training analysis: Changing word-span length had little impact on evaluation metrics in the initial experiments.The authors expect longer-context metrics may warrant more detailed analysis.
- TTS prompting: For TTS, [SPEECH] is replaced with [TTS] in models trained with parallel TTS data, while <speech:STOP> marks the spoken utterance stop token.
- Intent Classification: Intent Classification prompting maps speech token sequences to text outputs using paired examples such as “activate lights bedroom.”The output is postprocessed using special tokens and sequence boundary flags.
C Construction of Few-Shot examples for Sentiment Continuation
The sentiment-continuation benchmark constructs cross-modal few-shot prompts by pairing sentiment-consistent speech and transcription segments. Examples are balanced and reused across iterations, with generation settings varying by modality direction.
- Construction procedure: The construction uses S→T as an illustration, with the same process applied to the remaining modality directions.
- Data preparation: Speech samples longer than 200,000 waveform units are split into two equal parts, and the second segment is transcribed.The transcription uses WHISPER-MEDIUM.
- Data filtering: Pairs are retained only when the second segment’s transcription has the same sentiment as the first segment’s speech.This filtering creates sentiment-consistent continuation examples.
- Few-shot prompts: At each run, 3, 6, or 9 balanced sentiment samples are randomly selected, concatenated into an in-context prompt, and reused for subsequent iterations.
- Generation settings: Generation uses 50 maximum tokens for T→T and S→T, 200 for T→S, and 300 for S→S, with temperature 0.8 and top_p 0.95.All reported SPIRIT LM models were trained for 100k steps.
- Benchmark: Table 8 reports the statistics of the SPEECH-TEXT SENTIMENT PRESERVATION benchmark datasets.
- Examples: Table 1 presents generation samples from SPIRIT LM.
G Complementary Results
Complementary evaluations examine how aligned speech–text data, interleaving, modality alignment, toxicity, comprehension, and expressive resynthesis relate to Spirit LM’s behavior. The reported analyses compare Spirit LM variants and training setups across speech and text conditions.
- Aligned data: Figure 3 evaluates SPIRIT LM BASE on Topic-StoryCloze as the sampled share of aligned speech+text data varies from 0% to 100%.The full aligned corpus contains 8.4B tokens: 1.4B text and 7B speech.
- Training comparisons: Figure 4 compares SPIRIT LM BASE with a randomly initialized model and with a model trained without interleaved aligned data.The no-interleaving model uses separate raw-speech or raw-text sequences rather than interleaved aligned data.
- Additional evaluations: The complementary evaluations include toxicity distributions across 13 demographic axes, zero-shot comprehension in speech and text, and expressive speech resynthesis.The resynthesis tables compare Spirit LM tokenizers with HuBERT + HiFi-GAN and Encodec systems across Expresso, LibriSpeech, Fisher, and EmoV datasets.
- Feature alignment: Figure 6 measures speech–text feature similarity across model layers and compares Spirit LM with a model trained without speech–text interleaving.The analysis includes same-word maximum similarity averaged over a test set and pairwise cosine similarity for speech and text features from the same sentence.