PaperlayerView exact revision on arXiv

Source-linked AI summary

TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling

Liang-Hsuan Tseng, Yi-Chang Chen, Kuan-Yi Lee, Da-Shan Shiu, Hung-yi Lee

arXiv:2504.07053v3cs.CLcs.SDeess.AS

TL;DR

Joint text–speech modeling faces a length mismatch between speech and text tokens. TASTE creates text-aligned speech tokens through cross-attention, and its SLMs outperform other pretrained SLMs on speech continuation while performing comparably on SALMON and StoryCloze.

  • Problem

    Joint text–speech modeling faces length mismatch because speech token sequences are usually much longer than their textual counterparts.

    Source: §1 · pp. 1–2

  • Method

    TASTE uses specialized cross-attention to derive speech tokens from transcriptions for reconstruction, producing text-aligned tokens without explicit alignment.

    Source: §1 · p. 2, §6 · p. 10

  • Results

    TASTE-based SLMs outperform other pretrained 7B SLMs on speech continuation across human and machine evaluations, while performing comparably on SALMON and StoryCloze.

    Source: §1 · p. 2, §5.1.2 · p. 9

  • Takeaways & Limitations

    The results identify joint tokenization tailored to spoken language modeling as a key factor in joint modeling.

    Source: §1 · p. 3

    Evaluation so far is English-only; generalizability to other languages remains future work.

    Source: §6 · p. 10

Abstract

from arXiv · show

Recent efforts target spoken language models (SLMs) that not only listen but also speak for more natural human-LLM interaction. Joint speech-text modeling is a promising direction to achieve this. However, the effectiveness of recent speech tokens for joint modeling remains underexplored. To address this, we introduce Text-Aligned Speech Tokenization and Embedding (TASTE), a method that directly addresses the modality gap by aligning speech token with the corresponding text transcription during the tokenization stage. We propose a method that can achieve this through a attention-based aggregation mechanism and with speech reconstruction as the training objective. We conduct extensive experiments and show that TASTE can preserve essential paralinguistic information while dramatically reducing the token sequence length. With TASTE, we perform straightforward joint spoken language modeling by using Low-Rank Adaptation on the pre-trained text LLM. Experimental results show that TASTE-based SLMs perform comparable to previous work on SALMON and StoryCloze; while significantly outperform other pre-trained SLMs on speech continuation across subjective and objective evaluations. To our knowledge, TASTE is the first end-to-end approach that utilizes a reconstruction objective to automatically learn a text-aligned speech tokenization and embedding suitable for spoken language modeling. Our demo, code, and model are available at https://mtkresearch.github.io/TASTE-SpokenLM.github.io.

1 INTRODUCTION

The introduction motivates text-aligned speech tokens to address length mismatch and redundant content in joint spoken language modeling. It presents TASTE and reports speech-reconstruction and spoken-language-modeling results.

1 INTRODUCTION

  • Joint text–speech modeling faces length mismatch because speech token sequences are usually much longer than their textual counterparts.The introduction also motivates tokens that avoid redundantly encoding text content and correspond one-to-one with text tokens.Source: §1 · pp. 1–2
  • TASTE derives speech tokens from the transcription through specialized cross-attention for speech reconstruction, without requiring explicit speech-text alignment.Its text-aligned tokens have dynamic frequency rather than a fixed down-sampling rate.Source: §1 · p. 2
  • At ∼150 bps, TASTE achieves high-end speech reconstruction while retaining similarity to the original speech.The text-aligned tokens also allow duration and tone to be exchanged between same-content utterances through token swaps.Source: §1 · p. 2
  • TASTE enables text-aligned speech editing: swapping tokens between same-content utterances transfers duration and tone with the exchanged words, producing natural edited speech.The introduction identifies duration and tone as examples of paralinguistic information.Source: §1 · p. 2
  • TASTE-based SLMs outperform other pretrained 7B SLMs on speech continuation across evaluation aspects, while performing comparably on SALMON and StoryCloze.The continuation evaluation includes human and machine evaluations; the SALMON and StoryCloze results are compared with other text-speech joint modeling methods.Source: §1 · p. 2
  • TASTE supports straightforward joint modeling with Low-Rank Adaptation, and the pretrained SLM also performs spoken question answering in a few-shot scenario.The reported continuation experiments use 3-second speech prompts.Source: §1 · p. 2

2 RELATED WORK

Related work surveys speech-tokenization strategies and the shift toward joint speech-text modeling. It identifies limited discussion of tokenization effectiveness for that setting and distinguishes TASTE’s goal from cross-attention used for TTS style transfer.

2 RELATED WORK

  • Earlier speech tokenization uses discretized self-supervised representations or codec units, with codec models designed for resynthesis.The cited approaches include SSL-based methods and audio or speech codec models.Source: §2 · p. 3
  • Joint SLMs have used interleaving, reduced token frequency, and delayed or sequential generation to facilitate speech-text modeling.The passage describes Spirit LM’s interleaving strategy and moshi’s reduced token frequency.Source: §2 · p. 3
  • The authors report no prior discussion of current speech-tokenization effectiveness for joint speech-text modeling, motivating text-aligned tokenization at the tokenization stage.Recent efforts instead include reducing token frequency, adding alignment training, or using ASR-encoder representations.Source: §2 · p. 3
  • TASTE’s text-speech cross-attention targets text-aligned tokenization for joint SLM, unlike content-style cross-attention proposed for style transfer in TTS.The authors identify the tokenization and embedding for joint spoken language modeling—not the attention mechanism alone—as their main novelty.Source: §2 · p. 3

3 METHOD

TASTE aligns speech representations with transcription tokens, then uses them for speech reconstruction and joint spoken language modeling. Its tokenizer, decoder, training losses, and two modeling variants form the method’s main components.

3 METHOD

  • The method introduces TASTE as a joint tokenization for text-speech spoken language modeling, with tokenizer and decoder details organized around speech reconstruction.The framework’s overall stages are shown in Figure 2.Source: §3 · p. 3, §3.1 · p. 3, Figure 2 · p. 4

3.1 BUILDING TASTE

  • TASTE comprises a text-aligned speech tokenizer and a speech decoder trained to reconstruct speech from text and aligned speech tokens.Figure 2 depicts tokenization on the left and reconstruction on the right.Source: §3.1 · p. 3, Figure 2 · p. 4

3.1.1 TASTE SPEECH TOKENIZER

  • The tokenizer aggregates speech encoder representations into a compressed sequence matching the transcription length, then applies residual vector quantization to produce aligned codes and embeddings.Its attention uses transcription tokens as queries, the encoder’s last hidden states as keys, and shallow hidden states as values.Source: §3.1.1 · pp. 4–5

3.1.2 TASTE SPEECH DECODER

  • The speech decoder conditions autoregressive unit prediction on the transcription and aligned speech embedding, then converts predicted units into reconstructed speech.A speaker embedding is also used for global speaker voice control.Source: §3.1.2 · p. 5

3.1.3 TRAINING OBJECTIVE

  • TASTE is trained with next-unit cross-entropy reconstruction loss and a quantization commitment loss.The two losses are summed for the overall training objective.Source: §3.1.3 · p. 5

3.2 TASTE FOR SPOKEN LANGUAGE MODELING

  • TASTE’s text-aligned codes or embeddings are used with speech transcripts for joint speech-text pre-training of spoken language models.The paper describes token-based and embedding-based variants.Source: §3.2 · p. 5

3.2.1 MODELING TASTE TOKEN

  • TASLMtoken uses multiple prediction heads to predict the next text token and each RVQ code layer at every step.At inference, it samples text and codes together and sends the codes to the speech decoder.Source: §3.2.1 · p. 6

3.2.2 MODELING TASTE EMBEDDING

  • TASLMemb predicts a latent distribution with mean and log-variance vectors, using regularization and KL-divergence losses alongside text loss.The predicted latent is sampled using the mean, standard deviation, and Gaussian noise.Source: §3.2.2 · p. 6

4 EXPERIMENT SETUP

The experiments use Emilia and LibriTTS for training and LibriSpeech test-clean for evaluation. The setup specifies the tokenizer, reconstruction components, and spoken-language-model fine-tuning configuration.

Model Configuration

  • The tokenizer uses a frozen Whisper encoder, S3 speech units, a pretrained flow-model/HifiGAN vocoder, and an RVQ quantizer with four layers, 512 codes, and dimension 256.Spoken language models are initialized from a text LLM and fine-tuned with LoRA at rank 64 and α = 128.Source: §4 · p. 6

Evaluation

  • Table 1 reports speech-tokenization quality and similarity on LibriSpeech test-clean, with token frequency measured in tokens per second.Reported Text-only and TASTE results account for ASR errors; gray text marks the worst method for each metric.Source: Table 1 · p. 7

Dataset

  • Training uses about 40,000 hours of English Emilia and about 600 hours of LibriTTS; LibriSpeech test-clean is used for evaluation.Emilia transcriptions are pseudo-labeled, and the corpus is described as in-the-wild web-scale speech.Source: §4 · p. 7

5 RESULT

TASTE supports low-rate speech reconstruction while retaining quality and similarity, and TASTE-based SLMs improve speech continuation while remaining competitive on likelihood benchmarks.

5.1.1 TASTE FOR SPEECH RECONSTRUCTION

  • Reconstruction is assessed on quality (ASR-WER, UTMOS, DNS-MOS) and similarity (ViSQOL, duration consistency, speaker similarity, MUSHRA).Duration consistency counts same-word duration matches within a 50-millisecond tolerance.Source: §5.1.1 · p. 7
  • TASTE runs at the lowest token frequency and bitrate, matches or exceeds higher-bitrate methods on quality, and remains comparable on similarity except for weaker ViSQOL results.The authors report that the ViSQOL degradation is not reflected in the MUSHRA listening test, where TASTE attains competitive perceptual quality and similarity.Source: §5.1.1 · p. 8

5.1.2 TASTE FOR SPOKEN LANGUAGE MODELING

  • SLMs are tested with 3-second LibriSpeech prompts for continuation and with likelihood-based choices on SALMON and StoryCloze.Continuation evaluation combines GPT-4o semantic scores, UTMOS acoustic scores, and human overall MOS; the likelihood benchmarks report average accuracy across tasks.Source: §5.1.2 · pp. 8–9
  • TASLM substantially outperforms other pretrained SLMs on speech continuation across human and machine evaluations while remaining competitive on likelihood benchmarks.Table 2 reports speech continuation and likelihood-based next-speech selection results.Source: Table 2 · p. 8, §5.1.2 · p. 9

5.2.1 TASTE FOR TEXT-ALIGNED SPEECH EDITING

  • Swapping TASTE tokens at selected word positions shifts those words’ durations while untouched words preserve their timing, enabling text-aligned speech editing.The editing procedure extracts TASTE tokens, swaps them partially, then decodes the edited tokens into speech.Source: §5.2.1 · p. 9, Figure 3 · p. 9

5.2.3 ABLATION STUDY ON TASTE SPEECH TOKENIZER

  • The aggregator sharply reduces token rate for a small top-5 accuracy drop; quantization lowers accuracy further but stays above text-only, while shallow encoder states outperform final-layer states.Table 4 reports module-wise tokenizer ablations; top-5 accuracy is used as a proxy for information retention.Source: Table 4 · p. 10, §5.2.3 · p. 10

5.2.2 TASLM FOR SPOKEN QUESTION ANSWERING

  • Under few-shot evaluation, TASLM exhibits spoken question-answering ability and is the only compared approach that preserves its base text LLM’s performance.The comparison uses answer-containment accuracy on Web Questions and LLaMA-Questions, with performance compared across text and speech modalities.Source: §5.2.2 · pp. 9–10, Table 3 · p. 10

6 CONCLUSION

TASTE uses cross-attention and ASR initialization to learn text-aligned speech tokens end to end, without explicit word alignment. The conclusion identifies limitations in dialogue capabilities, language coverage, task scope, speech types, and real-time performance.

6 CONCLUSION

  • TASTE uses specialized cross-attention and ASR initialization to learn text-aligned speech tokenization end to end, without explicit word alignment.The authors present joint text–speech modeling with TASTE as straightforward and effective.Source: §6 · p. 10
  • The model lacks turn-taking and instruction following; evaluation is English-only, and its applicability beyond joint SLMs remains underexplored.The pipeline also focuses on single-speaker lexical speech, while latency and streaming performance remain unoptimized.Source: §6 · p. 10
  • TASTE can make voice mimicry and personalized speech synthesis easier, so the authors call for consent, provenance, and anti-abuse safeguards.The paper reports human listening tests with informed consent and no personal information collected.Source: §6 · p. 11

A.1.1 ABLATION STUDY ON THE EFFECT OF ASR

The ablations examine ASR’s influence on tokenization, spoken question answering, and GPT-4o semantic evaluation. They report negligible or insignificant ASR-related differences and better TASLM results than other pretrained SLMs in both evaluation setups.

A.1.1 ABLATION STUDY ON THE EFFECT OF ASR

  • ASR errors introduce almost negligible performance drops in both TASTE speech reconstruction and TASLM spoken question answering.The main-text experiments used ASR transcripts, not ground-truth transcriptions.Source: §A.1.1 · p. 16
  • Using nvidia-parakeet instead of Whisper for GPT-4o’s semantic-coherence evaluation produces no significant relative performance difference.nvidia-parakeet uses an RNN-T backbone.Source: §A.1.1 · p. 16
  • TASLMs achieve much better results than other pretrained SLMs in both GPT-4o evaluation setups.Source: §A.1.1 · p. 16

A.1.2 AGGREGATOR CROSS ATTENTION VISUALIZATION

The authors visualize aggregator cross-attention maps across layers and heads to inspect text–speech alignment. The first- and last-layer maps show different patterns, including aligned heads and a head attending to silence.

A.1.2 AGGREGATOR CROSS ATTENTION VISUALIZATION

  • The cross-attention maps across layers and heads are visualized to assess whether the aggregator learned text–speech alignment patterns.Source: §A.1.2 · p. 16
  • Many last-layer heads clearly show text–speech-aligned behavior in the aggregator cross-attention map.Source: Figure 4 · p. 16
  • The first-layer map differs from the last-layer map, with head 8 showing clear alignment and head 19 lighting up at silence.Source: Figure 5 · p. 17

A.1.3 DISCUSSION ON THE SELECTION OF SHALLOW HIDDEN LAYER

The authors select a shallow Whisper encoder representation as the cross-attention key because it carries acoustic information and aligns well with S3 targets. Their analysis reports that the final encoder layer still reconstructs better than the text-only baseline.

A.1.3 DISCUSSION ON THE SELECTION OF SHALLOW HIDDEN LAYER

  • Selecting Whisper layer 6 gives near-optimal S3 reconstruction, while even the final encoder layer performs better than the text-only baseline.The shallow-layer choice affects the attainable upper bound but is not a strict limitation.Source: §A.1.3 · p. 17
  • Layers 4–8 exhibit the highest CCA with S3 target embeddings, indicating that shallow representations contain the most target-relevant acoustic information.The correlation curve is shown in Figure 6.Source: Figure 6 · p. 17, §A.1.3 · p. 17
  • TASTE maintains good reconstruction quality across noise levels from 20 dB to 5 dB SNR.Source: Table 8 · p. 18

A.1.4 ROBUSTNESS ABLATION ON TASTE TOKENIZER

The appendix reports that TASTE remains stable across controlled noise levels and performs well on StoryCloze, while detailing evaluation protocols and methods for aligning and fusing speech and text representations.

A.1.4 ROBUSTNESS ABLATION ON TASTE TOKENIZER

  • TASTE remains stable from 20dB to 5dB SNR and achieves its best ASR-WER under the noisiest condition.The authors conclude that ASR is not a limiting factor for general applicability and serves as a reliable source of semantic information for reconstruction.Source: §A.1.4 · p. 18

A.1.5 DETAILS ON SALMON AND STORYCLOZE

  • TASLM is weaker on SALMON Room and Background, performs better on gender and speaker, and achieves the best results among pretrained SLMs on StoryCloze.SALMON evaluates likelihood-based detection of altered acoustic samples; StoryCloze tests selection between endings to assess narrative coherence and reasoning. Standard deviations for the reported tables are provided in the main text.Source: §A.1.5 · pp. 18–19, Table 9 · p. 18, §A.1.6 · p. 19

A.2.1 HYPERPARAMETERS AND CONFIGURATIONS

  • Tokenizer training updates the Aggregator, Quantizer, and UnitDecoder for five epochs, while spoken-language-model training uses AdamW and bfloat16 mixed precision.Quantization is off for the first two tokenizer-training epochs; the SLM is trained with LoRA as described in the paper context.Source: §A.2.1 · p. 20

A.4 TACKLING THE VOCABULARY MISMATCH

  • Word-level grouping, averaging, and alignment map ASR-aligned TASTE embeddings to LLM tokens despite differing vocabularies and token sequence lengths.The authors also use word-level averaging during tokenizer training to support high-quality reconstruction.Source: §A.4 · p. 21

A.5 THE WEIGHTED SUM MECHANISM FOR MODALITY FUSION

  • The speech decoder fuses normalized text-aligned speech and text embeddings using a weighted sum with learnable modality weights.The weights are converted to fusion proportions with softmax.Source: §A.5 · p. 22

A.6 DISCUSSION ON THE USAGE OF LLM

  • The authors used ChatGPT to polish prose and state that it did not generate technical content or results.They report reviewing and editing AI-assisted text and taking responsibility for the manuscript.Source: §A.6 · p. 22
Loading 2504.07053v3…