PaperlayerView exact revision on arXiv

Source-linked AI summary

Speaker-Disentangled Chunk-Wise Regression for Syllabic Tokenization

Ryota Komatsu, Kota Kawakita, Takuma Okamoto, Takahiro Shinozaki

arXiv:2607.04064v1cs.CLcs.AIcs.SDeess.AS

TL;DR

Speech tokenization faces a granularity mismatch with text and can entangle speaker characteristics with linguistic content. The paper introduces chunk-wise speaker-disentangled syllabic regression, achieving state-of-the-art syllable segmentation and clustering while improving syntactic and semantic understanding by 7% relative to phone-level SpiRit-LM.

  • Problem

    Speech tokens are typically higher-rate than text tokens, creating a granularity mismatch that can hinder speech-text alignment.

  • Method

    SylReg regresses speaker-perturbed representations toward syllabically structured targets over temporal chunks, followed by self-segmentation distillation.

  • Results

    SylReg achieves state-of-the-art syllable boundary detection and syllabic segment clustering, while SylReg-LM yields a 7% average relative improvement over phone-level SpiRit-LM.

  • Takeaways & Limitations

    Coarse speaker-disentangled syllabic tokens alleviate speech-text granularity mismatch and support higher-level linguistic abstraction in speech language modeling.

Abstract

from arXiv · show

Unsupervised syllabic tokenization aims to learn discrete syllabic tokens that capture latent linguistic content-related structure from raw speech. Recent syllabic tokenization methods employ teacher-student distillation of the pretrained HuBERT to organize latent speech frame representations into syllabic segments. However, when trained with an utterance-level cross-entropy objective, the model predicts speaker identity rather than linguistic content, thereby compromising the purity of syllabic tokens. To address this problem, we propose a speaker-disentangled syllabic tokenizer that regresses speaker-perturbed student representations toward clean teacher targets within fixed-length chunks. Experimental results demonstrate that our proposed method achieves state-of-the-art performance in syllable boundary detection and syllabic segment clustering. Moreover, a speech language model trained on our syllabic tokens achieves a 7% relative improvement in syntactic and semantic understanding over the phone-level SpiRit-LM.

I. INTRODUCTION

The introduction identifies speaker-related bias and prototype collapse in SD-HuBERT as threats to syllabic-token purity. It presents SylReg’s chunk-wise regression and SylReg-LM as solutions that improve syllabic tokenization and high-level linguistic understanding.

  • Motivation: Speech language models use learned speech tokens as pseudo-transcripts, but phonetic tokens occur at 12.5–50 Hz while text uses coarser subword tokens.This granularity mismatch motivates discovering linguistically meaningful syllabic tokens.
  • Problem: SD-HuBERT exhibits valid prototype collapse, with only a small subset of final softmax categories active, degrading syllabic features.Identical category signals can therefore be backpropagated to linguistically diverse utterances.
  • Problem: SD-HuBERT’s final softmax categories moderately correlate with speaker identities, indicating that utterance-level classification contaminates syllabic-token purity.The proposed analysis attributes this speaker-dominating problem partly to stationary speaker characteristics within utterances.
  • Proposed approach: SylReg enforces coherence over mid-level temporal chunks through speaker-disentangled regression, producing speaker-resilient yet syllabically structured representations.It extends frame-wise regression, whose locality did not sufficiently promote syllabic grouping.
  • Results: 7% average relative improvement in syntactic and semantic understanding is achieved by SylReg-LM over the phone-level-interleaved SpiRit-LM.The model is trained on interleaved syllabic and text tokens.

II. Related Work … III. Self-Supervised Syllabic Tokenization

Prior work contrasts phonetic and acoustic speech tokens, then examines coarser syllable-level representations that aggregate temporal information before language-model processing. Syllabic tokenizers preserve unchanged language-model architectures while avoiding some training complications of adapter-based aggregation.

  • A. Trade-off between phonetic and acoustic tokens: Speech representations use phonetic and acoustic tokens with distinct properties.Phonetic tokens can be quantized from self-supervised speech encoders or automatic speech recognition models.
  • A. Trade-off between phonetic and acoustic tokens: Phonetic tokens align strongly with linguistic content and support intelligible speech generation in speech language models.Their fine-grained acoustic details are largely marginalized.
  • B. Learning coarse phonetic tokens for speech LMs: Coarser subword- or syllable-level phonetic tokens form an orthogonal research direction.Some training-free methods apply subword tokenization or deduplication to phonetic tokens.
  • B. Learning coarse phonetic tokens for speech LMs: Training-free subword-token methods remain computationally fine-grained, operating at roughly 20 Hz token frame rates.The passage characterizes these methods as simple despite their relatively high frame rates.
  • B. Learning coarse phonetic tokens for speech LMs: Speech adapters aggregate frame-level tokens into coarser representations but can complicate speech-language-model training.Reported complications include curriculum learning requirements and risks of model collapse.
  • B. Learning coarse phonetic tokens for speech LMs: Syllabic tokenizers move temporal aggregation into tokenization, allowing language models to remain architecturally unchanged.This approach is presented as an alternative to front-end speech adapters.

A. Baseline method: SD-HuBERT · B. Speaker-dominating problem in SD-HuBERT

SD-HuBERT applies DINO-based teacher-student distillation to HuBERT, using utterance-level [CLS] classification to induce syllabic organization in frame representations. However, its predicted categories strongly encode speaker identity, compromising linguistic-content discrimination.

  • A. Baseline method: SD-HuBERT: SD-HuBERT finetunes pretrained HuBERT with DINO, prepending a learnable [CLS] token to the speech-frame sequence.The [CLS] token is aggregated through self-attention layers into an utterance-level embedding z^(L).
  • A. Baseline method: SD-HuBERT: A classification head computes logits as dot products between the final-layer [CLS] embedding and learnable prototypes.Each prototype corresponds to a softmax category.
  • A. Baseline method: SD-HuBERT: The student and moving-average teacher minimize an utterance-level cross-entropy objective to produce pseudo-categories.This distillation yields syllabic organization in student frame representations from the ninth Transformer layer.
  • A. Baseline method: SD-HuBERT: SD-HuBERT’s frame self-similarity matrix exhibits block structures with boundaries roughly matching ground truth.The minimum cut algorithm extracts syllabic segments, with an efficiency improvement.
  • B. Speaker-dominating problem in SD-HuBERT: Speaker-normalized mutual information measures the relative entropy reduction about speaker identity from predicted categories.It is defined as I(X; Y)/H(X) = 1−H(X | Y)/H(X).
  • B. Speaker-dominating problem in SD-HuBERT: 0.61 speaker-normalized mutual information was observed on the LibriSpeech test set.The value indicates substantial speaker information in the predicted categories.
  • B. Speaker-dominating problem in SD-HuBERT: The model therefore tends to discriminate speaker identity rather than linguistic content.This interpretation is consistent with findings that utterancewise mean frame representations capture relatively stationary speaker characteristics.
  • B. Speaker-dominating problem in SD-HuBERT: The self-attended [CLS] representation can be interpreted as a weighted global average of frame representations.This provides a mechanism for speaker information to dominate the utterance-level representation.

C. SylReg: Speaker-disentangled chunk-wise regression

SylReg addresses speaker-dominated syllabic tokenization by matching speaker-invariant representations from original and speaker-perturbed speech within fixed-length chunks. Its BYOL-style training uses chunk-level regression, normalized MSE, and an EMA-updated teacher, with constrained perturbations to preserve naturalness.

  • Objective and architecture: SylReg matches speaker-invariant representations between original and speaker-perturbed speech within fixed-length chunks to emphasize linguistic content.The method follows a BYOL-style architecture with student and teacher branches.
  • Chunk-wise regression: Chunk-wise regression encourages mid-level temporal grouping that empirically aligns with syllables, unlike BYOL’s global-level regression.Average pooling with chunk size C aggregates final-layer hidden states and reduces sequence length by a factor of C.
  • Optimization: The student minimizes mean squared error between ℓ2-normalized teacher and student outputs, while teacher gradients are stopped and its parameters follow an exponential moving average of the student.The CNN encoder is frozen in both branches for efficient training.
  • Speaker perturbation: Speaker perturbations use formant shifts and pitch changes restricted to male-to-female or female-to-male conversions to avoid unnatural speech.The speaker’s gender is estimated from each utterance’s average pitch.

D. Self-segmentation distillation

Self-segmentation distillation transfers SylReg’s emergent block-diagonal syllabic structure into data2vec 2.0, producing SylReg-Distill. The process uses pseudo-syllable boundaries, segment-averaged teacher targets, speaker-perturbed student inputs, and iterative teacher updates for self-refinement.

  • D. Self-segmentation distillation: SylReg-Distill transfers SylReg’s block-diagonal syllabic structure into pretrained data2vec 2.0 through self-segmentation distillation.Data2vec 2.0 is selected because its representations support better syllabic segment clustering than HuBERT’s.
  • D. Self-segmentation distillation: The first stage segments each utterance into pseudo-syllable boundaries, initializes a data2vec 2.0 Base teacher-student pair, and prunes the teacher’s 12th Transformer layer.Teacher frame representations are averaged within each pseudo-syllable segment to form regression targets.
  • D. Self-segmentation distillation: Speaker perturbation is applied to the student inputs during self-segmentation distillation.
  • D. Self-segmentation distillation: After each stage, the teacher copies the student parameters and is reused to extract and segment regression targets, enabling iterative self-refinement.

IV. Generative Spoken Language Modeling … C. Token-to-speech synthesis

The generative spoken language modeling pipeline combines a speech tokenizer, speech language model, and token-to-speech synthesizer. Its described components segment and quantize syllabic representations, train on interleaved text-syllable sequences, and synthesize acoustic features from speech tokens.

  • IV. Generative Spoken Language Modeling: The GSLM pipeline comprises a speech tokenizer, a speech LM, and a token-to-speech synthesizer.These three modules organize the system described in the section.
  • A. Syllable segmentation: Syllabic tokenization detects boundaries and quantizes representations through minimum cut segmentation, segment-wise average pooling, and two-step clustering.The procedure uses a PyTorch implementation for improved efficiency.
  • A. Syllable segmentation: M = ⌈T · F/50⌉ syllable segments are obtained from student Transformer-layer outputs using the HuBERT frame rate of 50 Hz.The upper-bound syllabic token frame rate is denoted by F.
  • A. Syllable segmentation: O(M)-complexity dynamic programming implements minimum cut segmentation efficiently after oversegmenting frames for fast speaking rates.Adjacent segments are then merged, according to the supplied passage.
  • B. Interleaved syllable-text language modeling: Following SpiRit-LM, the language model vocabulary is expanded with syllabic tokens and continually trained on sequences interleaving text and syllabic tokens.This creates a mixed textual and syllabic-token modeling setup.
  • B. Interleaved syllable-text language modeling: Word-level timestamps from a forced aligner associate words with segmented syllabic tokens, enabling chunk-wise random sampling of text or syllabic tokens.The sampling procedure constructs interleaved syllable-text data.
  • C. Token-to-speech synthesis: A conditional flow-matching Diffusion Transformer with a BigVGAN-v2 vocoder synthesizes speech from syllabic representations conditioned on log mel-spectrograms.The architecture supports parallel acoustic-feature generation from speech tokens.
  • C. Token-to-speech synthesis: Fewer than ten non-autoregressive denoising steps enable fast acoustic-feature generation, while a length regulator repeats input syllabic representations according to predicted duration.The length regulator operates on representations from the input embedding layer.

V. Experimental Setup · A. Datasets · B. Implementation details

The experiments train SylReg on large-scale English speech, distill syllable segments and evaluate tokenizers on LibriSpeech using syllable alignments. Implementation specifies chunked speech encoding, staged SylReg optimization, and DiT training and inference settings.

  • A. Datasets: SylReg is trained on Libri-Light’s 55k-hour audiobook corpus before syllable-segment distillation on LibriSpeech train-clean-100.Libri-Light provides 55k hours of audio book data.
  • A. Datasets: Quantizers are trained on the LibriSpeech train set and evaluated on its test set using syllable alignments.The evaluation uses syllable alignments.
  • A. Datasets: Language modeling uses 129k hours of English speech corpora, including LibriSpeech, Libriheavy, Emilia-Large, People’s Speech, VoxPopuli, and synthetic speech.The listed People’s Speech subsets are clean and clean_sa, while VoxPopuli uses its transcribed subset.
  • B. Implementation details: The speech encoder uses a default chunk size of C = 100 frames, with 2-layer MLP projector and predictor networks ending in 256-dimensional outputs.Each MLP has a 2048-output linear layer, batch normalization, GELU, and a second linear layer with output size 256.
  • B. Implementation details: SylReg is trained for 10k steps with AdamW, 0.01 weight decay, 1e-3 gradient clipping, 0.999 EMA decay, batch size 1024, and fixed LR 1e-4.The learning rate uses linear warmup over the first 100 steps.
  • B. Implementation details: Only the randomly initialized projector and predictor are updated during the first 2k training steps before self-segmentation distillation follows chunk-wise regression.The passage states that self-segmentation distillation is performed after chunk-wise regression to stabilize training.
  • B. Implementation details: The length regulator is a single kernel-size-3 convolution, while DiT is pretrained and finetuned for 200k and 50 steps, respectively.The corresponding gradient clipping values are 0.1, batch sizes are 400 and 14k sentences, and learning rates are 1e-3 and 1e-4.
  • B. Implementation details: During DiT training, entire syllabic token sequences are dropped with probability 0.2 for CFG; inference uses Euler step size 0.1 and CFG strength 0.7.The training schedule includes 1k linear warmup steps.

VI. Evaluation · A. Syllable segmentation and token quality

The evaluation measures syllable-boundary detection and syllabic-token quality, using multiple boundary and clustering metrics. On LibriSpeech, SylReg consistently outperforms HuBERT in both segmentation and token purity.

  • A. Syllable segmentation and token quality: Boundary detection uses precision, recall, F1, and R-value with a 50 ms tolerance.R-value balances recall against over-segmentation.
  • A. Syllable segmentation and token quality: Token quality is evaluated using syllable purity, cluster purity, and syllable-normalized mutual information.SNMI measures the fraction of syllable entropy explained by the tokens.
  • A. Syllable segmentation and token quality: Reference and predicted syllables are aligned through maximum weight matching over their segments’ temporal intersection-over-union matrix.This alignment supports evaluation of predicted syllabic segments against ground-truth syllables.
  • A. Syllable segmentation and token quality: Token quality is generally comparable among models with the same vocabulary size.The passage notes that vocabulary size affects token-quality comparisons.
  • A. Syllable segmentation and token quality: Increasing vocabulary size improves SP by reducing syllable mixing within clusters.The supplied passage ends mid-sentence after this observation.
  • A. Syllable segmentation and token quality: On the LibriSpeech test set, SylReg consistently outperforms HuBERT in syllable segmentation and token purity.The results attribute this advantage to avoiding degradation from naive segmental pooling of phone-level representations.

B. Ablation study · C. Effect of chunk size on SylReg · D. Layer-wise analysis of segmentation performances

The ablation study finds that DINO-based alternatives and removing speaker perturbation weaken syllabic tokenization, while larger chunks improve segmentation but not clustering uniformly. Layer-wise analysis selects the 8th Transformer layer because recall peaks there despite the 9th layer’s higher precision.

  • B. Ablation study: Across all metrics, replacing BYOL with DINO and using cross-entropy instead of MSE produces uniform degradation.The DINO configuration uses 2048 classes, student temperature 0.2, teacher temperature 0.05, and gradient clipping 0.5.
  • B. Ablation study: Removing speaker perturbation from the DINO variant increases TED, reducing token purity because speaking variations can split syllables across clusters.The passage attributes this effect to syllables with speaking variations being assigned to different clusters.
  • B. Ablation study: TED decreases from 16.6 to 13.9 with DINO, but this inherent property alone is insufficient for high-purity syllabic tokenization.The comparison concerns the DINO variant and follows the discussion of removing speaker perturbation.
  • C. Effect of chunk size on SylReg: Segmentation performance consistently improves as chunk size increases and saturates at a chunk size of 100.The analysis contrasts segmentation trends with clustering trends across chunk sizes.
  • C. Effect of chunk size on SylReg: SP and SNMI peak at a shorter chunk size of 20, roughly corresponding to the 90th percentile.The passage states that C = 1 is frame-wise regression and C = T is global average pooling, with T bounded by 250 frames.
  • D. Layer-wise analysis of segmentation performances: Recall peaks at the 8th Transformer layer and decreases at the 9th, although the 9th layer yields the highest precision.The study therefore uses the 8th layer, noting that 9th-layer precision can improve during subsequent self-segmentation distillation.

E. Learning dynamics of SylReg · F. Speech language modeling

SylReg’s learning dynamics favor 10k training steps, before boundary blurring and representation collapse emerge. Speech-language-model evaluation covers lexical, syntactic, and semantic understanding, generation quality, repetition, and compute, with speech-only SylReg-LM surpassing SyllableLM on selected high-level tasks.

  • E. Learning dynamics of SylReg: 10k training steps maximize recall on LibriSpeech development data, while overall F1 begins declining after 12k steps.Adjacent-syllable boundaries blur and the similarity matrix becomes more uniform as training progresses.
  • E. Learning dynamics of SylReg: 10k training steps are therefore chosen to help prevent representation collapse.This choice follows the observed decline in F1 and increasingly uniform similarity matrix.
  • F. Speech language modeling: Speech-language-model understanding is evaluated with sWUGGY, sBLIMP, and StoryCloze for lexical, syntactic, and semantic capabilities, respectively.Each contrastive metric tests whether linguistically correct speech receives higher likelihood than a minimally incorrect counterpart.
  • F. Speech language modeling: Spoken StoryCloze requires selecting a 10-second continuation given a 3-second prefix from LibriSpeech test-clean.The supplied passage introduces this generation setup while describing the speech-language-model evaluation protocol.
  • F. Speech language modeling: Generation uses a softmax temperature of 0.8, Whisper-large-v3 transcription, and OLMo 2 1B transcript perplexity evaluation.OLMo 2 1B is independent of all language models compared in Table 4.
  • F. Speech language modeling: Generation repetition is quantified with auto-BLEU using 2-gram evaluation.The metric is applied to generated speech outputs.
  • F. Speech language modeling: Estimated training compute is reported as 6ND, where N is the number of non-embedding parameters and D is the number of processed tokens.This compute estimate follows the stated FLOPs formulation.
  • F. Speech language modeling: Speech-only SylReg-LM surpasses a randomly initialized SyllableLM on the high-level linguistic tasks sBLIMP and tSC under the same computational budget.Both models are trained using the speech-only Libri-Light corpus and the SyllableLM Base architecture.

G. Token-to-speech resynthesis · H. Impact of the merge threshold on downstream tasks · VII. Conclusion

The paper evaluates syllabic-token resynthesis for content preservation, perceptual quality, and coding efficiency, and examines how merge-threshold choices affect downstream performance. It concludes that SylReg improves linguistic purity and syllabic-token quality while identifying expressive speech generation and continuous representations as future directions.

  • G. Token-to-speech resynthesis: Resynthesized speech is evaluated for content fidelity using WER and CER, perceptual quality using UTMOS, and coding efficiency using bitrate.Evaluation uses 4–10 second LibriSpeech test-clean utterances and Whisper-large-v3 transcripts.
  • H. Impact of the merge threshold on downstream tasks: Lowering τSylReg to 0.5 reduces the frame rate to 4.99 Hz by further merging adjacent segments.Coarser tokens align better with ground-truth boundaries and improve the R-value, which penalizes over-segmentation.
  • H. Impact of the merge threshold on downstream tasks: Coarser tokens degrade downstream performance, especially on sWUGGY, CER, and WER, which require fine-grained lexical discrimination.The low perplexity likely reflects reward hacking through word repetition, evidenced by high auto-BL.
  • VII. Conclusion: SylReg prioritizes linguistic content over speaker characteristics and achieves state-of-the-art syllable segmentation accuracy and syllabic-token quality.Its coarse syllabic tokens address the granularity mismatch between speech and text without modifying the downstream language-model architecture.
  • VII. Conclusion: SylReg-LM outperforms the phone-level token-based SpiRit-LM in syntactic and semantic understanding.The supplied conclusion passage reports the comparison but does not provide the improvement value.
  • VII. Conclusion: Future work will integrate acoustic features for expressive speech generation, while removing quantization may enable continuous syllabic representations and improve speech synthesis quality.The current work focuses on intelligibility through improved linguistic purity of syllabic tokens.
Loading 2607.04064v1…