PaperlayerView exact revision on arXiv

Source-linked AI summary

Speaker-Disentangled Chunk-Wise Regression for Syllabic Tokenization

Ryota Komatsu, Kota Kawakita, Takuma Okamoto, Takahiro Shinozaki

arXiv:2607.04064v1cs.CLcs.AIcs.SDeess.AS

TL;DR

Speaker-identity bias contaminates syllabic-token purity. The paper proposes SylReg, which learns speaker-disentangled syllabic representations through chunk-wise regression; it reports state-of-the-art syllable boundary detection and clustering, while SylReg-LM achieves a 7% average relative improvement over phone-level SpiRit-LM in syntactic and semantic understanding.

  • Problem

    SD-HuBERT is biased toward speaker identity, contaminating syllabic-token purity.

    Source: §I · p. 2

  • Method

    SylReg learns speaker-disentangled syllabic representations with chunk-wise regression that enforces coherence over mid-level temporal chunks.

    Source: §I · p. 2

  • Results

    SylReg outperforms state-of-the-art methods, in syllable boundary detection and syllabic segment clustering, while SylReg-LM achieves 7% average relative improvement over SpiRit-LM in syntactic and semantic understanding.

    Source: §I · p. 2

  • Takeaways & Limitations

    The paper concludes that coarse syllabic tokens alleviate speech-text granularity mismatch without modifying the downstream LM architecture.

    Source: §VII · p. 9

Abstract

from arXiv · show

Unsupervised syllabic tokenization aims to learn discrete syllabic tokens that capture latent linguistic content-related structure from raw speech. Recent syllabic tokenization methods employ teacher-student distillation of the pretrained HuBERT to organize latent speech frame representations into syllabic segments. However, when trained with an utterance-level cross-entropy objective, the model predicts speaker identity rather than linguistic content, thereby compromising the purity of syllabic tokens. To address this problem, we propose a speaker-disentangled syllabic tokenizer that regresses speaker-perturbed student representations toward clean teacher targets within fixed-length chunks. Experimental results demonstrate that our proposed method achieves state-of-the-art performance in syllable boundary detection and syllabic segment clustering. Moreover, a speech language model trained on our syllabic tokens achieves a 7% relative improvement in syntactic and semantic understanding over the phone-level SpiRit-LM.

I. INTRODUCTION

The paper introduces SylReg, a speaker-disentangled syllabic tokenizer trained with chunk-wise regression, and reports state-of-the-art syllable tokenization results. It also presents a syllable-text speech LM with improved syntactic and semantic understanding and a synthesizer with lower token bitrate.

  • Speech tokens occur at 12.5–50 Hz versus coarser text subwords, motivating syllabic tokens; SD-HuBERT uses self-distillation to organize frame representations into syllabic segments,,.Prior speech LMs built on syllabic tokens have outperformed phone-token LMs in syntactic understanding,,.Source: §I · p. 1
  • SD-HuBERT exhibits valid prototype collapse and speaker-correlated categories, contaminating syllabic-token purity with speaker identity.The authors attribute the speaker-dominating problem partly to SD-HuBERT's utterance-level objective, since speaker characteristics tend to remain stationary over an utterance.Source: §I · pp. 1–2
  • SylReg extends frame-wise speaker-disentangled regression with coherence over mid-level temporal chunks to promote syllabic grouping while retaining speaker resilience.Source: §I · p. 2
  • SylReg outperforms state-of-the-art methods, in syllable boundary detection and syllabic segment clustering.Source: §I · p. 2
  • 7% average relative improvement in syntactic and semantic understanding over the phone-level-interleaved SpiRit-LM is achieved by SylReg-LM.Source: §I · p. 2
  • At a 2.3× lower token bitrate, the token-to-speech synthesizer matches TWIST in character and word error rates.Source: §I · p. 2

II. Related Work … III. Self-Supervised Syllabic Tokenization

Phonetic tokens align strongly with linguistic content but marginalize fine acoustic detail, while acoustic tokens preserve waveform information but align weakly with text,,,,. This work advances syllabic tokenization with chunk-wise regression for more accurate English segmentation and contrasts its interleaved syllable-text LMs with ZeroSyl’s syllable-only approach.

A. Trade-off between phonetic and acoustic tokens

  • Phonetic tokens from self-supervised encoders, or ASR models align strongly with linguistic content and support intelligible speech generation, but marginalize fine acoustic detail.Acoustic tokens from waveform-reconstructing neural codecs can represent general audio, yet weak text alignment hinders lexical, syntactic, and semantic understanding,.Source: A. Trade-off between phonetic and… · p. 2

B. Learning coarse phonetic tokens for speech LMs

  • Training-free subword tokenization and deduplication of phonetic tokens, are simple but retain relatively high frame rates of around 20 Hz.The passage introduces the 20 Hz figure in the continuation of the preceding sentence.Source: B. Learning coarse phonetic tokens for… · pp. 2–3

III. Self-Supervised Syllabic Tokenization

  • Chunk-wise regression advances syllabic tokenization toward more accurate English segmentation, while interleaved syllable-text LMs use text knowledge transfer to enhance semantic modeling.This framework moves temporal aggregation into tokenization so LMs can remain architecturally unchanged.Source: B. Learning coarse phonetic tokens for… · p. 3

B. Learning coarse phonetic tokens for speech LMs

  • ZeroSyl directly uses an off-the-shelf speech encoder without distillation and focuses on LMs trained exclusively on syllabic tokens.This work instead uses interleaved syllable-text LMs for semantic modeling through knowledge transfer from text.Source: B. Learning coarse phonetic tokens for… · p. 3
  • Syllabic tokenizers aggregate temporally during tokenization, allowing speech LMs to remain architecturally unchanged,.This contrasts with speech-adapter approaches that aggregate frame-level tokens in the LM front-end.Source: B. Learning coarse phonetic tokens for… · p. 3
  • Speech adapters aggregate frame-level tokens in the LM front-end, [27], but can add training complexity through curriculum learning or risk model collapse [27].Source: B. Learning coarse phonetic tokens for… · p. 3
  • Sylber 2.0 adopts speaker-disentangled frame-wise regression and extends it to multilingual acoustic features for language-universal, expressive speech synthesis [28].The present work advances this framework with chunk-wise regression for more accurate English syllabic segmentation.Source: B. Learning coarse phonetic tokens for… · p. 3

A. Baseline method: SD-HuBERT · B. Speaker-dominating problem in SD-HuBERT

SD-HuBERT uses DINO self-distillation to induce syllabic organization in speech representations, but its predicted categories carry strong speaker information and tend to discriminate speaker identity rather than linguistic content.

A. Baseline method: SD-HuBERT

  • SD-HuBERT finetunes pretrained HuBERT with DINO, prepending a learnable [CLS] token whose frame-sequence information is aggregated by self-attention into an utterance-level embedding.Source: A. Baseline method: SD-HuBERT · p. 3
  • A classification head compares the utterance embedding with learnable prototypes to predict a pseudo-category from the resulting categorical distribution.Source: A. Baseline method: SD-HuBERT · p. 3
  • The student and its moving-average teacher minimize utterance-level cross-entropy, while syllabic organization emerges in student frame representations from Transformer layer 9.Source: A. Baseline method: SD-HuBERT · p. 3
  • A self-similarity matrix of layer-9 representations shows block structures with boundaries roughly matching ground truth, and minimum cut extracts syllabic segments [9],.The minimum-cut algorithm is modified for computational efficiency.Source: A. Baseline method: SD-HuBERT · p. 3

B. Speaker-dominating problem in SD-HuBERT

  • Speaker-normalized mutual information reached 0.61 on the LibriSpeech test set, indicating that SD-HuBERT’s predicted categories discriminate speaker identity rather than linguistic content.The measure quantifies the relative reduction in uncertainty about speaker identity after observing a predicted category.Source: B. Speaker-dominating problem in… · p. 3
  • This speaker dominance is consistent with Niekerk et al.: speaker characteristics remain relatively stationary over an utterance, and self-attended [CLS] can act as a weighted global average of frame representations.Source: B. Speaker-dominating problem in… · p. 3

C. SylReg: Speaker-disentangled chunk-wise regression

SylReg emphasizes linguistic content by matching representations of original and speaker-perturbed speech within fixed-length chunks. Its student regresses toward a teacher target, while constrained pitch and formant perturbations aim to disentangle speaker characteristics without producing unnatural speech.

C. SylReg: Speaker-disentangled chunk-wise regression

  • SylReg matches speaker-invariant representations of original and speaker-perturbed speech within fixed-length chunks to emphasize linguistic content.Its two-branch objective follows a BYOL-style framework.Source: C. SylReg: Speaker-disentangled… · p. 3
  • The student and teacher share a backbone initialized from HuBERT Base, with the last three Transformer layers pruned to form a nine-layer Transformer,.The pruning follows prior work, because the final pretrained layers have been shown to hinder learning linguistically coarse syllabic representations.Source: C. SylReg: Speaker-disentangled… · p. 3
  • Unlike BYOL’s global regression, SylReg averages final-layer hidden states in chunks of size C, encouraging mid-level temporal grouping that empirically aligns with syllables.It minimizes MSE between ℓ2-normalized teacher and student outputs; chunking reduces sequence length by a factor of C.Source: C. SylReg: Speaker-disentangled… · p. 3
  • Training stops gradients on teacher outputs, freezes both CNN encoders, and updates the remaining teacher parameters by exponential moving average of the student.Only the student is updated by backpropagation.Source: C. SylReg: Speaker-disentangled… · p. 3
  • SylReg adapts speaker perturbation from by restricting formant and pitch shifts to male-to-female or female-to-male conversions to avoid unnatural speech.Utterance-average pitch estimates gender: above a threshold triggers female-to-male conversion; otherwise male-to-female conversion, with perturbed speech fed to the student and original speech to the teacher.Source: C. SylReg: Speaker-disentangled… · p. 4

D. Self-segmentation distillation

SylReg-Distill transfers SylReg’s emergent syllabic structure into data2vec 2.0 through staged pseudo-syllable regression and iterative self-refinement.

D. Self-segmentation distillation

  • SylReg’s 8th-layer block-diagonal syllabic structure is distilled into data2vec 2.0 as SylReg-Distill using self-segmentation distillation (SylBoost),.Data2vec 2.0 is chosen because its representations enable better syllabic segment clustering than HuBERT.Source: D. Self-segmentation distillation · p. 4
  • SylReg pseudo-syllable boundaries define segments whose averaged data2vec 2.0 teacher representations serve as regression targets for the student, with speaker perturbation applied to student inputs.A new teacher-student pair is initialized from data2vec 2.0 Base, and its 12th Transformer layer is pruned.Source: D. Self-segmentation distillation · p. 4
  • After each stage, the teacher copies the student parameters and is used in subsequent stages to extract and segment targets, iteratively refining syllabic structure.Source: D. Self-segmentation distillation · p. 4

IV. Generative Spoken Language Modeling … C. Token-to-speech synthesis

The GSLM pipeline combines a speech tokenizer, a speech language model, and a token-to-speech synthesizer; this block describes syllable segmentation, interleaved syllable-text training, and synthesis.

IV. Generative Spoken Language Modeling

  • The GSLM pipeline comprises a speech tokenizer, a speech language model, and a token-to-speech synthesizer.Source: §IV · p. 4

A. Syllable segmentation

  • The tokenizer detects syllable boundaries with minimum-cut segmentation, averages features within segments, and assigns tokens through two-step clustering.Source: A. Syllable segmentation · p. 4
  • The segmentation algorithm sets its segment count from frame length and an upper-bound token frame rate, then merges adjacent segments when cosine similarity exceeds threshold τ.It uses an O(M)-complexity dynamic-programming loop and applies K-Means followed by agglomerative clustering.Source: A. Syllable segmentation · p. 4

B. Interleaved syllable-text language modeling

C. Token-to-speech synthesis

  • A CFM-based Diffusion Transformer with a BigVGAN-v2 vocoder generates acoustic features from speech tokens in fewer than ten non-autoregressive denoising steps –.Source: C. Token-to-speech synthesis · p. 5
  • A jointly trained length regulator repeats syllable embeddings according to predicted durations before DiT layers, with input embeddings initialized from and frozen at pretrained clustering centroids.Source: C. Token-to-speech synthesis · p. 5

V. Experimental Setup · A. Datasets · B. Implementation details

The setup covers syllabic tokenization, speech-language-model training, and token-to-speech synthesis, using large speech corpora and distinct configurations for each stage. It specifies the datasets, model settings, and training procedures used in these experiments.

A. Datasets

  • SylReg trains on 55k hours of Libri-Light before syllable-segment distillation on LibriSpeech train-clean-100; quantizers train on LibriSpeech train and are evaluated on test using syllable alignments.For language modeling, the study uses 129k hours of English speech, including LibriSpeech, Libriheavy, Emilia-Large, People’s Speech, VoxPopuli, and synthetic speech from TinyStories and Cosmopedia v2. The token-to-speech synthesizer is pretrained on LibriTTS-R and finetuned on the female subset of Hi-Fi-CAPTAIN.Source: A. Datasets · p. 5
  • Language-model data retain Emilia-Large utterances with MOS above 3.45 and duration over 10 seconds, filter transcription errors and codeswitching, and mix Cosmopedia v2 to mitigate text forgetting.Synthetic speech uses Kokoro TTS with the af_heart voice.Source: A. Datasets · p. 5

B. Implementation details

  • SylReg uses 100-frame chunks and two-layer projector and predictor MLPs with 2048-unit intermediate layers, batch normalization, GELU, and 256-dimensional outputs.Training runs for 10k AdamW steps with batch size 1024; only the randomly initialized projector and predictor are updated during the first 2k steps. Five self-segmentation-distillation stages follow chunk-wise regression, and male/female speaker perturbation uses specified formant and pitch conversions classified at a 155 Hz mean-pitch threshold.Source: B. Implementation details · p. 5
  • Tokenizer segments span 3–35 frames; SylReg uses layer 8 and threshold 0.7, while SylReg-Distill uses layer 11, a 6.67 Hz frame-rate bound, and merge threshold 0.95.SylReg’s threshold targets an approximately 6.25 Hz token frame rate. SylReg-Distill uses 24,576 K-Means centroids and 8,192 agglomerative clusters; comparison settings use 16,384 K-Means centroids and 4,096 agglomerative clusters.Source: B. Implementation details · p. 5
  • The speech-language-model experiments continually train Qwen2.5 7B for 15k steps and also train 85M models for 50k steps, with speech-only, text-only, and interleaved examples in an approximately 3:3:4 ratio.Interleaving samples syllabic tokens with probability approximately 0.3; training first updates only syllabic token embeddings for 5k steps, and the Qwen training takes 39 hours on 32 H100 GPUs.Source: B. Implementation details · p. 5
  • The token-to-speech synthesizer uses standardized 80-bin log-mel spectrograms and a DiT with a two-layer encoder and four-layer decoder, trained for 200k pretraining and 50 finetuning steps.DiT training drops full syllabic-token sequences with probability 0.2 for classifier-free guidance; inference uses Euler step size 0.1 and guidance strength 0.7.Source: B. Implementation details · pp. 5–6
  • BigVGAN-base synthesizes 16 kHz waveforms with modified upsampling kernels and strides, training for 1M steps on two A6000 GPUs.Training uses gradient clipping of 100 and batches of 20-second speech segments.Source: B. Implementation details · p. 6

VI. Evaluation · A. Syllable segmentation and token quality

The evaluation measures syllable-boundary detection, token quality, and sensitivity to speaking variation. SylReg and its distilled version perform strongly on segmentation and token-purity comparisons.

A. Syllable segmentation and token quality

B. Ablation study · C. Effect of chunk size on SylReg · D. Layer-wise analysis of segmentation performances

The ablations show that SylReg benefits from its speaker-perturbation design, while chunk-wise regression at an intermediate temporal scale balances segmentation and token purity. Layer-wise analysis selects the 8th Transformer layer for syllabic tokenization.

B. Ablation study

  • DINO with cross-entropy uniformly degrades across metrics, and only 32% of its final-softmax categories activate on LibriSpeech test data.The authors suggest category collapse may degrade syllabic representations because identical category signals are backpropagated to linguistically diverse utterances.Source: B. Ablation study · p. 6
  • Removing speaker perturbation from DINO increases TED and reduces token purity, while DINO alone lowers TED from 16.6 to 13.9.The passage attributes reduced purity to syllables with speaking variations being assigned to different clusters.Source: B. Ablation study · p. 7
  • Replacing SylReg’s speaker perturbation with random perturbation consistently degrades performance, supporting the efficacy of the proposed perturbation design.Source: B. Ablation study · p. 7

C. Effect of chunk size on SylReg

  • Segmentation improves with larger chunks and saturates at size 100, whereas SP and SNMI peak at size 20, roughly the 90th percentile of LibriSpeech syllable durations.Table 3 reports the chunk-size effects; the passage concludes that an intermediate temporal scale balances segmentation accuracy and token purity.Source: C. Effect of chunk size on SylReg · p. 7, Table 3 · p. 7
  • Both frame-wise regression (C = 1) and global average pooling (C = T, capped at 250 frames) yield suboptimal token purity; [CLS]-based aggregation also performs poorly relative to global average pooling.Source: C. Effect of chunk size on SylReg · p. 7

D. Layer-wise analysis of segmentation performances

E. Learning dynamics of SylReg · F. Speech language modeling

SylReg’s recall peaks at 10k training steps, while SylReg-LM 7B achieves a 7% average relative improvement over SpiRit-LM on sBLIMP and StoryCloze. The speech-LM experiments also show differences across tokenization, training, and compute settings.

E. Learning dynamics of SylReg

  • Recall peaks at 10k steps, and overall F1 begins declining after 12k, motivating 10k steps to prevent representation collapse.After further training, adjacent syllable boundaries blur and the similarity matrix becomes more uniform.Source: E. Learning dynamics of SylReg · p. 7

F. Speech language modeling

  • The study evaluates lexical, syntactic, and semantic understanding with sWUGGY, sBLIMP, and StoryCloze, respectively,.The contrastive metrics compare likelihoods of correct versus minimally incorrect samples; sSC tests story-ending selection and commonsense reasoning, while tSC tests topical coherence. Log-likelihoods are normalized by sequence length.Source: F. Speech language modeling · p. 7
  • For generation, the models produce 10-second continuations from 3-second LibriSpeech test-clean prefixes at temperature 0.8, with transcripts scored using OLMo 2 1B,.Whisper-large-v3 transcribes the generated speech, and auto-BLEU uses 2-grams to quantify repetition; compute is estimated as 6ND, where N is non-embedding parameters and D is processed tokens [62].Source: F. Speech language modeling · p. 8
  • 7% average relative improvement over SpiRit-LM on sBLIMP and StoryCloze is achieved by SylReg-LM 7B when scaling model and data sizes.Despite a 1-point drop on sBLIMP versus SylReg-LM 85M, the authors report efficacy in capturing high-level linguistic abstractions; the drop may relate to synthetic speech in training.Source: F. Speech language modeling · p. 8, Table 4 · p. 8
  • Speech-only SylReg-LM surpasses SyllableLM on sBLIMP and tSC but underperforms on sWUGGY, while C = 1 improves sWUGGY but worsens high-level metrics.The authors suggest tokenization improvements may not help distinguish subtle phone-level differences between words and nonwords; C = 1 uses finer-grained tokens.Source: F. Speech language modeling · p. 8
  • Speech-text interleaving with OPT 125M initialization boosts SylReg-LM’s StoryCloze scores, reflecting knowledge-transfer benefits for semantic tasks [63].Source: F. Speech language modeling · p. 8
  • SylReg-LM 85M outperforms SpiRit-LM 7B in continuation quality while using 0.2% of its compute.The authors attribute some of this efficiency to the low frame rate of syllabic tokens.Source: F. Speech language modeling · p. 8

G. Token-to-speech resynthesis · H. Impact of the merge threshold on downstream tasks · VII. Conclusion

The sections report resynthesis tradeoffs and merge-threshold effects, then conclude that speaker-disentangled syllabic tokens support content-focused speech modeling while identifying future synthesis and distillation directions.

G. Token-to-speech resynthesis

  • SylReg achieves the highest UTMOS and matches TWIST in CER and WER at 2.3× lower bitrate, while Sylber attains the lowest error rates with unbounded bitrate.Evaluation uses 4–10 second LibriSpeech test-clean utterances; SylReg can also resynthesize clean speech from recordings with background rain noise.Source: G. Token-to-speech resynthesis · p. 8
  • Chunk size 1 impairs resynthesis intelligibility despite its higher bitrate, which the passage attributes to inaccurate syllable segmentation.Source: G. Token-to-speech resynthesis · p. 8

H. Impact of the merge threshold on downstream tasks

  • Reducing τSylReg from 0.7 to 0.5 lowers token frame rate from 5.82 Hz to 4.99 Hz (ground-truth: 4.38 Hz), improving R-value but hurting downstream performance.The lower threshold merges adjacent segments, producing coarser tokens that align better with ground-truth boundaries.Source: H. Impact of the merge threshold on… · pp. 8–9, Table 6 · p. 9
  • Coarser tokens particularly degrade sWUGGY, CER, and WER; low perplexity likely reflects reward hacking through word repetition, consistent with high auto-BLEU.These effects qualify the downstream tradeoff from lowering the merge threshold.Source: H. Impact of the merge threshold on… · p. 9

VII. Conclusion

  • SylReg prioritizes linguistic content over speaker traits, yielding coarse syllabic tokens that bridge speech-text granularity without modifying LM architecture and achieve state-of-the-art syllable segmentation and token quality.The conclusion presents this as the method’s central contribution.Source: §VII · p. 9
  • SylReg-LM outperforms phone-level SpiRit-LM in syntactic and semantic understanding, supporting syllabic tokens for modeling high-level linguistic abstractions in speech.Source: §VII · p. 9
  • Future work targets expressive speech via acoustic features, potentially better synthesis from continuous SylReg representations without quantization, and improved multi-stage distillation for large-scale training.The stated ultimate goal is intelligible and expressive spoken dialogue agents.Source: §VII · p. 9

Related papers

Loading 2607.04064v1…