Source-linked AI summary
Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenization
Luca Della Libera, Cem Subakan, Mirco Ravanelli
TL;DR
Fixed-frame-rate speech codecs can produce unnecessarily long sequences because they allocate tokens uniformly in time. DyCAST uses character-aligned variable-rate tokenization, explicit duration modeling, and retrieval-augmented decoding, achieving competitive quality and downstream performance with significantly fewer tokens.
Problem
Existing speech tokenizers typically use fixed frame rates, allocating tokens uniformly and producing unnecessarily long sequences.
Method
DyCAST combines soft character-level alignment, explicit duration modeling, variable-rate tokenization, and retrieval-augmented decoding.
Results
DyCAST maintains competitive speech resynthesis quality and downstream performance while producing substantially shorter token sequences than fixed-frame-rate codecs.
Takeaways & Limitations
DyCAST offers content-adaptive frame rates and explicit control over duration, bitrate, and reconstruction quality for speech tokenization.
Abstract
from arXiv · showhide
Neural audio codecs are at the core of modern conversational speech technologies, converting continuous speech into sequences of discrete tokens that can be processed by LLMs. However, existing codecs typically operate at fixed frame rates, allocating tokens uniformly in time and producing unnecessarily long sequences. In this work, we introduce DyCAST, a Dynamic Character-Aligned Speech Tokenizer that enables variable-frame-rate tokenization through soft character-level alignment and explicit duration modeling. DyCAST learns to associate tokens with character-level linguistic units during training and supports alignment-free inference with direct control over token durations at decoding time. To improve speech resynthesis quality at low frame rates, we further introduce a retrieval-augmented decoding mechanism that enhances reconstruction fidelity without increasing bitrate. Experiments show that DyCAST achieves competitive speech resynthesis quality and downstream performance while using significantly fewer tokens than fixed-frame-rate codecs. Code and checkpoints will be released publicly at https://github.com/lucadellalib/dycast.
1 Introduction
Existing speech codecs typically tokenize at fixed frame rates, producing uniformly sampled sequences that can be unnecessarily long. DyCAST introduces character-aligned, variable-rate tokenization and retrieval-augmented decoding to reduce tokens while preserving speech quality and downstream performance.
- Fixed-frame-rate speech codecs allocate tokens at a constant temporal resolution, producing sequences that may be unnecessarily long.
- DyCAST dynamically groups compressed frame representations into variable-length chunks that are pooled and quantized into discrete tokens.
- Character-level boundaries from a frozen aligner supervise DyCAST during training, associating tokenization with linguistic units.
- Retrieval-augmented decoding improves reconstruction quality at low frame rates by using side information without increasing bitrate.
- DyCAST achieves competitive resynthesis and downstream performance while using significantly fewer tokens than fixed-frame-rate baselines.
2 Related Work
Prior speech tokenizers largely use fixed temporal resolutions, while recent work explores adaptive or text-aligned representations. DyCAST differs by targeting a standalone variable-rate tokenizer that supports text-free inference.
- Fixed-Frame-Rate Codecs: Fixed-frame-rate codecs discretize speech at a constant temporal resolution, while newer single-codebook designs jointly encode semantic and acoustic information.
- DyCAST departs from these approaches by providing a standalone speech tokenizer with variable-rate tokenization and text-free inference.
- Variable-Frame-Rate Codecs: Variable-frame-rate approaches dynamically adjust tokenization according to content complexity rather than fixing boundaries in advance.
- Text-Aligned Speech Representations: Text-aligned speech methods connect discrete speech tokens with text through alignment mechanisms or text-conditioned generation.
- Text-Aligned Speech Representations: Many text-aligned methods require ground-truth alignments or textual input at inference, limiting their use in text-free generative settings.
3 DyCAST
DyCAST combines dynamic chunking, character-level supervision, duration-based dechunking, and retrieval-augmented decoding. These components enable variable-rate tokenization with controllable reconstruction timing and improved low-rate fidelity.
- Dynamic Chunking: A compressor projects frozen self-supervised encoder features into a lower-dimensional latent space before dynamic chunking and quantization.
- Dynamic Chunking: The chunker uses a boundary predictor and downsampler, with a frozen character aligner providing training supervision for chunk boundaries.
- Dynamic Chunking: At inference, hazard predictions produce boundaries without textual input, while min gap and max gap regulate the resulting frame rate.
- Dechunking: The duration predictor estimates frames per token, and the upsampler expands token-level representations back to frame-level features for reconstruction.
- Dechunking: A normalized length penalty encourages globally consistent pacing while remaining invariant to utterance length.
- Inference: Inference supports token durations, utterance-length conditioning, and direct control over effective frame rate and duration.
- Retrieval-Augmented Decoding: Retrieval-augmented decoding refines discrete latents by similarity search over continuous speech latents before waveform reconstruction.
4 Experimental Setup
The experiments use pretrained speech components, modular predictors, scalar spherical quantization, and multistage training. Inference exposes multiple boundary and duration-control modes for evaluating DyCAST across speech tasks.
- Architecture: Experiments use WavLM-large encoder representations and Vocos waveform reconstruction within a modular codec architecture.
- Predictors: The boundary predictor operates on high-dimensional WavLM features, while the duration predictor uses pooled, quantized low-dimensional latents.
- Quantization: Scalar spherical quantization allows flexible bitrate selection while preserving spherical geometry and factorized codebook utilization.
- Training: DyCAST is trained on 16 kHz LibriTTS using a multistage procedure covering reconstruction, boundary prediction, and duration modeling.
- Inference: Inference can use character-aligner boundaries, a boundary predictor, transmitted durations, or utterance length, enabling frame-rate control.
5 Downstream Evaluation
DyCAST is evaluated across speech resynthesis, voice conversion, retrieval-augmented decoding, discriminative probing, and text-to-speech. It maintains competitive quality and downstream performance at substantially lower frame rates than fixed-frame-rate codecs.
- Speech resynthesis and voice conversion: On speech resynthesis, DyCAST maintains competitive naturalness, intelligibility, and speaker similarity while using 3–8x fewer frames than strong fixed-rate codecs on LibriSpeech.DyCAST variants closely match codecs such as FocalCodec and Stable Codec in reported resynthesis metrics.
- Speech resynthesis and voice conversion: DyCAST retains speaker identity across frame-rate variants and achieves voice-conversion speaker similarity and intelligibility comparable to FocalCodec, with graceful degradation at lower rates.In noisy conditions, aggressive compression increases dWER, while naturalness and speaker similarity remain stable.
- Speech resynthesis and voice conversion: Retrieval-augmented decoding reduces dWER and improves speaker similarity at moderate similarity thresholds, especially for lower-rate variants, while preserving naturalness.The strongest consistent gains occur at τ ∈[95, 97]; τ = 99 closely matches the no-retrieval baseline.
- Discriminative tasks: On discriminative probing, DyCAST-CA achieves the best ASR WER among evaluated codecs while using significantly fewer tokens, and predicted-boundary variants degrade gracefully as rates decrease.The evaluation covers ASR, speaker identification, and speech emotion recognition.
- Text-to-speech: DyCAST achieves strong text-to-speech performance at reduced frame rates, while DyCAST-CA performs best across metrics through a non-autoregressive one-to-one character-to-token architecture.Lower frame rates shorten autoregressive sequences; DyCAST-CA uniquely enables direct character-to-token mapping.
6 Conclusions
DyCAST enables variable-frame-rate speech tokenization with shorter sequences while maintaining competitive resynthesis and downstream performance. Its flexibility includes content-adaptive rates, duration and bitrate control, and retrieval-augmented reconstruction.
- DyCAST uses soft character-level alignment and explicit duration modeling for variable-frame-rate speech tokenization.
- DyCAST produces substantially shorter token sequences than fixed-frame-rate codecs while maintaining competitive speech resynthesis quality and downstream performance.
- DyCAST enables content-adaptive frame rates and explicit control over duration, bitrate, and reconstruction quality.
- Retrieval-augmented decoding improves speech resynthesis without increasing bitrate.
A Datasets
The appendix describes datasets spanning English, multilingual, noisy, separated, speaker-focused, emotion, and codec-baseline contexts. It also lists diverse fixed-frame-rate codecs used for comparison.
- Datasets: LibriSpeech is an approximately 1000-hour English read-speech corpus derived from LibriVox audiobooks.
- Datasets: LibriTTS provides 585 hours of transcribed speech for text-to-speech research.
- Datasets: MLS extends LibriSpeech to multiple languages and provides approximately 44,500 hours of transcribed English speech plus about 6000 hours in other languages.
- Datasets: VoiceBank contains noisy speech from 28 training speakers and 2 unseen test speakers for speech enhancement.
- Datasets: LibriMix provides mixtures of two or three speakers at different signal-to-noise ratios for speech separation and enhancement.
- Datasets: VCTK contains English recordings from 110 speakers with varied accents and supports speaker adaptation, text-to-speech, and voice conversion evaluation.
- Datasets: IEMOCAP contains scripted and improvised dialogues from 10 actors with audio, video, transcriptions, and emotion labels.
- Baseline codecs: The baseline set covers general-audio, speech-focused, single- and multi-codebook, acoustic, semantic, and hybrid fixed-frame-rate codecs.
C Hyperparameters and Training Details
DyCAST components use frozen pretrained aligner and encoder modules alongside trainable compression, chunking, quantization, dechunking, decompression, and decoding components. The appendix specifies their architectures, dimensions, and training settings.
- Training: All learnable modules are trained on 585 hours of LibriTTS speech with shared AdamW settings except the decoder.
- Character aligner: The frozen MMS character aligner supplies character-level durations through argmax decoding.
- Encoder: The frozen WavLM-large2 encoder extracts semantically rich, fine-grained acoustic representations from its sixth transformer layer.
- Compressor: The compressor processes 1024-dimensional WavLM features through three focal downscaling blocks and projects them to 32-dimensional latents.
- Chunker: The chunker predicts boundaries with a binary classification head, while dynamic downsampling uses parameter-free frame selection.
- Quantizer: The scalar spherical quantizer uses four levels per latent dimension, yielding an implicit codebook of size |C| = 432.
- Dechunker: The dechunker predicts token durations and performs parameter-free dynamic upsampling by frame repetition.
- Decoder: The decompressor mirrors the compressor with focal upscaling blocks, while the Vocos decoder reconstructs waveform output from 1024-dimensional WavLM features.
D Downstream Evaluation
DyCAST is evaluated across speech resynthesis, voice conversion, probing tasks, and text-to-speech. The setups measure reconstruction quality, speaker and linguistic information, emotion, and speech generation from text.
- Speech resynthesis: Speech resynthesis evaluates naturalness, speaker identity, and intelligibility on English, multilingual, noisy, and mixed-speech data.
- Voice conversion: Voice conversion tests whether single-codebook representations disentangle speaker identity from linguistic content using source-to-target conversion.
- Voice conversion: The voice-conversion evaluation contains 2,521 samples and uses UTMOS, dWER, and Sim.
- Probing: ASR probing uses a shallow bidirectional LSTM with a CTC head to assess linguistic content in fixed codec tokens.
- Probing: Speaker-identity probing groups LibriSpeech utterances by speaker and evaluates acoustic information in fixed codec tokens.
- Probing: Emotion probing evaluates four IEMOCAP classes using a shallow classifier trained on frozen codec tokens.
- Text-to-speech: Text-to-speech evaluation uses LibriSpeech and removes approximately 4% of test-clean utterances longer than 20 seconds to reduce train-test mismatch.
- Text-to-speech: Autoregressive TTS conditions next-token prediction on speaker, character, and speech-token embeddings using a Llama 3 decoder.
E.1 Decoding Mode
Explicit duration information improves intelligibility in DyCAST resynthesis, while naturalness and speaker similarity remain largely stable across decoding modes.
- E.1 Decoding Mode: Explicit durations consistently yield the lowest dWER across DyCAST variants, indicating more accurate temporal reconstruction and fewer linguistic distortions.The benefit is especially pronounced for predicted-duration variants and at low token rates.
- E.1 Decoding Mode: Tokens-only decoding preserves comparable or slightly higher UTMOS, but degrades intelligibility through temporal misalignment and increased recognition errors.The degradation is especially evident under aggressive compression.
- E.1 Decoding Mode: Tokens plus utterance length recovers most intelligibility gains while offering a favorable trade-off between timing information and decoding complexity.
E.2 Qualitative Analysis of DyCAST Chunk Boundaries
DyCAST supports four chunk-boundary variants with different average frame rates. Lower frame rates produce fewer, more widely spaced boundaries and therefore longer token durations.
- E.2 Qualitative Analysis of DyCAST Chunk Boundaries: The four variants are DyCAST-CA, DyCAST-BP1, DyCAST-BP3, and DyCAST-BP5, using character-aligner or boundary-predictor boundaries.The predictor variants use min gap values of 1, 3, and 5, respectively.
- E.2 Qualitative Analysis of DyCAST Chunk Boundaries: As frame rate decreases, chunk boundaries become less frequent and more widely spaced, reflecting longer token durations and coarser temporal resolution.
- E.2 Qualitative Analysis of DyCAST Chunk Boundaries: Average frame rates are approximately 14 Hz, 17 Hz, 9 Hz, and 6 Hz from left to right.