Source-linked AI summary
CoSTA: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models
Yingzhuo Deng, Chong Li, Bingxin Liu, Yi Shu, Tianyu Peng, Can Xie, Siyao Song, Shuo Ren, Jiajun Zhang
TL;DR
Spoken dialogue models typically use the same speech-token space for generation and synthesis, producing long, costly autoregressive sequences, while lower-rate tokenizers may introduce quality trade-offs and costly replacement. CoSTA adapts existing models to generate compressed speech tokens and deterministically expands them before synthesis, retaining the original tokenizer and decoder. It improves expanded speech-token generation speed and system efficiency while preserving response quality, with reported transfer across architectures and complementarity to low-rate tokenization.
Problem
Spoken dialogue models typically share a speech-token space for generation and synthesis, producing long autoregressive sequences, while lower-rate tokenizers may trade compression for quality and require adapting coupled components.
Source: §1 · p. 1Method
CoSTA post-hoc extends an existing model's generation vocabulary with compressed speech-token representations and deterministically expands generated sequences before synthesis, retaining the original tokenizer and decoder.
Source: §1 · p. 2Results
Takeaways & Limitations
CoSTA supports more sustained interaction under limited budgets, while vocabulary scaling reveals a practical trade-off among compression, decoding efficiency, and response quality.
Source: §6 · p. 8Dialogue-adaptation data uses synthesized speech-to-speech examples; how synthetic and real conversational speech data complement each other remains for future work.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Spoken Dialogue Models (SDMs) typically share a speech-token space for generation and synthesis, leading to long autoregressive sequences and costly inference. Lower-rate speech tokenizers shorten sequences but may introduce compression-quality trade-offs and costly replacement without breaking this coupling. To address this limitation, we propose CoSTA, a compatibility-preserving post-hoc framework that decouples generation and synthesis token spaces while retaining the original speech tokenizer and decoder. CoSTA extends the generation token space with compressed representations via mean-initialized vocabulary extension, pronunciation adaptation, and dialogue adaptation, followed by deterministic expansion back to the original speech-token space before synthesis. On bilingual speech-dialogue benchmarks, CoSTA achieves approximately $1.7\times$ expanded speech-token generation speed, lower full-response latency, and higher throughput while preserving response quality. Analyses show more sustained interaction under limited budgets and a practical trade-off among compression, decoding efficiency, and quality. Further experiments demonstrate cross-architecture transfer and complementarity to low-rate tokenization.
1 Introduction
CoSTA decouples speech-token generation from synthesis while preserving the original tokenizer and decoder, addressing long autoregressive sequences and the costs of replacing lower-rate tokenizers. It improves generation and system efficiency while preserving response quality across evaluated settings.
1 Introduction
- Existing SDMs generate long speech-token sequences that increase autoregressive cost, complicate text-speech alignment, slow responses, and consume generation budgets faster.Lower-rate tokenizers may trade compression for quality and can require adapting the tokenizer, paired decoder, and SDM together.Source: §1 · p. 1
- CoSTA extends the generation token space with compressed representations, then deterministically expands them before synthesis while retaining the original tokenizer and decoder.Adaptation uses mean-initialized vocabulary extension, pronunciation adaptation, and dialogue adaptation.Source: §1 · p. 2
- 1.736× in Chinese and 1.659× in English relative expanded speech-token generation speed, with preserved response quality, lower full-response latency, and higher throughput.The results are reported for Step-Audio 2 mini and supported by automatic and human evaluations.Source: §1 · p. 2
- Speech-token BPE compression follows an approximate log-linear scaling regularity across tokenizer configurations, data sources, and languages.The reported scaling spans 10K–320K added units across eight configurations.Source: §1 · p. 2
- CoSTA supports more sustained interaction under limited budgets and transfers across SDM architectures, complementing low-rate speech tokenization.Vocabulary scaling also reveals a practical trade-off among compression, decoding efficiency, and response quality.Source: §1 · p. 2
2 From Speech-Token BPE Scaling to Token-Space Decoupling
BPE compresses speech-token sequences reversibly, and its compression rate decreases approximately linearly with log2 vocabulary size. This scaling motivates generating compact BPE sequences and expanding them to the original token space for synthesis.
2 From Speech-Token BPE Scaling to Token-Space Decoupling
- BPE merges frequent adjacent speech-token pairs into an extended vocabulary, and deterministic expansion recovers the original sequence losslessly.The compression rate ρ indicates how much shorter the compressed sequence is; lower ρ means fewer autoregressive steps for the same decoder sequence.Source: §2 · p. 2
- CoSTA combines vocabulary extension with mean-based initialization, progressive pronunciation adaptation, and dialogue adaptation.Source: Figure 3 · p. 3
- ρ decreases approximately linearly with log2 M as added BPE vocabulary scales from 10K to 320K; the log-linear fit has R2 ≥0.98 across eight tokenizer configurations.The trend remains consistent across data sources and languages, with R2 ≥0.994.Source: §2 · p. 3
- The scaling regularity motivates decoupling generation from synthesis: the model generates compact BPE sequences that are expanded to the original space for the decoder.The speech decoder still requires the original token space.Source: §2 · p. 3
3 Method
CoSTA adapts an existing SDM to generate compressed speech sequences in an extended token space, then deterministically restores the original decoder-facing representation. Its method combines vocabulary extension, pronunciation adaptation, dialogue adaptation, and inference-time expansion before the unchanged speech decoder.
3.1 Vocabulary Extension with Mean-based Initialization
- CoSTA extends the model vocabulary with added BPE units and initializes each unit’s embedding and LM-head row by averaging the rows of its component tokens.The original vocabulary parameters remain unchanged; mean-based initialization is described as a more stable starting point than random initialization.Source: §3.1 · pp. 3–4
3.2 Pronunciation Adaptation
- Pronunciation adaptation trains the SDM to predict targets with BPE applied only to speech tokens, using a progressive two-stage fine-tuning strategy.The first stage uses half the data to tune only embeddings and the LM head; the second uses the remaining half and additionally unfreezes intermediate layers.Source: §3.2 · p. 4
3.3 Dialogue Adaptation
- Dialogue adaptation trains on compressed speech-to-speech dialogue while updating only LLM parameters and keeping the speech encoder, adaptor, and decoder frozen.The target response’s speech tokens are compressed, and the resulting stream is mapped back to the original speech-token space before synthesis.Source: §3.3 · p. 4
3.4 Inference with Deterministic Expansion
- At inference, the extended-vocabulary speech stream is deterministically expanded to the original speech-token sequence before waveform synthesis with the unchanged decoder.Original speech tokens remain unchanged; added BPE units expand into their corresponding sequences in the original vocabulary.Source: §3.3 · p. 4, Figure 4 · p. 4
- The method’s system- and model-efficiency tables cover bilingual speech-to-speech benchmarks and AlpacaEval settings, with the latter evaluated at concurrency 1, 4, and 8.The supplied table captions identify the efficiency measures and settings but do not provide table values.Source: Table 1 · p. 5, Table 2 · p. 5
4 Experiments
Experiments evaluate CoSTA’s efficiency and quality against baselines, analyze its adaptation components and vocabulary-size trade-offs, and test interaction sustainability and transfer to another SDM architecture.
4.1 Experimental Setup
- The evaluation uses Step-Audio 2 mini and matched Vanilla-FT baselines across 12 URO-Bench datasets, with model- and system-efficiency and response-quality metrics.System measurements use AlpacaEval-zh and AlpacaEval at concurrency 1, 4, and 8; quality measures include G-Eval, UTMOS, WER/CER, and MOS-style listening ratings.Source: §4.1 · pp. 5–6
4.2 Main Results
- CoSTA reaches 1.736× relative speed in Chinese and 1.659× in English, while automatic and human evaluations report preserved response quality and improved efficiency.It also reduces full-response latency and raises throughput relative to the base model; matched Vanilla-FT comparisons show lower latency and higher throughput at every tested setting.Source: §4.2 · pp. 6–7, Figure 5 · p. 6
4.3 Ablation Study
- Ablations show dialogue adaptation has the largest effect: removing it drops G-Eval from 72.27 to 24.35, while pronunciation adaptation strengthens compression and text-speech alignment.Timbre conversion mainly benefits response quality, progressive adaptation modestly improves compressed-token generation, and mean initialization converges faster and yields more stable gradients than alternatives.Source: §4.3 · p. 7, Table 4 · p. 7, Figure 6 · p. 7
4.4 Analyses
- CoSTA sustains 7.21/5.79 effective turns in Chinese/English, versus 3.71/1.84 for Vanilla-FT, while producing longer speech under limited generation budgets.The stress test measures generation-format sustainability, not conversational coherence.Source: §4.4 · p. 7, Table 5 · p. 7
- Increasing added vocabulary from 10K to 80K improves compression and decoding speed while maintaining quality; extending to 160K yields only 1.1% speed gain as G-Eval falls from 81.36 to 78.87.The analysis identifies 80K as a practical balance in this setting.Source: §4.4 · p. 7, Figure 7 · p. 8
- On OpenS2S, CoSTA reduces speech-token steps by 21.17% in Chinese and 18.64% in English while improving response-content quality and text-speech alignment over matched Vanilla-FT.UTMOS remains comparable, and the results suggest complementarity with the native 12.5-Hz tokenizer.Source: §4.4 · p. 7, Table 6 · p. 8
5 Related Work
Prior SDMs vary in how they couple text and speech generation, but typically retain a generation space tied to the speech decoder. CoSTA instead uses post-hoc adaptation to decouple these spaces while retaining the original synthesis components.
Spoken dialogue models
- Think–Talk, parallel-generation, and interleaved-generation SDMs differ in generation architecture, yet typically couple generation tokens to the speech decoder’s expected space.CoSTA instead enables compact, losslessly expandable generation through compatibility-preserving post-hoc adaptation.Source: §5 · p. 8
Discrete speech tokens
- Lower-rate tokenizers shorten speech sequences, but aggressive reduction may trade off quality and tokenizer replacement may require jointly adapting the tokenizer, decoder, and SDM.Prior acoustic-piece and BPE methods largely use native training of small, task-specific models, whereas CoSTA adapts existing large-scale SDMs post hoc and characterizes compression scaling and vocabulary-size trade-offs.Source: §5 · p. 8
6 Conclusion
The paper identifies an approximate log-linear scaling regularity in speech-token BPE and introduces CoSTA to decouple generation and synthesis while retaining the original tokenizer and decoder. CoSTA achieves approximately 1.7× expanded speech-token generation speed, improves system efficiency while preserving quality, and supports sustained interaction, architecture transfer, and combination with low-rate tokenization.
6 Conclusion
- The paper identifies an approximate log-linear scaling regularity in speech-token BPE and introduces CoSTA to decouple generation and synthesis while retaining the original tokenizer and decoder.CoSTA achieves approximately 1.7× expanded speech-token generation speed and improves system efficiency while preserving response quality.Source: §6 · p. 8
Limitations
The evaluation focuses on general speech-to-speech dialogue and efficiency, uses synthesized speech-to-speech examples for dialogue adaptation, and adopts BPE as its deterministic, lossless compression method.
Limitations
- Dialogue adaptation uses synthesized examples, BPE is the chosen lossless segmentation method, and evaluation covers general speech-to-speech dialogue and efficiency rather than finer-grained scenarios.The authors identify real conversational data, alternative lossless segmentation strategies, emotional or personalized speech, and diverse acoustic environments as areas for further study.Source: Limitations · p. 9
A Implementation Details and Hyperparameters
CoSTA keeps the original tokenizer and decoder while adapting the generation vocabulary to use compressed speech tokens. The section describes its adaptation, inference, and training-cost setup.
A Implementation Details and Hyperparameters
- The original speech tokenizer and decoder remain unchanged, while CoSTA adds 80K BPE speech tokens to the 6,656-token vocabulary.Integer token IDs are mapped to Unicode characters for BPE training and mapped back deterministically, preserving the underlying sequence.Source: A Implementation Details and… · p. 13
A.1 Pronunciation Adaptation
- Pronunciation adaptation uses two stages: first updating the embedding and LM head, then unfreezing intermediate layers at a lower learning rate.The stages use learning rates of 1.6 × 10−4 and 2.0 × 10−5, respectively.Source: §A.1 · p. 13, Table 7 · p. 13
A.2 Dialogue Adaptation
- Dialogue adaptation compresses only speech tokens in target responses, preserving text tokens and the base model’s text–speech interleaving ratio.The adaptation data are general speech-to-speech dialogue examples.Source: §A.2 · p. 13, Table 8 · p. 13
A.3 Inference
- At inference, original speech tokens pass through unchanged and added BPE tokens are deterministically expanded before the unchanged decoder synthesizes the waveform.Expansion applies only to the generated speech-token stream.Source: §A.3 · pp. 13–14, Table 9 · p. 14
A.4 Adaptation Cost
- CoSTA-ZH adaptation takes 9:56:35 of training time, approximately 79.5 A100 GPU-hours, excluding additional data preparation.Timbre conversion and dialogue synthesis each take approximately 1–2 days on eight A100 GPUs.Source: §A.4 · p. 14
B.1 Training Data
The paper constructs bilingual pronunciation- and dialogue-adaptation data and evaluates spoken-dialogue quality across URO-Bench tasks. Detailed results show average quality trends but also variation across datasets and metrics.
B.1 Training Data
- Pronunciation adaptation uses 300 hours each of Chinese and English Emilia speech, converted to a unified timbre and tokenized with the original Step-Audio 2 mini tokenizer.Text tokens remain unchanged, and compressed speech tokens follow the base model’s text–speech interleaving pattern.Source: §B.1 · p. 14, Table 10 · p. 14
- Dialogue adaptation uses 49,664 Chinese and 49,408 English instruction examples after filtering out queries or responses longer than one minute.Only target-response speech tokens are BPE-compressed; text tokens retain the original interleaving pattern.Source: §B.1 · p. 14
B.2 Evaluation Benchmarks
- Evaluation uses 12 aligned Chinese and English URO-Bench datasets spanning understanding, reasoning, and oral conversation in speech-to-speech settings.G-Eval scores response-content quality, UTMOS estimates speech quality, and WER/CER measures speech–text alignment.Source: Table 11 · p. 14, §B.2 · p. 15
C.1 Detailed Main Results
- Across detailed dataset-level results, average quality trends vary: improvements are not uniform for every metric and dataset.Tables 14 and 15 report per-dataset G-Eval, ASR-CER/WER, and UTMOS results.Source: §C.1 · p. 16
C.2 Initialization Strategy Analysis
The analysis compares random, select, and mean initialization for newly added BPE-unit embedding and LM-head rows. Mean initialization converges faster and produces more stable gradients, especially at the larger tested learning rate.
C.2 Initialization Strategy Analysis
- Mean initialization averages the embedding and LM-head rows of all original tokens in a BPE unit’s deterministic expansion.Random initialization ignores constituent tokens; select initialization copies one randomly chosen constituent’s rows.Source: §C.2 · pp. 16–17
- Mean initialization converges faster and maintains more stable gradients than random or select initialization, particularly at 1.6 × 10−4.The comparison evaluates two learning rates during the first phase of pronunciation adaptation.Source: §C.2 · p. 17, Figure 8 · p. 17
C.3 Data Scaling Analysis
The additional analyses examine data scaling, compression across corpora and languages, latency, human ratings, architecture transfer, adaptation ablations, vocabulary size, and output-layer cost. Together, they characterize trade-offs and evaluation conditions for CoSTA.
C.3 Data Scaling Analysis
- 300h of pronunciation-adaptation speech and 50K dialogue examples provide the selected balance between adaptation cost, compression behavior, and response quality.Additional TTS data generally improve compression and alignment with diminishing gains; 50K dialogue examples provide a strong balance.Source: §C.3 · p. 17
C.4 Compression Across Data Sources and Languages
- Across nine S3Tokenizer V2 corpus and language settings, held-out compression follows an approximate log-linear trend, with R2 = 0.9942–0.9997 and RMSE 0.10–0.47 pp.Each setting uses its own vocabulary learned on its corresponding training split; the comparison is not vocabulary transfer across settings.Source: §C.4 · p. 17, Figure 9 · p. 17, §D.1 · p. 20
C.5 First-Packet Latency
- First-packet latency falls 9.4% in Chinese and 9.6% in English, while LLM decode latency drops 25.2% and 29.8%, respectively.Prefill and speech-decoder latency remain largely unchanged; Table 18 separates these components.Source: §C.5 · pp. 17–18
C.6 Human Evaluation Protocol and Detailed Results
- CoSTA has more human-evaluation wins than losses in every aggregate comparison, while ties make up most fluency comparisons.Six evaluators rated responses on speech fluency and semantic quality across 60 sampled queries.Source: §C.6 · p. 18
C.7 Detailed OpenS2S Transfer Results
- On OpenS2S, CoSTA improves average content scores and alignment over matched fine-tuning while slightly lowering average UTMOS.The transfer experiments retain OpenS2S’s native GLM-4-Voice tokenizer and speech decoder.Source: §C.7 · p. 18
C.8 Why Dialogue Adaptation Matters
- Without dialogue adaptation, G-Eval drops from 72.27 to 24.35 as BPE coverage falls from 51.64% to 36.71%, alongside more original-token use and a query-repetition example.The analysis suggests dialogue adaptation connects TTS-learned compressed-token generation to appropriate spoken responses.Source: §C.8 · pp. 18–19
C.9 Full-Model Results Across Vocabulary Sizes
- Increasing added vocabulary from 80K to 160K improves compression but raises speed only 1.1%, while G-Eval and UTMOS decline.Under the tested conditions, the authors identify 80K as a balance among speed, compression, and quality.Source: §C.9 · p. 19
C.10 Computational Cost of Vocabulary Extension
- At 80K, the isolated normalized output-layer cost is 0.925; at 160K and 320K it rises to 1.152 and 1.616, respectively.These measurements exclude Transformer computation; full-model decoding speed can still rise slightly from 80K to 160K, so compression alone does not determine runtime.Source: §C.10 · p. 19
D.2 Fit Quality and Training–Test Consistency
The fitted log-linear scaling trend persists across held-out tokenizer, dataset, and language settings, with close training–test consistency. Case studies illustrate compact token generation with response quality preserved, and the paper reports artifact licenses and usage terms.
D.2 Fit Quality and Training–Test Consistency
- Test curves show 2.70–4.19 pp declines per vocabulary doubling across tokenizer configurations and 3.62–4.01 pp across dataset/language curves.The fitted tokenizer-curve R2 ranges from 0.9807 to 0.9997, while dataset/language-curve R2 ranges from 0.9942 to 0.9997.Source: §D.2 · pp. 20–21
- Across all eight tokenizer configurations, test slopes stay close to training counterparts, differing by 0.25–1.46 pp at 80K and 0.84–3.52 pp at 320K; scaling regularity transfers across data sources and languages.At 320K, gaps are at most 0.24 pp for Emilia and Emilia-TC, and 0.85–1.71 pp for dialogue speech.Source: §D.2 · p. 21
- The log-linear fits describe tested 10K–320K vocabulary settings, with largest absolute test residuals of 1.21 pp for Qwen3-TTS and 0.67 pp across dataset/language curves.The passage characterizes the fitted models as having high R2 across settings.Source: §D.2 · p. 21
E Case Study
- Five English AlpacaEval and WildchatEval case studies qualitatively show compact speech-token sequences while preserving response quality.They cover factual and commonsense questions, rewriting, open-ended dialogue, and an empathetic customer-service reply; BPE units are deterministically expanded before synthesis with the original decoder.Source: E Case Study · p. 21
F Licenses of Scientific Artifacts
- The paper reports the licenses or usage terms of its scientific artifacts in Table 13.Table 13 is captioned as listing those licenses and usage terms.Source: F Licenses of Scientific Artifacts · p. 21, Table 13 · p. 21