PaperlayerView exact revision on arXiv

Source-linked AI summary

CapTalk: Unified Voice Design for Single-Utterance and Dialogue Speech Generation

Xiaosu Su, Zihan Sun, Peilei Jia, Jun Gao

arXiv:2604.08363v1cs.SD

TL;DR

Natural-language voice design still has limited support for open-ended control, and systematic study of conversational dialogue remains limited. CapTalk combines caption-conditioned generation with hierarchical timbre conditioning for single utterances and dialogue. On InstructTTSEval-ZH, it achieves the best RP score and highest overall performance, surpassing the second-best Qwen3TTS-12Hz-1.7B-VD.

  • Problem

    Natural-language voice design lacks unified fine-grained attribute control, while dialogue generation and evaluation remain limited despite their relevance to conversational speech.

    Source: §1 · pp. 1–2

  • Method

    CapTalk uses caption-conditioned autoregressive generation, speaker-level dialogue captions and CoT turn controls, plus hierarchical variational conditioning for stable timbre and adaptive expression.

    Source: §1 · p. 2

  • Results

    CapTalk achieves the best RP score and highest overall performance on InstructTTSEval-ZH, surpassing the second-best Qwen3TTS-12Hz-1.7B-VD.

    Source: §5.1.1 · p. 7

  • Takeaways & Limitations

    The framework supports timbre reuse while keeping expression aligned with current text and, in dialogue, surrounding context.

    Source: §1 · p. 2

    Training data is mainly casual conversational speech, whose emotional intensity and expressive range are typically weaker than acted-style speech.

    Source: A Limitations and Future Work · p. 11

Abstract

from arXiv · show

Voice design from natural language descriptions is emerging as a new task in text-to-speech multimodal generation, aiming to synthesize speech with target timbre and speaking style without relying on reference audio. However, existing methods mainly focus on single-utterance generation, leaving conversational voice design largely unexplored. In this work, we extend voice design to dialogue, enabling better target speaker modeling and turn-level expressive control in natural conversational settings. We propose CapTalk, a unified caption-conditioned text-audio autoregressive framework for both single-utterance and dialogue voice design. CapTalk uses utterance-level captions for single-utterance voice design and speaker-level captions for dialogue speaker modeling, and further introduces a CoT control sequence in dialogue to explicitly plan turn-level dynamic attributes. To resolve the conflict between stable timbre preservation and context-adaptive expression, we propose a hierarchical variational conditioning module with an utterance-level speaker encoder to better balance stable timbre preservation and context-adaptive expression. This enables timbre reuse while keeping expression adaptive to the current utterance and, in dialogue, the surrounding context. We also build a comprehensive evaluation protocol for both single-utterance and dialogue settings. Experiments show that CapTalk achieves state-of-the-art performance on a single-utterance voice design benchmark and delivers better expression controllability and contextual appropriateness in multi-turn dialogue. Audio samples are available at: https://anonymous.4open.science/api/repo/Captalk-D601/file/index.html.

1 Introduction

Natural-language voice design still lacks unified control of fine-grained speaker attributes, and dialogue voice design remains little studied. CapTalk addresses these gaps with a shared framework, hierarchical timbre conditioning, and an evaluation protocol for single utterances and dialogue.

  • Current TTS systems lack a unified natural-language interface for fine-grained voice attributes, and early prompt-based methods remain focused on label-based control.Source: §1 · p. 1
  • Existing voice-design methods mainly target single utterances, while systematic study of natural multi-turn dialogue remains limited.Source: §1 · p. 1
  • Dialogue voice design must preserve speaker identity and timbre while adapting turn-level expression and prosody to conversational context.Source: §1 · pp. 1–2
  • CapTalk unifies single-utterance and dialogue voice design, using utterance-level captions for single utterances and speaker-level captions plus CoT control for dialogue.Source: §1 · p. 2
  • Its hierarchical variational conditioning uses an utterance-level speaker encoder and KL regularization to preserve stable timbre while attenuating segment-specific affective variation.Source: §1 · p. 2
  • The evaluation protocol covers both settings, combining single-utterance human evaluation with dialogue-oriented CoT prediction, controllability, and comparative evaluation.Source: §1 · p. 2

2 Related Work

Prior voice-design systems use natural-language prompts and related control approaches, but most voice-design-focused methods target single utterances. Broader TTS and audio-generation systems and factorized variational models provide adjacent approaches and inspiration.

  • Voice-design-focused methods use textual prompts to specify timbre, speaking style, and expressive attributes, but mainly focus on single-utterance generation.Source: §2 · p. 2
  • Related TTS and audio-generation systems extend language control to voice design, expressive scenarios, and multi-turn generation, while FHVAE separates stable factors from variation.Source: §2 · p. 2

3 Architecture

CapTalk uses a unified caption-conditioned text-audio autoregressive framework for single-utterance and dialogue generation, with hierarchical variational timbre conditioning. Its dialogue branch adds explicit turn-level expressive control, while the conditioning module separates stable speaker information from more local variation.

3.1 Overview

  • CapTalk combines a speech tokenizer, dual-transformer text-to-speech model, and hierarchical variational timbre conditioning in one framework for single-utterance and dialogue generation.The tokenizer converts 16 kHz speech into discrete acoustic tokens at 12.5 Hz with 16 codebooks.Source: §3.1 · p. 2, Figure 1 · p. 3

3.2 Single-Utterance and Dialogue Modeling

  • Single-utterance generation uses utterance-level captions, while dialogue generation uses speaker-level captions and dialogue context to model the target speaker and current turn.The dialogue branch also introduces CoT tokens to control turn-level emotion, tone, pitch, energy, and speaking rate.Source: §3.2 · pp. 3–4

3.3 Factorized Hierarchical Variational Timbre Conditioning

  • The hierarchical module combines a full-utterance speaker embedding with pooled segment and frame-level latents for acoustic generation.The design assumes speaker traits remain approximately constant across an utterance while emotion and tone can vary over time; emotion and tone are treated as locally stationary within a short segment.Source: §3.3 · p. 4
  • KL regularization toward an utterance-conditioned prior encourages shared stable speaker information in segment latents while attenuating segment-specific affective variation.The segment latent captures locally constant factors, including timbre and local affect, while the frame-level latent captures finer variation such as phonetic content and prosody.Source: §3.3 · p. 4
  • At inference, CapTalk can predict the utterance-level speaker condition from text or fix a previously obtained condition to reuse a designed timbre.The textual prediction is aligned with training-time representations, and the fixed condition supports timbre reuse.Source: §3.3 · p. 4

4 Task Formulation, Data Construction, and Evaluation Protocol

The section describes how CapTalk’s single-utterance and dialogue data are constructed and how expression attributes are represented. It also outlines an evaluation protocol that supplements single-utterance benchmarking with human ratings and dialogue-specific tests.

4.1 Data Sources and Caption Construction

  • Single-utterance experiments use 300 hours of public data and 5,000 hours of internal data, while dialogue experiments use full two-speaker sessions from internal speech data.The data cover both acted-style and natural conversational speech.Source: §4.1 · p. 5
  • Qwen3-Omni generates utterance-level captions for expressive characteristics and speaker-level captions for stable attributes and long-term traits, using ten segments per speaker.Speaker-level descriptions retain dynamic attributes such as pitch and volume variation.Source: §4.1 · p. 5
  • Training uses APS, DSD, and RP captions, offering structured attributes, free-form descriptions, and role- or scenario-based styles as conditioning variants.Single-utterance training uses utterance-level captions and retains filtered utterance-caption pairs.Source: §4.1 · p. 5
  • Dialogue sessions use speaker-level captions, eligible target speakers, and up to two target-speaker views when both participants pass quality filtering.Long sessions are divided using 20-turn sliding windows.Source: §4.1 · pp. 5–6

4.2 CoT Design and Attribute Extraction

  • Dialogue CoT represents five turn-level attributes: emotion, tone, pitch, energy, and speaking rate, linking high-level expression with lower-level prosody.Pitch, energy, and rate are modeled as relative to each speaker’s baseline rather than compared absolutely across speakers.Source: §4.2 · p. 6
  • Pitch, energy, and speaking rate are extracted relative to each speaker’s median baseline, while emotion and tone are extracted per utterance.The prosodic attributes are discretized into graded deviations from normal.Source: §4.2 · p. 6

4.3 Evaluation Protocol and Benchmark Analysis

  • Single-utterance evaluation combines InstructTTSEval with 1–5 human ratings across seven aspects, including description, identity, timbre, style, role intent, control stability, and naturalness.The benchmark is treated as a starting point because it does not cover dialogue context and turn-level control.Source: §4.3 · p. 6
  • Dialogue evaluation tests CoT plausibility, attribute controllability, and overall quality, using Gemini-assisted and human judgments as well as controlled generation comparisons.The protocol combines the dialogue tests with human evaluation and the existing single-utterance benchmark.Source: §4.3 · p. 6

5 Experiments

Experiments evaluate CapTalk separately on single-utterance and dialogue voice design, using benchmark and human evaluations for the former and CoT and context-aware evaluations for the latter. Results cover comparative performance, data scaling, and caption-loss ablation.

5.1 Comparative Experiments

  • Comparative experiments use a public benchmark and human evaluation for single utterances, and an extended protocol targeting CoT control and context-aware generation for dialogue.The settings are evaluated separately because their task forms and evaluation objectives differ.Source: §5.1 · p. 7

5.1.1 Single-Utterance Comparison.

  • CapTalk achieves the best RP score and highest overall performance on InstructTTSEval-ZH, while Qwen3TTS-12Hz-1.7B-VD leads on APS and DSD.Human evaluation also finds CapTalk best on overall score, role-intent consistency, control stability, and MOS; its timbre-consistency score is relatively lower.Source: §5.1.1 · p. 7

5.1.2 Dialogue Comparison.

  • With CoT, dialogue quality is preferred over the no-CoT variant by Gemini (72% vs. 28%) and human evaluators (65.5% vs. 34.5%).CoT attribute changes produce expected acoustic changes, and predictions are reasonable across five attributes, with pitch, energy, and speed most accurate; CapTalk also outscores Fish Speech S2 Pro on SIM, context coherence, and MOS.Source: §5.1.2 · pp. 7–8, Table 7 · p. 8

5.2.1 Scaling Law in Single-Utterance Voice Design.

  • AVG 73.73 is achieved by mixing 5000-hour conversational data with 300 hours of acted speech, the best reported single-utterance scaling result.At 1500 hours, conversational data underperforms 300-hour acted data; the results suggest broader coverage from conversational data and stronger expressive attributes from acted speech.Source: Table 5 · p. 7, §5.2.1 · p. 8

5.2.2 Scaling Law in Dialogue Voice Design.

  • Dialogue CoT prediction and controllability improve steadily across training sets of 17,258, 24,198, and 32,171 sessions.The reported trend spans all five evaluated attributes.Source: §5.2.2 · p. 8, Table 6 · p. 8

5.3 Ablation Study

  • Adding caption_loss raises DSD from 68.30 to 69.10 but lowers APS from 79.10 to 77.20, RP from 55.10 to 54.40, and AVG from 67.50 to 66.90.The authors conclude explicit caption supervision is not beneficial overall in this setting and disable caption_loss in the final model.Source: §5.3 · p. 8, Table 8 · p. 8

6 Conclusion

CapTalk unifies caption-conditioned generation for single utterances and dialogue, with hierarchical conditioning and CoT planning. Experiments report strong single-utterance performance and improved dialogue controllability and coherence.

6 Conclusion

  • CapTalk uses utterance-level captions for single-utterance design, speaker-level captions for dialogue, and CoT control for turn-level expression planning.Its hierarchical variational module uses an utterance-level speaker encoder to reinforce stable timbre while suppressing segment-specific affective variation, supporting timbre reuse with context-adaptive expression.Source: §6 · p. 8
  • Experiments show state-of-the-art overall performance on single-utterance voice design and better expression controllability and context coherence in multi-turn dialogue.The paper also presents an evaluation protocol for both settings and plans to release caption annotations and a subset of data upon acceptance.Source: §6 · p. 8

A Limitations and Future Work

The authors identify limits in caption quality and expressive range in the training data, and propose improvements to annotation, corpus coverage, and evaluation.

A Limitations and Future Work

  • Caption conditions depend on Qwen3-Omni audio understanding and caption generation, whose description-quality limitations affect their accuracy and stability.The authors propose multi-model cross-validation and human-in-the-loop correction to improve caption quality and reduce description bias.Source: A Limitations and Future Work · p. 11
  • Training data mainly comprise casual natural conversation, with weaker emotional intensity and expressive range than acted-style speech.The authors identify more expressive settings as an area for improvement and propose broader corpus coverage of age, accent, and expressive style, alongside a human-validated benchmark for both single-utterance and dialogue scenarios.Source: A Limitations and Future Work · p. 11

B Additional Evaluation Details

The appendix describes seven human-rated dimensions for single-utterance evaluation and points to a separate protocol summary for dialogue evaluation.

B.1 Single-Utterance Human Evaluation

  • Single-utterance human evaluation rates seven aspects on a 1–5 scale, including description, identity, timbre, expressive-style, role-intent, and control-stability consistency, plus naturalness.MOS averages annotator and sample scores; naturalness is judged regardless of caption condition.Source: §B.1 · p. 11

B.2 Dialogue Evaluation Protocol

  • The dialogue evaluation protocol is summarized in Table 9.The supplied appendix passage identifies the protocol summary but does not enumerate its criteria.Source: §B.2 · p. 11

B.2.1 Details of CoT Prediction Evaluation.

Dialogue CoT prediction is evaluated for whether it is a reasonable interpretation of the target speaker, dialogue history, and current utterance, rather than an exact match to one annotated answer. The protocol combines Gemini judgments with human verification.

B.2.1 Details of CoT Prediction Evaluation.

  • Predicted CoT is judged for contextual plausibility rather than exact agreement with a single ground-truth sequence, since one utterance can support multiple reasonable stylistic interpretations.The judge considers the target speaker profile, dialogue history, and target utterance; it is not shown the ground-truth CoT.Source: §B.2.1 · p. 11
  • Each target-turn CoT represents emotion, tone, pitch, energy, and speed, predicted from the caption, dialogue history, and current text.The evaluation judge also receives the speaker profile, recent dialogue text and audio, and target text.Source: §B.2.1 · p. 11
  • The 400 evaluation samples cover 100 validation sessions, with four turns per session selected from early through late dialogue positions.The four positions are early, early-middle, late-middle, and late.Source: §B.2.1 · p. 11
  • Gemini provides binary plausibility judgments for each of the five CoT attributes and an overall judgment; the paper primarily reports the five attribute rates.The overall judgment is auxiliary, and human evaluation on a smaller subset checks the automatic evaluation's reliability.Source: §B.2.1 · p. 11
  • Fifteen annotators judge 30 sampled predictions using the same reasonableness criterion, with majority votes compared against Gemini judgments.The annotators do not receive the ground-truth CoT; Table 9 summarizes the dialogue evaluation protocol.Source: §B.2.1 · p. 11, Table 9 · p. 12

B.2.2 Additional Human Evaluation Details.

Additional human evaluations assess single-utterance outputs across seven rating dimensions and dialogue outputs through CoT judgments and contextual pairwise preferences. Ratings or preferences are aggregated across annotators using averages or majority voting, depending on the evaluation.

B.2.2 Additional Human Evaluation Details.

  • Single-utterance evaluation uses 45 test cases and 25 annotators, who rate outputs on seven dimensions from description consistency to MOS naturalness.Each dimension uses a 1–5 scale, and model scores average ratings across annotators and samples.Source: §B.2.2 · p. 12
  • For dialogue CoT verification, 15 annotators judge 30 samples on five attributes and an overall result, with majority voting determining the final decisions.This supplements Gemini-based automatic evaluation.Source: §B.2.2 · p. 12
  • For overall dialogue evaluation, 20 annotators compare full-model and w/o-CoT sessions across 15 cases, choosing which target-speaker turns better fit the session context.Presentation order is randomized, and majority voting determines the reported context-coherence preference.Source: §B.2.2 · p. 12

C Analysis of Hierarchical Variational Timbre Conditioning

The hierarchical variational timbre module is examined for whether it preserves a designed voice across utterances while allowing local expression to adapt. The analysis distinguishes fixed-voice reuse from resampling a voice for each utterance, clarifying how to interpret human timbre ratings.

C Analysis of Hierarchical Variational Timbre Conditioning

  • Human timbre consistency is 3.59 for CapTalk, lower than several baselines, partly because independently sampled voices for one caption can be perceived as inconsistent.The authors describe voice design as one-to-many: different samples can be valid caption-consistent voices, while fixed-voice reuse tests preservation under one designed voice.Source: C Analysis of Hierarchical Variational… · pp. 12–13
  • In the reuse analysis, fixed e_spk reuses one caption-predicted speaker embedding across utterances, whereas resampled e_spk predicts a new embedding for each utterance.Multiple utterances with diverse text are generated, then their speaker embeddings are compared by average pairwise cosine similarity.Source: C Analysis of Hierarchical Variational… · p. 13
  • 0.92 SIM for fixed e_spk exceeds 0.42 SIM for resampled e_spk, showing stronger cross-utterance timbre consistency when the designed voice is held fixed.SIM is average pairwise cosine similarity between WavLM speaker embeddings; Table 10 reports the timbre reuse analysis.Source: C Analysis of Hierarchical Variational… · p. 13, Table 10 · p. 13
  • The authors argue that evaluation should distinguish caption-conditioned voice diversity from consistency when reusing one designed voice across utterances and dialogue contexts.They identify reuse consistency, rather than diversity, as the module's targeted property.Source: C Analysis of Hierarchical Variational… · p. 13

D Examples of Single-Utterance and Dialogue Captions

Single-utterance and dialogue captions serve different conditioning roles despite sharing three surface styles. Dialogue captions characterize stable speaker traits, while CoT represents transient turn-level expression.

D Examples of Single-Utterance and Dialogue Captions

  • Single-utterance captions describe how one recording is spoken, while dialogue captions characterize a reusable speaker across turns in a session.The transient state of each dialogue utterance is represented separately by CoT.Source: D Examples of Single-Utterance and… · p. 13
  • Dialogue APS captions capture relatively stable speaker traits and omit emotion and tone, which are modeled separately with other dynamic turn-level attributes in CoT.Single-utterance APS captions may include utterance-level expressive information.Source: D Examples of Single-Utterance and… · p. 13
  • Even when surface styles match, single-utterance captions contain more local recording information, whereas dialogue captions better support reusable speaker characterization.The authors pair speaker-level captions with CoT to separate long-term identity from short-term turn expression.Source: D Examples of Single-Utterance and… · p. 13
Loading 2604.08363v1…