PaperlayerView exact revision on arXiv

Source-linked AI summary

KABURI-TTS: Phoneme-Keyed Activity-conditioned Bi-channel Utterance Rendering for Interaction

Ryuichiro Higashinaka, Shinnosuke Takamichi, Tetsuji Ogawa

arXiv:2609.07200v1cs.SDcs.CL

TL;DR

Existing dialogue synthesis methods may not robustly represent or control overlap between speakers. KABURI-TTS conditions separate speaker channels on an externally supplied phoneme raster and derived voice activity. Evaluation reports stronger naturalness than baselines and increased overlap and turn-taking.

  • Problem

    Existing two-speaker synthesis may produce little simultaneous speech, and implicitly learned overlap is difficult to control.

    Source: §I · p. 1

  • Method

    KABURI-TTS generates separate speaker channels conditioned on per-frame phonemes, derived activity, and speaker references.

    Source: §I · p. 1, B. Model · p. 2

  • Results

    KABURI (stat) is best overall in Text-in Eval and significantly exceeds KABURI (pred) and Irodori (stat) in interaction naturalness.

    Source: E. Results · p. 5

  • Takeaways & Limitations

    Voice-activity analysis reports higher naturalness than strong baselines, with more frequent overlap and turn-taking.

    Source: §VI · p. 6

Abstract

from arXiv · show

Realizing full-duplex spoken dialogue requires large amounts of two-channel, one-speaker-per-channel conversational speech data. Although conversational text-to-speech (TTS) engines have been developed, they are not necessarily robust to two-party simultaneous phenomena such as backchannels, interruptions, and overlaps that occur while the interlocutor is speaking. In this work, aiming at conversational speech synthesis that reproduces human-like overlap, we propose KABURI-TTS. KABURI-TTS takes a per-speaker phoneme raster as input and renders the speech of the two speakers on separate channels, conditioned on the per-frame phonemes and the voice activity derived from them. Because the phoneme raster is supplied by a separate module, the proposed method enables controllable generation of one-speaker-per-channel, two-party spoken dialogue. A user evaluation shows that, compared with strong baselines, the proposed method attains higher naturalness at both the utterance and the interaction level. Furthermore, an analysis of voice activity confirms that the proposed method produces more overlap and more frequent turn-taking.

I. INTRODUCTION

Full-duplex dialogue systems need substantial two-channel, one-speaker-per-channel speech data, but existing synthesis approaches may produce little or uncontrollable overlap. KABURI-TTS uses an externally supplied phoneme raster to make simultaneous speech controllable.

I. INTRODUCTION

  • Full-duplex dialogue systems need substantial two-channel, one-speaker-per-channel speech data, and speech synthesis can adapt to a desired domain, [4].The passage also notes that speaker separation has been used to obtain two-channel dialogue data.Source: §I · p. 1
  • Existing methods may produce unnatural dialogue with little overlap, and their implicit learning of simultaneous speech makes it difficult to control.Source: §I · p. 1
  • KABURI-TTS conditions two-channel speech on an explicit, externally supplied phoneme raster to make simultaneous speech more controllable.The raster allows phonemes and utterance durations to be customized.Source: §I · p. 1, §II · p. 2

II. RELATED WORK

  • Single-speaker TTS can provide high audio quality, but placing utterances cannot naturally model interaction between speakers.Source: §II · p. 1
  • Serialized speaker-tagged methods can produce natural conversational speech, but simultaneous speech is not learned explicitly and remains difficult to control [5], [6],.CoVoMix handles per-speaker semantic token sequences but produces speaker-mixed audio.Source: §II · pp. 1–2
  • Text-to-stereo methods generate separate speaker channels, but learn overlap implicitly, so it may be insufficient or uncontrollable [2], [8],,.Source: §II · p. 2

III. METHOD

KABURI-TTS adapts a pretrained single-speaker synthesis backbone for two-speaker dialogue and conditions generation on synchronized per-channel phoneme, activity, and speaker-reference sequences. Its output is a pair of per-frame latent sequences decoded into separate speaker waveforms.

III. METHOD

  • KABURI-TTS adapts a pretrained single-speaker zero-shot TTS backbone to stereo dialogue using LoRA and updates selected weights.The model is trained on human-to-human dialogue annotated with phoneme rasters and uses DACVAE continuous latents as its audio representation.Source: §III · p. 2

A. Formulation

  • The dialogue is represented as synchronized frame sequences, with a per-channel phoneme raster and a binary activity raster marking speech.The speaker reference is a DACVAE latent encoded from each channel's reference speech.Source: A. Formulation · p. 2
  • The model outputs a DACVAE latent sequence for each speaker channel, which the decoder converts into separate waveforms.The acoustic model specifies the conditional distribution of stereo latents given phonemes, activity, and speaker references.Source: A. Formulation · p. 2

B. Model

The acoustic model derives activity and interaction-state conditions from the phoneme raster and injects them alongside phoneme and speaker-reference information. It processes the two speaker channels with shared weights, then generates and decodes stereo latents.

B. Model

  • The model derives activity from non-silence phonemes and maps each frame to one of ten interaction-state classes for silence, speaker activity, overlap, and speech boundaries.Boundary classes take precedence when a boundary occurs.Source: B. Model · p. 2
  • Phoneme, activity, and interaction conditions enter as projected additive biases, while phoneme information is re-injected in deep blocks and speaker references enter cross-attention.The phoneme condition is re-injected at layers 2, 4, 6, 8, and 10; speaker references are used in all twelve blocks.Source: B. Model · p. 2
  • The channels are stacked along the batch dimension and processed simultaneously with shared weights, without directly mixing the channels.Their interplay is represented through the conditioning signals.Source: B. Model · pp. 2–3
  • For 30-second chunks, inference integrates the predicted velocity field for 32 ODE steps and decodes the latent into 48-kHz stereo audio.At 25 fps, each chunk contains 750 frames; the left and right output channels correspond to speakers A and B.Source: B. Model · p. 3

C. Training

Training combines rectified-flow velocity regression with timing prediction and regularization. The timing auxiliary task supervises dialogue activity and boundaries, while silent frames receive reduced but nonzero weight.

C. Training

  • The model uses LoRA and selectively unfrozen weights, while L2-SP regularization pulls updated weights toward the pretrained backbone.New condition-injection layers are zero-initialized, and the L2-SP weight is 10^-4.Source: C. Training · p. 3
  • The main training objective is rectified-flow velocity regression, with activity-based weighting that emphasizes speech frames while retaining a nonzero weight for silence.The frame weight is a + 0.05(1−a), because silent-frame latents must also be generated.Source: C. Training · p. 3
  • An auxiliary binary-cross-entropy loss predicts activity, onset and offset, overlap, and turn-switch labels from intermediate DiT representations.Each of the five prediction types has weight 0.1; activity and onset/offset are assessed separately for both channels.Source: C. Training · p. 3
  • Training drops the phoneme condition with probability 0.15 and applies classifier-free guidance at scale 2.5 during inference.Source: C. Training · p. 3

D. Construction of the Phoneme Raster

The section describes statistical placement as a way to turn dialogue text into per-channel phoneme rasters by assigning utterance gaps and phoneme durations. The resulting rasters are fed to the acoustic model.

D. Construction of the Phoneme Raster

  • Statistical placement assigns each utterance to its speaker’s channel and determines both the gap between utterances and the durations of phonemes within them.The resulting two-channel phoneme raster is supplied to the acoustic model.Source: 1) Statistical Placement · p. 3

1) Statistical Placement:

  • Inter-utterance gaps are sampled from direction-specific empirical distributions, capturing observed gaps and overlaps while excluding extreme values and capping overlap at 0.3 s.The method uses inverse-CDF sampling with a uniform draw from 0.05–0.95; speaker changes can yield negative gaps.Source: 1) Statistical Placement · p. 3
  • Each utterance’s frame count is based on its phoneme count and the training median frames per phoneme, then apportioned according to training-average phoneme durations.If the resulting sequence exceeds the window, all durations are uniformly scaled to fit.Source: 1) Statistical Placement · p. 3

2) Duration Predictor:

The duration predictor estimates phoneme, silence, and utterance-gap timing from text and dialogue context, then assembles the predictions into phoneme and activity rasters. The paper evaluates synthesis using both test chunks and separately created dialogue text.

2) Duration Predictor:

  • The duration predictor is a 2-layer, approximately 0.8 M-parameter Transformer that classifies phoneme durations, silence durations, and inter-utterance gaps from phonemes, speaker, channel, and dialogue context.Its output bins represent negative gaps as overlap.Source: 2) Duration Predictor · p. 3
  • Training adds utterance-length, speaking-rate, and frame-state auxiliary losses; inference decodes predicted bins into timing values and assembles two-channel phoneme and activity rasters.Gaps receive a uniform +9-frame correction calibrated on validation data because expectation decoding tends to make turn-taking too tight.Source: 2) Duration Predictor · pp. 3–4

IV. EXPERIMENT

  • The evaluation includes synthesis from sampled test chunks and a practical text-in evaluation using separately created dialogue text.For the proposed system, Irodori TTS12 is the DiT backbone, with its text-input weights set to zero.Source: §IV · p. 4

A. Dataset

The dataset section describes the Japanese conversation corpus, its train and evaluation splits, and the preparation of phoneme rasters. It also specifies sampled test chunks and generated dialogue text for evaluation.

A. Dataset

  • LLM-jp-Zoom1 contains 2,000 approximately 30-minute Zoom conversations totaling 1,085 hours, recorded with 60 speakers across about 15 topics.The conversations were split into 30-second chunks before creating train, validation, and test splits.Source: A. Dataset · p. 4
  • The train split contains 1,760 dialogues and 113,310 chunks, while validation, test-seen, and test-unseen each contain 80 dialogues.The test-unseen subset has 10 unseen and 7 seen speakers because the 60-speaker corpus does not allow a split of entirely mutually unseen speakers.Source: A. Dataset · p. 4
  • Dialogue speech was segmented with Silero VAD, transcribed with Whisper-v3, converted with g2p, and aligned with Montreal Forced Aligner to create per-channel phoneme rasters.Boundary speech was included in the chunk transcription.Source: A. Dataset · p. 4
  • Test chunk evaluation uses 50 samples each from test-seen and test-unseen, for 100 chunks total, with their phoneme rasters as model input.Text-in evaluation instead uses LLM-generated Japanese casual dialogues of 16 utterances, including about three to five short backchannels and naturally occurring repeated turns by one speaker.Source: A. Dataset · p. 4
  • Text-in prompts request coherent casual conversation with short spoken-language utterances of roughly 5–25 characters, avoiding proper nouns, alphabetic characters, and numbers.Four dialogues were generated for each of the corpus’s 15 topics, yielding 60 inputs.Source: A. Dataset · p. 4

B. Systems for Comparison

The comparison includes KABURI-TTS variants, Irodori TTS baselines, and MOSS-TTSD for text-in evaluation. The systems differ in how they obtain timing and place utterances.

B. Systems for Comparison

  • KABURI (GT) uses the test chunk’s ground-truth phoneme raster, while GT provides the original human-to-human speech as an upper bound.The proposed acoustic model was trained for 30k steps, and its duration predictor for 12k steps; both use their final checkpoints.Source: B. Systems for Comparison · p. 4
  • KABURI (stat) builds a raster by statistical placement, whereas KABURI (pred) uses the trained duration predictor to construct it.Both variants start from the collapsed phoneme sequence obtained by applying g2p to the test-chunk text.Source: B. Systems for Comparison · p. 4
  • Irodori (GT) and Irodori (stat) synthesize utterances from text with ground-truth or statistically placed start times, respectively; MOSS-TTSD is an additional Text-in baseline.MOSS-TTSD generates single-channel speech, so its evaluation target differs from those of the other methods.Source: B. Systems for Comparison · pp. 4–5

C. Evaluation Metrics

KABURI performs well across test-chunk and text-in evaluations, with KABURI (stat) best overall in the practical text-in evaluation. The evaluations assess utterance naturalness, interaction naturalness, and intelligibility.

C. Evaluation Metrics

  • The human evaluation rates utterance-level naturalness, interaction-level naturalness, and intelligibility on five-point Likert scales.The items ask about conversational speaking style, naturalness of the exchange, and whether the conversation content can be heard without difficulty.Source: C. Evaluation Metrics · p. 5

E. Results

  • Test Chunk Eval comparisons use paired Wilcoxon signed-rank tests, with Benjamini–Hochberg correction controlling FDR at 5% within each evaluation item.Scores were averaged across two subjects for each item before comparison.Source: E. Results · p. 5
  • In Test chunk Eval, KABURI (pred) is significantly better than KABURI (stat) and Irodori-based methods in utterance naturalness, while KABURI variants exceed Irodori-based methods in interaction naturalness.KABURI (stat) is better than KABURI (pred) and Irodori-based methods in intelligibility, and numerically better than KABURI (GT); synthesized speech overall does not reach GT.Source: Table I · p. 5, E. Results · p. 5
  • KABURI (stat) is best overall in Text-in Eval, and KABURI achieves comparable or superior interaction naturalness to mixed-speech MOSS-TTSD.KABURI (stat) is significantly better than KABURI (pred) and Irodori (stat) on interaction naturalness; its advantage over MOSS-TTSD is a marginally significant tendency (corrected p = 0.071).Source: Table II · p. 5, E. Results · p. 5
  • Test-seen and test-unseen subsets show no significant differences for any system or evaluation axis.The comparison used the Mann–Whitney U test with Benjamini–Hochberg FDR correction.Source: E. Results · p. 5

V. ANALYSIS

Voice-activity statistics show that KABURI produces substantial overlap and frequent turn-taking, unlike the block-like, non-overlapping Irodori-based outputs. These aggregate measures do not establish frame-level adherence to the input raster.

V. ANALYSIS

  • The analysis measures overlap, silence, and switches per minute from binary voice activity derived using 20 ms energy windows at 25 fps.Frames at or above 5% of a channel’s peak energy count as speech; these aggregate statistics do not directly measure frame-level adherence to the input raster.Source: §V · p. 6
  • In Text-in Eval, KABURI produces much overlap and frequent turn-taking, whereas Irodori-based methods produce non-overlapping, block-like utterances.The activity statistics support the higher subjective naturalness of KABURI.Source: §V · p. 6
  • In Test chunk Eval, KABURI variants achieve turn-taking close to ground truth, though KABURI (pred) takes more pauses than KABURI (stat).Irodori-based methods show little overlap, much silence, and few turns; independently synthesized utterance lengths can obscure short responses and backchannels.Source: §V · p. 6

VI. SUMMARY AND FUTURE WORK

KABURI-TTS renders two-party, one-speaker-per-channel dialogue from per-speaker phoneme rasters conditioned on voice activity. Human evaluation and voice-activity analysis report higher naturalness than strong baselines, with more overlap and turn-taking; future work targets accuracy, broader comparisons, and applications.

VI. SUMMARY AND FUTURE WORK

  • KABURI-TTS generates two-party, one-speaker-per-channel dialogue from per-speaker phoneme rasters conditioned on voice activity, enabling controllable overlap.Source: §VI · p. 6
  • Human evaluation and voice-activity analysis report higher naturalness than strong baselines, with more overlap and more frequent turn-taking.The summary identifies these as the evaluation outcomes for KABURI-TTS.Source: §VI · p. 6
  • Future work includes improving placement accuracy, comparing other synthesizers and backbones, supporting arbitrary durations and languages, and using KABURI-TTS for data augmentation.Source: §VI · p. 6
Loading 2609.07200v1…