Source-linked AI summary
When the Persona Contradicts the Line: Diagnosing Plan-to-Synthesis Failures in Chain-of-Thought TTS
Joonyong Park, Jerry Li
TL;DR
When vocal affect must contradict lexical sentiment, existing evaluations do not locate whether intent is lost in planning, emotion control, or synthesis. This paper introduces a Japanese–English conflict benchmark and probes those stages while comparing control interfaces. Planner outputs are often direction-correct, but downstream agreement is lower, and interface results vary across languages and affective axes.
Problem
Existing evaluations use broad labels or overall ratings of final audio, leaving unclear whether conflicting-affect failures arise in planning, emotion control, or synthesis.
Source: §1 · p. 1Method
The authors build a Japanese–English benchmark with predeclared acoustic directions, probe planner and synthesis representations, and compare bypassed, supervised, and learned emotion-control interfaces.
Source: §1 · p. 1, §2.3 · p. 2Results
Planner representations were often direction-correct while downstream agreement was lower; supervision improved rates in both languages, whereas learned-interface results varied by language and affective axis.
Source: §3.1 · p. 4, §3.2 · p. 4Takeaways & Limitations
Learned repair of the plan-to-synthesis interface can recover counter-textual delivery, though its effect differs across languages and acoustic axes.
Source: §3.2 · p. 4The study covers one planner–synthesizer pipeline and two affective axes, and the reference-free acoustic scorer’s cross-lingual transfer remains imperfectly characterized.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Chain-of-thought TTS systems rely on language-model planners to translate persona and context into vocal delivery, but it is unclear where this intent is lost when the requested affect conflicts with lexical sentiment. We introduce a bilingual Japanese--English persona--line conflict benchmark in which each item pairs one line with congruent and conflicting personas and pre-declares the expected acoustic direction. We probe this direction in the planner-generated delivery phrase, the planner-written emotion vector, the QwenEmotion-generated vector, and the rendered waveform. Planner outputs are usually direction-correct, while downstream agreement can be substantially lower. Bypassing QwenEmotion provides only modest gains over the full planner-controlled Base. Supervising QwenEmotion raises the direction rate to 75% in Japanese and 84% in English. A learned text-to-latent interface reaches the highest English rate of 89% and the highest Japanese arousal rate of 88%. These results show that learned repair of the plan-to-synthesis interface can recover counter-textual delivery, although its effect differs across languages and acoustic axes.
1. INTRODUCTION
Persona–line conflicts require delivery to oppose a line’s sentiment, yet existing evaluations judge mainly the final audio and cannot localize where intent is lost. This work introduces a bilingual benchmark and probes planner representations and waveform outcomes while comparing emotion-control interfaces.
1. INTRODUCTION
- Existing evaluations of conflicting affect rely on broad labels or overall ratings of final audio, leaving failures among planning, emotion control, and synthesis difficult to localize.Speech emotion recognizers can also rely on lexical sentiment, which complicates evaluation when intended affect conflicts with the words.Source: §1 · p. 1
- The Japanese–English benchmark pairs each line with congruent and conflicting personas and predeclares an acoustic axis and direction for evaluating both representations and rendered speech.The declared target avoids requiring each persona to map to a single emotion category.Source: §1 · p. 1
- The study probes direction agreement in the planner’s delivery phrase and emotion vector, QwenEmotion’s vector, and the rendered waveform in a planner-controlled IndexTTS-2.5 pipeline.The stock route maps QwenEmotion’s vector through a speaker-matched emotion prototype bank.Source: §1 · p. 1
- Three controlled interface modifications bypass QwenEmotion, fine-tune it, or replace it and the prototype bank with a learned text-to-latent network, keeping the synthesis backbone fixed.Related work includes prompt-to-style embeddings and shared text–speech representations; this learned interface predicts the synthesizer’s own emotion latent from text using targets extracted from human acted speech.Source: §1 · pp. 1–2
2. METHOD
The method pairs congruent and conflicting personas with a predeclared acoustic direction, then checks whether that direction is preserved from planning through rendered speech. It compares emotion-control interventions while keeping the synthesis backbone fixed.
2.1. Conflict benchmark and scorer
- Each language has 150 test items, and a wav2vec2-based acoustic predictor scores rendered speech on the declared axis; its direction labels were validated on parallel human recordings.A fresh five-vote audit reproduced the declared target on 96% of 75 items; the predictor recovered labelled direction on 99.6% of 24,334 arousal pairs and 83.1% of 15,601 valence pairs.Source: §2.1 · p. 2
2.2. Pipeline and representation probe
- The fixed IndexTTS-2.5 pipeline is probed at four points: delivery phrase, planner emotion vector, QwenEmotion vector, and rendered waveform.The waveform probe fixes duration to 1.0 to isolate the stock emotion-control path; the separate duration factor is omitted from the representation probe.Source: §2.2 · p. 2
2.3. Interface treatments
- Three interface treatments bypass QwenEmotion, fine-tune it with render-selected supervision, or replace it and the prototype bank with a learned text-to-latent interface.The learned interface predicts continuous emotion-conditioning shifts from delivery descriptions using audio-derived targets, while the synthesis backbone and text encoder remain fixed.Source: §2.3 · pp. 2–3
3. EXPERIMENTS AND RESULTS
Planner representations often match the declared direction, but agreement is lower in downstream vectors and waveforms. Interface repairs show different outcomes across languages and acoustic axes.
3.1. Where direction agreement breaks down
- The protocol independently probes the declared direction in the delivery phrase, planner vector, QwenEmotion vector, and rendered waveform.Table 2 reports agreement at each probe point independently.Source: §3.1 · p. 4
- Planner delivery phrases and emotion vectors show 84% and 86% agreement in Japanese, and 98% and 89% in English; QwenEmotion vectors and stock-route waveforms score lower.QwenEmotion agreement is 46% in Japanese and 83% in English; stock-route waveform agreement is 55% and 66%, respectively. These representation-specific rates are not conditional stage-to-stage survival probabilities.Source: §3.1 · p. 4
3.2. Can the interface be repaired?
- The learned interface reaches 89% overall in English and 88% Japanese arousal, while SFT reaches 75% in Japanese and 84% in English; bypassing QwenEmotion reaches 68% and 79%.The learned route's Japanese overall rate is 66% (p = .60), and bypassing provides only modest evidence of improvement.Source: §3.2 · p. 4
3.3. Human validation
- Listener majorities agree with the predictor on 61/90 Japanese and 69/90 English item–condition pairs, while only the supervised English condition exceeds chance at item-majority level (22/30, p = .016).The listening test is descriptive and does not establish differences among treatments.Source: §3.3 · p. 4, Table 4 · p. 4
4. CONCLUSION
The study introduces a bilingual persona–line conflict benchmark and compares emotion-control interfaces in one planner–synthesizer pipeline. Its scope is limited by two affective axes and a reference-free scorer with imperfectly characterized cross-lingual transfer.
4. CONCLUSION
- The study is limited to one planner–synthesizer pipeline, two affective axes, and a reference-free acoustic scorer whose cross-lingual transfer remains imperfectly characterized.Future work should develop richer emotion captioners and evaluators and train models to reason from persona and context to continuous acting representations.Source: §4 · p. 4