Source-linked AI summary
Do You Hear What I Mean? Quantifying the Instruction-Perception Gap in Instruction-Guided Expressive Text-To-Speech Systems
Yi-Cheng Lin, Huang-Cheng Chou, Tzu-Chieh Wei, Kuan-Yu Chen, Hung-yi Lee
TL;DR
Natural-language instructions give ITTS users a way to control speech, but it remains unclear whether listener perception aligns with fine-grained prompts. This work evaluates graded expressive control using human perceptual data and finds that reliable fine-grained alignment is limited across the tested systems.
Problem
MOS and WER assess naturalness and intelligibility, respectively, but do not measure whether synthesized speech matches fine-grained instructions.
Source: §1 · p. 1Method
The study evaluates degree modifiers and graded emotion intensity, and collects human ratings for speaker age and word-level emphasis in the E-VOC corpus.
Source: §1 · p. 1, §2.2 · p. 2Results
Among the tested systems, gpt-4o reliably maps degree modifiers and graded emotion adjectives to ordered perceptual changes, while age and word-level emphasis remain inconsistently realized.
Source: §5 · p. 4Takeaways & Limitations
The findings indicate substantial room to improve fine-grained attribute control across the tested ITTS systems.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Instruction-guided text-to-speech (ITTS) enables users to control speech generation through natural language prompts, offering a more intuitive interface than traditional TTS. However, the alignment between user style instructions and listener perception remains largely unexplored. This work first presents a perceptual analysis of ITTS controllability across two expressive dimensions (adverbs of degree and graded emotion intensity) and collects human ratings on speaker age and word-level emphasis attributes. To comprehensively reveal the instruction-perception gap, we provide a data collection with large-scale human evaluations, named Expressive VOice Control (E-VOC) corpus. Furthermore, we reveal that (1) gpt-4o-mini-tts is the most reliable ITTS model with great alignment between instruction and generated utterances across acoustic dimensions. (2) The 5 analyzed ITTS systems tend to generate Adult voices even when the instructions ask to use child or Elderly voices. (3) Fine-grained control remains a major challenge, indicating that most ITTS systems have substantial room for improvement in interpreting slightly different attribute instructions.
1. INTRODUCTION
Instruction-guided TTS offers natural-language control, but it is unclear whether listener perceptions reliably match fine-grained instructions. This study evaluates that alignment and introduces a large-scale human-rating corpus for expressive speech attributes.
1. INTRODUCTION
- MOS assesses naturalness and WER intelligibility, but neither measures instructional fidelity: alignment between synthesized speech and fine-grained user prompts.The introduction frames this as an open question, especially for graded attributes such as emotion intensity.Source: §1 · p. 1
- The evaluation treats adverbs of degree and graded emotion intensity as explicit dimensions of ITTS controllability.Examples include “slightly” versus “extremely” and graded emotion words such as “ecstatic” versus “happy.”Source: §1 · p. 1
- The E-VOC corpus provides human perceptual ratings for speaker age and word-level emphasis, with judgments from over 165 raters.The study reports quality-controlled collection and plans public release of the corpus and analysis pipeline.Source: §1 · p. 1
2. RELATED WORKS AND BACKGROUND
The section surveys ITTS systems and evaluation approaches, then positions this study as measuring fine-grained listener-perceived control rather than relying only on coarse alignment or classifier predictions.
2.1. ITTS Systems and Selection
- The study selects five systems spanning open-source ITTS, commercial synthesis, and a general-purpose audio generation model.The selected systems are Parler-TTS, PromptTTS++, GPT-4o-mini-TTS, and UniAudio, with two Parler models represented.Source: §2.1 · p. 1
2.2. ITTS Evaluation Methods
- Prior ITTS evaluation includes attribute-based objective measures, audio-text or emotion embeddings, and instruction-following judgments from humans or automated evaluators.Examples include acoustic-attribute accuracy, Joint-CLAP similarity, listener ratings, and classifier-based assessment of age or emphasis.Source: §2.2 · pp. 1–2
- Unlike mostly coarse prior evaluations, this work measures graded expressive control and uses human ratings for age and emphasis, where prior work used automated methods.The authors note classifier predictions depend on training data and may inherit dataset biases, making them unreliable indicators of listener perception.Source: §2.2 · p. 2
3. EVALUATION FRAMEWORK
The evaluation framework combines control tasks, alignment metrics, and human perceptual data to systematically examine the instruction-perception gap in ITTS.
3. EVALUATION FRAMEWORK
3.1. Control Dimension
The framework evaluates four control dimensions, comprising two proposed fine-grained expressive tasks and two established speech-control tasks.
3.1. Control Dimension
- The four control dimensions are grouped into two proposed fine-grained expressivity tasks and two established dimensions of speech synthesis.Source: §3.1 · p. 2
3.1.1. Proposed Control Dimensions
- Adverbs of degree test prosodic and emotional scaling, while ordered adjectives test whether listeners perceive progressively stronger intensity within four emotion categories.The adjective sequences draw from a human-annotated emotion-intensity lexicon and are ordered by intensity; Table 1 identifies their levels and intensities.Source: §3.1.1 · p. 2, Table 1 · p. 2
3.1.2. Other Control Dimensions
- Speaker Age evaluates synthesis of a specified perceived age group, while Word-level Emphasis evaluates prominence on a target word through cues such as pitch excursion and duration.Source: §3.1.2 · p. 2
3.2. Evaluation Metrics
The evaluation combines objective acoustic analysis with human judgments to assess how instructions shape synthesized speech. It measures prosodic changes and listeners’ perceptions of emotion intensity, emphasis, and speaker age.
3.2. Evaluation Metrics
- The framework combines objective acoustic analysis with subjective human perceptual judgments across its evaluation.Figure 2 summarizes perceived emotion intensity across four emotions for both Adverbs of Degree and Emotion–Intensity Adjective tasks.Source: §3.2 · p. 2, Figure 2 · p. 3
3.2.1. Objective Measures
- Objective measures track loudness, pitch, and speaking rate across ITTS systems for Adverbs of Degree.Loudness is measured in LUFS, pitch as mean utterance F0, and speaking rate in words per second.Source: §3.2.1 · p. 3, Figure 1 · p. 3
3.2.2. Subjective Measures
- Human ratings assess perceived emotion intensity on a five-point scale and use forced-choice tasks for emphasis and speaker age.For emphasis, listeners select the most prominent word or “Unclear”; for age, they choose Child, Teenage, Adult, Elderly, or Unclear.Source: §3.2.2 · p. 3
3.3. Human Annotation Collection & the E-VOC dataset
The E-VOC corpus uses speech from five representative ITTS systems, generated from conversational transcripts paired with attribute-specific prompts. Human annotations are collected under quality controls, with at least five workers rating each utterance.
3.3. Human Annotation Collection & the E-VOC dataset
- E-VOC audio stimuli were generated with five ITTS systems: Parler-large, Parler-mini, PromptTTS++, UniAudio, and gpt-4o.The systems are Parler-TTS-large-v1, Parler-TTS-mini-v1, PromptTTS++, UniAudio, and gpt-4o-mini-tts.Source: §3.3 · p. 3
3.3.1. Transcripts Generation
- Eight conversational transcripts were paired with prompts specifying acoustic style, emotion intensity, emphasis targets, or speaker age.Examples include “Speak in an Ecstatic tone,” stressing a named word, and requesting a Child’s voice.Source: §3.3.1 · p. 3
3.3.2. Annotation and Quality Control
- Native English-speaking annotators completed training and quality checks, and at least five workers annotated every utterance.Ratings were retained only from annotators who performed highly on check items; Table 3 summarizes model performance on Age and Emphasis using accuracy and age-class F1-scores.Source: §3.3.2 · pp. 3–4, Table 3 · p. 4
4. EXPERIMENTAL RESULTS AND ANALYSES
Across the tested systems, gpt-4o most reliably follows graded instructions for acoustic changes and emotion intensity, while age and word-level emphasis remain difficult to control. Age judgments often default to adult voices, and even the best emphasis accuracy is low.
4.1. Adverbs of Degree
- gpt-4o is the only model that reliably translates degree modifiers into intended acoustic shifts and corresponding perceptual changes.It shows predictable loudness ordering, clearer pitch separation, broader speaking-rate variation, and stronger perceived-emotion gradation; other systems show weaker, compressed, or occasionally reversed distinctions.Source: §4.1 · p. 4
4.3. Speaker Age
- Table 4’s confusion matrices show that listeners most often perceived gpt-4o’s output as adult regardless of the prompt.Across systems, Parler-large and gpt-4o had the highest age accuracies, 0.294 and 0.289 respectively; child and elderly voices were harder to generate.Source: Table 4 · p. 4, §4.3 · p. 4
4.2. Emotion–Intensity Adjectives
- gpt-4o alone reliably controls graded emotional intensity across all four emotion categories.Other models show weaker distinctions: Parler variants have small adjacent steps, PromptTTS++ outputs are often nearly indistinguishable, and UniAudio sometimes reverses intensity trends.Source: §4.2 · p. 4
5. CONCLUSION AND FUTURE WORK
Across five ITTS models, fine-grained alignment with listener perception remains inconsistent, although gpt-4o reliably translated degree modifiers and graded adjectives into perceptually ordered changes. The E-VOC corpus’s 60,000+ human ratings may support future automated evaluation.
Conclusion
- Among the five systems tested, gpt-4o alone reliably mapped degree modifiers and graded adjectives to perceptually ordered changes in loudness, pitch, speaking rate, and emotion intensity.Fine-grained word-level emphasis and speaker age were inconsistently realized; most systems defaulted to adult-like voices and produced weak emphasis cues.Source: §5 · p. 4
- Consistent fine-grained alignment with human perception remains an open challenge, despite current ITTS systems following some high-level styles with coarse reliability.Source: §5 · p. 4
Future Work
- The E-VOC corpus contains over 60,000 human ratings and could support training and validating spoken language models such as Gemini as scalable proxies for perceptual judgments.The authors identify automated judging as a future direction for faster, more reproducible evaluation.Source: §5 · p. 5