Source-linked AI summary
Conversational Voice Aesthetic Model with Reinforcement Learning from Human Listeners
Xilin Jiang, Shun Zhang, Tejas Jayashankar, Yinghao Aaron Li, Osama Hanna
TL;DR
Speech LLMs often assess utterances without conversational context, while perceptual voice attributes have no definitive ground truth. CVAM combines synthetic-description finetuning with reinforcement learning on human votes; it outperforms evaluated speech LLMs and individual humans in agreement with listeners, while reward choice balances majority accuracy and class coverage.
Problem
Speech LLMs largely assess utterances without conversational context, while subjective perceptual attributes lack definitive ground truth and can elicit varied listener judgments.
Source: §1 · p. 1Method
CVAM is supervised-finetuned on synthetic descriptions and labels, then optimized with reinforcement learning on human votes to predict conversational voice aesthetics.
Source: §1 · p. 1, §2.2 · p. 3Results
Takeaways & Limitations
Reward design shapes the balance between majority-label accuracy and broader class coverage, with forward-KL rewards achieving the highest macro-F1.
Source: §3 · p. 4Some rare emotions remain difficult to identify, and aggregate gains do not improve every class uniformly.
Source: §3 · p. 4
Abstract
from arXiv · showhide
We introduce Conversational Voice Aesthetic Model, a speech large language model for describing the voice aesthetics of real or synthetic speech responses in natural conversational contexts. Given a context and a response speech, CVAM describes salient moments that characterize the voice and predicts nine categorical attributes spanning gender, pitch, pacing, emotion, and delivery. The key challenge lies in perceptual fields such as emotion and delivery, which are inherently subjective and lack definitive ground truth. Therefore, we collect ~10 human annotations for each of 3k real and synthetic responses derived from the CANDOR corpus. CVAM is supervised finetuned on synthesized aesthetic descriptions and labels, then optimized with Group Relative Policy Optimization on human judgments. Experiments show that CVAM better agrees with human listeners than Gemini 3.1 Pro and open-source speech LLMs, and outperforms single-human-vs.-rest agreement. Together, we demonstrate the importance of grounding voice aesthetics in human perception and propose a principled framework for human alignment.
1. INTRODUCTION
Voice aesthetics encompass physical traits and contextual qualities, but perceptual attributes are subjective and existing speech LLMs largely assess utterances without conversational context. CVAM addresses these gaps with human-grounded training and outperforms evaluated speech LLMs and single-human agreement.
1. INTRODUCTION
- Voice aesthetics include physical aspects such as pitch and pacing and contextual aspects such as emotion and delivery.The paper presents human-faithful voice description as relevant to conversation understanding and expressive voice synthesis.Source: §1 · p. 1
- Existing speech LLMs largely assess utterances without conversational context, although emotion and delivery depend on preceding turns.The paper also notes that models are increasingly expected to describe real and synthetic speech by the same standard.Source: §1 · p. 1
- Emotion, warmth, and confidence are subjective, lack definitive ground truth, and can receive different yet valid judgments across listeners.The paper motivates crowd supervision and objectives that capture both majority judgments and the distribution of listener votes.Source: §1 · p. 1
- CVAM uses synthetic-description supervised finetuning followed by reinforcement learning on human crowd judgments, and outperforms evaluated speech LLMs and single-human-vs-rest agreement.The human study covers 3k CANDOR responses.Source: §1 · p. 1
2. METHOD
CVAM predicts structured voice-aesthetic descriptions and labels from conversational context and response audio. Its data preparation combines objective acoustic measures, synthetic annotations, and crowd judgments; training first uses synthetic supervision, then aligns perceptual predictions to human votes.
2. METHOD
- CVAM predicts gender and four voice-aesthetic fields—pitch, pacing, emotion, and delivery—with an evidence-grounded description and two categorical labels for each field.Pitch and pacing are objective fields; emotion and delivery are perceptual fields judged by humans.Source: §2 · p. 2
2.1. Data Preparation
- CANDOR turns pair preceding-speaker context audio with a 3–30-second target response, then filter sensitive topics and exclude speaker or content overlap across splits.Each real response is supplemented with a voice-cloned and a voice-designed synthetic version.Source: §2.1 · p. 2
- Praat pitch, phoneme alignment, transcription, and other tool measures provide objective labels and an evidence log; CVAM receives these by default.The study also tests whether tool measures improve prediction of perceptual fields.Source: §2.1 · p. 2
- Gemini 3.1 Pro annotates 30k samples for supervised finetuning, while humans label 1.5k samples for reinforcement learning.Five raters judge each response version in pairwise comparisons; each response receives about ten votes, and perceptual labels are majority-voted.Source: §2.1 · p. 2
2.2. Two-Stage Training
- Two-stage training first uses supervised finetuning on 30k synthetic samples to establish output format and descriptions, then reinforcement learning aligns predictions with human votes.The paper motivates this sequence as addressing data scarcity in human-only training and annotator miscalibration in synthetic-only training.Source: §2.2 · p. 3
- Distribution-based rewards compare rollout predictions with human vote frequencies, aiming to match the diversity of listener judgments rather than reinforce only the majority label.The paper contrasts accuracy rewards with vote-distribution rewards and evaluates TVD-, forward-KL-, and reverse-KL-inspired alternatives.Source: §2.2 · p. 3
3. RESULTS
CVAM’s two-stage training performs better than the evaluated speech LLMs and individual listeners on human agreement, while reward choice shifts the balance between majority-label accuracy and broader class coverage.
3. RESULTS
- When inference is limited to raw audio without tools, SFT followed by GRPO achieves the best performance on most objective metrics.Tool measures essentially solve objective-label prediction and slightly improve perceptual-label prediction.Source: §3 · p. 4
- With all models initialized from the same SFT checkpoint, forward-KL GRPO reaches 50.1% macro-F1, exceeding 44.9% after continued human-label SFT and 43.5% from synthetic-data SFT alone.Source: §3 · p. 4
- Vote-fraction rewards achieve the highest accuracy at 77.8%, while forward-KL rewards achieve the highest macro-F1 at 50.1%.Both reward types exceed Gemini 3.1 Pro and single-human-vs.-rest agreement in average accuracy and F1.Source: §3 · p. 4, Table 3 · p. 4
- Voice-designed responses have the highest overall F1, voice-cloned responses the highest emotion F1, and real speech is most challenging overall.Performance varies by both response version and perceptual attribute.Source: §3 · p. 4, Table 4 · p. 4
- Synthetic SFT and subsequent GRPO each improve macro-F1 across all four perceptual labels, although some rare emotions remain difficult to identify.Aggregate gains do not imply uniform improvement for every class.Source: §3 · p. 4, Figure 4 · p. 4
4. CONCLUSION
The conclusion presents CVAM’s synthetic-SFT and human-vote-GRPO approach as a way to model voice aesthetics across real and synthetic conversational speech.
4. CONCLUSION
- CVAM combines SFT on synthetic data with GRPO on human votes and outperforms evaluated speech LLMs and individual humans in agreement with listeners.Source: §4 · p. 4
- Vote-fraction rewards favor majority-label accuracy, whereas distribution-based rewards improve macro-F1 and class coverage.Source: §4 · p. 4
- Future directions include scaling human annotations and using CVAM as a reward model for expressive speech generation and conversational voice agents.Source: §4 · p. 4