Source-linked AI summary
VStyle: A Benchmark for Voice Style Adaptation with Spoken Instructions
Jun Zhan, Mingyang Han, Yuxuan Xie, Chen Wang, Dong Zhang, Kexin Huang, Haoxiang Shi, DongXiao Wang, Tengtao Song, Qinyuan Cheng, Shimin Li, Jun Song, Xipeng Qiu, Bo Zheng
TL;DR
Spoken-language models have focused more on semantic accuracy than on adapting speaking style, and comprehensive frameworks for evaluating expressive dimensions remain limited. This paper introduces a bilingual benchmark and a staged audio-language-model judge; experiments show substantial performance differences across systems, task categories, and languages.
Problem
SLM research has focused more on semantic accuracy than expressiveness, while comprehensive evaluation frameworks for expressive dimensions remain lacking.
Source: §1 · p. 1Method
The paper introduces a bilingual benchmark for voice-style adaptation and a judge that progressively assesses content faithfulness, style consistency, and naturalness.
Source: §1 · p. 1Results
Commercial models clearly outperform open-source systems; GPT-4o achieves the best English-task overall performance at 4.05, with performance also varying across task categories and languages.
Source: §3.2 · pp. 3–4Takeaways & Limitations
VStyle provides a bilingual, multi-category diagnostic for model shortcomings, and LALM-as-a-Judge shows strong correlation with human assessment.
Source: §4 · p. 5The instruction distribution reflects annotator preferences and model-driven patterns that may diverge from real user demands.
Source: §4 · p. 5
Abstract
from arXiv · showhide
Spoken language models (SLMs) have emerged as a unified paradigm for speech understanding and generation, enabling natural human machine interaction. However, while most progress has focused on semantic accuracy and instruction following, the ability of SLMs to adapt their speaking style based on spoken instructions has received limited attention. We introduce Voice Style Adaptation (VSA), a new task that examines whether SLMs can modify their speaking style, such as timbre, prosody, or persona following natural language spoken commands. To study this task, we present VStyle, a bilingual (Chinese & English) benchmark covering four categories of speech generation: acoustic attributes, natural language instruction, role play, and implicit empathy. We also introduce the Large Audio Language Model as a Judge (LALM as a Judge) framework, which progressively evaluates outputs along textual faithfulness, style adherence, and naturalness, ensuring reproducible and objective assessment. Experiments on commercial systems and open source SLMs demonstrate that current models face clear limitations in controllable style adaptation, highlighting both the novelty and challenge of this task. By releasing VStyle and its evaluation toolkit, we aim to provide the community with a foundation for advancing human centered spoken interaction. The dataset and code are publicly available at \href{https://junzhan2000.github.io/VStyle.github.io/}{project's homepage}.
1. INTRODUCTION
Spoken-language-model research has emphasized what models say over how they say it, while comprehensive evaluation of expressive dimensions remains lacking. The paper introduces a bilingual voice-style-adaptation benchmark and a staged audio-language-model judge, then reports clear performance differences across systems.
1. INTRODUCTION
- Expressive dimensions such as speaker identity, emotion, and paralinguistic signals remain under-evaluated despite their importance to natural dialogue and user experience.Most SLM research instead emphasizes semantic accuracy over expressiveness.Source: §1 · p. 1
- VSA tests whether an SLM can change speaking style in response to spoken instructions; VStyle covers four categories with 1,523 Chinese- and English-language prompts.The categories are acoustic attributes, natural-language instructions, role-play, and implicit empathy.Source: §1 · p. 1
- The LALM-as-a-Judge framework progressively evaluates content faithfulness, style consistency, and naturalness to provide scalable, reproducible automatic assessment.It addresses the challenge of reliable quantitative evaluation when human assessment is costly and variable.Source: §1 · p. 1
- Tests on commercial and open-source SLMs show that VStyle distinguishes voice-style-adaptation performance and reveals a significant gap across systems.The dataset and toolkit are released to support progress toward expressive, controllable, human-centered spoken interaction.Source: §1 · p. 1
2. VSTYLE
VStyle evaluates whether spoken responses align with both instruction intent and desired speaking style, across four categories and two languages. It constructs its instructions through a hybrid human–LLM process and uses a staged LALM judge to score content, style, and naturalness.
2.1. Overview
- The benchmark pairs spoken intent with desired explicit or implicit style across acoustic controls, open-ended instructions, role-play, and empathy.Acoustic attributes include age, gender, rate, pitch, loudness, and emotion; open-ended instructions include emotion, style, and within-utterance variation.Source: §2.1 · p. 2
2.2. Data Construction
- The 1,523 bilingual instructions combine manually designed seeds, iterative LLM expansion, synthesized audio, and filtering of erroneous or ambiguous samples.Fixed instruction patterns were used to reduce ambiguity in spoken language.Source: §2.2 · p. 2
2.3. Large Audio-Language Model as Judge
- LALM judging gates a five-point MOS from textual adherence to style compliance and then naturalness, with final benchmark scores averaging categories equally.Content failure scores 1; style non-compliance or partial compliance scores 2 or 3, while naturalness scores 4 or 5.Source: §2.3 · pp. 2–3, Figure 2 · p. 3
3. EXPERIMENT
Commercial systems substantially outperform open-source systems on VStyle, while results vary by task and language. LALM-as-a-Judge scores show near-human consistency in both English and Chinese.
3.1. Experimental Setup
- The evaluation covered GPT-4o Audio, GPT-4o-Mini Audio, and Doubao commercially, alongside Step-Audio, Kimi-Audio, Baichuan-Audio, and Qwen-2.5 Omni.Gemini-2.5-pro served as the evaluator; fixed-speaker models were assigned one speaker per dialogue session, and Baichuan-Audio ran without a voice prompt.Source: §3.1 · p. 3
3.2. Results and Analysis
- 4.05 for GPT-4o in English and 4.10 for Doubao in Chinese led overall; open-source models generally scored 2–3 points, with Kimi-Audio best in Chinese at 3.11 and Step-Audio leading in English at 2.77.The authors attribute the gap partly to open-source models’ weaker expressive generation and instruction-following, while Baichuan-Audio showed stronger age control and some timbre control.Source: §3.2 · p. 3, Table 1 · p. 4
- Composite acoustic tasks were harder than single-attribute tasks, and most models struggled with style variation, though GPT-4o performed strongly across English instruction-following subtasks.GPT-4o did particularly well on the most difficult variation type.Source: §3.2 · p. 3
- GPT-4o performed well in scene and character role-play, Doubao was strong on Chinese scene tasks, and several models handled positive and negative emotions effectively in implicit empathy.Source: §3.2 · pp. 3–4
- Doubao, Kimi-Audio, and Qwen2.5-Omni performed better in Chinese than English, whereas the GPT-4o series showed the reverse pattern.The authors suggest language-data imbalance or pronunciation differences may contribute to the cross-language variation.Source: §3.2 · p. 4
3.3. Evaluation Consistency Analysis
- Model–consensus human score correlations reached 77.01% for English and 73.03% for Chinese, near inter-human agreement levels.The consistency analysis used five expert raters per instance on a random sample of about half the instructions.Source: §3.3 · p. 4, Table 2 · p. 4
4. LIMITATION AND CONCLUSION
VStyle’s instruction data may reflect annotator preferences and model-generated patterns rather than real user demands, and its audio-language evaluators can hallucinate. The authors report that prompting and guidelines improve consistency, while human-score correlations support the framework’s scalability and practicality.
- The instruction set is built from manually designed seeds and LLM expansion, so its distribution may diverge from real user demands; LALMs can also hallucinate.Step-by-step prompts and explicit guidelines help improve evaluation consistency and reliability.Source: §4 · p. 5
- The authors position VStyle as a foundation for voice style adaptation, with bilingual, multi-category coverage and model–human score correlations supporting its diagnostic and practical roles.They describe it as a diagnostic tool for model shortcomings and a catalyst for more natural, controllable, human-centered speech generation.Source: §4 · p. 5