PaperlayerView exact revision on arXiv

Source-linked AI summary

RoleBreak: Benchmarking Long-Horizon Role-Playing Robustness in Spoken Dialogue

Yuqi Wang, Fengyuan Liu, Haochen Luo, Zhiqi Yu, Qi Liu

arXiv:2609.16614v1cs.CLcs.AI

TL;DR

Existing spoken role-playing benchmarks are largely character-centric and short-horizon, leaving open whether models can sustain diverse roles over extended interactions. RoleBreak benchmarks that question across semantic behavior and vocal emotion, finding semantic adherence stronger but still brittle and vocal emotion weak.

  • Problem

    Existing spoken role-playing benchmarks are largely character-centric and short-horizon, leaving open whether models sustain diverse roles over extended conversations.

    Source: §1 · p. 1

  • Method

    RoleBreak evaluates nine speech dialogue configurations on an open benchmark of 310 character-based and user-centered roles, using long-horizon scenarios and fine-grained criteria.

    Source: §1 · p. 1

  • Results

    Semantic role adherence exceeds vocal emotion, but the strongest evaluated system's first persona and safety failures occur at 10.4 and 11.6 turns on average; LLM scaling improves robustness but only marginally improves vocal emotion.

    Source: §1 · pp. 1–2, §4.3 · p. 4

  • Takeaways & Limitations

    User vocal emotion affects role-playing behavior even when linguistic content is fixed, indicating that acoustic context matters beyond words.

    Source: §5 · p. 4

Abstract

from arXiv · show

Speech-to-speech dialogue models increasingly support persona control, yet existing spoken role-playing benchmarks remain largely character-centric and short-horizon. This leaves open whether spoken dialogue models can sustain diverse roles over extended interactions, especially beyond predefined fictional characters. We introduce RoleBreak, an open benchmark for long-horizon role-playing robustness in spoken dialogue. RoleBreak contains 310 character-based and user-centered roles, 6,688 human-verified dialogue turns, and 11,743 fine-grained evaluation criteria, with 1,856 turns carrying expressive emotion targets for evaluating vocal emotion. Its scenarios are designed to stress role consistency, interaction quality, safety, and affect over extended conversations. We evaluate nine configurations spanning full-duplex, omni-modal, and cascaded ASR--LLM--TTS paradigms. We find four key patterns. First, current systems are substantially stronger at semantic role adherence than at vocal emotion. Second, semantic robustness remains brittle over long interactions: even the strongest evaluated system encounters its first persona and safety failures after only 10.4 and 11.6 turns on average. Third, scaling the LLM substantially improves semantic robustness and delays failure, but yields little improvement in vocal emotion. Finally, user vocal emotion affects role-playing behavior even when linguistic content is fixed. These findings highlight persistent gaps in both long-horizon robustness and vocal expressiveness in spoken role-playing systems.

1. INTRODUCTION

Existing spoken role-playing benchmarks are largely character-centric and short-horizon, leaving open whether models sustain diverse roles over extended conversations. RoleBreak addresses this gap and finds semantic role adherence stronger than vocal emotion, but still brittle over long interactions.

  • Existing spoken role-playing benchmarks are largely character-centric and short-horizon, while general long-horizon benchmarks do not specifically test sustained spoken role-playing.The open question is whether models can sustain diverse roles beyond predefined fictional characters as context accumulates or targeted pressure is applied.Source: §1 · p. 1
  • RoleBreak provides an open benchmark with 310 roles, 6,688 human-verified turns, 11,743 criteria, and 1,856 turns carrying expressive emotion targets.Its scenarios stress role consistency, interaction quality, safety, and affect as conversational context accumulates.Source: §1 · p. 1
  • Systems are stronger at semantic role adherence than vocal emotion, yet the strongest evaluated system's first persona and safety failures occur at 10.4 and 11.6 turns on average.Source: §1 · pp. 1–2
  • Evaluation spans nine full-duplex, omni-modal, and cascaded ASR–LLM–TTS configurations.Source: §1 · p. 1

2. RELATED WORK

Prior work covers role-playing in text and spoken dialogue, but spoken benchmarks provide limited long-horizon, context-dependent robustness testing. RoleBreak bridges these evaluation directions with targeted stress interventions.

  • Spoken role-playing benchmarks remain largely character-centric and offer limited robustness evaluation over extended, context-dependent conversations.Related long-horizon and multi-round dialogue studies examine degradation, memory, instruction retention, and safety, but not role-conditioned behavior under targeted stress.Source: §2 · p. 2
  • RoleBreak bridges these directions by testing character-based and user-centered roles with context-dependent probes and targeted stress interventions.Source: §2 · p. 2
  • A 35-turn instance example pairs fine-grained rubric criteria with accepted emotion labels.Source: Figure 2 · p. 3

3. ROLEBREAK

RoleBreak evaluates whether spoken dialogue models sustain specified roles as context accumulates and targeted pressure challenges role-specific constraints. Its construction combines curated roles, coherent multi-turn scenarios, human review, and fine-grained evaluation criteria.

3.1. Benchmark Overview

  • RoleBreak tests whether a speech-to-speech model sustains a specified role as conversational context accumulates and targeted pressure challenges role-specific constraints.Source: §3.1 · p. 3
  • Each instance contains a role specification, scenario description, and coherent multi-turn interaction with context-dependent probes and stress interventions.The benchmark contains 310 roles, 6,688 turns, and 11,743 fine-grained criteria; 1,856 turns carry expressive emotion targets.Source: §3.1 · p. 3

3.2. Benchmark Construction

  • Five annotators combine roles from five prior benchmarks, collected characters, and original user-centered personas, manually translating non-English specifications into a unified representation.Source: §3.2 · p. 3
  • Scenarios make later turns depend on earlier facts, commitments, or events; LLMs draft turns and criteria under annotator design and revision.User speech is synthesized from LibriTTS-R and VCTK with CosyVoice in a neutral style; stress interventions span identity, knowledge, agency, affect, interaction, safety, and protected information.Source: §3.2 · p. 3
  • 4.42/5 was the average quality rating, and 90.36% of retained examples were rated at least 4 by five reviewers.Reviewers assessed role quality, scenario coherence, conversational naturalness, and consistency with evaluation targets.Source: §3.2 · p. 3

4. EXPERIMENTS

Across nine configurations, the experiments find that semantic role robustness remains limited over long interactions, scaling improves semantic performance more than vocal emotion, and user vocal affect changes role-playing behavior. The evaluation also compares system paradigms and tests user-emotion effects while holding linguistic content fixed.

4.1. Experimental Setup

  • The nine configurations span full-duplex, omni-modal, and cascaded ASR–LLM–TTS paradigms; cascaded systems vary Qwen3.5 from 2B to 27B with ASR and TTS fixed.The evaluated models include PersonaPlex, four omni-modal systems, and cascaded systems using Parakeet-TDT-0.6B-v3, Qwen3.5, and Qwen3-TTS.Source: §4.1 · p. 3

4.2. Evaluation Metrics

  • Semantic criteria are judged for interaction, persona, and safety, with first-failure turns measuring the earliest failed criterion; acoustic measures cover naturalness and vocal emotion.The LLM judge reached 94.55% agreement with human judgments on 312 sampled rubric–response pairs.Source: §4.2 · p. 3

4.3. Main Results

  • Even the strongest evaluated model first fails persona at 10.4 turns and safety at 11.6, versus 21.57 turns per conversation; all nine systems score below 14.7 on output emotion.The strongest cascaded system leads interaction, persona, and safety, while Qwen3-Omni is the strongest omni-modal model overall.Source: §4.3 · pp. 3–4, Table 2 · p. 4
  • Scaling Qwen3.5 from 2B to 27B substantially improves interaction, persona, safety, and both first-failure metrics, but output emotion improves only marginally.Naturalness remains nearly unchanged when the ASR and TTS components are fixed.Source: §4.3 · p. 4

4.4. Effect of User Speech Emotion

  • Changing only user speech emotion varies interaction by 4.2 points, persona by 5.6, and safety by 4.5 across conditions.Angry speech yields the highest persona score, while happy speech yields the highest interaction and safety scores; naturalness varies by 1.3 points and output emotion by 0.5 points.Source: §4.4 · p. 4

5. CONCLUSION

RoleBreak evaluates long-horizon robustness in spoken role-playing and reveals persistent weaknesses in semantic consistency and vocal emotion. The conclusion highlights semantic gains from scaling, the influence of user vocal affect, and the benchmark's role as a testbed for spoken agents.

5. CONCLUSION

  • Current systems remain brittle in semantic role adherence over extended interactions, while vocal emotion is uniformly poor.Scaling the LLM substantially improves semantic robustness and delays failure, but provides little improvement in vocal expressiveness.Source: §5 · p. 4
  • User vocal emotion affects downstream role-playing behavior even when linguistic content is fixed, showing that acoustic context matters beyond the words themselves.RoleBreak jointly evaluates sustained role consistency and vocal behavior as a testbed for developing spoken agents.Source: §5 · pp. 4–5

Related papers

Loading 2609.16614v1…