PaperlayerView exact revision on arXiv

Source-linked AI summary

ActorMind: Emulating Human Actor Reasoning for Speech Role-Playing

Xi Chen, Wei Xue, Yike Guo

arXiv:2604.11103v2cs.SDcs.AI

TL;DR

Existing role-playing research is concentrated in text, leaving speech-based role-specific and spontaneous interaction insufficiently addressed. The paper defines speech role-playing, introduces the hierarchical ActorMindBench benchmark, and proposes ActorMind’s theater-inspired multi-agent reasoning framework. Experiments on ActorMindBench demonstrate ActorMind’s effectiveness, while the benchmark remains limited in role and domain coverage.

  • Problem

    Existing role-playing work is largely confined to text, while speech is important for conveying emotion and information and current speech models struggle with role-specific spontaneous interaction.

  • Method

    The paper introduces ActorMindBench and ActorMind, an off-the-shelf multi-agent chain-of-thought framework using Eye, Ear, Brain, and Mouth agents for speech role-playing.

  • Results

    ActorMind demonstrates effectiveness against baseline models on ActorMindBench and produces spontaneous speech across different speech generation models.

  • Takeaways & Limitations

    ActorMindBench and ActorMind provide a benchmark and reasoning framework for studying speech role-playing without additional training.

  • Takeaways & Limitations

    ActorMindBench is derived entirely from Friends Season 1, limiting its roles and domain coverage to six characters in an urban comedy setting.

Abstract

from arXiv · show

Role-playing has garnered rising attention as it provides a strong foundation for human-machine interaction and facilitates sociological research. However, current work is confined to textual modalities, neglecting speech, which plays a predominant role in daily life, thus limiting genuine role-playing. To bridge this gap, we conceptualize and benchmark speech role-playing through ActorMindBench, and we present a corresponding reasoning framework, called ActorMind. Specifically, (1) Speech Role-Playing enables models to deliver spontaneous responses with personalized verbal traits based on their role, the scene, and spoken dialogue. (2) ActorMindBench is a hierarchical benchmark comprises Utterance-Level content with 7,653 utterances, Scene-Level content with 313 scenes, and Role-Level content with 6 roles. (3) ActorMind is an off-the-shelf, multi-agent, chain-of-though style reasoning framework that emulates how human actors perform in theaters. Concretely, ActorMind first reads its assigned role description via Eye Agent, then comprehends emotional cues within contextual spoken dialogues through Ear Agent. Subsequently, Brain Agent generates a descriptive emotional state, and finally, Mouth Agent delivers the scripts infused with corresponding emotion state. Experimental results demonstrate the effectiveness of ActorMind in enhancing speech role-playing.

1 Introduction

The paper extends role-playing from text to speech, defining speech role-playing around personalized, spontaneous verbal behavior. It introduces ActorMindBench and ActorMind, a multi-agent reasoning framework inspired by theatrical performance.

  • Existing role-playing benchmarks and methods primarily focus on text, despite speech conveying emotions and attitudes directly in daily interaction.
  • The paper conceptualizes speech role-playing, proposes the public ActorMindBench benchmark, and introduces ActorMind as a theater-inspired multi-agent method.
  • Speech role-playing generates spontaneous responses with personalized verbal attributes conditioned on roles, scenes, and spoken dialogue.
  • ActorMind coordinates Eye, Ear, Brain, and Mouth agents to read role information, interpret speech emotion, reason about emotion states, and deliver expressive scripts.
  • ActorMindBench contains 7,653 utterances, 313 scenes, and 6 roles across hierarchical utterance-, scene-, and role-level content.

2 Related Works

Prior role-playing research largely centers on text, while speech models still struggle to combine role-specific speaking styles with spontaneous, coherent responses. ActorMind extends chain-of-thought reasoning into speech through coordinated agents that connect role, scene, dialogue, and emotion.

  • Most role-playing work uses text-based LLM prompting or training, with some extension into language-vision settings but not speech-focused role-playing.
  • LLAMs support multimodal reasoning and instruction following, whereas TTS models support voice cloning and style transfer; both have complementary limitations for role-playing.
  • Speech generation models generally struggle to adopt role-specific speaking styles and respond spontaneously and coherently to dynamic scenes and dialogues.
  • ActorMind extends chain-of-thought reasoning to speech role-playing through an eye-ear-brain-mouth process inspired by theatrical acting.
  • Its coordinated agents use speech recognition and emotion captioning for perception, an LLM for emotion reasoning, and RAG for emotion-conditioned script delivery.

3 ActorMindBench

ActorMindBench is a hierarchical, authenticity-oriented benchmark built from Friends Season 1, organizing speech role-playing data at utterance, scene, and role levels. Its construction pipeline extracts speech segments, aligns scene boundaries and descriptions, and creates character profiles.

  • ActorMindBench organizes role-playing data into Utterance-Level, Scene-Level, and Role-Level content.
  • The benchmark uses human-written Friends Season 1 material to preserve consistent personas, stable character knowledge, and natural dialogue.
  • Construction Pipeline: The utterance pipeline denoises audio, diarizes speakers, and recognizes speech to produce labeled segments with textual content.
  • Construction Pipeline: Scene construction aligns online-script boundaries with utterance context and uses Llama3 to generate descriptive captions from dialogue.
  • Construction Pipeline: Role-Level profiles cover Rachel, Monica, Phoebe, Joey, Chandler, and Ross, summarized from Wikipedia pages using Llama3.
  • Its Utterance-Level data contains 7,653 utterances spanning 5 hours and 15 minutes, while Scene-Level data contains 313 scenes averaging 28.7 utterances and 4.23 roles per scene.

4 ActorMind

ActorMind models speech role-playing as a multi-agent reasoning process that combines role and scene context with spoken emotional cues. Its agents infer an emotional state and use retrieved expressive speech to deliver the next line.

  • 4.1 Notation: ActorMind represents utterances with role indicators, speech signals, and corresponding textual content, while scenes include descriptions and boundaries.Role profiles provide textual role information, and scene boundaries indicate utterance start and end points.
  • 4.2 Overview: Given a role, scene description, and preceding dialogue, the model must spontaneously perform the next target line orally.
  • 4.3 Eye Agent: The Eye Agent reads the role profile, scene description, and contextual dialogue before role-playing begins.It retains this preparatory information in memory.
  • 4.4 Ear Agent: The Ear Agent extracts speech tones and emotional descriptions from preceding dialogue using Speech Emotion Captioning tools.These textual emotional descriptions are logged for subsequent reasoning.
  • 4.5 Brain Agent / 4.6 Mouth Agent: The Brain Agent infers the next line’s emotional state from the role, scene, dialogue context, and perceived emotional cues.The Mouth Agent then uses RAG to retrieve a speech segment with a similar emotional state and renders the target text with its voice and tone.

5 Experiment

The experiments evaluate ActorMindBench and speech role-playing with data from Friends Season 1 using RP-MOS. The evaluation covers perceived quality, exact delivery, emotion expression, and comparisons with six baseline methods.

  • 5.1 Dataset: ActorMindBench uses Friends Season 1, training and deploying on episodes 1–10 and 15–24 while reserving episodes 11–14 for testing.The split spans relatively neutral to higher-intensity emotional expressions.
  • 5.2 Evaluation Metric: RP-MOS rates generated speech from 1 to 5 across Exact Delivery and Emotion Expression.Exact Delivery assesses character-voice impersonation and target-word articulation, while Emotion Expression uses prosodic cues such as tone, tempo, and intensity.
  • 5.2 Evaluation Metric: Exact Delivery is treated as a prerequisite: speech that fails to resemble the intended character’s voice or convey the correct content receives a score of 1.
  • 5.3 Baselines: ActorMind is evaluated against six baseline methods spanning one multimodal language model and multiple text-to-speech models.The listed baselines include Qwen_Omni, CosyVoice, SparkTTS, YourTTS, and F5-TTS.
  • 5.4 Results: Subjective evaluation using RP-MOS is reported in Table 1 for ActorMind and the baseline models.

6 Results and Analysis

ActorMind outperforms baseline LLM and TTS models in subjective speech role-playing evaluation, while ablations and cross-model tests assess its component importance and generality. Spectrogram comparisons further indicate more role-consistent prosody and expression.

  • 6.1 Main Result: ActorMind outperforms all baseline LLM and TTS models on average across roles in subjective RP-MOS evaluation.The evaluation uses subjective RP-MOS scores, while the supplied passages do not provide the numerical averages.
  • 6.2 Ablation Study: Removing any role profile, scene, or context input degrades performance, with role profile removal causing the largest drop.The ablation study identifies role profile information as the most critical component for role-conditioned speech generation.
  • 6.2 Ablation Study: Each ActorMind component is reported as necessary because removing components disrupts the sequential reasoning pipeline and reduces performance.Removing Brain also disables Mouth-agent retrieval-augmented generation, illustrating interdependence among agents.
  • 6.3 Generalization of ActorMind: Across speech generators, all but two ActorMind-plus-model configurations score above 0.5 relative to their baselines, and five score 1.The exceptions are ActorMind + CosyVoice on Ross and ActorMind + IndexTTS on Chandler.
  • 6.4 Qualitative Analysis: ActorMind produces more ground-truth-like spectrograms than randomly prompted TTS baselines, indicating more accurate role-consistent prosody and expression.The spectrogram comparison uses time on the x-axis and frequency-bin energy on the y-axis; Qwen_Omni fails to reproduce the target voice.

7 Conclusion

The paper formalizes speech role-playing, introduces a public benchmark and an off-the-shelf multi-agent reasoning framework, and evaluates ActorMind against baseline models. Results support its effectiveness and its use across different speech generation models.

  • 7 Conclusion: The paper introduces ActorMindBench and ActorMind to establish speech role-playing as a benchmarked task with an off-the-shelf reasoning framework.ActorMind can be applied without additional training.
  • 7 Conclusion: Experiments show ActorMind is effective against baseline models, generalizes across speech generation models, and produces spontaneous speech in qualitative spectrogram analyses.The conclusion combines benchmark comparisons, cross-model experiments, and qualitative analysis.

8 Limitations

ActorMindBench has limited role and domain coverage because it is derived from Friends Season 1, while ActorMind is training-free and may benefit from further training. Despite these constraints, the authors report strong performance and view ActorMind as a meaningful first step.

  • ActorMindBench covers only six roles within the urban comedy domain because it is derived entirely from Friends Season 1.
  • ActorMind is an off-the-shelf method that does not require training, but further training could improve its Mouth-agent retrieval and Brain-agent emotion reasoning.
  • The authors report strong performance from ActorMind and describe it as a meaningful step forward as the first system of its kind.

9 Ethical Considerations

The benchmark uses copyrighted Friends Season 1 material but distributes only structured annotations, not the original audio. Researchers must obtain episodes independently through legitimate channels.

  • ActorMindBench is built from Friends Season 1, whose copyrighted audio content the authors do not distribute.
  • The release contains only structured annotation files, allowing researchers to obtain the original episodes independently through legitimate channels.
  • The authors frame this copyright-safe release design as supporting reproducibility and extensibility without redistributing copyrighted media.

A.1 ActorMindBench Example Data

Appendix A.1 illustrates ActorMindBench data and identifies accompanying prompts and statistics for scene, role, utterance, and scene-level information.

  • Figure 4 provides an example of ActorMindBench data.
  • Figures 5 and 6 show the prompts used for scene summarization and role summarization, respectively.
  • Table 4 reports utterance-level counts and durations for a role in an episode.
  • Table 5 reports scene-level statistics, including average utterance and role counts per scene for each episode.

B Experiment Details

The experiment details define the speech role-playing prompt, benchmark statistics, human evaluation procedure, scoring criteria, and ActorMind’s agent components and retrieval-based speech generation.

  • Prompts and benchmark data: Figure 7 presents the Qwen_Omni prompt used for speech role-playing.
  • Human evaluation: Ten English speakers evaluated twelve rounds covering six roles, with two sets per role, using randomly selected utterances across model variants.
  • Human evaluation: Evaluators rated generated speech against reference recordings using emotional consistency, intensity alignment, naturalness, realism, and overall impression.
  • Scoring: Different voices or mismatched text content received a score of 1, while otherwise evaluators assigned a single integer from 1 to 5 after listening at least twice.
  • ActorMind agents: ActorMind uses Eye, Ear, and Brain agents to retain preparatory content, caption speech emotions, and reason about emotional states.
  • ActorMind agents: The Mouth Agent retrieves role-specific speech examples indexed by SECAP emotion descriptions and uses retrieved speech as the tone and emotion prompt for IndexTTS generation.
Loading 2604.11103v2…