PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning Does Not Necessarily Improve Role-Playing Ability

Xiachong Feng, Longxu Dou, Lingpeng Kong

arXiv:2502.16940v2cs.CLcs.AI

TL;DR

The paper asks whether reasoning techniques can improve LLM role-playing, as applications expand and reasoning capabilities advance. Across six benchmarks, 24 models, and three approaches, it finds that reasoning does not necessarily improve role-playing and identifies future research directions.

  • Problem

    The paper asks whether reasoning techniques enhance LLM role-playing, amid expanding applications and advancing reasoning capabilities.

    Source: §1 · p. 1

  • Method

    The study compares zero-shot, CoT, and reasoning-optimized role-playing across six benchmarks using 24 LLMs.

    Source: §1 · p. 2, §3.4 · p. 3

  • Results

    The findings include that CoT may reduce role-playing performance, reasoning-optimized LLMs are unsuitable, reasoning disrupts scaling, and Chinese performance surpasses English performance.

    Source: §7 · p. 8

  • Takeaways & Limitations

    The authors propose Role-Aware CoT and reinforcement learning as future research directions for role-playing LLMs.

    Source: §1 · p. 2

    The selected models and benchmarks may be incomplete, potentially biasing findings and limiting their robustness and generalizability.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The application of role-playing large language models (LLMs) is rapidly expanding in both academic and commercial domains, driving an increasing demand for high-precision role-playing models. Simultaneously, the rapid advancement of reasoning techniques has continuously pushed the performance boundaries of LLMs. This intersection of practical role-playing demands and evolving reasoning capabilities raises an important research question: "Can reasoning techniques enhance the role-playing capabilities of LLMs?" To address this, we conduct a comprehensive study using 6 role-playing benchmarks, 24 LLMs, and 3 distinct role-playing strategies, comparing the effectiveness of direct zero-shot role-playing, role-playing with Chain-of-Thought (CoT), and role-playing using reasoning-optimized LLMs. Our findings reveal that CoT may reduce role-playing performance, reasoning-optimized LLMs are unsuitable for role-playing, reasoning ability disrupts the role-playing scaling law, large models still lack proficiency in advanced role-playing, and Chinese role-playing performance surpasses English role-playing performance. Furthermore, based on extensive experimental results, we propose two promising future research directions: Role-aware CoT for improving role-playing LLMs and Reinforcement Learning for role-playing LLMs, aiming to enhance the adaptability, consistency, and effectiveness of role-playing LLMs for both research and real-world applications.

1 Introduction

Role-playing LLMs are increasingly relevant in academic and commercial applications, while reasoning capabilities are advancing. The paper asks whether reasoning techniques can enhance LLM role-playing and investigates the question across benchmarks and models.

1 Introduction

  • The study asks whether reasoning techniques can enhance LLMs’ role-playing capabilities, a question prompted by expanding applications and advancing reasoning capabilities.The paper identifies potential relevance to future applications of reasoning techniques in role-playing LLMs.Source: §1 · p. 1
  • Experiments compare direct zero-shot role-playing, role-playing with CoT, and reasoning-optimized LLMs across six benchmarks and 24 models.The benchmarks cover multiple-choice and text-generation tasks in English and Chinese, using traditional metrics and LLM-as-a-Judge methods.Source: §1 · p. 2
  • The reported findings are that CoT may reduce performance, reasoning-optimized LLMs are unsuitable, reasoning disrupts the scaling law, and Chinese performance exceeds English performance.The findings also include Qwen-series suitability and persistent difficulty with advanced role-playing in large models.Source: §1 · p. 2
  • The study proposes Role-Aware CoT and reinforcement learning as future research directions for role-playing LLMs.It also integrates datasets and experimental methods into OpenCompass to provide one-click execution and a standardized framework and codebase.Source: §1 · p. 2

2 Evaluation Framework

The framework evaluates role-playing performance by conditioning an LLM on a character profile and task, then assessing its output with a metric. It also incorporates reasoning techniques to examine their impact.

2 Evaluation Framework

  • The evaluation setup comprises an LLM, character profile, task, metric, and, when applicable, a reference answer.The study uses this framework to examine whether reasoning techniques enhance role-playing capabilities.Source: §2 · p. 2
  • The model generates responses conditioned on the character profile and task, and the metric assesses alignment with the intended role.When no reference answer is available or needed, the metric can use alternatives such as LLM-as-a-Judge.Source: §2 · p. 2
  • The framework scores role-playing as S = E(M(P, T, R), A), with higher S indicating stronger role-playing under the specified conditions.R represents incorporated reasoning techniques; A may be unavailable or unnecessary.Source: §2 · p. 2

3 Experimental Setting

The experimental setting combines six role-playing benchmarks, 24 models, multiple evaluation metrics, and three approaches to role-playing. The benchmarks span varied tasks and languages, and the methods include direct, CoT, and reasoning-optimized approaches.

3 Experimental Setting

  • The experiments use six benchmarks, 24 models, and three role-playing approaches, with relevant code integrated into OpenCompass for replication.The models include open-source and proprietary systems; the approaches are direct zero-shot, CoT, and reasoning-optimized LLMs.Source: §3 · p. 2, §3.2 · p. 3, §3.4 · p. 3

3.1 Role-playing Benchmarks

  • The six benchmarks cover varied role-playing tasks, including response generation, motivation recognition, social knowledge and style, and personality assessment.The benchmark set includes English and Chinese tasks and both multiple-choice and generation formats.Source: §3.1 · p. 3

3.2 Backbone LLMs

  • The study includes 24 models, comprising two closed-source and 22 open-source models from six companies.The selection includes GPT-4-Turbo and DeepSeek-R1.Source: §3.2 · p. 3

3.3 Evaluation Metrics

  • Evaluation combines automated metrics such as ROUGE, Accuracy, and Exact Match with pretrained-reward-model and prompting-based LLM-as-a-Judge scoring.The pretrained reward model scores specified dimensions, while prompting-based scoring generates scores using prompt engineering and LLM knowledge.Source: §3.3 · p. 3

3.4 Role-playing Methods

  • The three methods are direct zero-shot responses, explicit step-by-step CoT before role-playing, and reasoning-optimized models that reason autonomously.Figure 1 ranks models by scale horizontally, shows each dataset’s metric vertically, and uses color to indicate zero-shot versus CoT performance.Source: §3.4 · p. 3, Figure 1 · p. 4

4 Results and Findings

Across six benchmarks, the experiments compare zero-shot role-playing, CoT, and reasoning-optimized models. Results show that reasoning can weaken role-playing, while model scale and language also shape performance.

4.1 CoT May Reduce Role-Playing Performance

  • CoT more often degraded performance on CroSS-MR, HPD, SocialBench, and CharacterEval, though it improved performance on InCharacter and RoleBench.The authors attribute degradation primarily to attention diversion and linguistic style drift: reasoning competes with role-playing focus and can produce a formal style unlike vivid, character-consistent role-play.Source: §4.1 · p. 4

4.2 Reasoning-optimized LLMs Are Unsuitable for Role-Playing

  • Reasoning-optimized models, including o1-Mini and DeepSeek-R1, generally perform poorly at role-playing, and reasoning-distilled models fare worse than their originals.The authors connect this pattern to rational reasoning undermining emotional and intuitive aspects needed for role-playing.Source: §4.2 · pp. 4–5, Figure 2 · p. 5

4.3 Reasoning Ability Disrupts the Role-Playing Scaling Law

  • Five benchmarks generally show stronger role-playing with larger models, but scaling gains are modest and inconsistent, and reasoning makes the trend less predictable.InCharacter is the exception to the general scaling pattern; CoT and reasoning-optimized models are associated with greater fluctuations and variability across tasks and datasets.Source: §4.3 · p. 5

4.4 The Qwen Series Is Well-Suited for Role-Playing Tasks

  • Qwen2.5 outperforms Llama, Gemma, and Mistral across size ranges on role-playing accuracy, persona consistency, linguistic expressiveness, and contextual coherence.The authors recommend Qwen2.5-7B-Instruct as a cost-effective, balanced option for performance, computational efficiency, and adaptability.Source: §4.4 · pp. 5–6, Figure 3 · p. 6

4.5 Chinese Role-Playing Performance Surpasses English Role-Playing Performance

  • Across multiple benchmarks and metrics, current LLMs show stronger role-playing performance in Chinese than in English.The authors hypothesize that generalized character information internalized through extensive English training can interfere with context-sensitive role-playing.Source: §4.5 · p. 7

4.6 Large Models Still Lack Proficiency in Advanced Role-Playing

  • Current LLMs remain deficient in advanced role-playing dimensions, including character knowledge exposition, personality expression, and linguistic diversity.The analysis uses CharacterEval’s pretrained reward model to score 12 dimensions; direct zero-shot role-playing outperforms other approaches in most cases.Source: §4.6 · p. 7, Figure 4 · p. 7

5 Related Works

Prior work has developed LLM role-playing applications and evaluation, while reasoning methods have advanced especially on mathematics and coding. This paper examines the intersection of those strands.

  • LLM-based role-playing has opened possibilities in social simulation and applications such as psychological counseling, companionship, and game NPCs (Chen et al., 2024b; Tseng et al., 2024; Mou et al., 2024).Source: §5 · p. 8
  • Chain-of-thought, tree-of-thought, and o1-style reasoning methods have advanced LLM capabilities, particularly on mathematics and coding (Wei et al., 2022; Yao et al., 2024; Jaech et al., 2024; Guo et al., 2025; Team et al., 2025).Source: §5 · p. 8
  • Against prior research focused mainly on role-playing applications and evaluation, this study asks whether reasoning techniques enhance role-playing LLM capabilities.Source: §5 · p. 8

6 Future Directions

The paper proposes two directions for combining reasoning with role-playing: persona-aware reasoning and reward-designed reinforcement learning. Both aim to make reasoning more role-specific.

Role-aware Chain-of-Thought (CoT) for Improving Role-playing LLMs

  • Role-aware CoT would incorporate persona attributes, narrative constraints, and character perspectives, with dynamic memory retaining persona information across turns.The proposed approach aims to align deductions with predefined character traits and reduce breaking character or inconsistent reasoning; a historical figure should reason within the knowledge and biases of their era.Source: §6 · p. 8

Reinforcement Learning for Role-playing LLMs

  • Role-playing reinforcement learning could use carefully designed rewards to develop intrinsic, role-specific reasoning while accounting for task accuracy.The reward should discourage shortcuts and encourage authentic character-driven reasoning and decision-making.Source: §6 · p. 8

7 Conclusion

The study tests whether reasoning improves role-playing using six benchmarks, 24 LLMs, and three methods. It finds that reasoning does not necessarily help, and identifies limitations and future research directions.

  • Experiments across six role-playing benchmarks, 24 LLMs, and three role-playing methods address whether reasoning enhances LLM role-playing.Source: §7 · p. 8
  • Findings include possible CoT degradation, poor suitability of reasoning-optimized models, disrupted scaling, Qwen strengths, stronger Chinese than English performance, and remaining advanced-role-playing deficiencies.Source: §7 · p. 8
  • The code is integrated into OpenCompass to support reproducibility and further research.Source: §7 · p. 8

Limitations

The study’s model and benchmark selection may be incomplete, limiting the robustness and generalizability of its findings. It also uses different metrics across role-playing tasks.

Limitations

  • The model and benchmark selection may omit niche models or specialized benchmarks, potentially biasing findings and limiting their robustness and generalizability.The authors call for broader model coverage, additional benchmark evaluations, and more extensive validation.Source: Limitations · p. 9

B Detailed Metrics

  • Accuracy evaluates multiple-choice tasks in Cross-MR and SocialBench, while ROUGE evaluates text generation in HPD and RoleBench.Source: B Detailed Metrics · p. 10
  • Exact Match assesses alignment of key memory points in the SocialBench Generation Task.Source: B Detailed Metrics · p. 10
  • InCharacter uses LLM-as-a-Judge prompting to transform model responses into standardized answers.Source: B Detailed Metrics · p. 10
  • CharacterEval uses an LLM-as-a-Judge reward model to score responses across 12 evaluation dimensions.Source: B Detailed Metrics · p. 10

C Detailed Results on Role-palying Benchmarks

Tables 4–8 report results for three role-playing methods across six benchmarks; Table 3 lists the evaluated models, including open and reasoning-optimized models.

C Detailed Results on Role-palying Benchmarks

Closed-source

D Fine-Grained Results on CharacterEval

CharacterEval results are reported in fine-grained tables for three role-playing methods, with scores across 12 dimensions supplied by a pretrained reward model.

E Limitations of Reinforcement Learning for Role-Playing Tasks

A focused analysis of DeepSeek-R1 highlights tensions between reasoning-focused optimization and role-playing requirements. It identifies conflicts involving persona, style, reward design, and test-time computation.

E Limitations of Reinforcement Learning for Role-Playing Tasks

Loading 2502.16940v2…