Source-linked AI summary
CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation
Quan Tu, Shilong Fan, Zihang Tian, Rui Yan
TL;DR
Role-playing conversational agents lack a comprehensive benchmark and reliable evaluation data. CharacterEval addresses this gap with a Chinese benchmark, a human-filtered multi-turn dataset, and a thirteen-metric evaluation system, while experiments report stronger Chinese role-playing capabilities for Chinese LLMs than GPT-4. The paper also develops CharacterRM for subjective evaluation.
Problem
Role-playing conversational agents require evaluation across conversational ability, character consistency, attractiveness, and personality, but existing benchmarks and datasets are insufficiently comprehensive or reliable.
Method
The paper constructs a GPT-4-assisted, human-filtered Chinese dialogue dataset and proposes CharacterEval with thirteen metrics across four evaluation dimensions, plus the CharacterRM reward model.
Results
Chinese LLMs exhibit more promising capabilities than GPT-4 in Chinese role-playing conversation, while CharacterRM correlates with human judgments better than GPT-4.
Takeaways & Limitations
CharacterEval provides a comprehensive framework and reusable resources for assessing and comparing Chinese role-playing conversational agents.
Takeaways & Limitations
The dataset construction assumes that fidelity to source material and active human involvement are necessary for reliable role-playing evaluation.
Abstract
from arXiv · showhide
Recently, the advent of large language models (LLMs) has revolutionized generative agents. Among them, Role-Playing Conversational Agents (RPCAs) attract considerable attention due to their ability to emotionally engage users. However, the absence of a comprehensive benchmark impedes progress in this field. To bridge this gap, we introduce CharacterEval, a Chinese benchmark for comprehensive RPCA assessment, complemented by a tailored high-quality dataset. The dataset comprises 1,785 multi-turn role-playing dialogues, encompassing 23,020 examples and featuring 77 characters derived from Chinese novels and scripts. It was carefully constructed, beginning with initial dialogue extraction via GPT-4, followed by rigorous human-led quality control, and enhanced with in-depth character profiles sourced from Baidu Baike. CharacterEval employs a multifaceted evaluation approach, encompassing thirteen targeted metrics on four dimensions. Comprehensive experiments on CharacterEval demonstrate that Chinese LLMs exhibit more promising capabilities than GPT-4 in Chinese role-playing conversation. Source code, data source and reward model will be publicly accessible at https://github.com/morecry/CharacterEval.
1 Introduction
CharacterEval addresses the difficulty of systematically evaluating role-playing conversational agents with a Chinese benchmark, a quality-controlled dataset, and multidimensional assessment. The work also introduces CharacterRM and finds that existing Chinese LLMs show promising role-playing capabilities.
- CharacterEval is a Chinese benchmark designed to advance systematic evaluation of role-playing conversational agents.
- GPT-4 extracts dialogue scenes, utterances, and behaviors, after which human annotators retain high-quality dialogues and discard problematic instances.
- The evaluation covers conversational ability, character consistency, role-playing attractiveness, and personality back-testing through thirteen metrics.
- CharacterRM surpasses GPT-4 in correlation with human judgments, while Chinese LLMs show more promising Chinese role-playing capabilities than GPT-series models.
- The dataset contains 1,785 multi-turn dialogues, 11,376 examples, and 77 leading characters from diverse Chinese novels and scripts.
2 Related Work
Related work includes knowledge-based, personalized, and character-based dialogue systems, but role-playing dialogue requires jointly modeling personalized knowledge, style, behavior, and character traits. Existing character-based approaches either rely heavily on generated descriptions or lack sufficient human refinement.
- Role-playing dialogues require personalized knowledge, style, and behavior beyond the general knowledge handled by traditional knowledge-based systems.
- Character-based dialogue systems aim to mimic specific characters’ behavior and utterance style.
- Some approaches use ChatGPT to generate character profiles or dialogues, which may not accurately reflect characters’ true personalities.
- Extracting dialogues from novels, scripts, and games can preserve original character traits, but prior work lacks human-in-the-loop refinement and has limited data.
3 Problem Formulation
The RPCA generates responses by combining a character profile with the current multi-turn dialogue context. Its response includes both behavior and utterance, enabling evaluation of actions, expressions, tone, and spoken content.
- An RPCA uses a character profile and current dialogue context to generate a response consistent with the character.
- The dialogue context records alternating questions and responses across the preceding turns.
- Each generated response contains behavior and utterance components.
- Separating behavior from utterance supports fine-grained evaluation of actions, expressions, tone, and distinctive behavioral traits.
4 Data Collection
The dataset construction pipeline uses plot-aware GPT-4 extraction followed by structural filtering and human quality control. It prioritizes source fidelity, scenario diversity, multi-turn dialogue, and active human involvement.
- The construction principles require fidelity to source material, diverse scenarios, predominantly multi-turn dialogues, and human involvement.
- The data collection pipeline consists of plot division, dialogue extraction, quality filtering, and human annotation.
- Plot Division: GPT-4 identifies plot twists to divide novels and scripts into chunks representing complete plots rather than arbitrary sentence segments.
- Dialogue Extraction: GPT-4 extracts characters’ utterances, behaviors, and scenes from each plot chunk.
- Quality Filtering: The dataset preserves ABAB dialogues until a third character joins and retains dialogues exceeding five turns.
- Human Annotation: Human annotators assess dialogue coherence and quality to eliminate problematic instances caused by extraction randomness.
5 Evaluation Metric
CharacterEval evaluates role-playing conversational agents through four dimensions and thirteen metrics, covering basic dialogue quality, character consistency, attractiveness, and personality back-testing.
- Evaluation Framework: CharacterEval uses four dimensions and thirteen metrics to assess role-playing capabilities comprehensively.The dimensions are conversational ability, character consistency, role-playing attractiveness, and personality back-testing.
- Conversational Ability: Conversational ability measures fluency, coherency, and consistency in generated responses.These objectives address grammatical correctness, topic relevance, and stability across conversation turns.
- Character Consistency: Character consistency evaluates whether responses align with the character’s knowledge and persona.Knowledge consistency includes exposure, accuracy, and hallucination, while persona consistency covers behavior and utterance style.
- Role-Playing Attractiveness: Role-playing attractiveness assesses human-likeness, communication skills, expression diversity, and empathy.These criteria target the user’s experience of the RPCA during emotionally engaging conversation.
- Personality Back-Testing: Personality back-testing uses MBTI labels to assess role-playing capability through personality dimensions.Character MBTIs were collected from an archive website for this evaluation.
6 Experiment
The experiments evaluate CharacterEval across dataset properties, reward-model validation, ten LLM baselines, four dimensions, thirteen metrics, and conversation stages. Results show strong performance by specialized Chinese role-playing models, while performance generally declines as conversations become longer.
- Experimental setup: CharacterEval is split into training and test sets by examples rather than conversations, and its multi-turn design supports evaluation at longer turns.An example is a (Character, Context, Response) tuple.
- Experimental setup: Twelve subjective metrics are sparsely assigned across examples, with 12 annotators scoring model responses on a five-point scale.The human judgments support development of the CharacterRM reward model.
- Experimental setup: The experiments assess 10 open- and closed-source baselines, including specialized role-playing models and general-purpose LLMs.Open-source models are evaluated in chat versions, while closed-source models use official APIs.
- Overall performance: BC-NPC-Turbo outperforms in three of four dimensions, whereas GPT-4 is distinguished in personality back-testing.CharacterGLM receives no personality back-testing result because it cannot successfully complete the task.
- Overall performance: Specialized role-playing models such as Xingchen, MiniMax, and BC-NPC-Turbo achieve superior outcomes, while GPT-4 is less effective in Chinese role-playing conversations.InternLM-20B and Baichuan2-13B also show strong results across most dimensions despite lacking role-playing-specific customization.
- Detailed results: BC-NPC-Turbo leads conversational ability, character consistency, and role-playing attractiveness, while GPT-4 performs strongly in communication skills and personality back-testing.GPT-4 excels in knowledge exposure, and InternLM-20B leads empathy.
- Detailed results: Increasing parameter counts enhances conversational abilities within the Qwen and XVERSE model series.The comparison covers Qwen-7B versus Qwen-14B and XVERSE-7B versus XVERSE-13B.
- Robustness analysis: Most models decline as conversations progress, although InternLM-20B maintains consistent character consistency and conversational ability.GPT-4 also declines under longer conversations in complex Chinese role-playing scenarios.
7 Conclusion
The work introduces CharacterEval as a comprehensive benchmark for Role-Playing Conversational Agents, combining a high-quality multi-turn dataset, multidimensional evaluation, and a character-based reward model. Experiments indicate that Chinese LLMs show more promising Chinese role-playing capabilities than GPT-4.
- CharacterEval evaluates recent Role-Playing Conversational Agents with a comprehensive benchmark.
- The authors extract dialogues from novels and scripts with GPT-4, apply strict human filtering, and release a high-quality multi-turn role-playing dataset.
- The benchmark combines multidimensional RPCA evaluation with human annotations used to train a character-based reward model for convenient re-implementation.
- Chinese LLMs show more promising capabilities than GPT-4 in Chinese role-playing conversation.
A Evaluation Result by GPT-4
This appendix reports GPT-4-based evaluation results for CharacterEval, using a 2-shot setting for reference. The table covers subjective metrics across three dimensions, while personality back-testing lacks a CharacterGLM score.
- GPT-4 evaluation is reported in a 2-shot setting for reference because GPT-4 has lower correlation with human judgment.
- Table 5 reports 12 subjective metrics spanning conversational ability, character consistency, and role-playing attractiveness.
- CharacterGLM has no available score for personality back-testing and is represented by a dash in the table.