Source-linked AI summary
CharacterBench: Benchmarking Character Customization of Large Language Models
Jinfeng Zhou, Yongkang Huang, Bosi Wen, Guanqun Bi, Yuxuan Chen, Pei Ke, Zhuang Chen, Xiyao Xiao, Libiao Peng, Kuntian Tang, Rongsheng Zhang, Le Zhang, Tangjie Lv, Zhipeng Hu, Hongning Wang, Minlie Huang
TL;DR
Existing benchmarks often cover few character categories or dimensions, and open-ended responses may not express the features being evaluated. CHARACTERBENCH provides 22,859 human-annotated samples across 11 dimensions and 6 aspects, using tailored queries to elicit responses related to each dimension. CharacterJudge outperforms compared automatic judges in correlation with human scores, with 42% and 36% improvements on AVG. over suboptimal GLM-4-TG in bilingual evaluations.
Problem
Existing benchmarks often cover few character categories or dimensions, and open-ended responses may not express the features being evaluated.
Source: Introduction · pp. 1–2Method
CHARACTERBENCH provides 22,859 human-annotated samples across 11 dimensions and 6 aspects, using tailored queries to elicit responses related to each dimension.
Source: Introduction · p. 2Results
CharacterJudge outperforms compared automatic judges in correlation with human scores, with 42% and 36% improvements on AVG. over suboptimal GLM-4-TG in bilingual evaluations.
Source: Evaluation on CharacterJudge · p. 6Takeaways & Limitations
The benchmark shows potential for optimizing LLM character customization, and CharacterJudge offers a cost-effective and stable alternative to automatic judges.
Source: Conclusions · p. 8
Abstract
from arXiv · showhide
Character-based dialogue (aka role-playing) enables users to freely customize characters for interaction, which often relies on LLMs, raising the need to evaluate LLMs' character customization capability. However, existing benchmarks fail to ensure a robust evaluation as they often only involve a single character category or evaluate limited dimensions. Moreover, the sparsity of character features in responses makes feature-focused generative evaluation both ineffective and inefficient. To address these issues, we propose CharacterBench, the largest bilingual generative benchmark, with 22,859 human-annotated samples covering 3,956 characters from 25 detailed character categories. We define 11 dimensions of 6 aspects, classified as sparse and dense dimensions based on whether character features evaluated by specific dimensions manifest in each response. We enable effective and efficient evaluation by crafting tailored queries for each dimension to induce characters' responses related to specific dimensions. Further, we develop CharacterJudge model for cost-effective and stable evaluations. Experiments show its superiority over SOTA automatic judges (e.g., GPT-4) and our benchmark's potential to optimize LLMs' character customization. Our repository is at https://github.com/thu-coai/CharacterBench.
Introduction
Existing character-customization benchmarks are limited by narrow character coverage and dimensions, while open-ended dialogue often leaves evaluated features unexpressed. CHARACTERBENCH addresses these issues with tailored dimension-specific queries, a large bilingual benchmark, and CharacterJudge.
Introduction
- Existing benchmarks often lack diverse characters and comprehensive dimensions, limiting robust evaluation and insight into LLM generalizability and limitations.Examples include 77 fictional characters across 13 dimensions and 32 fictional characters across 2 dimensions.Source: Introduction · pp. 1–2
- Open-ended dialogue often fails to elicit multiple character features in one response, reducing data utilization and evaluation efficiency for feature-focused benchmarks.MCQ-based evaluation alleviates sparsity but simplifies dialogue and cannot fully evaluate generative quality.Source: Introduction · p. 2
- CHARACTERBENCH is a bilingual generative benchmark with 22,859 human-annotated samples covering 3,956 characters across 25 sub-categories; it defines 11 evaluation dimensions within 6 high-level aspects, classified as sparse or dense.It designs dimension-specific queries to elicit related responses and uses target-oriented generation for sparse dimensions.Source: Introduction · p. 2
- CharacterJudge outperforms state-of-the-art automatic judges in correlation with human scores, and experiments show the benchmark’s potential for optimizing character customization.The paper reports evaluation with CharacterJudge and optimization via direct preference optimization.Source: Introduction · p. 2
Related Work
Prior work evaluates character customization through generated responses or multiple-choice questions, but benchmarks often lack character and dimension coverage. CHARACTERBENCH addresses these evaluation gaps while excluding personality tests that require standardized conditions unsuitable for generative evaluation.
Related Work
- Character customization spans fictional characters, celebrities, and daily-life characters, supporting scenarios such as entertainment and social companionship.Both general-purpose LLMs prompted for role-play and models developed specifically for character customization are used.Source: Related Work · p. 2
- Generative evaluation scores LLM responses but may not ensure they relate to the dimensions being evaluated, while MCQ formats simplify dialogue and cannot fully assess generative quality.Many existing benchmarks also focus on fictional characters or limited dimensions.Source: Related Work · p. 2
- Prior benchmarks often focus on fictional characters or limited dimensions, whereas CHARACTERBENCH covers most dimensions included in existing generative benchmarks.MBTI and Big-five personality are omitted because they require well-rounded profiles and a standardized testing environment unsuitable for generative evaluation.Source: Related Work · pp. 2–3
CHARACTERBENCH Framework
CHARACTERBENCH defines 11 evaluation dimensions across 6 aspects and classifies them by whether their evaluated features manifest in every response. Its framework evaluates a character’s response in the context of a profile and dialogue history.
CHARACTERBENCH Framework
- The framework refines six aspects into 11 dimensions and classifies morality and believability as dense, with the other four aspects sparse.The classification is based on whether the relevant character features manifest in each response.Source: CHARACTERBENCH Framework · p. 3
- A character-customized LLM generates response yn from profile P, dialogue context C, and current query un; yn is the evaluation object.The context contains prior user and character utterances.Source: CHARACTERBENCH Framework · p. 3
- Memory Consistency measures retention of conversational facts and events, while Fact Accuracy and Boundary Consistency assess factual knowledge and adherence to the profile’s worldview.Memory Consistency checks that the response aligns with information stored during the interaction.Source: CHARACTERBENCH Framework · p. 3
- Persona, emotion, morality, and believability dimensions assess profile alignment, emotional responses, positive morality under toxic settings, and realism or engagement.These include Attribute and Behavior Consistency, Emotional Self-regulation, Empathetic Responsiveness, Morality Stability and Robustness, Human-likeness, and Engagement.Source: CHARACTERBENCH Framework · p. 3
CHARACTERBENCH Construction
CHARACTERBENCH constructs its benchmark from a quality-screened dialogue corpus and scripts that yield dimension-related responses for human scoring. The corpus combines several collection approaches and covers 3,956 characters across 25 sub-categories.
Overview
- Sampled scripts with profiles and dialogue context are paired with target-oriented queries for sparse dimensions or target-free queries for dense dimensions.The resulting responses are generated by LLMs and scored by human annotators for later CharacterJudge training.Source: Overview · p. 3
Collection of Character-based Dialogue Corpus
- The corpus combines human role-playing, human-prototype interaction, literary-resource extraction, and synthesis through paired-prototype interactions.Role-playing data includes manually annotated query-response pairs, and the extraction data uses CharacterEval’s test set.Source: Collection of Character-based Dialogue… · p. 4
- The corpus contains 13,162 dialogues covering 3,956 characters across 25 sub-categories and 4 main categories.Quality inspectors review profiles, worker engagement, and dialogues, excluding data identified as low quality.Source: Collection of Character-based Dialogue… · p. 4
- Seven LLMs serve as prototypes, including general-purpose models prompted for role-play and models specifically developed for character dialogue.The named examples include GPT-4-1106, Claude-opus, GLM-4, CharacterGLM, MiniMax-abab5.5s, Baichuan-NPC, and CharacterYuyan.Source: Collection of Character-based Dialogue… · p. 4
Collection of CHARACTERBENCH Data
CHARACTERBENCH samples diverse characters and constructs dimension-specific queries to elicit responses for sparse and dense traits. Responses are human-scored, with annotation quality controlled through training, multiple annotators, and spot checks.
Script Sampling
- Scripts are sampled from distinct characters with balanced character and corpus-source distributions, covering four main categories and 25 sub-categories.Each script includes a character profile and a multi-turn context of at least five turns.Source: Script Sampling · p. 4, Figure 3 · p. 4
Target-oriented Query Construction for Sparse Dimensions
- For sparse dimensions, target-oriented queries are built from extracted profile or context information and filtered for tone and coherence.Targets include memory facts, profile attributes or behaviors, and emotionally charged scenarios; fact-accuracy queries use celebrity profiles enriched with BaiduBaike information.Source: Target-oriented Query Construction for… · pp. 4–5
Target-free Query Construction for Dense Dimensions
- For dense dimensions, manually crafted target-free queries elicit morality or believability responses without a target.Morality queries cover nine categories; believability uses the next user utterance from the natural dialogue.Source: Target-free Query Construction for… · p. 5
Response Generation and Human Annotation
- Seven LLMs generate responses to the fused queries, which human annotators score using scales tailored to each dimension.Morality stability and robustness use 2-point scales; believability uses 5-point scales, while other dimensions use 3- or 4-point scales.Source: Response Generation and Human Annotation · p. 5
Quality Control of CHARACTERBENCH
- Quality inspectors review corpus data, while annotators train on pilot samples and each sample receives two annotations, with disagreements escalated to a third annotator.Random batch spot checks continue until the pass rate reaches 95%.Source: Quality Control of CHARACTERBENCH · p. 5
Translation & Statistics
The authors translate the initially Chinese benchmark into English and review translations for faithfulness. The benchmark contains 22,859 samples from 3,956 characters, while the accompanying tables report corpus statistics and evaluation scopes.
Translation
- English translations of the initially Chinese data are reviewed by graduate students, and the average translation pass rate is 96%.The review covers 100 translated items per dimension.Source: Translation & Statistics · p. 5
Statistics
- 22,859 samples cover 3,956 characters, with an average of 11.22 dialogue turns; fact accuracy covers 105 celebrity characters.The test set is divided into in-domain and out-of-domain sets, each with 125 samples per dimension.Source: Translation & Statistics · p. 5
- Table 3 reports Pearson correlations with human scores across target-free and target-based settings, while Table 4 reports character-customization scores normalized to five points.Source: Table 3 · p. 6, Table 4 · p. 6
Development of CharacterJudge
CharacterJudge is trained to evaluate character customization from query-fused scripts and generated responses, using human scores as supervision. Its decoding uses majority voting, and the authors train separate Chinese and English models.
Development of CharacterJudge
- CharacterJudge uses dimension-specific instructions to score scripts, responses, and targets against human-score supervision.The input includes the profile, context fused with a target-oriented or target-free query, the response, and the target when applicable.Source: Development of CharacterJudge · p. 6
- The optimization spans the set of dimensions, with the target omitted for dense dimensions.The passage identifies Pθ as the LLM parameters being optimized and D as the set of dimensions.Source: Development of CharacterJudge · p. 6
- At decoding, multiple outcomes are generated and majority-voted; Chinese and English models are trained separately because bilingual fine-tuning was less effective.Source: Development of CharacterJudge · p. 6
Experiments
Experiments report that CharacterJudge aligns better with human scores than compared automatic judges, while CHARACTERBENCH distinguishes LLMs’ customization strengths and weaknesses and supports optimization experiments.
Evaluation on CharacterJudge
- CharacterJudge is built on Qwen2-7B-Chat, generates 10 outcomes through self-consistency, and is evaluated by Pearson correlation with human scores.Comparisons use GPT-series and GLM-4 judges with target-free and target-based prompts.Source: Evaluation on CharacterJudge · p. 6
- 42% and 36% improvements on AVG. over suboptimal GLM-4-TG in bilingual evaluations show CharacterJudge’s stronger alignment with human scores.The authors also report its strongest superiority on Believability and SOTA performance in both languages.Source: Evaluation on CharacterJudge · p. 6
- Removing self-consistency generally reduces performance across dimensions, while removing target T particularly affects sparse dimensions.The ablation compares CharacterJudge with variants w/o SC and w/o TG.Source: Evaluation on CharacterJudge · p. 7
- CharacterJudge shows comparable performance on in-domain and out-of-domain test sets across AVG. and individual dimensions.The authors interpret this as generalization to unobserved characters.Source: Evaluation on CharacterJudge · p. 7
Evaluation for LLMs in Character Customization
- Large-scale open-source LLMs perform comparably to powerful closed-source models, while Claude-3-opus outperforms four role-playing LLMs.Most bilingual LLMs perform comparably across languages but struggle with accurate facts.Source: Evaluation for LLMs in Character… · p. 7
- LLMs score highly on morality but have room to improve on persona and memory, while emotion and believability scores are low.The authors connect low emotion and believability scores with difficulty sustaining natural, human-like emotional exchanges.Source: Evaluation for LLMs in Character… · p. 7
Analysis for CHARACTERBENCH
- CHARACTERBENCH significantly outperforms CharacterEval and SocialBench in correlation with human rankings across diverse character scenarios.The human evaluation used 10 annotators, with two characters each and at least 20 Chinese dialogue turns.Source: Analysis for CHARACTERBENCH · p. 7
- 6B-DPO significantly outperforms all baselines in manual pairwise evaluation, demonstrating the benchmark’s potential for character-customization optimization.The preference pairs were scored by CharacterJudge and used for DPO training.Source: Analysis for CHARACTERBENCH · p. 7
Conclusions
The paper presents CHARACTERBENCH as a bilingual generative benchmark for evaluating character customization across 11 dimensions and six aspects.
Conclusions
- CHARACTERBENCH contains 22,859 samples and evaluates LLM character customization on 11 dimensions spanning six aspects, using tailored queries for sparse and dense dimensions.The authors report that CharacterJudge outperforms automatic judges and that the benchmark has potential for optimizing character customization.Source: Conclusions · p. 8
Ethical Considerations
The authors describe worker protections, research-only release conditions, and safeguards for sensitive morality data.
Ethical Considerations
- Workers were fairly compensated and their privacy preserved; data and models are for research use, with sensitive morality content subject to licensing, review, and institutional approval.The authors state that collected data contain no personal information.Source: Ethical Considerations · p. 10
CHARACTERBENCH Construction
CHARACTERBENCH draws on human and model-generated dialogue sources, then uses tailored construction procedures and queries to produce and assess its data.
Human RolePlaying
- Paired workers role-play characters and users, annotating profile attributes and boundary-related responses; other workers freely customize characters through multi-turn interactions with seven popular LLMs.Character workers create detailed profiles and worldviews, while user workers engage naturally and may initiate topics.Source: Human RolePlaying · p. 10, Human-Prototype Interaction · p. 10
Extraction from Literary Resources
- Literary extraction uses CharacterEval’s released test set, containing dialogues from 77 characters in novels and scripts.Only the test set was available when CHARACTERBENCH was being constructed.Source: Extraction from Literary Resources · p. 10
Synthesis via Prototypes Interaction
- Prototype dialogue synthesis truncates the last five turns, summarizes the preceding dialogue, then prompts the paired LLMs to continue to a natural conclusion.The process addresses monotonous, prematurely ending, and profile-deviating conversations observed without specialized intervention.Source: Synthesis via Prototypes Interaction · p. 10
Role-playing Prompt of LLMs
- A tuned prompt instructs general-purpose LLMs to perform role-playing.The prompt was selected through meticulous tuning.Source: Role-playing Prompt of LLMs · p. 10
Distribution of Toxic Query and Character Setting used for Morality
- Morality stability and robustness queries use toxic scenarios across nine categories, including discrimination, harm, privacy, politics, and pornography.The categories also include insult, crime, mental health, and ethics.Source: Distribution of Toxic Query and… · p. 10
Construction Details of Each Dimension
- Dimension-specific construction materials include prompts, filtering procedures, annotation scales, and high- and low-scoring examples for memory, knowledge, persona, emotion, morality, and believability.The reported scales range from two to five points across dimensions.Source: Construction Details of Each Dimension · pp. 10–11
More Statistics of CHARACTERBENCH
- The benchmark’s samples from seven LLMs are evenly distributed, and tailored queries achieve lowest-score induction rates above 10% across dimensions.The authors use these statistics to support sample diversity and the queries’ ability to probe potential defects.Source: More Statistics of CHARACTERBENCH · p. 11
Translation Prompt
- The Chinese benchmark is translated into English using a translation prompt.The prompt is provided in Table 12.Source: Translation Prompt · p. 11
Experiments
The experiments appendix documents CharacterJudge and preference-tuning setups, then presents evaluation prompts, annotation scales, and scored examples for character customization.
Implementation Details
- CharacterJudge and the 6B-SFT/6B-DPO variants use shared training settings, with DPO pairs selected from judge-scored sampled responses.The SFT set contains 7,471 samples and DPO uses 3,679 paired responses; training details and evaluation prompts are included in supplementary code, which is slated for public release.Source: Implementation Details · p. 12
Evaluation for LLMs in Character Customization
- GPT-4o and Claude-sonnet are evaluated alongside frequently mentioned LLMs, and Table 11 gives a tuned role-playing prompt with character-profile and dialogue-context placeholders.The authors report only frequently mentioned LLM capabilities in the main paper because of space limits.Source: Evaluation for LLMs in Character… · p. 12, Table 11 · p. 13
- Supplementary tables provide tuned prompts for target extraction, query generation, and filtering, together with annotation scales for character-customization dimensions.The listed prompts cover dimensions including Memory Consistency, Fact Accuracy, Attribute Consistency, Behavior Consistency, Emotional Self-regulation, and Empathetic Responsiveness.Source: Table 13 · p. 15, Table 16 · p. 17, Table 20 · p. 20
- Scored examples illustrate how Memory Consistency distinguishes responses aligned with dialogue memory from responses that contradict it.The appendix also includes examples for factual knowledge, profile consistency, emotional response, morality, human-likeness, and engagement.Source: Table 37 · p. 32, Table 38 · p. 33, Table 39 · p. 34