PaperlayerView exact revision on arXiv

Source-linked AI summary

CoSER: A Comprehensive Literary Dataset and Framework for Training and Evaluating LLM Role-Playing and Persona Simulation

Xintao Wang, Heng Wang, Yifei Zhang, Xinfeng Yuan, Rui Xu, Jen-tse Huang, Siyu Yuan, Haoran Guo, Jiangjie Chen, Shuchang Zhou, Wei Wang, Yanghua Xiao

arXiv:2502.09082v3cs.CLcs.AI

TL;DR

Established-character role-playing needs authentic character data and evaluation that can capture nuanced portrayals. CoSER provides a literary dataset and uses given-circumstance acting to train and evaluate models in book scenes. Its models perform strongly across RPLA benchmarks, with CoSER 70B reaching 93.47% accuracy on LifeChoice.

  • Problem

    RPLA research lacks authentic character data and evaluation methods grounded in such data, while established characters require portrayals faithful to complex backgrounds and nuanced personalities.

    Source: §1 · p. 1

  • Method

    CoSER curates literary dialogue and contextual character data, then applies given-circumstance acting to train character-specific models and evaluate multi-agent scene simulations with penalty-based critics.

    Source: §1 · pp. 2–3, §4.2 · p. 7

  • Results

    CoSER models achieve state-of-the-art performance across four RPLA benchmarks and human evaluation, and CoSER 70B reaches 93.47% accuracy on LifeChoice.

    Source: §1 · p. 3, §5.2 · p. 10

  • Takeaways & Limitations

    The experiments demonstrate the value of the CoSER dataset for RPLA training, retrieval, and evaluation.

    Source: §1 · p. 3

    Applications must respect copyright policies, personal-data privacy, and consent, and use appropriate licensing.

    Source: Impact Statement · p. 14

Abstract

from arXiv · show

Role-playing language agents (RPLAs) have emerged as promising applications of large language models (LLMs). However, simulating established characters presents a challenging task for RPLAs, due to the lack of authentic character datasets and nuanced evaluation methods using such data. In this paper, we present CoSER, a collection of a high-quality dataset, open models, and an evaluation protocol towards effective RPLAs of established characters. The CoSER dataset covers 17,966 characters from 771 renowned books. It provides authentic dialogues with real-world intricacies, as well as diverse data types such as conversation setups, character experiences and internal thoughts. Drawing from acting methodology, we introduce given-circumstance acting for training and evaluating role-playing LLMs, where LLMs sequentially portray multiple characters in book scenes. Using our dataset, we develop CoSER 8B and CoSER 70B, i.e., advanced open role-playing LLMs built on LLaMA-3.1 models. Extensive experiments demonstrate the value of the CoSER dataset for RPLA training, evaluation and retrieval. Moreover, CoSER 70B exhibits state-of-the-art performance surpassing or matching GPT-4o on our evaluation and three existing benchmarks, i.e., achieving 75.80% and 93.47% accuracy on the InCharacter and LifeChoice benchmarks respectively.

1. Introduction

Established-character role-playing requires fidelity to complex backgrounds and nuanced personalities, but existing data and evaluation methods lack authenticity, context, and nuanced assessment. CoSER contributes literary data and models alongside given-circumstance acting (GCA), which trains and evaluates role-playing LLMs through characters’ original scenes.

Motivation

  • Existing datasets often lack dialogue context and other knowledge forms, while evaluation commonly uses single-turn questions and methods with limited nuance or bias concerns.The paper identifies LLM-judge bias and multiple-choice methods’ focus on specialized aspects as additional weaknesses.Source: §1 · p. 1

Dataset

  • CoSER extracts authentic multi-character dialogue from acclaimed books and adds plot summaries, character experiences, conversation backgrounds, actions, and internal thoughts.These varied representations support prompting, retrieval, training, and evaluation while preserving source fidelity.Source: §1 · p. 2

Method

  • GCA trains models on a target character’s authentic utterances and evaluates simulated multi-character conversations using penalty-based critics, detailed rubrics, and the original dialogue.CoSER 8B and 70B are built on LLaMA-3.1 models.Source: §1 · p. 3

Contributions

  • CoSER combines authentic character data, open models, and an evaluation protocol; its dataset contains 29,798 conversations from 771 renowned books.The dataset also includes plot summaries, character experiences, conversation backgrounds, actions, and internal thoughts.Source: §1 · pp. 2–3

2. Related Work

Prior RPLA research uses persona data and evaluation methods with distinct strengths and limitations. The paper situates CoSER against synthesized or difficult-to-scale datasets and evaluations that can be biased or limited to specific traits.

RPLAs

  • Early RPLA work includes fictional-character prototypes and multi-agent simulations of human society, alongside studies of role-play’s nature and limitations.Chen et al. (2024a) survey relevant research.Source: §2 · p. 3
  • RPLAs use LLMs to simulate personas from persona data, and established-character agents are expected to reflect complex backgrounds and nuanced personalities.The paper frames effective RPLAs as requiring comprehensive, high-quality data and advanced role-playing LLMs.Source: §2 · p. 4

Datasets for RPLAs

  • Existing persona datasets range from profiles and dialogues to experiences and multimodal information, but many are synthetic, difficult to scale, or limited in representation.Human-annotated datasets are costly to scale; authentic-dialogue datasets such as ChatHaruhi and HPD rely on human effort for individual sources.Source: §2 · p. 4

Evaluation for RPLAs

  • RPLA evaluations use LLM judges or multiple-choice questions, covering dimensions such as style, knowledge, personality, decision-making, and motivation recognition.LLM judges can exhibit length and position biases or lack character-specific knowledge; multiple-choice benchmarks assess particular aspects.Source: §2 · p. 4

3. CoSER Dataset

CoSER organizes authentic literary material into a broad dataset for role-playing, with character, plot, and conversation information. Its design adds contextual data and expressive message dimensions, while its curation pipeline selects, segments, extracts, and organizes book content.

3. CoSER Dataset

  • CoSER covers 17,966 characters from 771 renowned books, with authentic, non-synthesized dialogues and varied data representations.The dataset is intended to support multiple uses through its comprehensive representations.Source: §3 · p. 4

3.1. Design Principles

  • CoSER distinguishes itself through rich data types, internal thoughts and physical actions in messages, and the environment’s role as a dataset character.These design choices extend beyond the profiles and dialogues emphasized in previous work.Source: §3.1 · pp. 4, 6
  • The dataset structures book knowledge through interconnected plots, conversations, and characters, with plot summaries and experiences, conversation settings and motivations, and character profiles.The representations are designed to be comprehensive, complementary, and sufficiently contextual for portraying behavior in scenarios.Source: §3.1 · pp. 4–5
  • Messages distinguish speech, physical action, and internal thought, allowing role-players to represent nonverbal behavior and thoughts invisible to other characters.Actions can trigger downstream events in multi-agent systems, while thoughts create information asymmetry.Source: §3.1 · p. 5

3.2. Dataset Curation

  • The dataset draws on 771 books obtained from the Goodreads Best Books Ever top 1,000, selected for literary significance and diversity across genres, periods, and cultural backgrounds.The paper reports obtaining content for 771 of the 1,000 identified books.Source: §3.2 · p. 6
  • Curation combines chapter-based and plot-based chunking, with LLMs identifying truncated plots or trailing content so it can be joined to the next chunk.This dynamic step addresses plot integrity that static chunking may disrupt.Source: §3.2 · p. 6
  • LLMs extract plot and conversation data from book chunks; original dialogue supplies speech, while actions and thoughts may be extracted or inferred from context.The final 10% of plots in each book are held out for evaluation, and character data are organized by unifying aliases, aggregating relevant material, and generating profiles.Source: §3.2 · p. 6

4. Training and Evaluation via GCA

GCA adapts given-circumstance acting to train and evaluate role-playing LLMs through sequential portrayal of characters in a contextualized conversation. Training uses character utterances, while evaluation combines multi-agent simulation with penalty-based LLM judging.

4.1. Given-Circumstance Acting

  • Given-circumstance acting adapts an acting approach based on specified conditions: an LLM sequentially portrays each character in a conversation’s setting.The specified conditions in the acting approach include environmental context, historical events, and personal conditions.Source: §4.1 · p. 7

4.2. GCA Training and CoSER Models

  • Training samples pair a conversation with one character’s role-playing instruction, optimizing on that character’s utterances while using the remaining messages as inputs.The instruction includes the scenario, the character’s profile and motivation, and profiles of other involved characters.Source: §4.2 · p. 7
  • CoSER models use LLaMA 3.1 Instruct and training across 90% of the books, with varied roles, instruction formats, context combinations, and dialogue-thought settings.Training also extends to environment modeling and next speaker prediction; Tulu-3 data augments training to maintain general abilities.Source: §4.2 · p. 7

4.3. GCA Evaluation

  • GCA evaluation combines multi-agent simulation with penalty-based LLM judging to assess role-playing abilities.The approach addresses both scenario elicitation and performance assessment.Source: §4.3 · p. 7
  • Simulations use character agents, a next-speaker model, and an environment model, and end at an end signal or a maximum of 20 turns.Agents receive scenario and character context, while their motivations and inner thoughts are inaccessible to other agents.Source: §4.3 · pp. 7–8
  • A continue-from parameter k seeds simulations with the first k original messages, controlling story direction and language style.The paper says this also reduces the influence of differences in LLM language styles.Source: §4.3 · p. 8
  • LLM critics assess simulated dialogue by identifying rubric-specific role-playing flaws; each flaw receives severity 1–5, and each dimension starts at 100 minus five times the summed flaw severities.Source: §4.3 · p. 8
  • Evaluation covers individual-agent and overall-simulation scopes, each assessed as referenced against the source book or as standalone quality.The four dimensions are anthropomorphism, character fidelity, storyline quality, and storyline consistency; scores also receive length correction with λ set to 1.5.Source: Figure 3 · p. 8, §4.3 · p. 8

5. Experiments

The experiments evaluate models on CoSER Test, existing benchmarks, and human judgments, finding strong CoSER performance and benefits from authentic data and richer character context. Ablations, retrieval tests, and case studies further examine which data and evaluation choices matter.

5.1. Settings

  • CoSER Test uses 200 held-out conversations, and the experiments report LLM-judged dimensions, overall scores, and BLEU and ROUGE-L against original dialogues.The test includes 100 conversations from books used in training and 100 from books not used in training.Source: §5.1 · p. 9

5.2. Main Results

  • CoSER 70B outperforms all open models and is competitive with GPT-4o on LLM-judged metrics, exceeds second-best BLEU by 58%, and leads across three existing RPLA benchmarks.CoSER models significantly improve over their LLaMA 3.1 baselines; CoSER 70B reaches 93.47% LifeChoice accuracy, surpassing GPT-4o by 23%.Source: §5.2 · pp. 9–10, Table 2 · p. 10, Table 3 · p. 11
  • Starting simulations from three original messages raises scores and narrows model gaps; the Qwen-2 72B–7B average-score gap falls from 11.5% to 8.8%.The original messages guide story direction and language style, particularly benefiting smaller models.Source: §5.2 · p. 10

5.3. Human Evaluation

  • Human evaluation generally agrees with GCA evaluation, with CoSER 70B scoring 6.783 and winning 86.9% of comparisons.Human evaluators show less preference for GPT models than GPT-4o judges do, which the authors suggest may stem from self-preference bias in LLM judges.Source: §5.3 · p. 11, Table 4 · p. 11
  • All tested GCA evaluation components improve alignment with human preferences; DeepSeek-R1 achieves the highest alignment, while BLEU and ROUGE-L remain effective indicators.The compared components include dialogue references, scoring rubrics, length correction, and dimension separation.Source: §5.3 · pp. 11–12, Table 5 · p. 12

5.4. Ablation Studies

  • Including inner thoughts and motivations improves test-time scores across models, while models trained without inner thoughts consistently underperform regular CoSER models.The results compare overall CoSER Test scores with and without inner thoughts and motivations, and training variants with inner thoughts removed.Source: §5.4 · p. 12, Table 6 · p. 13

5.5. CoSER Dataset for Retrieval Augmentation

  • Retrieved character experiences and conversations benefit models, especially CoSER 70B, whereas retrieving raw plot text barely improves performance.The retrieval sources are related-conversation dialogues, character experiences, and raw text from related plots.Source: §5.5 · p. 12, Figure 4 · p. 13

5.6. Case Studies

  • Case studies find that CoSER models communicate more naturally, recall character knowledge better, and portray nuanced thinking such as Cersei Lannister’s suppressed anger.Other models, including GPT-4o and Claude-3.5-Sonnet, portray Cersei with stereotypical arrogance and pride in the cited cases.Source: §5.6 · p. 12

6. Conclusion

The conclusion presents CoSER as an authentic literary dataset, open role-playing models, and a training and evaluation protocol for established characters. Its experiments support the dataset’s value and report state-of-the-art performance across the authors’ evaluation and three existing benchmarks.

6. Conclusion

  • CoSER combines authentic data from 771 books with given-circumstance acting, in which LLMs sequentially portray multiple characters in book scenarios.The dataset includes dialogues, plot summaries, character experiences, and inner thoughts.Source: §6 · p. 13
  • Training LLaMA-3.1 models with CoSER data produces the open CoSER 8B and CoSER 70B role-playing models.The authors also use given-circumstance acting with multi-agent simulation and penalty-based LLM critics for evaluation.Source: §6 · p. 13
  • Experiments demonstrate dataset value for role-playing training, evaluation, and retrieval, while CoSER models achieve state-of-the-art results on the paper’s evaluation and three existing benchmarks.Source: §6 · p. 13

Impact Statement

The impact statement frames CoSER as a research resource while identifying copyright, privacy, consent, and responsible-use considerations.

Impact Statement

  • CoSER is intended for research use, and the authors say they will not distribute raw novel content.Users must follow copyright policies and obtain proper permissions for derivative works.Source: Impact Statement · p. 13
  • The authors note that source-literature content may raise ethical considerations and does not represent their viewpoints.Source: Impact Statement · p. 14
  • Applications to real-world individuals must respect personal-data privacy, obtain necessary consent, and use proper licensing.Source: Impact Statement · p. 14

A.1. Statistics and Analysis

CoSER combines a large literary character dataset with construction and comparison analyses. The appendix reports its scale and breadth, describes data handling and safety steps, and contrasts its character-profiling approach and goals with prior work and PersonaHub.

A.1. Statistics and Analysis

  • 17,966 characters and 29,798 conversations across 771 books comprise CoSER, with 392,298 utterances averaging 13.2 per conversation.The dataset also contains 30,069 unique plots.Source: §A.1 · p. 18, Table 7 · p. 18
  • The selected books span fiction genres including Fantasy, Historical, Science Fiction, Romance, and Mystery, alongside memoirs, biographies, and other nonfiction.Figure 5 presents the genre distribution based on Supersummary classifications.Source: §A.1 · p. 18, Figure 5 · p. 18

A.2. Data Splits

  • For evaluation, the last 10% of data from each book is held out from prompts, training, and retrieval, while CoSER models train on 90% of books.Source: §A.2 · p. 18

A.3. Implementation Details for Construction

  • Construction recovers raw text by matching extracted plot endpoints to exact book sentences, and uses JSON repair-and-retry when structured extraction fails.The pipeline retries up to five times, including handling outputs truncated by token limits.Source: §A.3 · p. 20
  • An additional LLM call refines conversation settings because extracted scenarios and motivations can omit context distributed across other fields.The refined setting is instructed not to reveal subsequent dialogue or plot developments.Source: §A.3 · p. 20

A.4. Comparison with Existing Methods for Character Profiling

  • CoSER’s multi-stage, extract-then-aggregate profiling pipeline is reported to improve precision and recall, capture evolving character arcs, and increase processing efficiency.Source: §A.4 · p. 20

A.5. Safety

  • Dataset safety checks truncate 110 sensitive conversations and remove 602 messages.Source: §A.5 · p. 20

A.6. Comparison with PersonaHub

B.1. Training Samples

This appendix describes how CoSER training samples and evaluation settings are constructed. It also specifies benchmark prompt adaptations and a length correction motivated by measured judge bias.

B.1. Training Samples

  • Each conversation-character pair yields a training sample: the target character’s messages are optimized from prompts and other characters’ messages, with adjacent inputs concatenated.Prompts vary phrasing, format, and whether they include other-character profiles, plot summaries, or inner thoughts.Source: §B.1 · pp. 21–22

B.2. Hyperparameters

  • LLaMA 3.1 models are fine-tuned for 8 epochs with a sequence length of 16,384 and a global batch size of 48.The learning rate is 1 × 10−5.Source: §B.2 · p. 22

C.1. Test Set Sampling

  • The 200-sample CoSER Test set contains 100 in-domain and 100 out-of-domain samples, weighted toward conversations involving characters with more plot appearances.Source: §C.1 · p. 22

C.3. Length Correction

  • −1.5909 points per additional simulation round is the fitted score change in 1,200 cases, motivating a correction for length bias.The analysis covers CoSER 70B, LLaMA-3.1 70B, and GPT-4o, with and without retrieval; the fitted line is score = −1.5909 × rounds + 59.0617.Source: Figure 6 · p. 23, §C.3 · p. 23

C.2. Prompting Strategies for Exisitng RPLA Benchmarks

  • InCharacter prompts encourage honest, detailed responses; LifeChoice and Cross-MR prompts ask models to reason before choosing.These adaptations address CoSER models’ tendency toward brief or declined answers in personality assessment and aim to elicit considered choices.Source: §C.2 · pp. 22–23

C.3. Length Correction

  • Length correction compensates for points deducted as rounds increase, using λ = 1.5 based on the regression analysis.The corrected score adds a rounds-based adjustment to the flaw-severity penalties.Source: §C.3 · p. 23

C.4. Human Evaluation

  • Judge-human alignment is measured by pairwise agreement frequency, excluding comparisons where human score differences are ≤1 or automatic-metric differences are ≤5%.Source: §C.4 · p. 24

D. Additional Results

Additional experiments examine judging consistency, generalization, conversation continuation, retrieval, and inner-thought ablations. The section also compares given-circumstance simulation with AI-driven storytelling.

  • CoSER models remain competitive or superior across judge settings, but GPT-4o, DeepSeek-R1, and DeepSeek-V3 show different model preferences.GPT-4o favors GPT models, while the two DeepSeek judges favor CoSER-70B and Claude-3.5-Sonnet over GPT-4o.Source: D. Additional Results · p. 24, Table 11 · p. 24
  • CoSER models maintain strong performance on both in-domain and out-of-domain character splits, with consistent trends across them.Table 13 separates results by whether the books appeared in CoSER training.Source: D. Additional Results · p. 24, Table 13 · p. 25
  • Conversation-continuation experiments start multi-agent simulations from the first three authentic messages.Table 14 reports performance in this k = 3 setting.Source: D. Additional Results · p. 24, Table 14 · p. 26
  • Retrieval experiments compare character conversations, plot experiences, and raw plot text, including combinations of experiences and conversations.The tested settings include one plot’s raw text, one conversation, experiences from three or ten plots, and combinations with a conversation.Source: D. Additional Results · pp. 24, 27, Table 16 · p. 28
  • The inner-thought ablation compares CoSER Test performance with and without inner thoughts during inference and model training.The supplied passages identify the comparison but do not state its outcome.Source: D. Additional Results · p. 27, Table 15 · p. 27
  • 59.2% versus HollmWood’s 50.2% is the average score for GCA simulation and HollmWood, respectively, on 30 samples judged with GCA evaluation.GPT-4o served as both actor and writer in both frameworks.Source: D. Additional Results · p. 27

E. Authentic v.s. LLM-Generated Role-playing Data

The paper compares authentic literary dialogue with generated alternatives and presents case studies of CoSER 70B simulations. It also identifies where detailed prompts are provided.

E. Authentic v.s. LLM-Generated Role-playing Data

  • 85.1% versus 76.2% is the GCA score for authentic dialogues and GPT-4o-generated dialogues, respectively.The comparison uses 100 CoSER Test samples and DeepSeek-R1 as judge, without supplying ground-truth dialogue as a reference.Source: E. Authentic v.s. LLM-Generated… · p. 29
  • 55.3% versus 54.7% is the GCA score for models trained on authentic CoSER dialogues and GPT-4o-generated dialogues, respectively.Both LLaMA-3.1-8B models were fine-tuned for four epochs and evaluated on 100 new samples.Source: E. Authentic v.s. LLM-Generated… · p. 29

F. Examples and Case Study

  • In the A Storm of Swords example, CoSER 70B reproduces Lysa Arryn’s iconic line to Sansa Stark, illustrating recall and use of character-related knowledge.Source: F. Examples and Case Study · p. 29
  • In the A Dance with Dragons walk-of-atonement scene, CoSER 70B captures Cersei’s suppressed anger, while GPT-4o and Claude-3.5-Sonnet portray her through familiar arrogance and pride.Source: F. Examples and Case Study · p. 29

G. Prompts

  • The appendix lists prompts for dataset curation, RPLA and multi-agent simulation, and penalty-based LLM judging.Source: G. Prompts · pp. 29–30

H. Limitations

The paper identifies limitations in LLM-based evaluation, generated character thoughts, and data-extraction recall. Copyright restrictions and uneven model familiarity with books further constrain data access and evaluation comparisons.

H. Limitations

  • LLM-based judging retains length bias and may lack the knowledge needed to assess character fidelity, despite penalty-based scoring and detailed rubrics.The simulation stage effectively elicits role-playing performance, but the judging stage still relies on LLM judges.Source: H. Limitations · p. 30
  • Character thoughts are often sparse in the novels and inferred by LLMs from limited context, so generated thoughts may miss sophisticated thinking processes.The authors identify improving these thoughts as future work.Source: H. Limitations · p. 30
  • The extraction pipeline does not ensure coverage of all plots, conversations, or characters in the source material.The authors identify improving recall as future work.Source: H. Limitations · p. 30
  • Copyright concerns prevent release of raw novel content, limiting studies that explore raw text for RPLA development; the dataset is intended for research purposes only.The authors urge developers to respect copyright policies and use proper licensing.Source: H. Limitations · p. 30
  • Different actor LLMs may have different familiarity with the selected books, making cross-model comparisons potentially unfair; comparisons within the same model series are appropriate.The authors cannot confirm whether a model has thoroughly learned a particular book, even though the novels are renowned.Source: H. Limitations · p. 30
Loading 2502.09082v3…