PaperlayerView exact revision on arXiv

Source-linked AI summary

Character is Destiny: Can Role-Playing Language Agents Make Persona-Driven Decisions?

Rui Xu, Xintao Wang, Jiangjie Chen, Siyu Yuan, Xinfeng Yuan, Jiaqing Liang, Zulong Chen, Xiaoqing Dong, Yanghua Xiao

arXiv:2404.12138v2cs.AI

TL;DR

Whether language agents can make decisions aligned with a persona remains underexplored beyond tone and knowledge imitation. The paper evaluates this with LIFECHOICE, a benchmark of novel-based decisions, and proposes CHARMAP’s persona-based memory retrieval. CHARMAP improves accuracy over previous baselines, though performance remains below human performance.

  • Problem

    Whether RPLAs can reliably make persona-aligned decisions remains underexplored beyond their tone and knowledge fidelity.

    Source: §1 · p. 1

  • Method

    The paper constructs LIFECHOICE from expert analyses of high-quality novels and proposes CHARMAP, which uses persona-based memory retrieval.

    Source: §1 · p. 2

  • Results

    67.95% accuracy for CHARMAP on LIFECHOICE exceeds previous baselines by 5.03%, while human performance is 92.01%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LIFECHOICE provides a benchmark for evaluating RPLA decision simulation and informing future personal LLM assistant development.

    Source: §6 · p. 8

    The study focuses primarily on fictional characters and choices in books, which may differ from real-world humans and include illogical author-designed choices.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Can Large Language Models (LLMs) simulate humans in making important decisions? Recent research has unveiled the potential of using LLMs to develop role-playing language agents (RPLAs), mimicking mainly the knowledge and tones of various characters. However, imitative decision-making necessitates a more nuanced understanding of personas. In this paper, we benchmark the ability of LLMs in persona-driven decision-making. Specifically, we investigate whether LLMs can predict characters' decisions provided by the preceding stories in high-quality novels. Leveraging character analyses written by literary experts, we construct a dataset LIFECHOICE comprising 1,462 characters' decision points from 388 books. Then, we conduct comprehensive experiments on LIFECHOICE, with various LLMs and RPLA methodologies. The results demonstrate that state-of-the-art LLMs exhibit promising capabilities in this task, yet substantial room for improvement remains. Hence, we further propose the CHARMAP method, which adopts persona-based memory retrieval and significantly advances RPLAs on this task, achieving 5.03% increase in accuracy.

1 Introduction

The paper benchmarks whether role-playing language agents (RPLAs) can make persona-aligned decisions from stories, a capability less studied than tone, knowledge, or personality fidelity. It introduces LIFECHOICE and CHARMAP to evaluate and improve decision simulation.

1 Introduction

  • Whether RPLAs can reliably make persona-aligned decisions remains underexplored compared with their tone and knowledge fidelity.The paper frames decision simulation as requiring understanding of characters’ mindsets and reasoning about unobserved behavior.Source: §1 · pp. 1–2
  • LIFECHOICE evaluates decisions from high-quality novels using 1,462 character decisions from 388 novels and expert-written character analyses.Each sample provides preceding context and a multiple-choice decision point.Source: §1 · p. 2
  • LIFECHOICE challenges agents to find sparse motivations in long contexts, track changing characters and environments, and reason through entangled motives.The authors present these as long-context understanding, temporal intelligence, and intricate motives.Source: §1 · p. 2
  • CHARMAP reaches 67.95% accuracy on LIFECHOICE, 5.03% above previous baselines, while human performance is 92.01%.Existing RPLAs reach up to 62.92%; the authors identify room for improvement.Source: §1 · p. 2
  • The paper proposes CHARMAP, which uses persona-based memory retrieval, and reports that well-summarized character descriptions and accurate retrieval are crucial for RPLAs.The work also evaluates varied LLMs and RPLA frameworks on LIFECHOICE.Source: §1 · p. 2

2 Related Work

Prior character role-playing work largely evaluates dialogue, speaking style, and role-specific knowledge, while this study evaluates behavior and decision-making. The paper also situates its novel-based character histories alongside personal LLM assistants that use user histories.

2 Related Work

  • Earlier role-playing studies focus mainly on dialogue imitation, speaking style, and role-specific knowledge, whereas this work evaluates behavior and decisions.The related work characterizes dialogue-focused imitation as shallow relative to decision-making.Source: §2 · p. 3
  • Personal assistants can model profiles and preferences from user histories, but obtaining real user memory data is difficult and raises privacy concerns.The paper uses character histories from novels as a benchmark setting for restoring choices from preceding text.Source: §2 · p. 3

3 Dataset and Task Setups

LIFECHOICE contains multiple-choice character decisions grounded in novel context and expert analyses, with questions generated and manually examined. The paper categorizes decision motivations and frames the task as selecting the narrative-consistent choice from the supplied context.

3.1 Dataset Construction

  • Decision-point selection filters out nonfiction, non-linear narratives, and overly popular books before GPT-4 extracts choices and motivations from expert-written content.The stated selection criteria also require first- or third-person narration.Source: §3.1 · pp. 3–4
  • GPT-4 generates questions whose correct option matches the character’s original choice and whose distractors are plausible for an arbitrary person.Ten native English-speaking university students assess whether questions are challenging and reasonable and filter low-quality data.Source: §3.1 · p. 4
  • LIFECHOICE contains 1,462 characters from 388 books, with samples built from preceding context, a scenario, a question, options, the correct answer, and motivation.The data source includes literary experts’ character descriptions, chapter summaries, and book analyses.Source: §3.1 · pp. 3–4

3.2 Dataset Analysis

  • The dataset analysis groups motivations into character-driven motives rooted in inner traits and plot-driven motives arising from external events and conflicts.Each topic receives one motivation category, with several accompanying sub-motivations.Source: §3.2 · pp. 4–5

3.3 Task Setups

  • LIFECHOICE models decision prediction as P(y|x), where x includes context, scenario, character, question, and options, and evaluates multiple-choice accuracy.The task requires locating relevant scene information across extensive personal data.Source: §3.3 · p. 5

4 Experiments

The experiments build character profiles from descriptions and preceding-text memories, then use those profiles to test LLM decision-making. CHARMAP makes memory retrieval scenario-specific by using the description to identify relevant plot episodes first.

4 Experiments

  • Because inputs generally exceed 100k, the experiments construct a character profile before asking an LLM to answer decision questions.The profile contains a character description and memories from the preceding text.Source: §4 · p. 5, §4.1 · p. 5

4.1 Character Profile Construction

  • Character descriptions are built through hierarchical merging or incremental updating, or taken from expert-written Supersummary descriptions truncated to text before decision points.GPT-3.5 summarizes the automated methods’ chunks; descriptions are kept within 5k tokens.Source: §4.1 · p. 5
  • CHARMAP uses the description and question to locate relevant plot episodes, then uses those episodes to retrieve related memories for the decision profile.The retrieved memories are combined with the description for inference.Source: §4.1 · p. 6, Figure 3 · p. 6

4.2 Reasoning for Decisions

  • After compressing original input x into a character profile, experiments feed it to LLMs: description- or memory-only methods use GPT-3.5, GPT-4, and LLaMA-3; both-method experiments also include Claude-3, Gemini, and Mixtral.Source: §4.2 · p. 6

5 Analysis

The experiments assess whether RPLAs can make decisions from historical data and examine how motivations, genres, and the amount of available story context relate to accuracy.

5.1 Can LLMs Make Decisions Based on Historical Data?

  • CHARMAP outperforms direct concatenation of Description and Memory by 5.03%, and gold motivations consistently yield accuracy above 80%.The authors also report that performance gaps among LLMs are not significant when reasoning from profiles, suggesting generated profiles are the main factor.Source: §5.1 · p. 6
  • 92.01% human accuracy on raw text indicates substantial room for improvement in RPLA methods.In a test using six unfamiliar novels, humans slightly outperform GPT-4 when reasoning from profiles.Source: §5.1 · pp. 6–7
  • For books with fewer than 5,000 reviews, CHARMAP results resemble those for books absent from the LLMs’ training corpus; accuracy improves significantly above 5,000 reviews.The authors interpret this as evidence that data leakage has little impact for low-review books and use 5,000 reviews as a filtering threshold.Source: §5.1 · p. 7, Figure 4 · p. 7
  • Long-context models show potential on LIFECHOICE, but their performance is below CHARMAP.The paper presents their results in Table 5; the supplied table passage does not provide the values.Source: §5.1 · p. 7, Table 5 · p. 7

5.2 What Influences the Decision-making of LLMs?

  • Puzzles and mysteries challenge all four profile methods; plot-driven questions fare worse with descriptions alone, while character-driven questions challenge memory-only profiles.The authors suggest descriptions capture characters’ overall essence, while memories provide direct access to relevant events.Source: Figure 5 · p. 7, §5.2 · p. 8
  • Science fiction, fantasy, and romance show relatively high accuracy, whereas crime and mystery perform poorly.The authors suggest stylized characters and fixed archetypes may help in the former genres, while complex logical chains and abnormal actions may challenge the latter.Source: §5.2 · p. 8, Figure 6 · p. 8
  • Early temporal inputs leave most character decisions near random accuracy (25%); character-driven accuracy tends to stay stable, while plot-driven accuracy may shift abruptly.The authors suggest sudden events may strongly influence plot-driven choices.Source: §5.2 · p. 8, Figure 7 · p. 8

6 Conclusion

The conclusion presents LIFECHOICE as a benchmark for reconstructing characters’ decisions from historical data and CHARMAP as a persona-based memory retrieval method. Experiments show promising RPLA performance on the task.

6 Conclusion

  • LIFECHOICE evaluates whether RPLAs can reconstruct decisions from historical data, and CHARMAP uses persona-based memory retrieval to enhance decision-making.The dataset includes 1,462 characters from 388 books; the authors hope the work provides better RPLA evaluation benchmarks and directs development of personal LLM assistants.Source: §6 · p. 8

Limitations

The study primarily tests whether fictional characters can recreate their choices within a book, and notes that author-designed plots may contain illogical choices.

Limitations

  • The study focuses primarily on fictional characters’ choices within books, which may not transfer directly to real-world humans.Author-designed storylines may also produce illogical choices, and fictionalized story backgrounds may affect model generations.Source: Limitations · p. 9

Ethics Statement

The ethics statement notes unresolved risks from both source materials and GPT-4-generated data. The appendices document prompts for dataset construction and role-playing, alongside a classification of novel genres.

Ethics Statement

  • The dataset cannot be guaranteed free of toxic or discriminatory language, and GPT-4-generated data may introduce biases or hallucinations.The statement identifies both novel and literary-expert materials and GPT-4 generation as risk sources.Source: Ethics Statement · p. 9

A.1 Selecting Decision Points

  • Decision-point selection prompts ask GPT-4 to identify decisions, gold motivations, and related chapters from character descriptions, chapter summaries, and book analysis.The examples and prompt are associated with the 2024 novel A Calamity of Souls.Source: §A.1 · p. 11

A.2 Locating the Node

  • A separate prompt uses previously identified decision points and chapters to locate the corresponding position in the original book and help avoid data leakage.Source: §A.2 · p. 11

A.3 Constructing Multiple-Choice Questions

  • GPT-4 constructs multiple-choice questions from the preceding step’s inputs, outputting a scenario, question, and four options.Source: §A.3 · p. 11

A.4 System Prompts for Role-Playing

B.1 Categories of novel

  • The novel-genre classification includes crime, mystery, thriller, science fiction, romance, fantasy, action-adventure, speculative, and mystery-thriller categories.Source: §B.1 · p. 11
  • The classification also includes young adult, new adult, horror and supernatural, historical, western, family saga, women’s, and magical realism novels.Source: §B.1 · pp. 11–12

B.2 Categories of motivations

The motivation categories distinguish character-driven behavior, grounded in a character’s inner world and development, from plot-driven behavior, propelled by external conflicts. The appendix lists sub-motivations for both.

B.2 Categories of motivations

  • Character-driven motivation centers on inner world, growth, and transformation, with subcategories covering traits, emotions, relationships, values, and desires.Source: §B.2 · p. 12
  • Plot-driven motivation emphasizes external conflicts and events, with external conflicts listed as a sub-motivation.Source: §B.2 · pp. 12–13
  • Other listed motivations include tasks and goals, puzzles and secrets, pursuits and escapes, exploration and discovery, power and control, and intrigue and betrayal.Source: §B.2 · p. 13

C Manual Annotation

The manual examination process has annotators score GPT-4-generated questions against literary-expert materials, retaining only data above an average-score threshold.

C Manual Annotation

  • Annotators assess GPT-4-generated questions using character descriptions, chapter summaries, and book analyses written by human literature experts.Source: C Manual Annotation · p. 13
  • Only data with an average examination score greater than 6 points are retained.Annotation participants are compensated at least at the local minimum hourly wage.Source: C Manual Annotation · p. 13

D Future direction

The authors identify fictional-character data, reasoning efficiency, and the breadth of decision tasks as future research concerns. The manual examination guidelines assess question comprehensiveness, consistency, challenge, and alignment with character motivation.

D Future direction

  • Fictional characters’ author-designed behavior can contain logical inconsistencies, unlike real-world human behavior, motivating exploration of real-human historical and behavior data.Source: D Future direction · p. 13
  • Improving CHARMAP requires balancing reasoning efficiency, which depends on input-token length, against achieving superior outcomes.Source: D Future direction · p. 13
  • The selected fictional decisions create a large decision space without a fixed task framework, making more systematic tasks informed by social sciences a future challenge.Source: D Future direction · p. 13
  • The comprehensiveness rule checks whether questions reflect the character’s background, context, and motivations.Source: D Future direction · p. 20, Table 14

2. Logical Consistency

  • The logical-consistency rule checks whether each question fits the narrative and the character’s plausible decision process.Source: §2 · p. 20

3. Challenge Level

  • The challenge-level rule checks whether incorrect options are believable, while the alignment rule checks whether questions clearly test the character’s story decision and motivations.Source: §3 · p. 20, §4 · p. 20
Loading 2404.12138v2…