Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Works

1 Introduction

• Character profiling offers a more nuanced generative assessment than prior character classification and characteristic-imitation tasks.The paper frames profile generation as a challenging, understudied practice already used in role-playing-agent development.Source: §1 · p. 1
• CROSS and two evaluation tasks address the lack of high-quality datasets and protocols for evaluating generated character profiles.The dataset draws on literature experts; evaluation compares profiles with references and tests whether they support motivation recognition.Source: §1 · pp. 1–2
• Experiments across summarization methods and LLMs find profiles satisfactory but leave room for improvement, with all four profile dimensions important for motivation recognition.The results also suggest possible information loss in role-playing agents built on generated profiles.Source: §1 · p. 2
• The paper presents an evaluation framework, a literature-expert-sourced dataset, and experiments demonstrating the promise of LLM-based character profiling.These are the stated contributions of the work.Source: §1 · p. 2

2 Related Work

• RPA development approaches include manual construction, online resource collection, and automatic extraction using LLMs.The paper explores LLMs’ ability to generate character profiles for RPA construction.Source: §2 · pp. 2–3
• Prior character-understanding work covers identification, personality prediction, and question answering, but not motivation recognition based on character profiles.The proposed task evaluates whether models identify the motivations behind each character decision.Source: §2 · p. 3
• Character motivation reflects personality and surroundings, linking decisions in narratives to a character’s inner traits and external relationships.The paper uses this connection to motivate assessing decision motivations from profiles.Source: §2 · p. 3

3.1 Task Formulation

• Given a character name and fictional work, the model generates a profile covering core character information.The profile is represented through attributes, relationships, events, and personality.Source: §3.1 · p. 3, §3.2 · p. 3

3.2 Character Profile Dimensions

• The four profile dimensions are attributes, relationships, events, and personality, with events ordered chronologically even when narratives use complex timelines.Attributes include background, skills, talents, objectives, and gender; relationships connect to experiences and personality.Source: §3.2 · p. 3

3.3 Summarization Methods

• Long-book processing compares hierarchical merging, incremental updating, and one-go summarization, with the latter used for books under 120,000 tokens.Hierarchical merging combines segment summaries iteratively; incremental updating carries prior context forward, addressing the risk that context-free segment summaries may misinterpret later material.Source: §3.3 · pp. 3–4, Figure 2 · p. 4

4.1 Evaluation Tasks

• Factual consistency examination compares four-section generated profiles with reference profiles, while Motivation Recognition tests whether profiles help models identify decision motivations.Source: §4.1 · pp. 4–5

4.2 Evaluation Metrics

• Consistency Score uses Llama-3-70B to rate factual alignment from 1–5; Motivation Recognition is measured by dataset-wide accuracy.Source: §4.2 · p. 5

4.3 CROSS Dataset Construction

• The dataset uses 126 novels published in 2022 and 2023, focuses on each main character, and limits one-go books to fewer than 120,000 tokens.Source: §4.3 · p. 5
• CROSS draws on expert SuperSummary summaries and analyses, reorganized by GPT-4 into four dimensions and checked by two annotators for information integrity.Source: §4.3 · p. 5
• GPT-4 generated 641 motivation questions across 126 characters with structurally similar alternative motivations; annotators retained 445 of 641 questions.Source: §4.3 · p. 6

• Incremental and hierarchical summarization tested nine LLMs with 3,000-token chunks and profiles capped at 1,200 words; one-go used GPT-4-Turbo and Claude-3-Sonnet.All models used temperature 0 and their original instruction formats.Source: §5 · p. 7
• Motivation Recognition used GPT-4-Turbo-0409; two annotators also answered all questions to establish human accuracy and standard deviation.Source: §5 · p. 7
• Dimension ablations used incremental GPT-4 profiles, with each experiment repeated three times and mean and standard deviation reported.Source: §5 · pp. 7–8

6.1 Can LLMs generate character profiles from fictional works precisely?

• GPT-4 consistently outperforms other models across summarization methods, while larger models such as Qwen1.5-72B-Chat tend to score higher in consistency.Across dimensions, models generally capture personality more consistently than events.Source: §6.1 · p. 8
• Profiles from summarizing-in-one-go achieve the highest consistency scores in every dimension, surpassing methods that process book content in segments.The authors attribute this to preserving narrative coherence and reducing information loss by processing the whole book at once.Source: §6.1 · p. 8
• Incremental updating slightly trails hierarchical summarizing in average consistency but performs better for events.Its iterative passes allow the model to revise its understanding as more information arrives or errors are corrected; this accords with Chang et al. (2023), who report greater detail in incremental book summaries.Source: §6.1 · p. 8
• Case analysis identifies five profile errors: character and relationship misidentification, key-information omission, event misinterpretation, and character misinterpretation.The event category includes failures to revise earlier interpretations after later revelations.Source: §6.1 · p. 8
• Complex narrative structures can confuse models: shifting perspectives lead to Benjamin Rask/Andrew Bevel trait confusion, while a plot twist in The Housemaid’s Secret yields an incorrect final summary.The examples appear in the character-misidentification and event-misinterpretation cases.Source: §6.1 · p. 8

6.2 Can LLMs recognize the character’s motivation for a specific decision?

• Consistency scores strongly positively correlate with Motivation Recognition accuracy, supporting the validity of profiling and the link between accurate profiles and motivation understanding.The authors interpret this relationship as evidence that accurate profiles help models understand motivations behind character behavior.Source: §6.2 · p. 9
• Ablations show all four profile dimensions contribute to Motivation Recognition, with removing events alone lowering accuracy by 9.21%.The authors say events supply plot context for decisions and integrate information from other dimensions; reducing profile information also correlates with greater outcome variance.Source: §6.2 · p. 9

7 Conclusion

• The paper introduces a character-profiling evaluation task using 126 character profiles from novels and evaluates factual consistency and motivation recognition.Source: §7 · p. 9
• LLMs generally perform well, but even advanced models sometimes hallucinate or err, particularly on complex narratives.The authors identify a need for further improvement.Source: §7 · p. 9

Limitations

• The study evaluates only four common profile dimensions, leaving other possible dimensions and their downstream effects outside its scope.The authors identify broader dimensions as a direction for future work.Source: Limitations · p. 9
• Evaluation may be biased if models encountered the selected books during training or if evaluator-LLM errors distort profile-consistency estimates.The authors also note that the three tested summarization methods could be improved.Source: Limitations · p. 9

Ethics Statement

• Annotators were paid above the local minimum wage, their privacy was respected, and they consented to research use of processed data.The institution recruited annotators for motivation-recognition dataset construction.Source: Ethics Statement · p. 9
• The authors cannot guarantee publicly sourced CROSS data are free of harmful or toxic language, and motivation-recognition data quality judgments may vary across people.The latter quality evaluation is based on common sense.Source: Ethics Statement · p. 9

A.1 Dimension

• Character-summary studies often target a single dimension, while multidimensional RPA summaries lack systematic quality assessment.The cited dimensions include role, relationships, personality, and mental states; multidimensional summaries include attributes, appearance, relationships, and storyline.Source: §A.1 · p. 12

A.2 Evaluation

• The framework covers character information across dimensions, including objectives, emotions, beliefs, relationships, and interactions with the external world.Objectives reflect desires and intentions, while emotions and beliefs can emerge through reactions and behavior in event profiles.Source: §A.2 · p. 12

A.3 Application

• Profiles may provide information for fictional-character chatbots, interactive narratives, and study guides, but the total word limit can leave some dimensions less detailed.The profiles may be less detailed on some dimensions than summaries devoted solely to those dimensions.Source: §A.3 · p. 12

B.1 Dataset Construction

• The dataset comprises 126 fictional novels, selected from 2022–2023 publications and excluding biographical novels and works based on real historical figures.The selection also excludes sequels unless they can be regarded as independent works.Source: §B.1 · p. 12
• Experts’ SuperSummary book summaries and character analyses were used with developer permission, and only reorganized profiles are released to protect copyright.The materials are used for academic research purposes.Source: §B.1 · p. 12

B.2 Integrity Verification of CROSS dataset

• 3.60 vs 3.62 average consistency scores for 2022- and 2023-published books, and 3.58 vs 3.62 for best-selling and other books, are described as very close.The authors conclude that publication year and sales volume do not significantly affect task performance within CROSS.Source: §B.2 · pp. 12–13, Table 5 · p. 13, Table 6 · p. 13
• Consistency scores are much higher on ten top-rated 20th-century classics than on CROSS, which the authors say suggests their selection reduces data-leakage impact.They attribute the classic-book performance to accumulated related training corpora, including literary analyses and fan creations.Source: §B.2 · p. 13, Table 7 · p. 14
• −0.037 Spearman correlation (p = 0.799) indicates no significant association between publication year and average consistency across 50 books from the past ten years.The authors infer that prior exposure to some recent books is insufficient for strong performance based on memory alone.Source: §B.2 · p. 13, Figure 3 · p. 14
• The authors speculate CROSS will remain effective for updated LLMs for at least the next few years and describe a framework for continually expanding it.They state they will keep updating the dataset and evaluating new LLMs.Source: §B.2 · p. 13

C.1 Incremental Updating

• Incremental updating processes book chunks sequentially, repeatedly incorporating each new chunk into the running summary and compressing it when it exceeds the maximum length.The chunk size is 3,000 tokens and the maximum summary length is 1,200.Source: C Detailed Information of… · p. 13, §C.1 · p. 13

C.2 Hierarchical Merging

• Hierarchical merging first summarizes chunks separately, then repeatedly merges summaries in context-window-limited groups until a final summary is produced.Subsequent merges may use the prior level-2 summary as context.Source: §C.2 · pp. 13–14

C.3 Summarizing in One Go

• Summarizing in one go provides the whole book to the model and generates the final summary at once, when the book and prompt fit the context window.This condition is stated for GPT-4 and Claude-3-Sonnet.Source: §C.3 · p. 14

D.1 Reference Profile Examination

• Two annotators found no plot inconsistencies or misjudgments of character traits in any CROSS reference profile they checked.They compared GPT-4-reorganized profiles with the original plot summaries and character analyses.Source: §D.1 · p. 14

D.2 Manual Evaluation of FCE

• 0.752 Pearson correlation (p = 4.3e−12) between average human and Llama-3-70B consistency scores is reported as evidence of comparable evaluation ability.The comparison uses 50 sampled profiles and the same scoring prompt for people and the model.Source: §D.2 · p. 14

D.3 Motivation Recognition MCQs filtering

• 445 motivation-recognition questions remained after filtering, with Fleiss’s κ = 0.91; annotators checked decision-maker, motivation focus, story significance, and leakage criteria.Options were rearranged to distribute correct answers fairly.Source: §D.3 · p. 15

E Traditional Metrics on Generated Profiles

• Traditional metrics broadly match consistency-score patterns: larger, more complex LLMs perform better, event profiles often score lowest, and one-go summarization outranks hierarchical and incremental methods.Relative model performance varies by metric because ROUGE-L and BLEU emphasize lexical matching, BERTScore semantic similarity, and consistency score factual accuracy.Source: E Traditional Metrics on Generated… · p. 15

F.1 Summarizing in One Go

• Claude-3-Sonnet sometimes forgets the requested four-section format when the prompt precedes novel content, so the prompt is placed after the content.The authors attribute this behavior to limitations in the model’s long-context capabilities.Source: §F.1 · pp. 15–16