Source-linked AI summary
Do LLMs Understand Personality? Rethinking Persona Fidelity Evaluation through Structured Behavioral Inference
Mengfan Li, Zesheng Wei, Xuanhua Shi, Yang Deng
TL;DR
Persona fidelity evaluation must capture whether responses reflect a character’s psychological and stylistic behavior, not merely reproduce persona facts. PRISM addresses this with structured multidimensional inference, outperforming holistic judges in accuracy, reliability, and stability.
Problem
Existing evaluations emphasize factual consistency or holistic judgments, leaving nuanced psychological and stylistic persona alignment insufficiently captured in dynamic dialogue.
Method
PRISM reformulates persona fidelity as inverse structured inference across task framing, interpersonal stance, and linguistic style, producing interpretable dimension-specific evidence.
Results
PRISM consistently outperforms holistic judges across three benchmarks, with greater reliability, robustness, and stability across evaluators and scoring rubrics.
Takeaways & Limitations
Persona fidelity is more reliably assessed through structured dimension-level evidence than through a single holistic judgment.
Takeaways & Limitations
PRISM requires token-level log-probabilities, making it most naturally applicable to open-source evaluator backbones rather than closed-source models.
Abstract
from arXiv · showhide
As large language models are increasingly deployed to simulate diverse human characters, ensuring persona fidelity, defined as the extent to which an agent's behavior consistently reflects the psychological and stylistic characteristics of a target persona, has become a critical requirement. However, existing evaluation paradigms primarily rely on either holistic LLM-based judges, which are prone to "holistic appraisal hallucination'', or static psychometric inventories, which fail to capture the context-dependent fidelity required in dynamic dialogue. To address these limitations, we propose PRISM (Persona Reasoning with Inverse SFL-based Modeling), a psycholinguistically grounded framework that reformulates persona fidelity evaluation as a structured inverse inference task. Inspired by Systemic Functional Linguistics (SFL), PRISM decomposes persona fidelity into three functional dimensions: Task Framing, Interpersonal Stance, and Linguistic Style. It estimates dimension-specific evidence over a persona-conditioned label space and aggregates these signals into an interpretable and auditable evaluation process. Experiments show that PRISM yields more accurate and stable judgements than traditional holistic judging, providing a more reliable framework for persona fidelity evaluation.
1 Introduction
The paper defines persona fidelity as multidimensional behavioral consistency rather than factual recall or surface similarity, and introduces PRISM to evaluate it through interpretable, dimension-specific inference. PRISM consistently outperforms holistic judges, reducing appraisal hallucination while remaining stable across evaluator backbones and scoring rubrics.
- Persona fidelity concerns alignment with a character’s underlying personality and behavioral style, which factual recall, semantic similarity, and simple matching may miss.An agent can mention persona-related facts correctly while still deviating in nuanced psychological or stylistic ways.
- PRISM decomposes persona consistency into Task Framing, Interpersonal Stance, and Linguistic Style, treating fidelity as a structured multidimensional behavioral-consistency problem.These dimensions capture how an agent frames goals, negotiates relationships, and expresses language, not only what it says.
- PRISM reformulates evaluation as inverse structured inference, estimating dimension-specific posterior distributions over Aligned, Indeterminate, and Contradictory labels conditioned on the target persona.Aggregating these signals produces interpretable, psycholinguistically grounded, and auditable persona evidence.
- The authors construct Big5-Persona-EASY, Big5-Persona-HARD, and Social-Persona benchmarks with contextually plausible hard negatives to evaluate persona-fidelity assessors.The hard negatives subtly violate the target persona while remaining contextually plausible, enabling controlled reliability tests.
- PRISM consistently outperforms traditional holistic judges in persona-fidelity reliability and robustness, with greater stability across evaluator backbones and scoring rubrics.Its functional decomposition also mitigates holistic appraisal hallucination, where fluent but out-of-character responses are overrated.
2 Related Work
Related work distinguishes factual persona consistency from persona fidelity, arguing that accurate recall does not ensure behaviorally faithful role-playing. PRISM addresses this gap by replacing single-score appraisal with an interpretable, multidimensional evaluation grounded in SFL.
- Personalization for Role-playing Agents: Existing evaluation mainly checks whether agents reproduce persona-related facts, but factual consistency alone does not ensure persona-consistent behavior.This motivates shifting evaluation from “what the agent knows” to “how the agent behaves.”
- Methodologies for Persona Fidelity Evaluation: Persona fidelity evaluation has followed holistic appraisal and psychological probing, with LLM judges assigning scalable single consistency scores but risking surface-fluency and helpfulness biases.The cited passages identify holistic appraisal as susceptible to “holistic appraisal hallucination” and helpfulness bias (Zheng et al., 2023).
- Methodologies for Persona Fidelity Evaluation: PRISM formulates persona fidelity as a structured, interpretable consistency problem that decomposes persona-consistent behavior into multiple functional dimensions for fine-grained diagnosis.This design targets subtle behavioral deviations that single-score holistic judgments may miss.
- Systemic Functional Linguistics: Drawing on SFL’s view of language as social-contextual meaning-making, PRISM organizes evidence into Task Framing, Interpersonal Stance, and Linguistic Style.These dimensions capture activity or communicative goal, relational position toward the interlocutor, and characteristic linguistic expression, respectively.
3 PRISM Evaluation Framework
PRISM formulates persona fidelity as a persona-conditioned inverse structured evaluation problem, decomposing it into task framing, interpersonal stance, and linguistic style. It estimates interpretable dimension-level evidence from restricted persona-conditioned labels and averages these signals into a diagnostic fidelity score.
- 3 PRISM Evaluation Framework: PRISM decomposes persona fidelity into three interpretable, psycholinguistically grounded dimensions: task framing, interpersonal stance, and linguistic style.This formulation follows Systemic Functional Linguistics.
- Persona-Conditioned Label Space Construction: For each dimension, PRISM uses persona-conditioned labels for aligned, indeterminate or mixed, and opposite or non-aligned latent states.The labels are dimension-specific and relative to the target persona, allowing a response to align on some dimensions while deviating on others.
- Inverse Posterior Estimation: Given context and response, PRISM estimates normalized model support over the restricted label space through a dimension-specific inverse prompt.Randomly permuting displayed label order and remapping outputs to canonical states reduces label-position bias.
- Inverse Posterior Estimation: PRISM uses aligned-state probability as each dimension’s consistency signal and averages the three dimension scores into the final persona fidelity score.The resulting dimension scores quantify how strongly a response expresses persona-consistent latent states along each dimension.
- 3 PRISM Evaluation Framework: The framework replaces opaque end-to-end ratings with interpretable sub-decisions while preserving diagnostic granularity about aligned and misaligned persona aspects.Individual dimension scores reveal which aspect of persona realization is aligned or misaligned beyond the final fidelity score.
4 Experimental Details
The experiments introduce three persona-fidelity benchmarks, evaluate nine LLM-based judges with ranking metrics, and show that PRISM consistently improves over direct judging baselines on open-source backbones.
- Dataset construction: The study constructs Big5-Persona-EASY, Big5-Persona-HARD, and Social-Persona because no persona-fidelity benchmarks were available.The Big5 datasets derive from Big5-CHAT, while Social-Persona derives from SocialBench’s role-style subset.
- Dataset construction: Big5-Persona-EASY uses opposite-profile substitutions, whereas Big5-Persona-HARD introduces cross-trait and response-level near-miss negatives requiring fine-grained behavioral discrimination.The experiments sample 2,000 EASY and 3,000 HARD instances, and reformulate SocialBench candidates as independent profile-context-response triplets.
- Main results: On Social-Persona, PRISM with Qwen raises P-AUC from 84.34 to 91.08 and G-Acc from 49.32 to 78.78 over Vanilla.Across all three open-source backbones, PRISM consistently outperforms Vanilla and CoT on P-AUC and G-Acc.
- Main results: PRISM reaches 68.20 G-Acc on Big5-Persona-HARD, exceeding GPT-5.4 with CoT at 46.6 despite GPT-5.4 being a stronger judge.The result indicates that stronger external judges do not necessarily outperform PRISM.
5 Analysis of Persona Fidelity Evaluation
The analysis finds that holistic LLM judges are unstable across evaluator backbones, rubrics, and decoding temperatures, whereas PRISM is more stable through structured persona-conditioned dimensions. Individual dimensions provide informative but dataset-dependent signals, while aggregation is needed because each captures only part of the relevant evidence.
- Diagnostic value of persona dimensions: Single-dimension scoring outperforms holistic judging, but the most informative dimension varies by dataset and no dimension consistently dominates Big5-Persona-HARD.Interpersonal stance and linguistic style are particularly effective on Big5-Persona-EASY, while linguistic style is strongest for most models on Social-Persona.
- Necessity of multi-dimensional aggregation: On Big5-Persona-HARD, Vanilla judging often overlaps positive and negative score distributions, while single dimensions improve but capture only part of the relevant evidence.This overlap indicates that direct judging can assign overly high scores to contextually plausible hard negatives, motivating multi-dimensional aggregation.
- Stability of holistic judges: PRISM achieves stronger mean Pair-AUC performance with smaller variance across evaluator backbones, and remains less sensitive to evaluator choice than holistic direct judging.The same qualitative trend holds under G-Acc, indicating that PRISM’s stability is not limited to Pair-AUC.
- Stability of holistic judges: Direct LLM-as-a-judge scores change across 5-point versus 7-point rubrics and are also affected by evaluator models and decoding stochasticity.These shifts occur across datasets and evaluator families, with the same rubric pattern under G-Acc.
6 Conclusion
PRISM is a persona-conditioned inverse structured framework for assessing persona fidelity through three Systemic Functional Linguistics dimensions. Across three benchmarks, it consistently outperforms direct LLM-as-a-judge baselines, particularly on harder benchmarks and under stricter group-level metrics.
- PRISM is introduced as a persona-conditioned inverse structured evaluation framework for persona fidelity assessment.
- Grounded in Systemic Functional Linguistics, PRISM models persona-relevant behavior through task framing, interpersonal stance, and linguistic style.
- Across three benchmarks, PRISM consistently improves over direct LLM-as-a-judge baselines, especially on harder benchmarks and under stricter group-level metrics.
Limitations
PRISM’s limitations concern the theoretical scope of its SFL-based dimensions, its dependence on internal probability access, and its focus on diagnosis rather than generative alignment. These constraints limit the framework’s universality, applicability to closed-source models, and role in training pipelines.
- Theoretical Scope of Functional Dimensions: PRISM’s three SFL-based dimensions are theoretically principled and empirically robust in the experiments, but alternative sociolinguistic or discourse-theoretic frameworks may characterize persona expression differently.The framework organizes persona-relevant behaviors into task framing, interpersonal stance, and linguistic style.
- Requirement of Internal Probability Access: PRISM requires token-level log-probabilities to estimate posteriors over structured labels, making it most naturally applicable to open-source evaluator backbones.Further research could investigate structured Chain-of-Thought reasoning or explicit dimensional decomposition for closed-source models.
- Diagnostic Evaluation vs. Generative Alignment: PRISM diagnoses persona misalignments but does not investigate integrating its signals into training or alignment pipelines, including use as an RLHF reward signal.The work evaluates persona fidelity rather than actively improving persona-consistent generation.
Ethical Considerations · A Dataset Construction · B Experimental Details
The paper uses openly licensed benchmarks and models for academic research, with ChatGPT limited to paraphrasing and language polishing. Its datasets convert persona-evaluation cases into labeled positive and negative triplets, including minimally perturbed hard negatives and distractor-based negatives.
- Ethical Considerations: Open-source Big5-CHAT, SocialBench, Qwen2.5, Llama3.1, and Mistral are used under their licenses and intended academic-use conditions.
- Ethical Considerations: ChatGPT is used only for limited paraphrasing and language polishing of author-written text.
- A Dataset Construction: The dataset construction section provides representative examples and summarizes how positive and negative candidates are created.
- A Dataset Construction: Big5-Persona-EASY and Big5-Persona-HARD use persona-conditioned positive responses for high and low Big Five trait levels under the same scenario.
- A Dataset Construction: Hard negatives are constructed by minimally disturbing each positive triplet (p, c, r), with examples shown in Tables 3 and 4.
- A Dataset Construction: Social-Persona converts each open-ended character-profile case into labeled triplets, treating the gold option as positive and each distractor as negative.
B.1 Experimental Setup · B.2 Prompt Templates · B.3 Label Space Construction
The experiments use deterministic inference on four NVIDIA L40 GPUs, while PRISM evaluates persona fidelity through dimension-specific latent label spaces rather than direct scalar prediction. Prompt templates support the main experiments and rubric sensitivity analysis, and label construction differs between trait-polarity and profile-cue personas.
- B.1 Experimental Setup: Main experiments run on a single server with four NVIDIA L40 GPUs using CUDA 12.6 and vLLM for efficient inference across open-source backbones.The main results use deterministic decoding: greedy generation for open-source models and temperature 0.0 for closed-source APIs.
- B.2 Prompt Templates: Figures 8 and 9 provide the direct LLM-as-a-judge prompts for the main experiments, while Figures 10 and 11 provide 7-point prompts for rubric sensitivity analysis.The prompt figures distinguish the main evaluation templates from those used to test rubric sensitivity.
- B.3 Label Space Construction: PRISM constructs a latent label space for each functional dimension and performs inverse posterior estimation instead of directly predicting a scalar consistency score.Across dimensions, the label space uses three states: aligned, weakly marked or underspecified, and non-aligned.
- B.3 Label Space Construction: For Big5-Persona-HARD and Big5-Persona-EASY, aligned labels represent the target trait polarity, whereas contradictory labels represent the opposite polarity of that trait.The representative EASY case keeps the dialogue context and response fixed while replacing the target profile with the opposite Big Five trait polarity.
- B.3 Label Space Construction: Big5-Persona-HARD pairs positive responses with two hard negative types, as illustrated by its representative case.The supplied passage identifies the hard-negative pairing but does not specify the two negative types.
- B.3 Label Space Construction: For Social-Persona, aligned labels capture profile-specified personality cues, while non-aligned labels capture states outside those cues rather than opposite trait polarities.The example profile includes innocent, naive, and adventurous cues; weakly marked, mixed, flat, generic, or hard-to-read responses form the intermediate state.
C Supplement Results · D Human Evaluation
The supplement examines PRISM’s sensitivity and score distributions across backbones, rubrics, temperatures, and benchmarks, while human evaluation verifies persona consistency using three annotators and a 5-point rubric.
- C Supplement Results: The supplement reports backbone and rubric sensitivity analyses under G-Acc in Figures 12 and 13.These analyses assess how evaluation scores vary with the selected backbone and rubric.
- C Supplement Results: Temperature sensitivity of Vanilla evaluation is reported on Big5-Persona-HARD for Qwen, Mistral, and Llama backbones in Figures 16–18.The three figures provide backbone-specific temperature analyses for the same benchmark and evaluation method.
- C Supplement Results: Table 5 gives a representative Social-Persona case with an open-ended character description, one gold response, and multiple distractors under the same dialogue context.This case differs from Big5-based benchmarks because the target profile is specified as an open-ended character description.
- D Human Evaluation: Table 6 reports human verification results from a stratified sample spanning all three benchmarks.The table evaluates persona consistency rather than presenting an automated sensitivity analysis.
- D Human Evaluation: Three annotators rate persona consistency using a 5-point rubric in the human verification study.The supplied table description specifies the annotation design but does not provide the numerical results.
- C Supplement Results: Figures 14 and 15 present score distribution analyses for Big5-Persona-EASY and Social-Persona, respectively.The supplement therefore examines score distributions across both a Big5-based benchmark and the Social-Persona benchmark.
D.1 Benchmark-level Human Verification · D.2 Dimension-level Diagnostic Validation
The authors validate both the benchmark-level persona-fidelity contrast and PRISM’s ability to diagnose which behavioral dimension a response violates. Human studies use stratified contrastive samples, shared annotation procedures, and reliability or diagnostic metrics, with results reported in Tables 6 and 7.
- D.1 Benchmark-level Human Verification: Benchmark construction starts from persona-consistent responses with explicitly specified dimensions, perturbs them across traits, and post-validates samples to reduce dependence on specialized psychological knowledge.The retained samples are intended to be more general and suitable for role-playing evaluation.
- D.1 Benchmark-level Human Verification: The benchmark-level study samples 50 contrastive groups from each of three benchmarks, totaling 150 groups, with Big5-Persona-HARD balanced across negative construction types.The study is designed to test whether reformulated instances preserve the intended contrast between persona-consistent and persona-inconsistent responses.
- D.1 Benchmark-level Human Verification: Three NLP graduate-student annotators rate each candidate on the same 5-point persona-fidelity rubric used in direct judging, with Krippendorff’s α reported using ordinal distance.Annotators receive the target profile, dialogue context, and candidate response; results are reported in Table 6.
- D.2 Dimension-level Diagnostic Validation: PRISM’s dimension-level validation tests whether its scores identify the specific persona-fidelity aspect in which a response deviates.This complements benchmark-level verification of overall persona-fidelity contrast.
- D.2 Dimension-level Diagnostic Validation: The dimension-level study annotates 450 responses across Social-Persona, Big5-Persona-EASY, and Big5-Persona-HARD, balancing the hard benchmark across its two negative construction types.The sample comprises 200 Social-Persona, 100 Big5-Persona-EASY, and 150 Big5-Persona-HARD responses.
- D.2 Dimension-level Diagnostic Validation: Annotators assign three-way labels for task framing, interpersonal stance, and linguistic style, while PRISM ranks dimensions by aligned-state probabilities from the Llama evaluator backbone.Option order is randomized and mapped back to canonical categories, and majority votes are used for analysis.
- D.2 Dimension-level Diagnostic Validation: Dimension-level diagnostics use Top-1 Accuracy, Top-2 Recall, and Macro-F1 to evaluate whether PRISM’s lowest-scoring dimensions match human-identified violations.Top-2 Recall accommodates responses that violate multiple persona-fidelity aspects, while Macro-F1 treats persona-inconsistent labels as violations.
- D.2 Dimension-level Diagnostic Validation: Human-label reliability for the three behavioral dimensions is measured with Krippendorff’s α using ordinal distance, with diagnostic results reported in Table 7.Table 7 is identified as the dimension-level human diagnostic validation of PRISM using the Llama evaluator backbone.