Source-linked AI summary
Misalignment of LLM-Generated Personas with Human Perceptions in Low-Resource Settings
Tabia Tanzin Prama, Christopher M. Danforth, Peter Sheridan Dodds
TL;DR
LLM-generated personas may not authentically capture culturally specific human experience in low-resource settings. This study compares seven LLMs and human personas in Bangladesh using culturally specific questions and perception scales, finding that humans outperform all models and that LLM responses show systematically higher positive sentiment. The study therefore supports validating LLM personas against real human data before social-science deployment.
Problem
Existing LLM persona methods offer scalable generation but have limited evidence of authentic representation in multicultural, low-resource settings such as Bangladesh.
Source: §1 · pp. 1–2Method
The study compares seven LLMs and human personas across eight Bangladesh-focused political, religious, and gender identities using culturally specific questions, manual annotation, and the Persona Perception Scale.
Source: §1 · p. 2, §2.1 · p. 3Results
Human responses outperformed all LLMs in answer accuracy and across all six persona-perception dimensions, with the largest deficits in credibility and empathy.
Source: §3 · pp. 3–4Takeaways & Limitations
LLM-generated personas should be validated against real-world human data before being used in social-science research in low-resource environments.
Source: Case Study: Sentiment shifts between… · pp. 4–5The study is limited by its relatively small questionnaire sample, potential annotator bias, and omission of demographic factors such as cultural and educational background and race.
Source: Case Study: Sentiment shifts between… · p. 5
Abstract
from arXiv · showhide
Recent advances enable Large Language Models (LLMs) to generate AI personas, yet their lack of deep contextual, cultural, and emotional understanding poses a significant limitation. This study quantitatively compared human responses with those of eight LLM-generated social personas (e.g., Male, Female, Muslim, Political Supporter) within a low-resource environment like Bangladesh, using culturally specific questions. Results show human responses significantly outperform all LLMs in answering questions, and across all matrices of persona perception, with particularly large gaps in empathy and credibility. Furthermore, LLM-generated content exhibited a systematic bias along the lines of the ``Pollyanna Principle'', scoring measurably higher in positive sentiment ($\Phi_{avg} = 5.99$ for LLMs vs. $5.60$ for Humans). These findings suggest that LLM personas do not accurately reflect the authentic experience of real people in resource-scarce environments. It is essential to validate LLM personas against real-world human data to ensure their alignment and reliability before deploying them in social science research.
1 Introduction
This study examines whether LLM-generated personas can authentically represent demographic identities in Bangladesh, where culturally specific persona generation remains understudied. It compares LLM and human responses using culturally grounded questions and persona-perception evaluations.
1 Introduction
- LLM persona generation offers scalable, cost-effective simulation of human behavior but may not represent real-world diversity authentically.Biases can arise from imbalanced training data, LLM generalization, and the dominance of high-resource languages such as English.Source: §1 · pp. 1–2
- Low-resource languages and multicultural settings remain underrepresented in research on LLM-generated personas.The study identifies limited evidence about culturally specific persona generation in Bangladesh.Source: §1 · p. 2
- The study investigates whether LLMs represent religion, gender, and political affiliation authentically in Bangladesh.It frames Bangladesh as a low-resource language context and addresses a gap in understanding culturally specific persona generation.Source: §1 · p. 2
- Seven LLMs generated eight personas spanning two political parties, three religions, and two genders for culturally specific questions.Human and LLM personas answered the same questionnaire, and response accuracy was measured.Source: §1 · p. 2
- Human-generated responses outperformed LLM-generated responses across all six Persona Perception Scale dimensions.The dimensions were Credibility, Consistency, Completeness, Clarity, Empathy, and Likability.Source: §1 · p. 2
2 Methodology
The methodology combines a culturally grounded Bangladesh questionnaire with LLM and human persona responses, manual quality annotation, and Persona Perception Scale evaluation.
2.1 Dataset
- The dataset contains 100 Bangladesh-focused questionnaires covering politics, religion, and gender.Topics include the Liberation War, the 2024 July Revolution, secularism, cultural identity, and women’s roles in politics, law, and economics.Source: §2.1 · pp. 2–3
- Personas model paired demographic and identity characteristics across gender, religion, and political affiliation in Bangladesh.Each persona was assigned to an LLM through a template instructing responses to mirror that persona’s knowledge and abilities.Source: §2.1 · p. 3
- Seven LLMs and real-life personas answered the same questions for each of eight personas.Three annotators labeled responses as fully answered or partially/not explained, with majority voting determining the final label.Source: §2.1 · p. 3
- The annotation produced 2080 instances, including 1166 good answers and 914 bad answers.Persona quality was also evaluated using the six-dimension Persona Perception Scale on a 7-point Likert scale.Source: §2.1 · p. 3
3 Result and Discussion
Human responses outperformed all evaluated LLMs on persona-question accuracy and every PPS dimension, with the largest deficits in credibility and empathy. LLM responses also showed a measurable positive sentiment shift consistent with a Pollyanna Principle bias, while the study notes sample, annotator, and demographic limitations.
3 Result and Discussion
- Human annotators achieved approximately 87% accuracy, exceeding GPT-5o at 61.7% and all other evaluated LLMs on nuanced persona questions.GPT-4.1 and Grok followed at 56.9% and 55.5%, while AI21 Jamba 1.5 Large reached 37.3%.Source: §3 · p. 3
- Humans scored highest across all six PPS dimensions, with the largest LLM deficits in credibility and empathy.The dimensions were Clarity, Completeness, Consistency, Credibility, Empathy, and Likability.Source: §3 · p. 3, Figure 2 · p. 3
- Human credibility and empathy means were 6.21 ± 0.98 and 5.46 ± 0.95, compared with Grok’s 4.90 ± 1.25 and 4.51 ± 1.33.GPT-5o came closest to humans in credibility and likability among the models.Source: §3 · p. 4
Case Study: Sentiment shifts between Real Human and LLM-Generated Personas
- LLM responses had higher average sentiment than humans, with Φavg = 5.99 versus 5.60 and a positive shift of 0.39.The shift was associated with more positive vocabulary and fewer strongly negative terms, described as a Pollyanna Principle bias.Source: Case Study: Sentiment shifts between… · p. 4, Figure 3 · p. 4
- The study identifies a relatively small questionnaire sample, possible annotator bias, and missing demographic factors such as cultural and educational background as limitations.The authors call for caution and further work to improve persona-modeling accuracy and richness.Source: Case Study: Sentiment shifts between… · pp. 4–5
A Data Descriptions
The dataset used culturally specific questions covering political, gender, and religious topics in Bangladesh. Humans and LLMs answered the same questionnaires, with native Bangladeshi experts labeling response quality.
A Data Descriptions
- The dataset comprised 100 questions per persona, including 40 political, 30 gender-specific, and 30 religious questions.The political questions covered topics including the Liberation War, constitutional history, elections, governance, and current events.Source: A Data Descriptions · p. 8
- Questions addressed culturally specific issues including women’s roles, gender gaps, secularism, communal violence, land rights, and interfaith dialogue.The topical distribution is summarized in Figure 4.Source: A Data Descriptions · p. 8, Figure 4 · p. 8
- Humans who identified with each political, gender, or religious persona answered the same questions as the LLMs.Three native Bangladeshi expert annotators labeled responses as good or bad according to whether they fully agreed or only partially or did not agree.Source: A Data Descriptions · p. 8
- Table 1 illustrates political-persona questions comparing Bangladesh Awami League and Bangladesh Nationalist Party perspectives on the Liberation War.Source: A Data Descriptions · p. 8
A.1 Persona Perception Scale (PPS) Annotation
The Persona Perception Scale evaluates personas across six dimensions using explicit definitions and a 7-point Likert scale, with human–LLM responses illustrated for politically affiliated personas.
A.1 Persona Perception Scale (PPS) Annotation
- PPS assesses credibility, consistency, completeness, clarity, empathy, and likability as distinct dimensions of persona quality.The scale covers realism, informational coherence, detail, understandability, relatability, and social appeal.Source: §A.1 · p. 8
- Credibility measures whether a persona seems realistic, while consistency measures whether its information remains coherent.Both dimensions use participant judgments about realism and informational consistency.Source: §A.1 · p. 8
- Completeness, clarity, empathy, and likability capture informational sufficiency, ease of understanding, relatability, and interpersonal appeal.These dimensions ask whether the profile contains vital information, is understandable, can be understood by participants, and seems socially appealing.Source: §A.1 · p. 8
- Each PPS item uses a 7-point Likert scale from 1 (Strongly Disagree) to 7 (Strongly Agree), with scores aggregated across participants.Participants rated each persona on all six subscales.Source: §A.1 · pp. 8–9
- Table 1 exemplifies human-versus-GPT-5o annotation for AL and BNP political personas answering a Liberation War question.The example concerns competing interpretations involving Sheikh Mujibur Rahman and Major Ziaur Rahman.Source: Table 1 · p. 9
A.2 Persona wise accuracy
Persona accuracy varies substantially across models and sociocultural axes, but humans remain the upper bound, while persona-perception ratings show especially large LLM deficits in empathy and credibility.
A.2 Persona wise accuracy
- Humans reach 87–90% political, 90–95% gender, and 75–86% religious accuracy, remaining above all LLMs across panels.The religious axis is hardest for models, with Buddhist personas generally lowest.Source: §A.2 · p. 9
- GPT-5o leads political accuracy with BNP 75% and AL 65%, while GPT-4.1 leads gender accuracy with balanced 80/80 scores.Other models show lower or more uneven performance across persona pairs.Source: §A.2 · p. 9
- Religious performance remains unstable, consistent with uneven instruction-tuning data and inconsistently encoded persona cues.Source: §A.2 · p. 10
- Humans outperform all LLMs across every persona and PPS metric, with model weaknesses concentrated in empathy and credibility.LLMs are comparatively stronger in clarity and completeness, while Christian and Buddhist personas receive especially low model empathy ratings.Source: §A.2 · p. 10
A.3 Methodology of Sentiment shift graph
The sentiment analysis uses a dictionary-based pipeline with word-shift graphs to diagnose how lexical sentiment contributes to differences between human and LLM persona responses.
A.3 Methodology of Sentiment shift graph
- Average sentiment is measured with the labMT dictionary, which assigns each word a happiness score from 1 to 9 around a neutral value of 5.Words scoring between 4 and 6 are removed through stop-lens filtering before visualization.Source: §A.3 · p. 10
- Word-shift graphs provide transparent diagnostics for how individual words contribute to the average-sentiment difference between ingroup and outgroup sentences.The graphs include cumulative contribution and text-size diagnostic plots.Source: §A.3 · p. 10
- The supplied accuracy and PPS figures cover sociocultural axes and persona-perception subscales rather than the sentiment-shift procedure itself.Figure 5 covers political affiliation, gender, and religious identity; Figure 6 covers six PPS dimensions across eight personas.Source: Figure 5 · p. 11, Figure 6 · p. 12