PaperlayerView exact revision on arXiv

Source-linked AI summary

Limited Ability of LLMs to Simulate Human Psychological Behaviours: a Psychometric Analysis

Nikolay B Petrov, Gregory Serapio-García, Jason Rentfrow

arXiv:2405.07248v1cs.CLcs.AIcs.CYcs.HC

TL;DR

It remains difficult to determine whether LLM text responses to standardized questionnaires reveal latent psychological traits suitable for simulating human behaviour. The study evaluates GPT-3.5 and GPT-4 with generic and silicon persona prompts using psychometric analyses of standardized personality measures. GPT-4 exhibited better psychometric properties than GPT-3.5 across the board, while fine-tuning influenced LLM trait responses.

  • Problem

    It remains difficult to determine whether LLM text responses to standardized questionnaires reveal latent psychological traits suitable for simulating human behaviour.

    Source: §1 · pp. 1, 4

  • Method

    The study evaluates GPT-3.5 and GPT-4 with generic and silicon persona prompts using psychometric analyses of standardized personality measures.

    Source: §4 · p. 17

  • Results

    GPT-4 exhibited better psychometric properties than GPT-3.5 across the board, while fine-tuning influenced LLM trait responses.

    Source: §4 · pp. 17–18

  • Takeaways & Limitations

    Current silicon-persona responses are poor signals of underlying latent traits, casting doubt on individual-level human behaviour simulation across multiple-choice tasks.

    Source: §3 · p. 14, §4 · p. 17

    The study used a mono-method approach based only on self-report data.

    Source: §4 · p. 18

Abstract

from arXiv · show

The humanlike responses of large language models (LLMs) have prompted social scientists to investigate whether LLMs can be used to simulate human participants in experiments, opinion polls and surveys. Of central interest in this line of research has been mapping out the psychological profiles of LLMs by prompting them to respond to standardized questionnaires. The conflicting findings of this research are unsurprising given that mapping out underlying, or latent, traits from LLMs' text responses to questionnaires is no easy task. To address this, we use psychometrics, the science of psychological measurement. In this study, we prompt OpenAI's flagship models, GPT-3.5 and GPT-4, to assume different personas and respond to a range of standardized measures of personality constructs. We used two kinds of persona descriptions: either generic (four or five random person descriptions) or specific (mostly demographics of actual humans from a large-scale human dataset). We found that the responses from GPT-4, but not GPT-3.5, using generic persona descriptions show promising, albeit not perfect, psychometric properties, similar to human norms, but the data from both LLMs when using specific demographic profiles, show poor psychometrics properties. We conclude that, currently, when LLMs are asked to simulate silicon personas, their responses are poor signals of potentially underlying latent traits. Thus, our work casts doubt on LLMs' ability to simulate individual-level human behaviour across multiple-choice question answering tasks.

1 Introduction

The introduction frames LLM-based human simulation as promising but difficult to validate, because text responses may not reliably reveal latent psychological traits. The study therefore applies psychometric analysis to GPT-3.5 and GPT-4 using generic and silicon personas.

Contribution

  • The study uses psychometrics to assess whether questionnaire responses reveal latent constructs, addressing limitations of analyses based mainly on summary scores.Psychometrics is presented as a framework for evaluating underlying traits that generate behaviour across tasks.Source: §1 · pp. 2–4

Motivation

  • LLMs could support experiments and agent-based simulations if they reproduce human psychological and behavioural responses.Potential applications include testing social-media, organisational, and country-wide interventions before implementation.Source: §1 · p. 2

Related work

  • Existing studies report mixed evidence about whether LLMs can simulate human behaviour across cognition, politics, economics, and theory-of-mind tasks.The introduction situates personality questionnaires within this broader research programme.Source: §1 · pp. 2–3

Problem

  • Repeatedly prompting an LLM with the same survey item can produce near-uniform responses, limiting inferential analysis, and changing temperature may not solve this problem.Generic persona prompting was proposed to introduce variation by assigning randomly sampled PersonaChat descriptions.Source: §1 · p. 3

Prior approaches

  • Prior work found that generic personas produced reliable and valid personality responses from PaLM models, while demographic silicon personas reproduced human voting preferences in GPT-3.5.These approaches motivate comparing generic descriptions with targeted demographic profiles.Source: §1 · p. 3

Study design

  • The study analyzes LLM abilities to simulate human responses using generic persona prompting and silicon persona prompting, including whether outputs closely match targeted personas.Source: §1 · p. 4

2 Methods

The methods query GPT-3.5 and GPT-4 with generic or silicon personas across standardized personality questionnaires, then extract numeric answers for psychometric analysis. The design also tests how response-format choices affect the resulting distributions.

Persona prompts

  • Researchers used 150 generic PersonaChat descriptions and 1,000 silicon personas sampled from a UK BBC personality dataset.Silicon descriptions used demographics including age, sex, ethnicity, education, occupation, income, relationships, and children.Source: §2 · pp. 4–5

Measures

  • The study administered the Big Five Inventory and eight personality-related tests comprising 104 items.Measures included affect, aggression, and creative self-efficacy or identity subscales.Source: §2 · p. 5

Prompting and sampling

  • Each prompt combined a personality instruction, persona description, test instruction, item statement, and item postamble, producing 239,200 separate API calls.The study used GPT-3.5 and GPT-4 with temperature 0.Source: §2 · pp. 4–5

Methodological constraint

  • The experiment’s interpretation was constrained because the OpenAI API did not provide log probabilities for alternative completions at the time of data collection.The study therefore used text responses and numeric extraction rather than probability-based response measures.Source: §2 · p. 5

Response processing

  • Allowing free-form responses revealed sensible answers even when the first token was non-numeric, whereas digit-only filtering produced near-uniform responses dominated by “3”.The researchers therefore extracted the first digit appearing anywhere in each response rather than discarding non-digit-first responses.Source: §2 · p. 5

3 Results

The results evaluate the psychometric properties of LLM responses across reliability, validity, response distributions, and factor structure, comparing GPT-3.5 and GPT-4 under generic and silicon personas with human data. They also examine whether the responses recover the expected personality factor structure and how trait bias varies across models and personality traits.

Overall results

  • GPT-4 with generic personas generally produced the strongest psychometric results, while both models using silicon personas showed poorer reliability and validity across the examined measures.The analyses covered internal consistency, construct validity, item-response frequencies, and confirmatory factor structure, with comparisons to human data where available.Source: §3 · p. 7

Internal consistency

  • Across LLMs and prompting variations, reliability indices generally differed little, but silicon sampling produced very low reliability (<.70) for some measures, below the psychometric threshold of .70.Source: §3 · p. 7, Figure 3 · p. 8

Internal validity

  • Big Five intercorrelations were substantially higher than human values for generic personas in both models; silicon-persona correlations remained higher but were more ambiguous.These comparisons used human data from a large representative sample.Source: Figure 4 · p. 9, §3 · p. 10

Criterion validity

  • Generic-persona data adequately correlated with personality-related criteria, with GPT-4 marginally better, while silicon-persona data showed weaker correlations than generic data and than reported by Serapio-García et al. (2023).The authors note that poor internal consistency could contribute to the weaker criterion-validity results, highlighting problems with silicon-persona prompting.Source: §3 · p. 10, Figure 5 · p. 11

Item-response distributions

  • GPT-3.5 generic responses produced close-to-uniform distributions, with most responses at “3”, while silicon prompting for GPT-3.5 and GPT-4 produced distributions skewed toward desirable Big Five traits.Source: §3 · p. 10

Structural validity

  • The GPT-4 silicon-persona confirmatory factor analysis failed because its covariance matrix was not positive definite; across models and prompting styles, LLMs showed smaller path loadings and higher latent-factor covariances than human data.Source: §3 · p. 10, Figure 7 · p. 13

Trait bias

  • Average trait bias across all traits was .63 for GPT-3.5 and .62 for GPT-4, with a statistically significant difference only for BFI Agreeableness.Trait bias is the absolute difference between human and LLM-assumed personality scores for the same demographic information.Source: Figure 8 · p. 15

Trait-bias correlates

  • Higher average trait bias was unrelated to demographic variables, negatively correlated with agreeableness, conscientiousness, extraversion, and openness, and positively correlated with neuroticism across personality traits.Source: Figure 9 · p. 16

4 Discussion

The discussion concludes that LLMs’ psychometric signals depend strongly on model and persona type, with specific demographic personas performing poorly. It also identifies instruction fine-tuning and methodological limits as important qualifications to the findings.

Generic personas

  • GPT-4, but not GPT-3.5, showed promising psychometric properties with generic personas, although GPT-4 still failed to fully recover Big Five latent factors.GPT-4’s responses had higher intercorrelations between Big Five traits than human data and lacked solid structural validity.Source: §4 · p. 17

Specific personas

  • Both GPT-3.5 and GPT-4 showed poor psychometric properties when prompted with specific demographic personas.The authors contrast this latent-trait result with work finding that LLMs can simulate political opinions from demographic data, suggesting surface-level associations may be easier to reproduce than latent traits.Source: §4 · p. 17

Model differences

  • GPT-4 exhibited better psychometric properties than GPT-3.5 across analyses, consistent with its stronger performance on broader LLM benchmarks.The authors remain agnostic about whether this reflects parameters, architecture, training data, fine-tuning, or a combination.Source: §4 · pp. 17–18

Instruction fine-tuning

  • Instruction fine-tuning was associated with more socially desirable personality responses, including greater agreeableness, conscientiousness, openness, and extraversion and lower neuroticism.The authors speculate that alignment toward congenial behavior may partly explain this trait-bias pattern.Source: §4 · p. 18

Limitations

  • The study’s mono-method design relied only on self-report data, so scenarios, multi-turn conversations, and generated text could provide complementary evidence.The authors describe these additional data sources as ways to enrich understanding of latent-trait representation.Source: §4 · p. 18

Persona construction

  • Demographic variables have weak associations with personality, whereas generic personas contain more explicit references to characteristic patterns of thinking, feeling, and behaving.This may partly explain why generic personas supported more consistent personality inferences than demographic profiles.Source: §4 · p. 18

Future research

  • The authors suggest future persona seeds should combine ground-truth personality scores with demographics and characteristic information such as hobbies or interests.This proposed direction is intended to improve the basis for creating silicon personas.Source: §4 · p. 18

Conclusion

  • Overall, the findings contradict some earlier reports of successful human-behaviour simulation and support recent work questioning whether LLMs can reliably simulate individual perspectives.The paper therefore concludes that current LLMs are not yet ready to fully simulate individual-level human psychological behaviour.Source: §4 · p. 18

A Variables and transformations used to create silicon persona descriptions from the variables in the BBC dataset

The study used variables from the BBC personality dataset to construct silicon persona descriptions, with their descriptions and possible values documented in Table 2.

B Psychological scale instruments used

The study assessed personality, creativity, aggression, and affect using standardized multi-item psychological scales with Likert-style ratings.

C Confirmatory factor analyses on the BPAQ, SSCS and PANAS scales

Confirmatory factor analyses tested whether LLM responses recovered the intended factors of the BPAQ, SSCS, and PANAS scales across models and prompting styles. Overall, the analyses found weak structural validity, despite some favorable fit or loading patterns in specific conditions.

Related papers

Loading 2405.07248v1…