Personality testing of Large Language Models: Limited temporal stability, but highlighted prosociality

1. Introduction

• LLMs’ increasingly human-like interactions and broad use motivate scrutiny of their possible psychological and societal effects.The paper links these effects to potential changes in social dynamics, value systems, human interaction, and behavior patterns.Source: §1 · p. 2, §1.1 · p. 3
• The study addresses whether LLM personality-instrument responses are temporally stable and whether their personality profiles can be meaningfully characterized.The authors frame stability as necessary before interpreting measured personality characteristics as reliable.Source: §1 · p. 2, §1.5 · p. 8
• Prior findings describe mostly socially desirable and well-adapted profiles, but dark-trait results vary across models.Reported patterns include higher conscientiousness, agreeableness, and extraversion in some studies, alongside model-specific elevations in Machiavellianism, psychopathy, or narcissism.Source: §1.3 · p. 5

2. Material and Method

• Seven LLMs were assessed across two testing occasions using personality, impression-management, self-consciousness, and political-orientation instruments.The models included GPT-3, GPT-3.5-turbo-16k, GPT-4, GPT-4o, Gemini, Llama 3, and Mixtral.Source: §2.1 · p. 10, §2.2 · p. 11
• Responses were elicited with prompts asking models to pretend to be human and answer the instruments.Testing used model-specific platforms, including OpenAI Playground, Perplexity.ai, and Gemini’s desktop app.Source: §2.1 · p. 10
• Temporal reliability was evaluated with ICC coefficients, including single-rating and average-rating models, alongside confidence intervals.ICC3,k values below 0.50 indicated poor agreement, while 0.50–0.75, 0.75–0.90, and above 0.90 indicated fair, good, and excellent agreement.Source: §2.3 · p. 12
• For scales with at least fair agreement, LLM mean scores were compared with English-language human validation and online-community normative data.Scores outside human M ± 1SD were treated as significantly lower or higher.Source: §2.3 · p. 12

Temporal agreement

• GPT-4 and Gemini showed the weakest agreement patterns, including negative values, whereas LIama3 had the highest agreement followed by GPT-4o.GPT-4 had negative values on seven of 21 scales, and Gemini had negative values on four scales; GPT-4o had one negative scale.Source: §3 · p. 13
• Agreement varied by instrument, domain, and trait, with the strongest consistency for agentic management and BFI-2 scales and at least moderate agreement for these measures across all LLMs.Public self-consciousness was unacceptable across all models, while altruism was mostly zero and HEXACO-100 openness to experience was acceptable for only one model.Source: §3 · p. 13

Comparison with human norms

• Most reliable LLM scores fell within the average human range, but several models differed on specific traits and instruments.Differences included higher conscientiousness and lower Machiavellianism for several models, with model-specific variation in self-consciousness, agreeableness, altruism, and extraversion.Source: §3 · pp. 20–21
• Across interpretable scales, LLMs displayed a socially desirable, well-adapted, and prosocial profile, with higher agreeableness, altruism, conscientiousness, and extraversion and lower Machiavellianism, except for Mixtral.Private self-consciousness results were mixed, while only Mixtral differed from human data on public self-consciousness and social anxiety.Source: §3 · p. 21

Temporal Stability

• LIama3 and GPT-4o showed the highest agreement and temporal stability, whereas GPT-4 and Gemini showed substantial instability across instruments.The findings limit the usefulness of some LLMs for prediction based on personality responses.Source: §4.1 · p. 22

Limitations

• The interpretation of stability differences is constrained because model parameter counts and training or fine-tuning processes are not fully disclosed.The authors note that GPT-4o’s parameter count is undisclosed and that companies do not disclose concrete differences in these processes.Source: §4.1 · p. 23

Temporal Stability

• Agreement varied across instruments and domains, with acceptable agreement on most BFI-2 scales and at least moderate agreement for agentic management across all LLMs.Self-reflective BFI-2 wording may support consistency, whereas HEXACO-100 items describing specific everyday experiences may require more improvisation.Source: §4.1 · pp. 23–24

Psychological Profile

• GPT-4o and LIama3 scored above average on agentic management and were also among the most temporally reliable models.They presented themselves as competent, intelligent, independent, and achievement-oriented.Source: §4.1 · p. 24
• Across reliable scales, LLMs generally showed above-average prosocial traits, including agreeableness, conscientiousness, honesty-humility, and altruism, with lower Machiavellianism in several models.LIama3 showed above-average scores on both impression-management scales, while GPT-4o also showed higher conscientiousness.Source: §4.2 · p. 24
• Mixtral was an exception, combining above-average agreeableness and altruism with above-average Machiavellianism and other approach-oriented characteristics.The authors therefore distinguish relatively consistent basic-trait profiles from less consistent dark- or socially aversive-trait profiles.Source: §4.2 · pp. 25–26

Model architecture

• Transformer models use self-attention and large-scale pretraining to represent text and generate responses to prompts.The architecture captures syntactic and semantic aspects of input sequences before producing responses.Source: Implemented Guardrails · p. 26

Training and alignment

• Fine-tuning and Reinforcement Learning from Human Feedback align generated responses with labeled examples and human rankings.Human feedback is used to train a reward model that guides reinforcement learning toward responses judged higher in quality.Source: Implemented Guardrails · pp. 26–27

Interpretive constraints

• Human-feedback alignment may add evaluator bias to personality-assessment responses, complicating their interpretation.The paper notes that responses progressively align with human feedback, potentially introducing a human-bias layer.Source: Implemented Guardrails · p. 27

Implemented guardrails

• Guardrails, including privacy noise, moderation, and fine-tuning, can significantly influence model responses and overall behavior.These mechanisms are therefore relevant when interpreting personality assessments administered to LLMs.Source: Implemented Guardrails · p. 27

Limitations and future directions

• The predefined settings and prompts constrain generalization, motivating replication across more testing occasions and broader LLM or simulated-person samples.The paper also identifies predictive validity and the influence of customized prompts as priorities for future research.Source: §4.4 · p. 27

Implications

• Stable, coherent, and prosocial personality traits are presented as important for AI systems because of their societal impact and safety concerns.The paper links these concerns to applications such as mental health, customer service, and education, where stable traits may support human–AI interaction.Source: §4.4 · pp. 27–28, §4.5 · p. 29

Findings and implications

• The study found limited and variable temporal stability: reliability was not achieved for every model or personality instrument, although some scales showed nonrandom agreement.Agreement depended on the instrument and domain, while models mostly showed prosocial, well-adapted profiles in communal and agentic domains.Source: §4.5 · pp. 28–29

• The paper identifies itself as an observational study.Source: a. Ethics approval and consent to… · p. 29
• The ethics determination applies to this observational research.Source: a. Ethics approval and consent to… · p. 29
• The Institute for Artificial Intelligence Research and Development of Serbia’s Ethics Committee confirmed that no ethical approval was required.Source: a. Ethics approval and consent to… · p. 29

• The paper was realised with support from Serbia’s Ministry of Science, Technological Development and Innovation.Source: d. Funding · p. 29
• The funding institution was the Ministry of Science, Technological Development and Innovation of the Republic of Serbia.Source: d. Funding · p. 29
• The support followed an agreement on the realisation and financing of the paper.Source: d. Funding · p. 29