PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting the Reliability of Psychological Scales on Large Language Models

Jen-tse Huang, Wenxiang Jiao, Man Ho Lam, Eric John Li, Wenxuan Wang, Michael R. Lyu

arXiv:2305.19926v5cs.CL

TL;DR

Whether human-designed psychological scales reliably assess LLMs remains contested, particularly because responses can vary with input conditions. The paper tests the Big Five Inventory across varied settings and finds stable responses for several models, while also examining prompt-based personality variation.

  • Problem

    The reliability of human-designed psychological scales on LLMs is contested because LLM responses can vary with input conditions.

    Source: §1 · p. 2

  • Method

    The study evaluates BFI responses across 2,500 settings varying instruction templates, item wording, language, choice labels, and choice order.

    Source: §1 · p. 2, §3.2 · p. 6

  • Results

    Across 2,500 experimental settings, GPT-3.5-Turbo, GPT-4-Turbo, and Gemini-Pro generated stable BFI responses.

    Source: §6 · p. 10

  • Takeaways & Limitations

    GPT-3.5-Turbo’s prompt-conditioned personality variation indicates potential for simulating diverse human populations.

    Source: §1 · p. 2, §4.2 · p. 9

    Reliability alone does not establish validity: models may respond consistently to scales yet behave inconsistently.

    Source: §5.1 · p. 9

Abstract

from arXiv · show

Recent research has focused on examining Large Language Models' (LLMs) characteristics from a psychological standpoint, acknowledging the necessity of understanding their behavioral characteristics. The administration of personality tests to LLMs has emerged as a noteworthy area in this context. However, the suitability of employing psychological scales, initially devised for humans, on LLMs is a matter of ongoing debate. Our study aims to determine the reliability of applying personality assessments to LLMs, explicitly investigating whether LLMs demonstrate consistent personality traits. Analysis of 2,500 settings per model, including GPT-3.5, GPT-4, Gemini-Pro, and LLaMA-3.1, reveals that various LLMs show consistency in responses to the Big Five Inventory, indicating a satisfactory level of reliability. Furthermore, our research explores the potential of GPT-3.5 to emulate diverse personalities and represent various groups-a capability increasingly sought after in social sciences for substituting human participants with LLMs to reduce costs. Our findings reveal that LLMs have the potential to represent different personalities with specific prompt instructions.

1 INTRODUCTION

The paper examines whether human-designed personality scales produce reliable LLM responses under varied conditions and whether prompting can elicit different personalities. It presents reliability analysis and personality simulation as contributions relevant to psychological assessment and social science.

1 INTRODUCTION

  • The authors frame psychological analysis as relevant to understanding LLM behavior as assistants and identifying potential biases or harmful behaviors.Source: §1 · p. 2
  • Across 2,500 settings that varied instructions, item wording, language, choice labels, and choice order, the study reports sufficient BFI reliability across various LLMs.The authors describe this as a comprehensive analysis of five distinct factors.Source: §1 · p. 2
  • GPT-3.5-Turbo can represent varied personalities in response to specific prompt adjustments.The study explores emotional contexts, assigned personalities, and embodying a character as three approaches.Source: §1 · p. 2
  • The study develops a framework for assessing scale reliability on LLMs, intended to support future validation of more scales across more models.Source: §1 · p. 2
  • The authors made experimental results and code publicly available to promote transparency and facilitate further research.Source: §1 · p. 3

2 PRELIMINARIES

Personality tests quantify aspects of individual character and behavior, with the Big Five among several established models. The section distinguishes reliability from validity and identifies reliability assessment as this study’s focus.

2.1 PERSONALITY TESTS

  • Personality tests quantify character, behavior, thoughts, and emotions; the Big Five OCEAN model is one prominent framework alongside MBTI and EPQ.Source: §2.1 · p. 3

2.2 RELIABILITY AND VALIDITY OF SCALES

  • Reliability concerns consistency and stability, while validity concerns whether a scale measures what it should; reliability is necessary but insufficient for validity.The study focuses on reliability as a foundational step in evaluating LLM personality scales.Source: §2.2 · p. 3

3 THE RELIABILITY OF SCALES ON LLMS

The study evaluates whether psychological scales yield reliable personality responses from LLMs by varying prompt and scale conditions. GPT-3.5-Turbo showed satisfactory reliability on the BFI, with consistent responses across perturbations and over time.

3.1 FRAMEWORK DESIGN

  • Crossing instruction, item, language, choice-label, and choice-order factors yielded 2,500 configurations rather than varying one factor at a time.The authors argue that one-factor-at-a-time designs may restrict observation and generalizability.Source: §3.1 · pp. 4–5
  • The framework varied five prompt templates, five item versions, ten languages, five choice-label formats, and ascending or descending choice order.Items were rephrased to assess consistency across paraphrases, and translated versions were manually reviewed for correctness.Source: §3.1 · pp. 4–5

3.2 EXPERIMENTAL RESULTS

  • The experiments used the 44-item, five-point Big Five Inventory and evaluated four LLMs, reporting GPT-3.5-Turbo results in the main section.Items were randomized, with 17 to 27 presented simultaneously; each of the 2,500 settings yielded a five-dimensional OCEAN vector.Source: §3.2 · pp. 5–6
  • 77 outliers (3.08%) appeared in the upper-right BFI region; nearly all involved Arabic-numeral labels, descending choice order, and Arabic or Chinese.The authors hypothesize these anomalies reflect diminished ability to associate numerical labels with natural-language descriptions in those contexts.Source: Figure 2 · p. 5, §3.2 · p. 6
  • Only 7 of 135 factor comparisons exceeded 0.15; GPT-3.5-Turbo’s OCEAN standard deviations were 0.3, 0.3, 0.4, 0.3, and 0.4, versus crowd data’s 0.7, 0.7, 0.9, 0.7, and 0.8.All reported ANOVA p-values were < 0.0001.Source: §3.2 · p. 6

3.3 TEST-RETEST RELIABILITY

  • Across GPT-3.5-Turbo versions 0613 and 1106, analysis found no variation attributable to model updates during the study period.The authors interpret this as high test-retest reliability.Source: §3.3 · p. 6
  • GPT-3.5-Turbo showed satisfactory internal-consistency and test-retest reliability on the BFI, with stable responses across perturbations and over time.The study also reports lower response variance than in crowd data.Source: §3.3 · p. 6, §3.2 · p. 6

4 REPRESENTING DIVERSE GROUPS

The study tests whether GPT-3.5-Turbo’s personality distribution can be steered through environmental contexts, assigned traits, or character prompts. Environmental changes leave the distribution near default, while personality and character prompts can produce broader variation.

4.1 APPROACHES

  • The study compares low-directive environments, moderate-directive personality assignment, and high-directive character embodiment as routes to changing personality distributions.It tests positive and negative environments, three personality-assignment methods, and prompts for fictional and real-world characters.Source: §4 · p. 7, §4.1 · pp. 7–8

4.2 RESULTS

  • Environmental contexts leave GPT-3.5-Turbo’s personality distribution close to its default, while assigned personalities diverge and distinguish high from low trait values.The study notes that heroic characters remain similar to default and hypothesizes that this reflects the model’s positive bias.Source: §4.2 · p. 8, Figure 3 · p. 8
  • Only POR among QA, BIO, and POR effectively alters GPT-3.5-Turbo’s personality distribution; CoT does not significantly influence the results.Figure 4 compares personality-assignment distributions with and without CoT.Source: §4.2 · p. 8, Figure 4 · p. 9
  • For more accurate character simulation, the authors recommend detailed descriptions of the character’s traits, habits, temperaments, and experiences.The character prompts use a name alone or add detailed experiences through CoT.Source: §4.1 · p. 8, §4.2 · p. 9
  • GPT-3.5-Turbo can adopt varied personalities after specific prompt adjustments, with distinct clusters at opposite ends of the same dimension.The study interprets these clusters as evidence that the model distinguishes high and low values within BFI dimensions.Source: §4.2 · p. 9

5 DISCUSSIONS

The discussion identifies limits in interpreting scale reliability and in the methods tested. It also situates the findings against prior work, noting that differences in response aggregation may help explain conflicting results.

5.1 LIMITATIONS

  • Scale modifications and translations may affect reliability and validity, and the changes prevent using Cronbach’s alpha to assess internal consistency.The authors argue that prompt variations still need study when evaluating scales on LLMs.Source: §5.1 · p. 9
  • The study tests three representative methods for influencing personality results, leaving broader approaches for future research.The authors describe their selection as a way to test whether LLMs can mirror diverse human populations.Source: §5.1 · p. 9
  • The authors distinguish reliability from validity: consistent scale responses do not establish that a model behaves consistently.They identify scale validity as a future research direction.Source: §5.1 · p. 9

5.2 RELATED WORK

  • The study varies instructions, items, languages, choice labels, and order across 2,500 data points, reporting satisfactory BFI reliability for GPT-3.5-Turbo.The authors present this as a detailed reliability analysis distinct from earlier personality assessments.Source: §5.2 · p. 10
  • The authors suggest differing model sizes and response aggregation methods may explain conflicting findings about personality stability.They contrast selecting the most likely answer with using the average response, which they describe as more standard.Source: §5.2 · p. 10

6 CONCLUSION

The study evaluates human-designed psychological scales on LLMs across 2,500 settings. It reports stable BFI responses for three models and finds that prompts can steer GPT-3.5-Turbo toward diverse personalities.

6 CONCLUSION

  • Across 2,500 settings, GPT-3.5-Turbo, GPT-4-Turbo, and Gemini-Pro generate stable BFI responses, while prompt adjustments enable GPT-3.5-Turbo to represent diverse personalities.The authors report that comparison with human standard deviations suggests the model exhibits tendencies toward specific personality traits rather than random responses.Source: §6 · p. 10

ETHICS STATEMENTS

The paper frames its contribution as supporting scientific inquiry into LLMs from a psychological standpoint, not establishing suitability for counseling or companionship.

ETHICS STATEMENTS

  • Benchmark performance does not establish applicability or certify automated counseling or companionship use.The passage also notes that negative personas can produce more toxic, unsafe, and misleading outputs on TruthfulQA and SafetyQA.Source: ETHICS STATEMENTS · p. 10

A RELIABILITY TESTS ON OTHER LLMS

GPT-4-Turbo and Gemini-1.0-Pro show response consistency on the BFI, while LLaMA-3.1-8B is more decentralized. The models have distinct profiles, with their distributions centered in a similar region.

A RELIABILITY TESTS ON OTHER LLMS

  • GPT-4-Turbo and Gemini-1.0-Pro show satisfactory BFI response consistency, unlike the more decentralized LLaMA-3.1-8B distribution.Their outlier rates were 5.6%, 4.2%, and 4.4%, respectively, for GPT-4-Turbo, Gemini-1.0-Pro, and LLaMA-3.1-8B.Source: A RELIABILITY TESTS ON OTHER LLMS · p. 15
  • Each model has a distinct BFI personality profile, though distributions center in a similar region; GPT-4-Turbo and Gemini-1.0-Pro are each described as closer to GPT-3.5-Turbo.The figures visualize the 2-D projections for GPT-4-Turbo, Gemini-1.0-Pro, and LLaMA-3.1-8B.Source: A RELIABILITY TESTS ON OTHER LLMS · p. 15, Figure 5 · p. 15, Figure 6 · p. 16

B COMPARISON OF THE TWO EXTREMUMS ON EACH DIMENSION

The assigned extreme BFI personalities form separate clusters, and t-tests report statistically significant differences across all five dimensions. The accompanying tables describe personality-modification prompts and translated test instructions.

B COMPARISON OF THE TWO EXTREMUMS ON EACH DIMENSION

  • The two assigned extreme personalities form separate clusters, with statistically significant differences (p < 0.001) across all five BFI dimensions.The visualization notes that the separation is less obvious for one dimension because it is compressed.Source: Table 4 · p. 17, Figure 8 · p. 17

C.1 MODIFYING PERSONALITIES

  • Table 5 lists prompts for positive and negative environments, assigned personalities, and embodied characters, with optional chain-of-thought scenarios.The table marks LLM responses in Italian.Source: Table 5 · p. 18

C.2 TRANSLATED MULTILINGUAL INSTRUCTIONS

  • Table 6 contains test instructions translated from English into nine other languages.Source: Table 6 · p. 19

D IMPACT OF ITEM ORDER

Across three BFI item-order scenarios, means and standard deviations differ negligibly, and pairwise tests do not reject equal means. The experiment used a subset of 100 configurations.

D IMPACT OF ITEM ORDER

  • The test compared original order, one fixed shuffle, and 100 random shuffles across 100 configurations.All possible item orders were impractical to evaluate; the paper gives their number as 44! ≈ 2.65 × 10^54.Source: D IMPACT OF ITEM ORDER · p. 20
  • BFI item-order variations produce negligible mean and standard-deviation differences, and pairwise tests fail to reject equal means.Table 8 reports that no null hypothesis is rejected at the 0.05 significance level.Source: D IMPACT OF ITEM ORDER · p. 20, Table 8 · p. 20

E QUANTITATIVE RESULTS OF FACTOR COMPARISON

The quantitative comparison table reports differences in means and t-test p-values for a selected factor against the remaining factors. Additional tables document personality-distribution interventions and their BFI results.

E QUANTITATIVE RESULTS OF FACTOR COMPARISON

  • Table 9 compares a selected factor with the remaining factors, reporting mean differences and t-test p-values.Its example compares T1 (500 data points) with T2 to T5 (2,000 data points).Source: Table 9 · p. 21

F DETAILS FOR CHANGING THE PERSONALITIES DISTRIBUTION

  • Tables 10–11 list positive and negative environments and report the resulting GPT-3.5-Turbo BFI means ± standard deviations.Source: Table 10 · p. 22, Table 11 · p. 22
  • Tables 12–13 list assigned minimum and maximum BFI personalities and report their GPT-3.5-Turbo dimension means ± standard deviations.Source: Table 12 · p. 23, Table 13 · p. 23
  • Tables 14–15 list positive and negative fictional or historical characters and report their GPT-3.5-Turbo BFI means ± standard deviations.Source: Table 14 · p. 24, Table 15 · p. 24
Loading 2305.19926v5…