PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Self-knowledge and Action Consistent or Not: Investigating Large Language Model's Personality

Yiming Ai, Zhiwei He, Ziyin Zhang, Wenhong Zhu, Hongkun Hao, Kai Yu, Lingjun Chen, Rui Wang

arXiv:2402.14679v2cs.CLcs.CY

TL;DR

The paper asks whether LLMs’ personality questionnaire responses reflect their behavioral tendencies, a question the authors say remains unresolved. It tests alignment using personality statements and practical scenarios, and finds substantial gaps between claimed personality and exhibited behavior. For most LLMs, the proportion of consistent pairs falls below 55%, compared with more than 84% for human respondents.

  • Problem

    Whether LLMs’ questionnaire responses reflect genuine personality inclinations and align with behavior in real-world scenarios remains unknown.

    Source: §1 · p. 1

  • Method

    The authors compare LLMs’ responses to bilingual personality questionnaires with their choices in practical scenarios, using human-generated matched statement–scenario pairs.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    For most LLMs, the proportion of consistent pairs falls below 55%, while human respondents’ proportion exceeds 84%.

    Source: §3.2 · p. 4

  • Takeaways & Limitations

    LLMs show some capacity to mimic human-like tendencies, but significant gaps remain between stated personality and exhibited behavior.

    Source: §4 · p. 4

    The initial experiments covered only several LLMs, the reasons for the inconsistency remain under investigation, and the study has not identified an effective way to improve congruence.

    Source: Limitations · p. 5

Abstract

from arXiv · show

In this study, we delve into the validity of conventional personality questionnaires in capturing the human-like personality traits of Large Language Models (LLMs). Our objective is to assess the congruence between the personality traits LLMs claim to possess and their demonstrated tendencies in real-world scenarios. By conducting an extensive examination of LLM outputs against observed human response patterns, we aim to understand the disjunction between self-knowledge and action in LLMs.

1. Introduction

The paper asks whether LLMs’ questionnaire-based personality claims reflect their behavioral tendencies, and introduces an evaluation of the gap between self-knowledge and action. It tests LLMs against human response patterns and reports substantial disparities between claimed personality knowledge and behavioral inclinations.

1. Introduction

  • LLMs’ responses to personality questionnaires may not reflect genuine inclinations or align with their behavior in real-world scenarios.The paper notes that the reliability of these responses and their behavioral alignment remain unknown.Source: §1 · p. 1
  • Self-knowledge means awareness of internal states, while action refers to behavior in actual situations.The paper treats them as interacting aspects; for humans, action is how self-knowledge is expressed externally.Source: §1 · p. 1
  • The study addresses the stated gap in research on the validity of psychological scales for LLMs by examining alignment between claimed traits and behavior.Prior work had investigated questionnaire reliability, while the validity of psychological scales on LLMs had not yet been explored, according to the authors.Source: §1 · p. 1
  • The authors design a behavior-tendency questionnaire based on real-world situations and evaluate LLMs’ self-knowledge–action congruence against observed human response patterns.They report substantial disparities between LLMs’ personality knowledge and behavioral inclinations.Source: §1 · pp. 1–2

2. Corpus Design

The corpus pairs bilingual personality statements with practical scenarios to compare professed traits with behavioral choices. The authors draw on established questionnaires, gather and refine human-generated scenarios, and review the resulting pairs for consistency.

2. Corpus Design

  • The authors construct 180 personality statements from TDA-100, BFI-44, and 16 Personalities questionnaires grounded in the Big Five and MBTI models.They selected the instruments for reported reliability and validity in English and Chinese.Source: §2 · p. 2
  • Sixteen individuals, each representing a distinct MBTI type, supplied personal scenarios with contrasting actions that aligned with or contradicted each statement.The authors describe the approach as sampling test behaviors assumed to be a subset of actual behaviors of interest; participants were native Chinese speakers with CEFR C1 English proficiency.Source: §2 · p. 2
  • For 19 statements with substantially varying scenarios, the authors combined cases and selected the highest-rated case as the final scenario.The 16 contributors rated whether each case reflected its corresponding personality statement.Source: §2 · p. 2
  • Ten reviewers assessed the 180 personality–scenario pairs, with each pair receiving a consistency approval rate above 90%.Source: §2 · p. 2
  • The corpus contains 180 matched pairs in a bilingual English–Chinese test set linking personality knowledge statements with action scenarios.The authors describe it as a tool for evaluating LLMs’ understanding of and action on personality traits.Source: §2 · p. 2

3. Experiment on LLMs’ Self-knowledge-Action Congruence

The experiment compares LLMs’ questionnaire-based personality knowledge with their scenario responses using repeated prompts and congruence metrics. Human responses align more closely across the two questionnaires than responses from the five LLMs analyzed.

3.1. Experiment

  • Of 12 queried models, seven produced valid responses and five passed reliability assessment for further personality analysis.Some models failed to grasp the prompts’ intended meaning and produced unreasonable responses.Source: §3.1 · p. 3
  • Five questionnaire prompts were used to reduce prompt-specific influence; each LLM answered each prompt ten times, and valid Likert responses were averaged and rounded.Personality-knowledge responses used a 7-point Likert scale.Source: §3.1 · p. 3
  • The behavior questionnaire used a 7-point graded forced-choice format to rate scenarios with contrasting actions.The prompt retained the meanings of 1, 4, and 7: action A, equal applicability, and action B, respectively.Source: §3.1 · p. 3

3.2. Results

  • Four measures assess congruence: cosine similarity, Spearman’s rank correlation, value mean difference, and the proportion of response pairs differing by no more than one.The consistent-pair measure focuses on similarity of tendencies rather than exact matches.Source: §3.2 · pp. 3–4
  • LLMs’ average Cosine Similarity and Spearman’s Rank Correlation Coefficient were substantially below human results, with a difference exceeding 0.42; most LLMs had under 55% consistent pairs.Human averages exceeded 0.75 for both correlation measures, and their Proportion of Consistent Pairs exceeded 84%.Source: Table 1 · p. 4, §3.2 · p. 4

4. Conclusion

LLMs can mimic some human-like tendencies, but their stated personalities and exhibited behaviors remain significantly incoherent. The authors suggest further work on improving psychological realism in LLM interactions.

  • LLMs exhibit some capacity to mimic human-like tendencies.Source: §4 · p. 4
  • Significant gaps remain between LLMs’ stated personalities and exhibited behaviors.Source: §4 · p. 4
  • The authors call for further exploration of ways to improve the psychological realism of LLM interactions.Source: §4 · p. 4

Limitations

The study examines alignment between LLMs’ claims and their behavior, but its initial experiments covered only a selection of models. The authors have not identified an effective strategy for improving that alignment.

Limitations

  • The initial experiments covered only a selection of LLMs, and future work will broaden the model set.The study also says it has not identified an effective strategy for improving self-knowledge–action congruence.Source: Limitations · p. 5
  • The study treats the observed inconsistency as one of several hypotheses about its underlying causes, which remain for further investigation.Source: Limitations · p. 5

Impact Statement

The study uses established personality questionnaires in Chinese and English and preserves their original content. Its survey procedures also address inclusion, anonymity, and informed consent.

Impact Statement

  • The personality survey uses TDA-100, BFI-44, and the 16 Personalities Test, which are available in Chinese and English and have reliability and validity analyses.Source: Impact Statement · p. 5
  • The questionnaires’ original content was retained, and their design avoids gender bias and racial content.Source: Impact Statement · p. 5
  • Participants’ anonymity was preserved, and all were informed of the study’s purpose and consented to scientific use of their responses.Source: Impact Statement · p. 5

A. LLMs’ Unreasonable Responses

The paper classifies unreasonable questionnaire responses into five types, including uniform answers, out-of-range values, and responses that cannot be scored.

A. LLMs’ Unreasonable Responses

B. Reliability of LLMs’ Responses

The authors test questionnaire-response reliability using logical consistency and split-half reliability, then identify five LLMs whose responses meet their stated reliability criteria. They also describe the corpus and the tables used to report verification and experimental resources.

B. Reliability of LLMs’ Responses

  • Logical Consistency checks whether positive and reverse-scored items show consistent response directions after negative items are adjusted; Table 2 presents the distribution of these item types.Source: B. Reliability of LLMs’ Responses · pp. 7–8, Table 2 · p. 8
  • Split-half reliability compares equal-length questionnaire halves balanced by assessment dimension, correlates their scores, and applies the Spearman-Brown formula; values around the human level qualify for further investigation.Source: B. Reliability of LLMs’ Responses · p. 8
  • 0.73 ± 0.13 for Consistency and 0.69 ± 0.09 for Reliability were the human averages; ChatGLM3, GPT-3.5-turbo, GPT4, Vicuna13b, and Vicuna33b were deemed sufficiently reliable for further analysis.The 16 human participants had minimum values of 0.49 for Consistency and 0.57 for Reliability.Source: B. Reliability of LLMs’ Responses · p. 8

C. Several Examples of Our Corpus

  • The corpus pairs 180 personality-knowledge statements with 180 corresponding practical-scenario cases.Source: C. Several Examples of Our Corpus · p. 8
  • Table 3 is titled as verification results for LLM and human personality-cognition responses, while Table 4 lists LLM resources and corresponding cognition-action congruence experiments.Source: Table 3 · p. 9, Table 4 · p. 9

D. Experiment Setup

The experiment setup details are presented in Tables 4 and 6.

E. Meaning of numbers in 7-point graded forced-choice

The seven-point scale runs from complete applicability of Action A through an equal midpoint to complete applicability of Action B.

F. Significance Tests

Permutation tests found statistically significant LLM–human differences across metrics in both Section 3.2 experiments; the merged material also describes participant recruitment and study procedures.

F. Significance Tests

G.1. Recruitment of Scenario Providers, Reviewers and Human Respondents

  • The study recruited 16 reviewers and 45 human respondents, and recruited 16 people with distinct MBTI types for the review tasks.Respondents were undergraduate, postgraduate, and PhD students; participation and anonymized use of reviews required consent.Source: §G.1 · p. 11, §G.2 · p. 11

G.2. Instructions Given to Scenario Providers

  • Scenario providers supplied a real-life case for each personality statement, pairing similarly sized actions that aligned with or contradicted it.Source: §G.2 · p. 11

G.3. Instructions Given to Reviewers

  • Reviewers judged whether each practical scenario matched its personality statement and rated it 1 or 0.They were asked to suggest improvements when all scenarios for a statement received a 0.Source: §G.3 · p. 12

G.4. Instructions Given to Respondents

  • Respondents were not told the questionnaire types or question relationships in advance, and their answers were anonymized with consent.Humans received the same prompts as LLMs, but answered once after reading semantically similar prompts together; LLMs answered separately by prompt type.Source: §G.4 · p. 12
Loading 2402.14679v2…