PaperlayerView exact revision on arXiv

Source-linked AI summary

Open Models, Closed Minds? On Agents Capabilities in Mimicking Human Personalities through Open Large Language Models

Lucio La Cava, Andrea Tagarelli

arXiv:2401.07115v3cs.AIcs.CLcs.CYcs.HCphysics.soc-ph

TL;DR

Prior personality research had not specifically examined Open LLMs, despite their widespread adoption. This study assesses 12 Open LLM agents with MBTI and BFI tests and personality and role conditioning, finding distinct intrinsic profiles but uneven mimicry: a few models mirror prompts while most retain their intrinsic traits.

  • Problem

    Personality detection and conditioning had not been specifically studied in Open LLMs, despite their widespread adoption.

    Source: §1 · pp. 1–2

  • Method

    The study assesses 12 Open LLM agents using MBTI and BFI tests, examining intrinsic traits and conditioning with specified personalities and roles.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    Mimicry varies across models: SOLAR, Dolphin, NeuralChat, and Llama3-8 show notable capabilities, while most models retain their intrinsic traits.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The findings may advance understanding of the interplay between NLP and human psychology and potentially inform responsible development of human-like computational agents.

    Source: §5 · p. 8

    The study excludes some larger model variants because their deployment would require excessive quantization on the available hardware.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The emergence of unveiling human-like behaviors in Large Language Models (LLMs) has led to a closer connection between NLP and human psychology. Scholars have been studying the inherent personalities exhibited by LLMs and attempting to incorporate human traits and behaviors into them. However, these efforts have primarily focused on commercially-licensed LLMs, neglecting the widespread use and notable advancements seen in Open LLMs. This work aims to address this gap by employing a set of 12 LLM Agents based on the most representative Open models and subject them to a series of assessments concerning the Myers-Briggs Type Indicator (MBTI) test and the Big Five Inventory (BFI) test. Our approach involves evaluating the intrinsic personality traits of Open LLM agents and determining the extent to which these agents can mimic human personalities when conditioned by specific personalities and roles. Our findings unveil that $(i)$ each Open LLM agent showcases distinct human personalities; $(ii)$ personality-conditioned prompting produces varying effects on the agents, with only few successfully mirroring the imposed personality, while most of them being ``closed-minded'' (i.e., they retain their intrinsic traits); and $(iii)$ combining role and personality conditioning can enhance the agents' ability to mimic human personalities. Our work represents a step up in understanding the dense relationship between NLP and human psychology through the lens of Open LLMs.

1 Introduction

Despite growing interest in LLM personalities, studies had not specifically examined personality detection and conditioning in Open LLMs, despite their widespread adoption. This study assesses 12 Open LLM agents with MBTI and BFI tests and examines personality and role conditioning.

1 Introduction

  • Open LLMs had not been specifically studied for personality detection and conditioning, despite their widespread adoption and practical advantages over closed models.The paper identifies cost-effectiveness, data control, transparency, and customization as advantages of Open LLMs.Source: §1 · pp. 1–2
  • The study explores Open LLMs’ intrinsic personalities and whether prompts and roles can shape them around specified personalities.It frames this exploration through MBTI and BFI and asks whether agents know the tests, exhibit consistent traits, mimic specified traits through system prompting, and improve with role instructions.Source: §1 · p. 2
  • The authors create 12 agents based on representative Open LLMs and assess their personalities through human-like interviews under different experimental scenarios.Source: §1 · p. 2

2 Background

The background introduces MBTI’s categorical personality types and BFI’s graded traits, then reviews research on LLM agents and personality. It situates the paper among work on personality extraction, conditioning, tailored agents, and assessment.

2.1 The MBTI and BFI Personality Tests

  • MBTI groups respondents into 16 types across four dichotomies, whereas BFI measures five personality factors on a scale.Source: §2.1 · p. 2, Table 1 · p. 3

2.2 Related Work

  • Prior work spans prompted LLM agents for varied tasks and research on personality extraction, intrinsic traits, conditioning, tailored agents, and assessment.Source: §2.2 · pp. 2–3

3 Methodology

The methodology checks models’ familiarity with MBTI and BFI, administers both tests under unconditioned and conditioned prompts, and varies temperature and repetitions. It uses 12 Open models deployed as interview agents.

3.1 Awareness Check

  • The awareness check compares each model’s descriptions of MBTI types and BFI factors with domain-expert reference descriptions using lexical and semantic similarity.Source: §3.1 · p. 3

3.2 Administering the tests

  • MBTI answers map to one of 16 types, while BFI answers on a five-point Likert scale yield scores for five factors; questions are presented individually in random order.The question sets contain 60 MBTI questions and 44 BFI questions.Source: §3.2 · p. 3

3.3 Personalities of Open LLMs

  • The study tests unconditioned prompts, personality-conditioned prompts using type traits or factor details, and prompts pairing each target personality with selected human roles.Psychologists selected three pertinent roles per personality or factor from a catalog of 120 professions.Source: §3.3 · pp. 3–4, Figure 1 · p. 4, Figure 2 · p. 4

3.4 Temperature and Repetitions

  • For RQ1, models undergo repeated MBTI and BFI tests at temperatures τ = 0.01 and 0.7; RQ2 and RQ3 use 30 repetitions per model, temperature, and target personality.The conditioned tests total 11,520 MBTI and 3,600 BFI assessments.Source: §3.4 · pp. 4–5

3.5 Models

  • The study selects 12 representative Open LLMs spanning model sizes and architectures, including Dolphin, Gemma, Falcon, Llama variants, Mistral, Mixtral, and others.Source: §3.5 · p. 5, Table 2 · p. 5

3.6 Agent Creation and Models Deployment

  • AutoGen configures each model as interviewer or interviewee, with system messages assigning personality or role; default top_p and top_k settings are retained for reproducibility.Experiments ran locally using text-generation-webui, with temperature varied while top_p and top_k remained unchanged.Source: §3.6 · p. 5

4 Results and Discussion

Open LLM agents recognize MBTI and BFI, exhibit distinct baseline personality patterns, and vary in how well they follow personality prompts. Adding role conditioning helps some models mimic personalities, while many retain intrinsic traits or show limited response.

4.1 RQ0: Models’ awareness of the tests

  • Agents show moderately high semantic similarity but low lexical overlap with MBTI and BFI descriptions, suggesting awareness expressed in different wording.Detailed awareness results are deferred to Appendix A.Source: §4.1 · p. 5

4.2 RQ1: LLM-inherent Personalities

  • At low temperature, no model has a clear single BFI factor; Neuroticism generally does not emerge, while higher temperature chiefly shifts Neuroticism scores with model-specific increases and decreases.At baseline, Conscientiousness emerges for Llama3-8, Agreeableness and Extraversion for Llama2-13, and Openness for several models.Source: §4.2 · p. 6
  • At low temperature, J is constant and ENFJ is typical, but Mixtral, Mistral, and Vicuna prefer INFJ, ESFJ, and ENTJ; higher temperature broadens some models’ types, not Llama, Gemma, or Phi3.No model produced ISTP, INFP, or INTP across temperatures.Source: §4.2 · p. 6

4.3 RQ2: Prompt-conditioned Personalities

  • Most agents disregard personality conditioning or retain intrinsic traits; SOLAR, Dolphin, NeuralChat, and Llama3-8 are exceptions on both tests, with Llama2-13 additionally responding on BFI.Higher temperature especially weakens MBTI conditioning for many models.Source: §4.3 · p. 7
  • MBTI conditioning accuracy is 0.74–0.79 for SOLAR and 0.63–0.65 for Dolphin, while most agents show limited mimicry and often default to a different type.NeuralChat and Llama3-8 also show some success; performance tends to worsen at higher temperature.Source: §4.3 · p. 7, Table 3 · p. 8
  • BFI conditioning raises scores for all models except Mistral, Mixtral, and Falcon; Neuroticism is most responsive, with Llama3-8 exceeding a 230% increase.Several models reach a score of 5.0 on selected factors, but those perfect outcomes occur only at τ = 0.01.Source: §4.3 · p. 7, Table 3 · p. 8
  • Role-plus-personality conditioning improves MBTI mimicry for SOLAR, NeuralChat, Llama3-8, and Dolphin, with ENFJ nearly captured by 9 of 12 models; BFI gains coexist with regressions.Mistral and Mixtral remain insensitive to BFI conditioning, and both Llama2 models lose Agreeableness while Falcon declines in most cases.Source: §4.4 · p. 8

5 Conclusions

The study assesses intrinsic and prompted personalities in Open LLM agents using MBTI and BFI, finding distinct intrinsic traits and varied capacity to mimic imposed personalities. Its findings may inform understanding of human-like computational agents and responsible development.

5 Conclusions

  • The work assesses intrinsic personality types and tests whether prompts specifying personalities and associated human professions can shape 12 Open LLM agents.The study uses the Myers-Briggs and Big Five tests.Source: §5 · p. 8
  • Across 12 Open LLM agents, intrinsic personality types are distinguishable, but mimicry varies: SOLAR, Dolphin, NeuralChat, and Llama3-8 are singled out, while most models show closed-mindedness.The assessment used MBTI and Big Five tests and conditioned agents with personality traits and representative roles.Source: §5 · p. 8
  • The authors propose that the findings may advance understanding of the NLP–human psychology relationship and potentially support responsible development of human-like computational agents.They describe these implications as potential rather than demonstrated outcomes.Source: §5 · p. 8

Limitations

The study’s scope and interpretation are bounded by deployment resources, context lengths, English-only evaluation, and incomplete access to model training details. The authors also describe the MBTI assessment tool choice and ethical concerns around personalized agents.

Limitations

  • Larger models, including Falcon 180B and Llama3-70B, were excluded because deployment required excessive quantization on available hardware, risking performance degradation.The authors therefore focused on models with more manageable deployment requirements.Source: Limitations · p. 9
  • Limited model context lengths led the authors to present questions individually, repeating the conditioning for each prompt rather than submitting a whole test at once.They state this also mitigates potential ordering bias.Source: Limitations · p. 9
  • The evaluation was conducted exclusively in English, and the authors caution that results may differ in other languages.They suggest multilingual testing could reveal outcome variations based on linguistic differences.Source: Limitations · p. 9
  • The MBTI evaluation used 16personalities.com instead of the Myers & Briggs Foundation test because about 20,000 assessments at roughly $60 each would be impractical.The authors say this cost constraint does not apply to BFI, which can be scored offline using predefined rules.Source: Limitations · p. 9
  • The precise effects of model training methods and data on responses cannot be quantified because model owners do not provide full underlying-model details.The authors identify this as a constraint on explaining their findings.Source: Limitations · p. 9
  • The authors warn that personalized LLMs may be misused and that stronger emulation of human behaviors could make agents harder to distinguish and more effective at their goals.They urge caution and responsibility in deployment and use.Source: Ethics Statement · p. 9

R Bryan Kennedy and D Ashley Kennedy. 2004. Using

The supplied material is a bibliography excerpt listing work on LLM personality assessment, personality editing, and role-playing, alongside broader agent research. Its table caption identifies a comparison of MBTI- and BFI-awareness scores.

R Bryan Kennedy and D Ashley Kennedy. 2004. Using

A Models’ Awareness of the MBTI and BFI Tests

The authors first check whether selected Open LLMs can describe MBTI and BFI personalities, comparing generated descriptions with reference descriptions. Their prompts specify interview roles and personality conditions, while the supplied excerpt does not report the detailed prompt content beyond selected templates.

A Models’ Awareness of the MBTI and BFI Tests

  • For awareness assessment, models generated descriptions of all 16 MBTI types and five BFI factors, which were compared with references using lexical and semantic similarity.The descriptions were generated at temperature 0.01; lexical overlap used preprocessed text and semantic similarity used Sentence Transformer embeddings.Source: A Models’ Awareness of the MBTI and… · p. 14
  • Average cosine similarity was 0.67 for MBTI and 0.66 for BFI, despite lower word overlap averages of 0.21 and 0.26, respectively.The authors interpret this as awareness expressed with a different lexicon; awareness did not appear significantly influenced by architecture or parameter size.Source: A Models’ Awareness of the MBTI and… · p. 14

B MBTI Detailed Prompts and System Messages

  • The interview prompt templates include personality-only conditioning and role-plus-personality conditioning, with placeholders for the personality, role, or traits.The supplied templates also instruct the agent to answer the assessment questions using the best-fitting option.Source: B MBTI Detailed Prompts and System… · p. 15

C MBTI Test Questions

The MBTI test questions are listed in Table 5, while the accompanying BFI prompts describe assessment, personality conditioning, and role-and-personality conditioning procedures.

C MBTI Test Questions

  • Table 5 lists the 60 questions for the MBTI personality test.The table is identified as the full question set available on 16personalities.com.Source: Table 5 · p. 16

D BFI Detailed Prompts and System Messages

  • The BFI assessment prompt asks agents to select how much they agree or disagree with each statement and answer without overthinking.The response options range from “Disagree strongly” to “Agree Strongly.”Source: D BFI Detailed Prompts and System… · p. 17
  • The BFI prompts also ask agents to identify three representative professions for each of the five factors from a supplied list.The prompt supplies a catalog of professions from which to choose.Source: D BFI Detailed Prompts and System… · p. 17
  • Personality conditioning presents a personality factor and its details as the agent’s stated human personality.The role-and-personality variant adds a specified role to that conditioning prompt.Source: D BFI Detailed Prompts and System… · p. 17

E BFI Test Questions

Table 6 presents the full set of 44 questions for the Big Five Inventory.

E BFI Test Questions

  • Table 6 lists all 44 questions in the BFI test.The table identifies John and Srivastava (1999) as the source of the test questions.Source: Table 6 · p. 18

F Scoring the BFI Test

The BFI scoring procedure averages factor-specific question values after reverse-scoring negatively related items. The accompanying figures display model personality scores across temperature settings.

F Scoring the BFI Test

  • The final score for each BFI factor is the average of its associated question values, with negatively related items reverse-scored.For example, an answer of 4 to a reverse-scored question becomes 2 using 6 − 4.Source: F Scoring the BFI Test · p. 18

G MBTI LLM-inherent Personality Dichotomies

  • Figure 6 illustrates N-averaged scores for each MBTI dichotomy across models and temperature settings.Source: Figure 6 · p. 19

H BFI LLM-inherent Personality Traits

  • For RQ1, Figures 7–11 show distributions of BFI scores for the extraversion, agreeableness, conscientiousness, neuroticism, and openness factors, covering all models and temperature settings.Source: Figure 7 · p. 20, Figure 8 · p. 20, Figure 9 · p. 21

I Further Results from the Personality-Conditioned MBTI tests

The further results compare personality-conditioned MBTI behavior across models and relate differences to architectural commonalities. The role-conditioned results are tabulated for both MBTI accuracy and BFI score increases.

I Further Results from the Personality-Conditioned MBTI tests

  • SOLAR outperforms Llama2-7 despite being built on it, a difference the authors associate with its pretraining, depth up-scaling, and dataset optimizations.The proposed explanations are presented as possible contributors.Source: I Further Results from the… · p. 22
  • Dolphin’s improvement over Mistral is attributed by the authors to absent alignment and training on Airoboros and Samantha DNA; NeuralChat’s improvement is linked to SlimOrca fine-tuning and DPO.The authors suggest these differences may support greater flexibility or human-preference alignment, respectively.Source: I Further Results from the… · p. 22
  • The authors report no strong correlation between model size and performance: Llama2-13B is slightly worse than Llama2-7B, while Llama3-8 outperforms both.They also note Mixtral’s inferior performance relative to SOLAR, which has a quarter of its parameters.Source: I Further Results from the… · p. 22

J Role&Prompt-conditioned Personality Traits

  • Table 7 reports role-and-personality-conditioned MBTI average accuracy, while Table 8 reports BFI percentage increases relative to unconditioned BFI results.Source: Table 7 · p. 29, Table 8 · p. 30

K MBTI Personality Descriptions and Traits (from www.myersbriggs.org)

These parts compile contrasting MBTI personality profiles, describing varied personal styles and their characteristic strengths, development risks, preferred work settings, stress responses, and relationship styles.

  • Duty-focused profiles emphasize thoroughness, dependability, order, and loyalty, while some may become rigid or overly cautious and face stress from disrupted routines or feeling unappreciated.Source: K MBTI Personality Descriptions and… · p. 31
  • Visionary profiles seek meaning and a common good, while strategic profiles pursue original, long-range ideas; they may be private, difficult to understand, or seem cold and distant.Source: K MBTI Personality Descriptions and… · p. 32
  • Practical problem-solvers favor adaptable, logical action and immediate results, but may lose sight of the big picture, struggle with collaboration, or defer decisions.Source: K MBTI Personality Descriptions and… · p. 33
  • Values-centered profiles are idealistic and compassionate, aiming to help others grow, whereas theoretical profiles focus on abstract analysis and strategic models and may find teamwork difficult.Source: K MBTI Personality Descriptions and… · p. 34
  • Action-oriented and outgoing profiles prefer practical activity, spontaneity, and working with others, but may lose interest in long projects, miss deadlines, or become distracted.Source: K MBTI Personality Descriptions and… · p. 35
  • Another outgoing profile is described as imaginative and enthusiastic, quickly connecting ideas and relying on affirmation, appreciation, flexibility, and verbal improvisation.Source: K MBTI Personality Descriptions and… · p. 36

Imaginative, energetic profile

  • One profile is imaginative, energetic, and flexible, thriving on new people and possibilities but risking overcommitment and intense stress.It is described as creative and supportive, with roles involving teamwork and helping others; in relationships, it is perceptive and emotionally expressive.Source: K MBTI Personality Descriptions and… · p. 36

Resourceful, analytical profile

  • Another profile is quick, resourceful, and analytical, generating strategic ideas but sometimes avoiding decisions or pursuing impractical possibilities.It favors autonomous, high-energy work and stimulating debate, while routine, excessive detail, and inefficiency can cause stress.Source: K MBTI Personality Descriptions and… · p. 36

Practical, decisive profile

  • A practical, decisive profile organizes people and projects efficiently, but may overlook others’ ideas or act before gathering enough information.It prefers stable settings with clear roles and is described as dependable in relationships; uncertainty and lack of control can provoke inflexibility.Source: K MBTI Personality Descriptions and… · p. 37

Warm, conscientious profile

  • A warm, conscientious profile seeks harmony and completes tasks reliably, but may defer to others’ preferences and struggle to adapt plans.It prefers caring, close-knit workplaces and is attuned to partners’ emotional needs; lack of appreciation or disrupted procedures can prompt rigidity and self-doubt.Source: K MBTI Personality Descriptions and… · p. 37

Empathetic, collaborative profile

  • An empathetic, responsive profile fosters others’ growth through collaborative leadership and consensus, but may struggle with conflict and overlook factual realities.It favors shared-goal environments and helping professions; disrupted harmony, criticism, or lack of appreciation can lead to pessimism and self-doubt.Source: K MBTI Personality Descriptions and… · p. 38

Strategic, decisive profile

  • A strategic, decisive profile leads organizational problem-solving and long-term planning, but may overlook contributors’ needs or drive others too hard.It favors challenging leadership roles; its relationships are energized by stimulating interaction, while inefficiency and loss of control are stressors.Source: K MBTI Personality Descriptions and… · p. 38

L BFI Personality Descriptions and Traits (from (John et al., 2008))

The section describes five BFI personality factors through conceptual definitions, verbal labels, and behavioral examples.

Agreeableness Personality Factor

  • Agreeableness contrasts a prosocial, communal orientation with antagonism and includes altruism, tender-mindedness, trust, and modesty.Its labels include Altruism and Affection; examples include praising others, lending possessions, and consoling an upset friend.Source: L BFI Personality Descriptions and… · p. 39

Conscientiousness Personality Factor

  • Conscientiousness describes socially prescribed impulse control that supports task- and goal-directed behavior, planning, organization, and prioritization.Examples include arriving on time, studying hard, and proofreading; letting dishes pile up is a reversed item.Source: L BFI Personality Descriptions and… · p. 39

Extraversion Personality Factor

  • Extraversion is an energetic approach to the social and material world, including sociability, activity, assertiveness, and positive emotionality.Labels include Energy and Enthusiasm; examples include approaching strangers, leading a project, and keeping quiet when disagreeing as a reversed item.Source: L BFI Personality Descriptions and… · p. 39

Neuroticism Personality Factor

  • Neuroticism contrasts emotional stability and even-temperedness with negative emotionality, including anxiety, nervousness, sadness, and tension.Examples include becoming upset when someone is angry; accepting life's good and bad and relaxing are reversed items.Source: L BFI Personality Descriptions and… · p. 39

Openness Personality Factor

  • Openness describes the breadth, depth, originality, and complexity of a person's mental and experiential life.Labels include Originality and Open-Mindedness; examples include learning for pleasure, watching educational programs, creating novel living spaces, and seeking stimulating activities.Source: L BFI Personality Descriptions and… · p. 39

Related papers

Loading 2401.07115v3…