Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancement

The evaluation challenge

• LLM evaluation faces challenges in measuring psychological constructs, keeping benchmarks current, ensuring robustness and validity, and extending human-centered evaluation to agentic and multimodal systems.The review identifies personality, values, and cognitive biases among the constructs that traditional task-specific benchmarks may not capture.Source: §1 · p. 4

LLM Psychometrics

• LLM Psychometrics applies psychometric instruments, theories, and principles to evaluate, understand, and enhance LLMs’ measured human-like behaviors.Its scope includes non-cognitive traits such as personality and values, as well as cognitive abilities such as reasoning and learning.Source: §1 · p. 4

Recent research

• The field examines dynamic, construct-oriented evaluation, non-cognitive measurement, adaptive assessment, validation, human-centered evaluation, and evaluation of agentic and multimodal systems.The review describes these as directions in recent research, including efforts to improve protocol reliability and validity.Source: §1 · p. 4

Scope and caveats

• LLM constructs refer to systematic, measurable response patterns, not claims that models possess subjective experience, sentience, or genuine psychological states.The review notes unresolved questions about whether psychometric instruments capture meaningful latent patterns or sophisticated statistical mimicry, and whether prompt sensitivity strains assumptions of trait stability.Source: §1 · p. 4

Motivation for the review

• Research on LLM Psychometrics spans isolated constructs, diverse methods, and distinct validation techniques, motivating a systematic synthesis.The review situates this fragmentation within the field’s broad interdisciplinary contributions.Source: §1 · p. 4

Motivation

• Fragmentation across studies of disparate constructs motivates a systematic review to integrate the field’s insights.The passage identifies a lack of cohesion among contributions from different academic communities.Source: Psychometrics for LLM Enhancement · p. 5

Review framework

• The review organizes LLM Psychometrics across evaluation, validation, and enhancement, with evaluation framed around what to measure, how to measure it, and how well results measure it.It describes psychometric insights as informing the development and refinement of LLMs.Source: Psychometrics for LLM Enhancement · p. 5

Scope

• The review treats LLMs as subjects of psychometric tests, excluding work that uses LLMs to assess or profile human respondents.That adjacent field is referred to Ye et al. [2025a].Source: Psychometrics for LLM Enhancement · p. 5

Inclusion criteria

• Included studies must use psychometric approaches or principles and characterize latent behavioral tendencies, rather than report only scalar performance metrics.The review distinguishes minimal scale administration from more rigorous work that assesses measurement properties such as reliability and construct validity.Source: Psychometrics for LLM Enhancement · p. 6

2 Preliminary and methodological foundations

• The section presents preliminary and methodological foundations intended to make later discussions self-contained and accessible to a cross-disciplinary audience.This introduction frames the material that follows on language models, psychometrics, and earlier psychometric approaches to AI.Source: §2 · p. 6

2.1 Large language models

• LLMs generate text autoregressively by modeling token probabilities conditioned on preceding context, and their outputs may vary with sampling, decoding, prompts, or in-context examples.Training exposure to test items can contaminate evaluation, while prompting can influence reasoning performance and exhibited traits.Source: §2.1 · pp. 6–7

2.2 Psychometrics

• Psychometrics uses items to measure behavior and psychological characteristics, with scores interpreted in context and measurement quality assessed through reliability, validity, and related principles.The section distinguishes personality tests of typical tendencies from cognitive tests of speed or accuracy, and also describes standardization, fairness, and measurement invariance.Source: §2.2 · p. 7

2.3 Psychometric evaluation of AI before the era of LLMs

• Before LLMs, psychometric AI work largely focused on building systems that could solve cognitive tests, while later proposals broadened the target toward intelligence and mental abilities.The passage traces this history from early heuristic programs through calls to include emotional aspects in AI.Source: §2.3 · p. 7

3.1 Fundamental differences between psychometrics and AI benchmarking

• Psychometrics prioritizes defined latent constructs, item quality, and evidence about reliability, validity, prediction, and explanation, unlike benchmarks’ emphasis on task performance and scale.Psychometric development may use theory-driven design and statistical models, which generally require relatively large participant samples for stable estimates.Source: §3.1 · pp. 8–9
• Conventional benchmarks often operationalize ambiguous abilities through ad hoc task selection and can conflate task performance with deeper cognitive capacities.Their scale and rapid model development can leave item quality difficult to maintain, while results may generalize poorly beyond the target task.Source: §3.1 · p. 9

3.2 Benchmarking with psychometrics-inspired principles

• Construct-oriented evaluation uses factor analyses or theory-driven scales to model abilities and explain or predict performance across tasks.Studies differ on whether LLM performance is best represented by one general factor or multiple factors, and the review notes that model and benchmark choices may account for the discrepancy.Source: §3.2 · p. 9
• Psychometrically grounded benchmark-development approaches include ECBD’s five justified modules, PATCH’s eight-step process, and adaptations of Evidence-Centered Design.Reported applications include validity-threat identification, results diverging from traditional benchmarks in an 8th-grade mathematics pilot, and improved interpretability with reduced contamination in a pedagogy application.Source: §3.2 · p. 10
• IRT-based evaluation estimates latent ability and item properties, supports adaptive item selection, and can place estimates on a unified scale for comparisons across systems and human norms.Related methods calibrate item difficulty or generate items at specified difficulty levels.Source: §3.2 · p. 10

Values

• LLMs show systematic value patterns, including Self-Transcendence and Conservation preferences, Self-Expression over Survival, and Humane and Performance Orientation with moderate Assertiveness.The review also reports stronger inclinations toward Universalism, Benevolence, Conformity, and Security, and opposition to Power and Achievement.Source: Table 2 · p. 11, §4 · p. 12
• Value orientations vary across models and model versions, and profiling prompts can elicit different values, challenging assumptions of stability.The review also reports blended cultural values and a general tendency toward Western liberal values.Source: §4 · p. 12

Morality

• LLMs often show rationalist, consequentialist, harm-avoidant moral profiles, yet moral reasoning and preferences diverge, and evaluations do not establish stable beliefs or ethical suitability.Some studies attribute alignment with human moral standards to exposure to conventional ethical values during training; others report that underlying reasoning is mostly imitation rather than conceptual understanding.Source: §4 · p. 12

Political attitudes

• Studies commonly find misalignment with human opinions and left-leaning political bias, alongside Western-centric tendencies and limited understanding of non-English or multi-partisan perspectives.The degree and manifestation of bias vary across contexts and domains.Source: §4 · p. 13
• Prompt design, calibration, and fine-tuning can produce opinion distributions that closely approximate human group distributions, while reports differ on whether larger models mitigate or amplify bias.Some studies report improved reasoning and bias mitigation in newer, larger, chain-of-thought-enabled models; others argue that complexity without deliberate mitigation can amplify existing biases.Source: §4 · p. 13

Emotional intelligence

• Advanced LLMs can match or exceed humans on some emotional-intelligence tasks, but may express empathy mechanically, lack deep reflexive analysis, and misalign with human emotional behavior.The review describes the emotional task findings as limited to some tasks of emotional intelligence.Source: §4 · p. 13

Cognitive capabilities

• Advanced LLMs surpass humans on some psycholinguistic tasks, and show strong verbal comprehension, working memory, and analogical reasoning.Reported examples include pragmatic reasoning and creative writing.Source: §4 · p. 13
• LLMs also show notable cognitive deficits, particularly on benchmarks such as WAIS-IV and ARC.Source: §4 · p. 14

4.1.1 Personality traits

• Researchers use established Big Five, HEXACO, and Dark Triad inventories, adapting tests for conversational settings and developing LLM-specific tools; some question whether human-derived traits apply.Adaptations include behavioral tests, the Machine Personality Inventory, and PersonalityEdit.Source: §4.1.1 · p. 15
• Modern LLMs tend toward high Openness and Agreeableness and low Neuroticism, while prompt steering produces a reliable, valid synthetic personality that can predict downstream task performance.The review contrasts this profile with elevated Dark Triad traits reported in early models and notes that the default compliant-assistant persona may trade personality diversity for safety.Source: §4.1.1 · p. 15

4.1.2 Values

• Across value frameworks, LLMs show a broadly prosocial profile, but orientations vary across model versions and retain a Western-liberal tilt.Studies use Schwartz, WVS, VSM, GLOBE, and SVO frameworks; the review also notes that alignment training compresses distributional pluralism and that questionnaire profiles can be context-dependent and fragile.Source: §4.1.2 · pp. 15–16

4.1.3 Morality

• LLMs generally exhibit rationalist, consequentialist, harm-avoidant moral profiles, yet evidence also indicates prompt-calibrated mimicry, inconsistent judgments, and Western-centric deviations.Assessments use Moral Foundations instruments and other moral theories; stated principles can diverge from applied judgments, and nominal post-conventional performance degrades with language and prompt variation.Source: §4.1.3 · pp. 16–17

4.1.4 Attitudes and opinions

• Political-attitude studies use standardized questionnaires, election surveys, political scales, and LLM-tailored datasets for US and cross-national assessment.Examples include ANES, ATP, GLES, PCT, OpinionQA, IssueBench, and GlobalOpinionQA.Source: §4.1.4 · p. 17
• Studies report misalignment with human opinion distributions, left-libertarian tendencies, and Western-centric limits; survey conclusions vary by response format and can become statistically random after option-position correction.The silicon-sampling hypothesis proposes that calibrated models can approximate human group opinion distributions, but evidence also reports underrepresentation, misportrayal, and flattening of identity groups.Source: §4.1.4 · p. 17

4.2.1 Heuristics and biases

• LLMs show human-like cognitive biases, but newer models may mitigate them while bias can persist or increase without deliberate mitigation.Studies also use cognitive theories and mechanistic interpretability to explain or trace these patterns.Source: §4.2.1 · p. 18

4.2.2 Social interactions

• LLMs show ToM-like reasoning in structured or prompted tasks, but performance can rely on surface cues and remains fragile on complex, open-ended, or modified tasks.Studies report declines on higher-order belief attribution and faux pas detection, and comparisons suggest narrow-context approximation rather than general human-like social cognition.Source: §4.2.2 · pp. 18–19
• Advanced LLMs can match or exceed humans on some emotional-awareness tasks, but show mechanical empathy and limitations in emotional reflection and alignment.Studies assess EI with theory-driven benchmarks and adapted psychometric instruments, including multimodal evaluations.Source: §4.2.2 · p. 19
• LLMs show rule-based social competence, yet struggle with implicit social understanding, high-level needs, and some emotionally reflective or human-aligned responses.They can follow norms, pursue multi-agent goals, and make prosocial decisions, while ToM-like performance remains context-dependent and brittle under task changes or open-ended reasoning.Source: §4.2.2 · pp. 18–19

4.2.3 Psychology of language

• Language abilities vary by task: advanced models can match or exceed humans in some pragmatic, grammatical, and creative tasks, while retaining linguistic and developmental mismatches.Reported limits include prompt sensitivity, weak implicature understanding, non-human-like generated language, and mixed alignment with human language acquisition.Source: §4.2.3 · p. 20

4.2.4 Learning and cognitive capabilities

• Adapted WAIS-IV and Raven’s tests suggested human-like abilities, but later analyses identify shortcut learning, brittle transfer, and uneven performance across specialized and intuitive tasks.The review argues that cognitive evaluation should assess reasoning robustness, process, and validity rather than accuracy alone.Source: §4.2.4 · pp. 20–21

5 Psychometric evaluation methodology

• The methodology framework covers test formats, data and task sources, prompting, output and scoring, and inference parameters.Figure 3 presents the framework’s components.Source: §5 · p. 21, Figure 3 · p. 22

5.1 Test format

• Psychometric evaluations use structured tests, open-ended conversations, and agentic simulations, with representative formats varying by construct.Table 4 collects examples of these formats across constructs.Source: §5.1 · p. 21

5.1.1 Structured tests

• Structured tests retain predefined instructions and response formats, enabling scalable, objective, often automated scoring but risking weak real-world applicability, bias, contamination, and validity problems.Adaptations commonly preserve original items while reformatting them as prompts; examples include Likert scales, scenario choices, and short answers.Source: §5.1.1 · p. 22

5.1.2 Unstructured tests

• Unstructured tests use free-form conversations or contextualized agent simulations to elicit nuanced behavior, but are harder to standardize, score, and reproduce.Examples include narrative generation, advice-seeking, moral dilemmas, and multi-agent social or cognitive scenarios.Source: §5.1.2 · p. 23

5.2 Data and task sources

• Established inventories standardize assessment across constructs, but applying them to LLMs raises concerns about contamination, response bias, and ecological validity.Examples include BFI and HEXACO for personality, SVS and PVQ for values, and CRT for heuristics and biases.Source: §5.2 · p. 24
• Custom-curated items can better fit LLM capabilities and novel constructs, but their labor-intensive development limits scalability and diversity.Examples include AI-informed value statements, voting-advice questionnaires, open-world moral scenarios, and dialogue-based ToM tests.Source: §5.2 · p. 24
• Synthetic items can scale test content and improve contextual fit, but require careful prompt engineering and validation.Studies adapt inventories or generate tests from scratch, including tests whose scores correlate with human performance on established tests.Source: §5.2 · p. 24

5.3 Prompting strategies

• Role-playing prompts simulate multiple participants and probe trait flexibility, but unstable values and perspectives can compromise validity and complicate alignment.Researchers use persona or profiling prompts to examine reliability, validity, value systems, and simulated social interactions.Source: §5.3 · pp. 24–25
• Performance-enhancing prompts—including Chain of Thought, few-shot, and construct-specific strategies—are used to improve evaluation performance and reasoning.Reported applications include bias reduction, cognitive and social-intelligence improvements, and ToM or moral-reasoning enhancements.Source: §5.3 · p. 25
• Prompt perturbations test robustness to reordered options, rephrasing, altered formats, languages, and adversarial inputs; sensitivity to them raises questions about reliability under standard conditions.Studies examine whether measured traits and abilities remain stable across these variations.Source: §5.3 · p. 25

5.4.1 Closed-ended output and scoring

• For closed-ended tests, researchers use token-level logits to infer traits or uncertainty and compare output distributions with human data, or score explicit responses using predefined rubrics and criteria.Examples include surprisal for predictability and cognitive effort, averaged Likert scores, and accuracy against ground-truth labels.Source: §5.4.1 · p. 25

5.4.2 Open-ended output and scoring

• Rule-based scoring detects keywords or response patterns, but lexicon-based methods can miss semantic nuance.Some methods instead identify deviations from equivalent treatment in responses.Source: §5.4.2 · p. 25
• Model-based scoring offers flexibility and scalability through trained evaluators, LLM judges, and embedding comparisons.Evaluators may score item-level or whole responses and may be tailored to a framework or trained as generalists.Source: §5.4.2 · pp. 25–26
• Human scoring is used when evaluation must adhere closely to standard psychometric manuals or require contextual judgments.Examples include psychologist ratings using LEAS and human assessment of cognitive biases, ToM, or MoCA performance.Source: §5.4.2 · p. 26

5.5 Inference parameters

• Greedy decoding is deterministic and reproducible but limits diversity, whereas sampling can reveal broader response variation while complicating validation.Parameter choice therefore affects which traits, opinions, or strategies appear in test results.Source: §5.5 · p. 26

6 Psychometric validation

• LLM psychometric validation prioritizes theoretical grounding, standardized protocols, and reproducibility, but the field lacks standardization in test design and administration.Figure 4 organizes the landscape around reliability and consistency, validity, and standards and recommendations.Source: §6 · p. 26, Figure 4 · p. 27

6.1 Reliability and consistency

• LLM reliability must address both instrument flaws and stochastic, context-sensitive responses, unlike human measures that primarily assess instrument quality.Researchers adapt reliability measures to repeated trials, prompt variations, languages, parallel forms, option-position robustness, and adversarial robustness.Source: §6.1 · p. 26
• Reliability findings are mixed: closed-choice personality tests can show high internal consistency while prompt sensitivity undermines parallel-form reliability.Reliability varies with construct, test format, prompting, model, and context; structured tests with strong prompt control tend to be more consistent.Source: §6.1 · pp. 27–28

6.2.1 Content validity

• Content validity is threatened by contamination in established tests and by insufficiently vetted custom or generated items that may miss the construct or introduce extraneous factors.Evaluating the content validity of custom-curated and model-generated items is described as crucial but rare.Source: §6.2.1 · p. 28

6.2.2 Construct validity

• Procedural validity asks whether response-producing processes align with the measured construct; high accuracy can coexist with shortcut reasoning and does not alone establish intelligence.The review calls for scrutiny of reasoning processes beyond output accuracy.Source: §6.2.2 · p. 28
• Human-oriented constructs may not transfer to LLMs: studies find differing latent structures, and the review describes a move toward AI-native tests and operational definitions.Reported examples include LLM personality patterns that differ from human trait structures and value factors that outperform Schwartz’s system across tasks.Source: §6.2.2 · pp. 28–29
• Response sets and social desirability can distort self-reports: option-position bias may make corrected political responses uniformly random, while direct reports can suppress less desirable values.The review notes that these patterns raise questions about the suitability of self-report tests for LLM psychometrics.Source: §6.2.2 · p. 29

6.2.3 Criterion and ecological validity

• Forced-choice measurements can diverge from human-LLM interactions, prompting work to connect personality, values, and morality measures to real-world interactions and external standards.Recommendations include open-ended dialogue and links between value measurement, safety prediction, and alignment tasks.Source: §6.2.3 · p. 29

7 Psychometrics for LLM enhancement

• The review groups psychometrically grounded enhancement into trait manipulation, safety and alignment, and cognitive enhancement.Psychometric constructs can specify enhancement targets, provide training or reward signals, or inform architecture and prompt design.Source: §7 · p. 30

7.1 Trait manipulation

• Validated inventories and belief networks guide prompts that elicit or modulate personality traits and support controlled persona simulation.Other methods intervene on hidden representations at inference or fine-tune models and parameters for trait control.Source: §7.1 · p. 30
• Fine-tuning can help LLMs simulate survey-response distributions for specific subpopulations, including on unseen questions, though effectiveness varies by domain.Psychometric alignment trains models on human response data to improve matching on novel items.Source: §7.1 · p. 30
• Current personalization methods can produce more than superficial output changes, but robust, authentic, context-independent personalization remains challenging.Source: §7.1 · p. 30

7.2 Safety and alignment

• Personality traits measured with MBTI-M and HEXACO scales correlate with safety, and modifying personalities can improve safety, especially for privacy and fairness.Studies also use value orientations to distinguish safe from unsafe responses and predict safety scores; reinforcement learning is used to align models with human values.Source: §7.2 · pp. 30–31

7.3 Cognitive enhancement

• Psychology-inspired prompts, architectural modules, and specialized training are used to improve cognition, empathy, and communication.Examples include emotional and role-playing prompts, a theory-based emotional-support dialogue framework, a neural listener module, and preference or reward signals based on emotion theories and empathy judgments.Source: §7.3 · p. 31

8.1 Psychometric validation

• Structured ability tests adapted from human instruments can lack parallel-forms reliability, construct validity, and evidence of real-world generalizability; simulation-based tests may also lack rigorous psychometric development.Defining core abilities and designing valid, comprehensive ability tests remains open.Source: §8.1 · p. 31

8.2 From human constructs to LLM constructs

• Human personality and value structures often do not transfer to LLMs, while newly adapted constructs may vary with tools, tasks, and models, leaving foundational behavioral structures unresolved.Studies report differing value structures and ability factors.Source: §8.2 · p. 31

8.3 Perceived vs. aligned traits

• Perceived traits in text can differ from traits aligned with annotators’ personal values, raising questions about which traits matter and how judges’ subjectivity affects evaluation.A response may be perceived as expressing Power but align more closely with a Conformity-valuing annotator.Source: §8.3 · pp. 31–32

8.4 Anthropomorphization challenges

• LLM traits may reflect intrinsic expression or prompted persona simulation, so evaluation can treat a model as one entity, a population of prompted profiles, or a hybrid.These choices differ in target and validation: intrinsic alignment, role-playing fidelity and steerability, or consistency under prompt permutations.Source: §8.4 · p. 32

8.5 Expanding dimensions in model deployment

• Evaluation beyond single-turn English text remains limited: language, conversation history, modalities, and agent environments each introduce distinct psychometric challenges.Traits can vary across languages and turns; multimodal tests and systematic methods for agents and multi-agent systems remain underdeveloped.Source: §8.5 · pp. 32–33

8.6 Item response theory

• Item response theory can improve item analysis, adaptive testing, bias and robustness assessment, and benchmark design, but its use remains narrow and often limited to standard models.Polytomous, hierarchical, and multidimensional models are little explored, and automated item-generation methods can be too complex for practical use.Source: §8.6 · p. 33

8.7 From evaluation to enhancement

• Psychometric insights could inform prompting, inference-time control, data selection, reward design, and methodological development, but enhancement methods remain nascent.Statically defined value targets may limit integration of context-dependent value states.Source: §8.7 · p. 33

8.8 Standardization and norming

• LLM psychometrics lacks consensus on standardized inference and prompting protocols, while its appropriate reference population for norming remains undefined.Heterogeneous protocols hinder cross-study comparison and reproducibility; rapidly changing models and prompt-sensitive profiles complicate norming.Source: §8.8 · pp. 33–34

• Attributing human-like traits or consciousness to LLMs may mislead researchers and the public, fostering unwarranted trust, emotional attachment, or misplaced expectations.The review urges precise terminology that distinguishes metaphorical from literal attributions.Source: §9 · p. 34
• Unrepresentative data or instruments can amplify bias or enable manipulation, while tools reflecting particular groups’ values may marginalize others.The review recommends inclusive, representative data and instruments, with limitations and potential biases reported clearly.Source: §9 · p. 34
• Human psychometric data may be reused beyond participants’ original consent, including as comparison baselines, training signals, or repurposed test items.The review calls for reporting data provenance and consent scope, preferring open, freely licensed instruments or purpose-built LLM benchmarks.Source: §9 · p. 34
• Psychometric comparisons risk reducing complex human traits to algorithmic terms, so the review calls for collaboration with psychologists, ethicists, and social scientists.Researchers should recognize limits of the methods and differences between human and machine cognition.Source: §9 · p. 34

• The review presents LLM Psychometrics as an approach integrating psychometric instruments, theories, and principles into LLM evaluation.Source: §10 · p. 34
• Psychometric methods can capture broad latent personality and cognitive constructs, but vary in formats, data and task sources, prompting, outputs, and scoring.The review states that methods have distinct strengths, weaknesses, and application scenarios, and should adhere to reliability, validity, and fairness principles.Source: §10 · p. 35
• Psychometrics-inspired techniques address trait manipulation, safety and alignment, and cognitive capabilities as part of developing more powerful and responsible AI systems.The review argues that psychometrics will contribute principles, dimensions, techniques, and insights to evaluation-driven AI development.Source: §10 · p. 35