Source-linked AI summary

Human Psychometric Questionnaires Mischaracterize LLM Behavior

Woojung Song, Dongmin Choi, Yoonah Park, Jongwook Han, Eun-Ju Lee, Yohan Jo

arXiv:2509.10078v5cs.CLcs.AI

TL;DR

Human psychometric questionnaires may not reliably characterize LLM behavior in everyday user interactions. This study compares questionnaire-based profiles with generation-probability profiles across eight open-source models and finds substantial divergence, including absent consistency and persona shifts in generation behavior.

  • Problem

    Evidence is limited on whether human psychometric questionnaires reliably characterize LLM behavior in realistic everyday user interactions.

    Source: §1 · p. 1
  • Method

    The study compares Likert questionnaire profiles with generation-probability profiles from everyday queries across eight open-source LLMs, including persona-prompted conditions.

    Source: §1 · pp. 1–2, §4.4 · p. 9
  • Results

    Questionnaire and generation-probability profiles diverge substantially: questionnaire consistency and human-aligned persona shifts do not appear in generation probabilities.

    Source: §1 · p. 2, §5 · p. 9
  • Takeaways & Limitations

    Questionnaire scores alone are insufficient evidence of LLM response tendencies in realistic interactions and should be complemented by generation-probability profiling with validated realistic items.

    Source: §1 · p. 2, §5 · p. 9

    The analysis relies on a single benchmark covering Schwartz values and Big Five traits, and the generation-probability framework requires token-level log-probabilities.

    Source: Limitations · p. 10

Abstract

from arXiv · show

We examine whether human psychometric questionnaires can serve as reliable tools for characterizing and predicting LLM behavior in everyday user interactions. We analyze eight open-source LLMs by comparing their value and personality profiles derived from two different methods: Likert self-reports on established questionnaires (PVQ-40/21 and BFI-44/10) and generation probabilities over value-laden responses to everyday user queries. The two profiles diverge substantially. Within-construct item consistency, often cited as evidence of stable LLM dispositions, disappears in generation probabilities. We find that established questionnaire items contain explicit lexical cues that allow models to recognize the target construct and respond in alignment-consistent, socially desirable ways, whereas realistic user queries contain far less recognizable cues. In addition, demographic persona prompts shift models' responses to human questionnaires in ways consistent with real human patterns, but no such shifts appear in the generation setting, highlighting that human questionnaires overestimate LLMs' ability to faithfully reproduce expected psychological traits when role-playing demographic personas. Overall, our study indicates that questionnaire scores alone should not be treated as evidence of LLMs' response tendencies in realistic user interactions, and supports generation-probability profiling with ecologically valid items as a complementary behavioral measure.

1 Introduction

This study tests whether human psychometric questionnaires reliably characterize and predict LLM behavior in realistic user interactions. Comparing questionnaire self-reports with generation-probability profiles, it finds substantial divergence and argues that questionnaire scores require complementary behavioral measures.

Motivation

  • The study addresses the safety need to characterize LLM values and traits as models are deployed in high-risk tasks, while questioning whether human questionnaires provide reliable behavioral prediction.Prior work applies established human measures such as the PVQ and BFI to LLMs, but often studies non-everyday tasks or reflective self-reports.Source: §1 · p. 1

Methods

  • The authors compare Likert questionnaire scores from PVQ and BFI instruments with generation-probability scores derived from validated, realistic user queries and plausible responses.The Value Portrait dataset links responses to Schwartz values and Big Five traits, enabling psychometrically grounded measurement of generative behavior through response log-probabilities.Source: §1 · p. 2

Findings

  • Questionnaire-based profiles and generation-probability profiles show low construct-ranking agreement across eight open-source LLMs.The comparison directly tests whether established questionnaires and actual generation behavior produce different psychological profiles.Source: §1 · p. 2
  • Intra-construct consistency appears in questionnaire responses but disappears in generation probabilities, challenging its interpretation as evidence of stable LLM dispositions.Established questionnaire items contain explicit construct cues, whereas realistic queries and responses contain fewer recognizable cues; obscuring those cues reduces consistency.Source: §1 · p. 2
  • Persona-induced, human-aligned shifts in questionnaire profiles do not emerge in generation-probability profiles, so questionnaire scores should be interpreted cautiously and complemented with validated realistic items.The findings indicate that role-playing demographic personas can produce expected questionnaire patterns without corresponding shifts in realistic generation behavior.Source: §1 · p. 2

2 Related Works

Prior work uses established psychometric questionnaires to assess LLM values and personality, while newer studies examine whether questionnaire-derived profiles generalize to behavioral probes. This work extends that literature by using generation probabilities over validated items and examining questionnaire item transparency.

2.1 Psychometric Assessment of LLMs

  • Prior studies apply BFI, IPIP-NEO, and PVQ questionnaires to LLMs, reporting reliability, differentiation, or construct-related validity.These instruments include BFI variants (John et al., 1991; Soto and John, 2017), IPIP-NEO, and PVQ, applied by Miotto et al. (2022), Serapio-García et al. (2025), and Kovaˇc et al. (2024).Source: §2.1 · p. 2

2.2 Psychometric Profiles vs. LLM Behavior

  • Behavioral-profiling studies use diverse probes, but often lack independently validated construct links or remain structurally close to questionnaires rather than realistic interactions.This work complements them with generation probabilities over psychometrically validated items, preserving questionnaire constructs and examining item transparency.Source: §2.2 · pp. 2–3

2.3 Response Probability as a Measure of Model Behavior

  • Prompt-sensitive self-reports contrast with more robust probability-based measures that read models’ candidate-text scoring directly from generation distributions.The study derives profiles from candidate-response log-probabilities, avoiding reliance on accurate metajudgments about models’ own dispositions.Source: §2.3 · p. 3

3 Profiling Methods and Models

The section contrasts questionnaire-based profiling with generation-probability profiling of Big Five traits and ten Schwartz values. It introduces validated everyday scenarios, scoring procedures, methodological scope, and eight open-source models used for comparison.

3.1 Established Questionnaire Profiling

  • For established profiling, the study uses PVQ-40/21 and BFI-44/10 questionnaires with gender-neutral wording, reversed Likert-scale prompt orders, and construct scores averaged across items and variants.PVQ responses use a 1–6 scale, BFI responses use a 1–5 scale, and opposite option orders address known order sensitivity.Source: §3.1 · p. 3

3.2 Generation Probability Profiling

  • The Value Portrait dataset contains 520 scenario-response pairs from 104 real-world queries, with five candidate responses per query and 286 value-tagged and 228 trait-tagged pairs validated against human ratings.The queries span everyday requests and value-laden interpersonal dilemmas, and construct tags require human-validated correlations of ρ ≥.30.Source: §3.2 · p. 3
  • Generation profiles average within-scenario response log-probabilities across construct-tagged scenarios, producing ten-dimensional value and five-dimensional personality profiles per model.The macro-average gives each scenario equal weight regardless of its number of tagged responses, and probabilities use summed token log-probabilities without length normalization.Source: §3.2 · pp. 3–4
  • Generation-probability profiling scores models’ relative construct preferences from fixed, psychometrically validated candidate responses in realistic scenarios rather than measuring unconstrained real-world behavior.The method averages tagged responses across scenarios to retain inter-construct contrasts while reducing item-level variation, and its candidate constructs have been psychometrically validated.Source: §3 · p. 3, §3.2 · p. 4

3.3 Models

  • The evaluation covers eight open-source models from four families, each represented by smaller and larger variants: Gemma3, Qwen 2.5, Qwen 3, and GPT-OSS.The set includes dense and mixture-of-experts models, with configurations listed in Appendix C.Source: §3.3 · p. 4

4 Research Questions

Across four research questions, questionnaire-based LLM profiles diverge from generation-probability behavior: construct rankings and item consistency are strong for established questionnaires but largely absent in realistic scenarios. LLMs recognize transparent questionnaire cues and reproduce demographic patterns there, while these effects do not reliably carry over to generation behavior.

4.1 RQ1: Do established questionnaires and actual generation-probability produce different LLM profiles?

  • Cross-method agreement is substantially lower than within-questionnaire agreement, with generation correlations as low as 0.11 and NDCG gaps across both values and traits.Within-method Spearman ρ averages 0.74 for PVQ-40↔PVQ-21 and 0.77 for BFI-44↔BFI-10; cross-method ρ is 0.31/0.28 for Schwartz values and 0.26/0.11 for Big Five, while NDCG is 0.69–0.81 versus 0.89–0.91 within-method.Source: §4.1 · pp. 4–5, Table 2 · p. 5
  • Generation-probability profiles diverge from questionnaire profiles across all model families, with direction varying by model and construct and no uniform post-hoc correction supported.Hedonism decreases and Power increases relative to PVQ-40 in all eight models, while Conscientiousness never increases relative to BFI-44; other divergence directions are inconsistent.Source: §4.1 · p. 5

4.2 RQ2: Does intra-construct response consistency hold for generation-probability profiles?

  • Questionnaire items show strong construct structure, but generation-probability scores do not: η2 averages 0.526/0.492 and WMV .603/.592 for PVQ-40/BFI-44, whereas generation values remain at permutation baselines.For generation probabilities, η2 is indistinguishable from baseline for PVQ-40 (p = 0.604) and BFI-44 (p = 0.726), with WMV near 1.0; held-out human ratings recover value structure (η2 = .084, WMV = .908, p ≤.004).Source: §4.2 · p. 6, Table 3 · p. 7

4.3 RQ3: Can LLMs recognize the target construct from item text?

  • LLMs recognize established questionnaire constructs but not VP scenarios: mean F1 is .69–.83 versus .05–.11, and sentence embeddings assign the correct construct to 77–81% versus 11–26%.Established items show diagonal item–definition and within-construct similarity, whereas VP items do not; reducing construct cues weakens BFI-44 structure from η2 = .492 to .241 and PVQ-40 from .526 to .432.Source: §4.3 · pp. 7–8, Figure 1 · p. 7

4.4 RQ4: Do persona-induced shifts in questionnaire profiles carry over to generation behavior?

  • Persona-induced shifts align with human demographic value patterns on questionnaires but not generation probabilities: questionnaire direction matches are 62/80 and 55/80, versus VP’s 40/80 at chance.PVQ-40 cosine is positive in all eight conditions, while VP mean cosine is −0.03, aggregate cosine is +0.007 with bootstrap 95% CI [−0.22, +0.24], and contrasting personas can produce opposite-sign shifts.Source: §4.4 · p. 9, Table 5 · p. 9

5 Conclusion

Established questionnaire profiles diverge from generation-probability profiles of LLM behavior, limiting what questionnaire scores alone can establish. The findings support supplementing questionnaires with generation-probability evaluation for behavioral characterization.

  • Questionnaire-derived psychological profiles do not align with profiles derived from generation probabilities, showing that questionnaire scores alone are insufficient evidence of LLM behavior.Reducing explicit cues to the target construct makes responses less consistent across items intended to measure the same construct.Source: §5 · p. 9
  • Instruction tuning strengthens value-specific response patterns on original PVQ-40 items but not on cue-reduced items.This pattern emerges from matched comparisons between base and instruction-tuned checkpoints.Source: §5 · p. 9
  • Demographic personas shift questionnaire responses in human-consistent directions but produce directionally incoherent generation-probability shifts that do not track human demographic patterns.These results question whether questionnaire scores demonstrate faithful expression of the psychological traits expected of demographic personas.Source: §5 · p. 9
  • Future evaluations should supplement established questionnaires with generation-probability measures when the goal is to characterize model behavior rather than self-report responses.Source: §5 · p. 9

Limitations · A Construct Definitions

The framework’s main limitations concern token-level log-probability access, reliance on a single benchmark, and its fixed-response design. Tables 6 and 7 define the psychological constructs measured in the study.

Limitations

  • The generation-probability framework requires token-level log-probabilities, limiting external evaluation when model APIs do not expose likelihoods.Model developers can apply it to open-weight and proprietary models using internal logits.Source: Limitations · p. 10
  • Because the behavioral analysis uses a single benchmark, its claims are restricted to the Schwartz values and Big Five traits covered by Value Portrait.Source: Limitations · p. 10
  • The RQ4 persona comparison uses the European Social Survey as a human reference, which provides Schwartz value data but not Big Five measures.Source: Limitations · p. 10
  • The generation-probability score lies between Likert questionnaires and unconstrained generation by evaluating fixed, construct-validated candidate responses.This design avoids sampling-temperature and open-ended-paraphrasing variation but does not capture how psychological tendencies appear in freely generated text.Source: Limitations · p. 10
  • Analyzing open-ended outputs is identified as a natural next step for complementing the fixed-response generation-probability design.Source: Limitations · p. 10

A Construct Definitions

B Value Portrait Dataset Details · C Inference Configuration · D Prompt Templates

The study builds a validated Value Portrait dataset from diverse conversational and advisory queries, then specifies separate inference procedures and prompt templates for questionnaires, generation probabilities, and construct recognition. These materials operationalize comparisons between established psychometric instruments and ecologically grounded user interactions.

B Value Portrait Dataset Details

  • The 104 source queries combine human–LLM conversations from ShareGPT and LMSYS (Zheng et al., 2024) with human–human advisory exchanges from Reddit (Lourie et al., 2021) and Dear Abby archives.Sources cover opinion questions, brainstorming, open-ended discussion, interpersonal dilemmas, and value-laden decision-making.Source: B Value Portrait Dataset Details · p. 13
  • Validation by 681 participants retained responses whose similarity ratings correlated with participant construct scores at ρ ≥.30, producing 286 value-associated and 228 trait-associated tags.The tags derive from 221 value-tagged and 174 trait-tagged unique items, with approximately 46 ratings per query–response pair.Source: B Value Portrait Dataset Details · p. 13

C Inference Configuration

  • Inference used locally served open-weight models on 4× NVIDIA A100 80 GB GPUs, with bfloat16 weights except Qwen3-235B-A22B’s officially released FP8 checkpoint and no additional post-training quantization.Experiments consumed roughly 110 node-hours, equivalent to approximately 500 A100 GPU-hours.Source: C Inference Configuration · pp. 13–14
  • Established questionnaires used deterministic native chat completions, whereas Value Portrait scoring used echoed sequence log-probabilities over response tokens extracted from fully templated prompt–response sequences.Questionnaire runs used temperature = 0.0 and max_tokens = 1024; Value Portrait scoring used echo = True, logprobs = 1, max_tokens = 1, and temperature = 1.0.Source: C Inference Configuration · p. 13

D.1 Established Questionnaire Prompts

  • Established questionnaire prompts preserved original PVQ-40, PVQ-21, BFI-44, and BFI-10 wording while reversing response-option order between high-to-low and low-to-high variants.The prompt structures are illustrated in Figures 2–9.Source: §D.1 · p. 15, Figure 2 · p. 15, Figure 3 · p. 15

D.2 Value Portrait Generation Prompts

  • Value Portrait generation prompts cover both human–human advisory scenarios and human–LLM conversation scenarios.These templates are shown in Figures 10 and 11.Source: Figure 10 · p. 17, Figure 11 · p. 17

D.3 Item Construct Recognition Prompts

E Sampling Log-Probability Validity … F.3 Comparison Metric Details

Supplementary analyses validate the generation-probability method while clarifying aggregation, scoring, and comparison-metric choices. VP candidates occupy plausible regions of model distributions, and the main within-versus-cross-method discrepancy remains under alternative scoring and aggregation schemes.

E Sampling Log-Probability Validity

  • VP candidates had a median best rank of 4 and mean rank of 5.7/15, placing them around the 69th percentile and within models’ normal generation distributions.Across 832 model–scenario pairs, candidates were neither anomalously high- nor low-probability.Source: E Sampling Log-Probability Validity · p. 18, Table 8 · p. 18
  • VP placement varied by model, from near-top ranks for GPT-OSS, Qwen2.5-7B, and Qwen3-30B to middle-to-lower ranks for Gemma-3, Qwen2.5-72B, and Qwen3-235B.These comparisons support relative construct ordering but do not show how often exact candidates would arise under open-ended sampling.Source: E Sampling Log-Probability Validity · p. 18

F.1 Generation Probability Scoring: Aggregation Choice

  • The main analysis uses macro-averaging because it weights scenarios equally and parallels equal per-item weighting in established questionnaire profiles.This reduces amplification from scenarios containing multiple candidates tagged to the same construct.Source: §F.1 · pp. 18–19
  • Macro- and micro-aggregation produced highly correlated Schwartz profiles, with per-model Spearman ρ ranging from 0.79 to 1.00 and mean ρ = 0.92.The headline within-versus-cross-method gap remained robust, although individual Table 2 cells shifted by up to approximately 0.31.Source: §F.1 · p. 19

F.2 Length-Normalized Scoring

  • Length normalization altered some correlation magnitudes, but all four cross-method means remained below within-method references and pooled Spearman ρ and NDCG gaps stayed significant.Under normalized scoring, separate value and trait tests were also significant for Spearman ρ (p = .004 and .012) and NDCG (p = .004 and .016).Source: §F.2 · p. 19, Table 9 · p. 19
  • Summed log-probability remains primary because it measures complete-response probability, whereas per-token means reduce the mechanical penalty for longer responses.Table 10 provides the full per-model normalized results, while summed-score counterparts appear in Table 2.Source: §F.2 · p. 19, Table 10 · p. 20

F.3 Comparison Metric Details

  • Spearman’s ρ measures monotonic agreement and NDCG emphasizes top-ranked constructs, with full-list NDCG equal to 1 indicating perfect ranking recovery.Because profiles contain only 10 Schwartz or 5 Big Five constructs, significance is tested at the aggregate level across all eight models.Source: §F.3 · p. 19
  • For tied ideal-profile scores, constructs receive distinct relevance grades using deterministic NumPy argsort order, imposing a strict order on ties.For example, Qwen3-30B’s PVQ-40 profile contains five values tied at 6.00.Source: §F.3 · p. 20

F.4 Statistical Testing: Within-Method vs. Cross-Method Agreement … G.1 Permutation Test Details

Exact sign-flip tests show that within-method agreement significantly exceeds agreement between generation-probability and established measures across Schwartz values, Big Five, and overall analyses. Supplementary materials provide per-model scores, rank trajectories, and permutation-test details with important interpretive limitations.

F.4 Statistical Testing: Within-Method vs. Cross-Method Agreement

  • The primary inference used exact one-sided sign-flip permutation tests, enumerating 256 assignments per construct group and 65,536 assignments for the pooled 16-observation analysis.Bootstrap 95% confidence intervals used 10,000 resamples but were treated descriptively because n = 8 per construct group makes percentile intervals somewhat anticonservative.Source: §F.4 · p. 20, §F.5 · p. 21
  • All eight models had positive within-minus-cross differences for Schwartz values and seven of eight for Big Five, with Qwen3-235B’s BFI exception linked to an unusually low within-method baseline (ρ = 0.205).Strongly anti-correlated generation profiles, including Qwen3-30B and Gemma3-27B for Big Five, contribute disproportionately large differences.Source: §F.4 · p. 20, Table 12 · p. 21

F.5 Per-Model Construct Scores

  • Per-model construct tables provide Likert-scale established scores and mean total log-probability generation scores for rank-based Spearman ρ and NDCG analyses.Generation-probability scores are not comparable in absolute magnitude across models; only within-model ordering is meaningful, and rounded values may differ from raw results by 1 ULP.Source: §F.5 · pp. 20–21

F.4 Statistical Testing: Within-Method vs. Cross-Method Agreement

  • Within-method agreement significantly exceeded cross-method agreement for Spearman ρ across Schwartz values (∆= 0.446, p = 0.004), Big Five (∆= 0.584, p = 0.016), and overall (∆= 0.515, p < 0.001), with analogous significant NDCG gaps overall (∆= 0.132, p = 0.002).Table 11 reports these paired significance tests; both construct groups were individually significant for NDCG.Source: §F.4 · p. 20, Table 11 · p. 21

F.6 Construct-Level Rank Trajectories

  • Figures 15 and 16 visualize construct-rank trajectories across PVQ-40/PVQ-21 or BFI-44/BFI-10 and generation-probability measurements, with line crossings indicating rank reversals.The figures compare two within-method columns against the generation-probability column for each model.Source: §F.6 · p. 21

G.1 Permutation Test Details

  • Per-model permutation tables report observed η2, one-sided p-values, and null-distribution means and standard deviations for PVQ-40, BFI-44, and VP.PVQ-21 and BFI-10 are omitted because their two to three items per construct make within-construct permutation tests underpowered.Source: §G.1 · p. 21

G.2 Per-Construct Within-Variance

Per-construct within-variance measures how tightly items cluster within each value or trait, with values below 1.0 indicating tighter-than-random clustering. Lower z-scored variance likewise indicates stronger within-construct clustering across questionnaire and generation-probability profiles.

G.2 Per-Construct Within-Variance

  • Values below 1.0 indicate that items within each construct cluster more tightly than expected under random assignment.Source: §G.2 · p. 21
  • Lower z-scored within-construct variance indicates tighter clustering within a construct.Source: §G.2 · p. 24
  • The generation-probability analysis defines each scenario–trait or scenario–value score by averaging construct-tagged responses meeting the human-validated threshold ρ ≥.30.The VP trait analysis includes N = 140 scenario–trait pairs, while the VP value analysis includes N = 226 scenario–value pairs.Source: §G.2 · p. 25

H Held-Out Human Reference Analysis … I.2.3 Within-Construct Item Similarity

Held-out human ratings validate construct structure for values but not traits, while questionnaire items show strong lexical transparency that largely disappears in VP response texts. Embedding analyses likewise find substantial within-construct similarity for established questionnaires but near-zero clustering for VP items.

H Held-Out Human Reference Analysis

  • Human questionnaire congruence predicts held-out similarity ratings for both values (β = .116, partial r = .060) and traits (β = .083, partial r = .034), with p = .001 in both domains.Cross-fitted analyses use batch-balanced folds, fixed effects, and within-fold permutation tests.Source: H Held-Out Human Reference Analysis · p. 26
  • Held-out human value ratings show greater construct structure than within-scenario permutation references, whereas trait results are not significant.This positive control validates sensitivity only in the value domain and is not an absolute human–LLM effect-size comparison.Source: H Held-Out Human Reference Analysis · p. 26

I RQ3: Supplementary Materials

  • The supplementary materials report per-construct recognition and sentence-embedding analyses that independently assess LLM construct recognition and textual transparency.These analyses complement main-text instrument-level recognition results and evaluate item–definition and within-construct similarity.Source: I RQ3: Supplementary Materials · p. 26

I.1 LLM Item–Construct Recognition: Per-Construct Breakdown

  • BFI items are highly recognizable: GPT-OSS-120B and Qwen3-235B approach perfect recognition, while Gemma3-4B’s weakest pairs remain at F1 = .53.Gemma3-4B’s lowest BFI-44 scores occur for Agreeableness and Neuroticism, and still exceed every VP result.Source: §I.1 · p. 26
  • PVQ Achievement and Hedonism are most recognizable, whereas Security, Power, and Tradition are hardest, with F1 often below .60.Most models reach F1 ≥.85 for Achievement and F1 ≥.75 for Hedonism.Source: §I.1 · p. 26
  • VP construct recognition is pervasive weak: Agreeableness reaches F1 from .09 to .27 and Stimulation from .07 to .25, while no model exceeds F1 > .27 for any construct.Interpersonal conflict and adventure- or novelty-themed scenarios may provide weak signals for these two constructs; all others hover near zero.Source: §I.1 · pp. 26–27

I.2.1 Sentence Encoder

  • Sentence embeddings show established questionnaires achieve 77–81% Top-1 item–definition accuracy and discrimination of 0.13–0.22, while VP responses are near chance with near-zero discrimination.The primary encoder is all-mpnet-base-v2, selected to measure surface lexical-semantic cues rather than domain-specific features.Source: §I.2.1 · p. 27, §I.2.2 · p. 28

I.2.3 Within-Construct Item Similarity

  • Within-construct clustering gaps are 0.07–0.15 for established questionnaires but −0.001 to +0.004 for VP responses, indicating questionnaire wording consistency absent from VP texts.Nearly all established constructs exceed inter-construct baselines, with strongest similarity for PVQ-40 Stimulation and Hedonism and BFI-44 Agreeableness; Security and Tradition are borderline in PVQ-40.Source: §I.2.3 · p. 28

I.2.4 Encoder Robustness Check

Across five sentence encoders from multiple model families, established questionnaire items consistently show stronger construct cues and within-construct clustering than VP items. Although absolute magnitudes vary by encoder quality, the qualitative established ≫VP pattern remains stable.

Encoder Robustness Check

  • The robustness check spans five encoders from multiple model families: MPNet-base, MiniLM-L12, MiniLM-L6, BGE-base, and E5-base.E5-base shows lower discrimination overall, indicating that encoder quality affects magnitude without changing the qualitative pattern.Source: §I.2.4 · p. 28
  • Across all five encoders, established questionnaires consistently outperform VP items in item–definition discrimination and within-construct clustering gap.The absolute magnitudes vary with encoder quality, but the established ≫VP pattern holds across MPNet, MiniLM, BGE, and E5 encoders.Source: §I.2.4 · p. 28, Table 30 · p. 30
  • Established questionnaires show positive within-construct clustering gaps, whereas VP outputs show near-zero gaps across the five encoders.This pattern indicates stronger textual similarity among established items within the same construct than among VP items.Source: Figure 18 · p. 30

J Construct-Cue Reduction Experiment … L.1 Persona Prompt Templates

Cue reduction weakened construct recognition while leaving overall construct structure largely unchanged, whereas instruction tuning strongly shifted questionnaire scores but not consistently VP scores or questionnaire–VP agreement. Persona prompts were applied identically to questionnaire and VP-probability settings across eight demographic conditions.

J Construct-Cue Reduction Experiment

  • Construct-recognition F1 fell from .70 to .63 for PVQ-40 and from .85 to .68 for BFI-44 after cue reduction.Sentence-encoder top-1 matching likewise fell from .775 to .425 and .773 to .477, respectively, toward the VP reference of .08.Source: J Construct-Cue Reduction Experiment · p. 31
  • Cue reduction left mean construct structure nearly unchanged, with η2 changing .230 →.231 for PVQ-40 and .094 →.093 for BFI-44.Mean WMV changed from 1.025 →1.025 for PVQ-40 and 1.022 →1.023 for BFI-44; paired sign-flip p-values were .078 and .004.Source: J Construct-Cue Reduction Experiment · p. 31, Table 31 · p. 32

K Base vs. Instruction-Tuned Model Pairs

  • Instruction tuning increased Benevolence, Universalism, and Agreeableness in 11 of 12 comparisons, averaging +0.88 Likert points, while standardized VP changes averaged .00.The questionnaire scores used constrained next-token probabilities, and Table 32 reports the base-to-instruction changes.Source: K Base vs. Instruction-Tuned Model Pairs · p. 32, Table 32 · p. 32
  • On original PVQ-40 items, η2 rose in every matched pair by +.196 on average, but cue-reduced gains averaged −.006 and were smaller in all four pairs.The cue-reduced contrast had one-sided sign-flip p = .0625; BFI-44 gains were inconsistent, with positive contrasts in 2/4 pairs and p = .44.Source: K Base vs. Instruction-Tuned Model Pairs · p. 32, Table 33 · p. 33
  • Questionnaire–VP agreement did not consistently improve after post-training: value-profile ρ changed from .318 to .339, while trait-profile ρ changed from .525 to .275.Value-profile agreement rose in two pairs and fell in two; trait-profile agreement rose in none, fell in two, and was unchanged in two.Source: K Base vs. Instruction-Tuned Model Pairs · p. 32, Table 34 · p. 33
  • Post-training produced a consistent VP signature across pairs, with all six ∆VP cross-pair cosines positive, but ∆VP and ∆PVQ aligned inconsistently within pairs.Cross-pair cosines ranged from .61 to .97 with mean .78, whereas within-pair cosine alignment averaged −.10 and ranged from −.30 to +.14.Source: K Base vs. Instruction-Tuned Model Pairs · p. 32, Table 35 · p. 33

L.1 Persona Prompt Templates

  • Table 38 specified eight demographic persona system prompts for both Likert questionnaire responses and VP candidate-response probabilities, using the vanilla configuration as the RQ1 baseline.The same prompt wording was used across PVQ-40, PVQ-21, and VP-probability conditions.Source: §L.1 · p. 38

L.2 Methodology Details … L.7 Cross-Model Variance

Across demographic persona conditions, Likert questionnaire shifts align with human effects, whereas generation-probability shifts do not. The analyses quantify this divergence using direction agreement, cosine similarity, consensus, normalized magnitudes, value-level profiles, and cross-model variance.

L.2 Methodology Details

  • The methodology compares centered human and LLM demographic deltas using cosine similarity, direction match, bootstrap and permutation tests, and model-consensus categories.Deltas are computed separately for PVQ-40, PVQ-21, and VP across seven models; strong consensus requires at least 6 of 7 models, while exactly 5 of 7 is intermediate.Source: §L.2 · pp. 38–39

L.4.5 Between-Value Normalized Magnitude

  • Normalized magnitude compares shifts with each source profile’s within-value baseline spread, while value-level VP deltas use unit-length normalization and support direction-only interpretation.The normalized magnitude is not a classical Cohen’s d, and VP magnitudes are not directly comparable with Likert magnitudes.Source: §L.2 · p. 39, §L.4.5 · p. 41, Table 45 · p. 42

L.3 Direction Match Statistical Tests

  • 77.5% of PVQ-40 and 68.8% of PVQ-21 directions matched human effects significantly, whereas VP matched 50.0% without significant agreement.Binomial-test p-values were 4.07 × 10−7, 5.26 × 10−4, and 5.44 × 10−1, respectively.Source: §L.3 · p. 39, Table 40 · p. 40

L.4.1 Cosine Similarity by Condition

  • PVQ-40 had positive human–LLM cosine similarity in all eight conditions, PVQ-21 in seven, while VP showed four positive and four negative conditions.Across models, PVQ-40 cosine ranged from +.184 to +.600, whereas VP ranged from −.101 to +.196 and was negative for four of seven models.Source: §L.4.1 · p. 39, §L.4.2 · p. 39, Table 41 · p. 40

L.4.2 Per-Model Cosine Similarity

  • Per-model results show that weak VP alignment is distributed across models rather than caused by one outlier, while every model has positive PVQ-40–human cosine.PVQ-40–human cosine ranges from +.184 to +.600; VP–human cosine ranges from −.101 to +.196 and is negative for four of seven models.Source: §L.4.2 · p. 39, Table 42 · p. 40

L.4.3 Cross-Model Consensus

  • Cross-model consensus is evaluated over 80 condition–value pairs by the proportion of seven models sharing each shift’s majority direction.The accompanying consensus table distinguishes strong agreement at ≥6/7 models from weak agreement below 5/7, excluding intermediate 5/7 cases.Source: §L.4.3 · p. 41, Table 43 · p. 42

L.4.4 Bootstrap Confidence Intervals and Permutation Tests

  • Bootstrap intervals resample eight conditions, while permutation tests independently shuffle ten value labels within each condition to assess aggregate cosine reliability.Both procedures use 10,000 iterations.Source: §L.4.4 · p. 41, Table 44 · p. 42

L.7 Cross-Model Variance

  • Political conditions show the greatest cross-model PVQ-40 delta variance, whereas Gender and Education conditions are more stable; per-model tables expose individual questionnaire shifts.Variance uses the population standard deviation across seven models.Source: §L.7 · p. 41, §L.6 · p. 41, Table 49 · p. 46
Loading 2509.10078v5…