Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents

Benchmark design

• The benchmark uses 263 Lithuanian employees, 68 items, 12 subscales, and three validated organisational instruments to test psychometric fidelity under cross-linguistic conditions.The dataset includes the Dunham Attitudes Toward Change, UWES-17, and Koopmans IWPQ instruments, whose published structures provide external constraints.Source: Figure 1 · p. 2, §1 · p. 3

Psychometric motivation

• LLM survey simulation must preserve covariance, factor structure, reliability, mediation, and group differences because matching item means alone can distort measurement and downstream inference.Over-coherent synthetic responses can inflate internal-consistency statistics, while incorrect inter-construct relations can alter latent-variable and mediation analyses.Source: §1 · pp. 2–3

Evaluation framework

• The study evaluates 37 LLMs across persona disclosure, presentation, reasoning effort, language, and demographic-swap conditions using a six-dimensional PSS anchored to five statistical baselines and a human ceiling.The design also includes Tucker’s φ with an item-permutation null, respondent-bootstrap confidence intervals, and a verbatim-recall probe.Source: Figure 1 · p. 2, §1 · p. 3

Downstream consequences

• The paper extends raw-fidelity evaluation with fairness, inter-LLM agreement, ensemble, response-style, predictive-validity, and mediation-negative-control diagnostics to quantify substitution risks.It also reports that explicit debiasing instructions and few-shot conditioning on real answer vectors do not close the gap, indicating a structural limitation rather than a prompting artifact.Source: §1 · p. 4

Central claim

• The central claim is that LLMs reproduce the theoretical direction of human psychometric relationships but tend to generate over-coherent, low-variance, and demographically stereotyped respondents.The paper therefore distinguishes dimensions of fidelity and examines their dependence on persona-conditioning regimes rather than treating synthetic respondents as simply good or bad.Source: §1 · p. 4

Prior evidence

• LLM respondent studies show plausible human-like answers and qualitative behavioral or demographic effects, but primarily assess single items, marginal contrasts, or isolated aggregate effects rather than construct validity.Argyle et al. [2], Horton, Aher et al. [1], Park et al. [17], and Dillion et al. establish feasibility, while Santurkar et al., Durmus et al. [9], Bisbee et al., and Wang et al. [24] document systematic population and identity-group biases.Source: §2 · p. 4

Contribution

• The study isolates failures across joint distributions, correlations, reliability, mediation, demographic effects, and factor structure while testing persona disclosure, presentation, reasoning effort, and stochastic stability.It combines these dimensions rather than treating plausibility as sufficient evidence of psychometric fidelity.Source: §2 · pp. 4–5

Experimental design

• Persona prompting varies across profile granularity, presentation mode, and reasoning effort, so the study tests a five-step disclosure ladder, three presentation modes, and two reasoning-effort levels.The presentation-mode ablation is described as lacking direct precedent in LLM-as-respondent research, despite related educational test-item work [1].Source: §2 · p. 5

Counterfactual fairness

• The counterfactual design swaps one demographic field in a fixed multi-attribute persona, administers a 68-item instrument, and estimates paired Cohen’s d at the subscale level.This makes LLM effects comparable with human contrasts and separates stereotype amplification by demographic axis; Section 5.4 reports stronger education than gender amplification.Source: §2 · p. 5

Cross-cultural psychometrics

• Cross-language evaluation treats LLM translation as a measurement-invariance problem, using Lithuanian factor structures as external constraints on construct-fidelity statistics.The LLM ablation reports mean LT↔EN per-item drift of 0.26 Likert points and per-item Pearson 0.889, while no human Lithuanian-English invariance test is run.Source: §2 · p. 5

Psychometric framework

• The evaluation combines established measures of reliability, mediation, covariance similarity, factor congruence, invariance, discriminant validity, and bifactor structure into one harness anchored to statistical baselines and a held-out human ceiling.The framework uses Cronbach’s α, McDonald’s ω, Hayes mediation, modified RV2, Tucker’s φ, ∆CFI, HTMT, and bifactor decomposition.Source: §2 · p. 5

Original study and consent provenance

• The human ground truth contains n = 263 Lithuanian business-organisation employees recruited through an anonymous online questionnaire under informed-consent and reuse permissions.The records were collected in March–April 2020 and released in de-identified form under CC-BY-NC-4.0.Source: §3 · p. 6

Demographic composition

• The sample spans diverse sectors, organisation sizes, roles, ages, and tenure, but is 76% women and has demographic contrasts with a median minimum-detectable d of 0.45.The dataset includes 33% managers and 67% non-managers; 81% of tested contrasts are underpowered for small effects.Source: §3 · p. 6

Instrument battery and translation history

• The battery combines IWPQ, ATC, UWES-17, and the weaker ChangeEng auxiliary scale, totaling 68 items across 12 subscales with Lithuanian translations and reported internal-consistency reliabilities.IWPQ and ATC contain 18 items each, UWES-17 contains 17, and ChangeEng contains 15; ChangeEng is intentionally included as a less-validated construct.Source: §3 · pp. 6–7

Mediation reproduction

• ab = 0.16 (95% bootstrap CI [0.10, 0.24], nboot = 5000), reproducing the published partial mediation of attitudes toward change through engagement to performance.The indirect path exceeded the smaller positive direct path, and its direction, magnitude, and significance define the LLM-side mediation benchmark.Source: §3 · p. 7

Profile cards and de-identification

• LLM conditioning uses 11-field profile cards containing demographics, tenure, sector, organisation size, and change-context ratings, while withholding respondents’ actual item responses.The released data are de-identified through coarse demographic categories, removal of free-text industry fields, and rounded timestamps, with an audit trail and checksum.Source: §3 · p. 7

4.1 Persona conditioning, prompts, presentation

• C3 is the headline persona condition, exposing all structured respondent fields without free text, while C0–C4 and C9 test disclosure and profile-presentation effects.Prompts enforce Lithuanian JSON responses across the full Likert range without revealing hypotheses, subscales, or construct labels; M1–M3 and English provide presentation and language ablations.Source: §4.1 · p. 8

4.2 Generation procedure

• The headline grid samples n = 100 stratified respondents per model-condition cell across 37 models, uses one temperature-0.7 repeat, and fixes reasoning effort at minimal.Stratification matches the human joint gender × age-band × role distribution; parser retries, caching, checkpoints, and provider-specific infrastructure support validation and reproducibility.Source: §4.2 · pp. 8–9

4.3 Evaluation framework

• The six-dimension framework scores distributions, correlations, reliability, mediation, demographic effects, and construct-level fidelity, with PSS weighted toward distribution and correlation.Construct fidelity includes factor alignment, Tucker’s φ with an item-permutation null, bifactor indices, inter-subscale gaps, and HTMT; PSS weights are (0.25, 0.25, 0.20, 0.20, 0.10).Source: §4.3 · pp. 10–12
• Five statistical baselines and an 80/20 held-out human split anchor interpretation, including Gaussian copula as the key combined distribution-and-correlation benchmark and a ceiling around 0.83.The baselines separately stress distribution, covariance, demographic effects, and retrieval, while the human ceiling reflects sampling variability at nH = 263.Source: §4.3 · p. 11

5.1 Human baseline replication

• The human-baseline pipeline reproduces published composites, reliability, mediation, role effects, and gender nulls, establishing the reference target for evaluating LLM samples.Composite means differ by at most 0.04, Cronbach’s α by at most 0.02, role effects satisfy p < 0.005, and gender effects remain p > 0.4.Source: §5.1 · p. 12

5.2 Anchored PSS leaderboard

• Gaussian-copula and MVN baselines outperform every LLM on sample-driven components, while gpt-5.4-mini leads LLMs at PSS 0.714 versus the human ceiling of 0.825.The copula reaches c = 0.95 and r = 0.99, compared with c = 0.52 and r = 0.94 for the best LLM; its combined PSS is 0.688.Source: §5.2 · pp. 12–13, Table 2 · p. 14

5.3 Construct-level fidelity with permutation null

• Permutation-corrected factor congruence is strongest for IWPQ, weaker for ChangeEng, and misleading for UWES, where eight models fall inside the chance distribution despite raw φ up to 0.856.IWPQ has mean φ=0.821 with 32/37 rejections, whereas ChangeEng has φ=0.458 with 22/37; gpt-5.4-mini lies inside the null for ChangeEng and UWES.Source: §5.3 · pp. 14–15

5.4 Counterfactual stereotype amplification on three axes

• Education swaps produce the largest counterfactual effects, with mean |d| = 0.563 versus 0.176 for role and mean |dz| = 0.119 for gender.The education≫role > gender ordering is consistent across models; gender effects are near-null for most models, although they exceed the human null in some cases.Source: §5.4 · p. 16

5.5 Ablations: reasoning effort, cross-language, presentation mode, cohort PSS

• Presentation mode materially changes responses, while reasoning effort does not uniformly improve fidelity and Lithuanian–English drift remains bounded.Per-subscale presentation reaches |d| = 1.20 for llama-4-maverick; LT–EN correlation is 0.889 with mean absolute difference |∆| = 0.26 Likert points.Source: §5.5 · p. 17

5.6 Memorization probe and inter-LLM homogeneity

• Verbatim recall is rare and unrelated to psychometric performance: the maximum high-recall rate is 4.69%, 22 models have zero recall, and recall–PSS correlation is 0.00.Models more commonly refuse or confabulate plausible Lithuanian items than reproduce published wording, ruling out verbatim memorization as the leaderboard driver.Source: §5.6 · p. 18
• The inter-LLM mean PSS is 0.733 versus 0.71 for the best LLM against humans, while the wisdom-of-LLMs ensemble scores only 0.305 and collapses toward the no-profile baseline.Correlated errors mean averaging strengthens shared distortions and erodes model-specific persona signals; vendor clustering and outlier models are consistent with this failure.Source: §5.6 · pp. 18–19
• Test–retest ICC(1) ranges from 0.47 to 0.92, with median 0.85; 21 of 30 complete-repeat models meet or exceed the human reference ICC of 0.78.Reliability tracks model scale rather than vendor and confirms that the R = 1 headline grid is not dominated by sampling noise, but consistency alone does not ensure human-like answers.Source: §5.6 · p. 19

5.7 Downstream consequences: response style, predictive validity, and fabricated mediation

• A mean acquiescence shift of +0.84 SD, alongside extreme-rate −0.23 and midpoint use +0.12, shows LLMs compress responses toward an agreeable centre.The shift reaches +1.46 for deepseek-3-2, while mistral-large-3 is the clear exception at −0.14.Source: §5.7 · pp. 19–20
• R2_human = 0.28 versus R2_synth = −0.18, a validity gap of ΔR2 = −0.46, showing synthetic-trained regressors generally fail to predict held-out human outcomes.Only gemini-3.1-pro nearly preserves human-level utility (ΔR2 = −0.03), while the worst gap is −1.57.Source: §5.7 · p. 20
• On 3 of 10 placebo mediation paths with null human indirect effects, pooled LLMs produce significant indirect effects, indicating fabricated rather than purely recovered mediation structure.The result cautions against using LLM respondents for exploratory structural-equation or causal-pathway discovery.Source: §5.7 · p. 20

6.1 What works

• Persona conditioning recovers positive mediation, role contrasts, item-correlation block structure, and improved group means, with PSS gains up to ∼0.58 units.The mediation pattern is recovered on 36 of 37 models at C3, while role differences are correctly signed for every model.Source: §6.1 · p. 20

6.2 What does not

• The median model restricts response variability to σLLM/σH = 0.53 for IWPQ and 0.44 for UWES, inflating reliability and inter-item correlations.Range restriction contributes to over-coherence, where the best LLM’s correlation PSS component is 0.52 versus 0.95 for the Gaussian-copula baseline.Source: §6.2 · pp. 20–21

6.3 What is no longer plausible

• Recall rate and PSS correlate at 0.00, with 22/37 models showing zero recall, ruling out memorization of the specific Lithuanian items as the leaderboard driver.The worst-case high-recall rate is 4.7%, although paraphrased exposure to English instrument versions cannot be ruled out.Source: §6.3 · p. 21

6.4 Theoretical implications for psychometrics

• The Gaussian-copula baseline reproduces sample-driven distribution, correlation, and reliability components without construct understanding, while LLM advantages appear mainly in mediation and demographic effects.This implies Cronbach’s α and Tucker’s φ primarily measure item dependencies in a sample rather than whether responses come from real respondents.Source: §6.4 · p. 21

6.5 Methodological implications for LLM evaluation

• Evaluation should anchor LLMs to statistical baselines and human-vs-human ceilings, using permutation nulls to correct upward-biased similarity metrics such as Tucker’s φ.The recommended permutation procedure is cheap, with typically K = 500 permutations sufficient for a proper significance test.Source: §6.5 · pp. 21–22

6.6 Implications for AI safety and alignment evaluation

• Education-swap amplification is far larger than gender-swap amplification, while inter-LLM PSS of 0.733 shows that ensemble safety evaluations provide limited behavioral diversity.The off-diagonal inter-LLM PSS range is 0.6–0.8, supporting population-level evaluation against statistical baselines and human references.Source: §6.2 · p. 21, §6.6 · p. 22

6.7 Practitioner recommendations

• PSS 0.71 for the best LLM remains below the 0.83 human-vs-human ceiling, so the benchmark does not support replacing human samples.Failure modes compound rather than cancel, while the best LLM is only approximately 0.03 PSS units above the Gaussian-copula baseline.Source: §6.7 · p. 22
• Use C3-conditioned LLMs for pilot-study effect directions, statistical baselines for power calculations, LLM-human gaps for bias audits, and neither LLMs nor their leaderboard for human-sample replacement.PSS varies by ∼0.35 across models, but the best-LLM versus Gaussian-copula gap is only ∼0.03 PSS units.Source: §6.7 · p. 22

Limitations and future work

• External validity is unestablished beyond the Lithuanian organisational sample (n = 263), although the LT↔EN ablation showed bounded drift (|∆| = 0.26 Likert points; per-item Pearson 0.889).Follow-up studies should test non-Indo-European, clinical-psychology, and educational-psychology samples.Source: §7 · pp. 22–23
• Self-report ground truth prevents separating LLM over-reporting of work performance from human over-reporting, including inherited social-desirability bias on IWPQ task performance.The resulting LLM–human gap on that subscale cannot be decomposed into model versus human bias.Source: §7 · p. 23
• Instrument memorisation was not the leaderboard driver—high-recall rate was 4.7% at worst, 22/37 models had zero recall, and recall–PSS rank correlation was 0.00—but paraphrased exposure remains possible.A bespoke novel-instrument benchmark would test performance without prior exposure to the instrument’s wording or construct literature.Source: §7 · p. 23
• Headline LLM cells used n = 100 and R = 1; increasing to n = 263 and R = 3 would tighten bands by approximately 2.8×, but at roughly 8× cost.The leaderboard bootstrap CI mean width was 0.046 PSS units, comparable to differences within the top-five cluster.Source: §7 · p. 23
• Minimal reasoning effort was generally a small, model-specific factor, while provider-version drift remains uncontrolled despite archived response logs enabling future detection.Only gpt-5.4 (|d|=0.37) and claude-opus-4-7 (|d|=0.31) shifted clearly beyond the resampling-noise floor.Source: §7 · pp. 23–24
• The causal interpretation is limited to algorithmic asymmetry under controlled persona prompting, and documented failure modes may not generalise beyond organisational-psychology constructs.Future work should test other survey domains, open-text responses, longitudinal stability, demographic interaction effects, and instruction-tuning or RLHF sensitivity.Source: §7 · p. 24

Harms

• Inflated Cronbach’s α and inter-item correlations can overestimate effects, producing under-powered human follow-ups and over-confident published conclusions.These distortions arise from over-coherence and range restriction relative to the human reference.Source: §8 · p. 24
• Education effects averaged |d| = 0.56, while fabricated gender effects on human-null contrasts risk making biased systems appear fair.Most gender effects were at the resampling-noise floor, but a handful of frontier models clearly fabricated them.Source: §8 · p. 24
• Republishing synthetic respondents as human-equivalent data can bake over-coherence, range restriction, and stereotype amplification into future training corpora.Once the human reference contains LLM-generated data, these failures may no longer be recoverable from the LLM-versus-human gap.Source: §8 · p. 24

Mitigations

• Statistical baselines, counterfactual swaps, memorization probes, inter-LLM matrices, and ensemble analyses expose distinct failure modes for deployment-specific auditing.The released harness, dataset, configurations, and generation logs enable benchmarking new models against the same human ground truth without rerunning prior models.Source: §8 · pp. 24–25

9 Conclusion

• A Gaussian-copula baseline beats every LLM on sample-driven PSS components, while Tucker’s φ falls inside the permutation null for 8 of 37 models on UWES.LLMs reproduce qualitative relationship directions but generate over-coherent, low-variance, demographically stereotyped, and inter-model-homogeneous respondents.Source: §9 · p. 25
• Current frontier LLMs are not drop-in replacements for human psychometric samples but can serve as cheap pilots, sensitivity-analysis lower bounds, and bias-audit instruments.The conclusion also reports education-driven stereotype effects around |d| ≈0.6, versus gender-driven effects around |d| ≈0.1, and fairness gaps up to 0.07 across cohort-stratified PSS surfaces.Source: §9 · p. 25

A Parser and validation pipeline

• The parser strips Markdown fences, recovers the first JSON object, coerces string values to integers, clips out-of-range values, and re-issues invalid responses with stricter instructions.This retry procedure resolves >95% of parsing failures in pilot runs while storing both raw text and parsed dictionaries.Source: A Parser and validation pipeline · p. 27

B Reproducibility, resumability, and incremental extension

• The four-command pipeline supports offline mock smoke tests, YAML-captured configurations, cached resumability, incremental model addition, and analysis reruns without regenerating prior models.New models require a one-line configuration edit, and completed respondent-condition-model cells are skipped on reruns.Source: B Reproducibility, resumability, and… · p. 27

C Instrument English translations

• The bundled English translations cover all 68 items, anchors, and instructions, with LT↔EN per-item Pearson correlation of 0.889 and mean |∆|=0.263 Likert points.The translations were drafted by the authors and checked against published English instrument versions where available.Source: C Instrument English translations · p. 27

D Tucker’s φ permutation null distribution

• The Tucker’s φ correction permutes item labels in the LLM loading matrix and computes p=(k+1)/(K+1), converting greedy alignment into a significance test against chance.Table 3 summarizes mean null φ from 500 random permutations and counts models exceeding the null’s 95th percentile by instrument.Source: D Tucker’s φ permutation null… · pp. 27–28, Table 3 · p. 28

PSS leaderboard

• d≥0.48: the 37-model PSS leaderboard is well-powered at n=100 under Holm–Bonferroni, below the observed PSS gaps.Power was assessed at α=0.05 and 0.80 power using post-hoc minimum-detectable Cohen’s d.Source: E Power analysis · p. 28

Counterfactual contrasts

• |d|=0.56: education-swap contrasts are well-powered, whereas |d|=0.12 gender-swap contrasts are underpowered and should be interpreted directionally rather than as per-cell-significant.The counterfactual family comprised 3 axes × 12 subscales and used Holm–Bonferroni correction.Source: E Power analysis · p. 28

• Education swaps produce the largest counterfactual shifts, role swaps are intermediate, and gender swaps are generally near the paired-d resampling-noise floor.The amplification is measured as mean |paired Cohen’s d| across 12 composite subscales.Source: Table 5 · p. 29, Figure 6 · p. 31
• Synthetic data performs worse than the mean (synthetic R2 < 0), while human performance is R2 = 0.277; almost every model loses substantial predictive validity in downstream modelling.LLM respondents are not a drop-in replacement for human survey data in downstream modelling.Source: F Counterfactual, ablation, cohort,… · p. 37
• LLMs fabricate 3 of 10 placebo mediation paths, while also systematically over-agreeing and overusing the scale midpoint relative to humans.The mediation result indicates that some reproduced structural relationships are confabulated rather than recovered.Source: Table 12 · p. 36, Table 14 · p. 38
• Median discriminator AUC reaches 0.999 for LLMs, whereas Gaussian-copula and MVN baselines are statistically indistinguishable from humans.This provides the sharpest separation between statistical baselines and LLM synthetic respondents.Source: Table 16 · p. 39
• The copula rises from 0.69 at n=10 to 0.82 at n=200, whereas the best LLM plateaus near 0.62 by n≈50; debias prompting improves style but leaves overall PSS flat.The widening fidelity gap shows that more synthetic respondents do not close the gap, while correlation remains the binding constraint for prompting.Source: Table 17 · p. 39, Table 20 · p. 40
• LLMs inflate general-factor ECV across instruments and collapse specific-factor reliability, including Change-engagement ∆ωh = −0.94, while test–retest reliability is benchmarked against human ICC = 0.78.These findings indicate over-coherence and reduced multidimensional fidelity; gpt-oss-120b is omitted from the repeatability table because of rate limits.Source: F Counterfactual, ablation, cohort,… · p. 38, Table 19 · p. 39

G Robustness ablations: numerical detail

• 0.67 mean recall accuracy at C3 across 37 models, while claude-haiku-4-5 rises from 0.32 at C0 to 0.85 at C2.Recall peaks when work context is most salient.Source: G Robustness ablations: numerical detail · p. 41
• Anthropic, Google, and OpenAI form within-vendor inter-model PSS clusters, while llama-4-maverick and grok-4-20 are outliers.Figure 8 additionally presents cohort-stratified PSS and worst-axis disparity by model.Source: Figure 7 · p. 42, Figure 8 · p. 43
• r<0 for forward-versus-reverse responses across all 37 models, and item-order shuffling changes PSS by <0.02.These checks support reverse-key consistency and rule out positional cue exploitation as major confounds.Source: G Robustness ablations: numerical detail · p. 42
• API availability, not parsing or refusal, drives the main data loss, with qwen3-7-plus retaining n=72–94 and four models lacking C9.grok-4-20 and llama-4-maverick rank low because of genuine response degeneracy; grok-4-20 has r≈0 on reliability.Source: G Robustness ablations: numerical detail · pp. 42–43

H Psychometrics primer for machine-learning readers

• Psychometric instruments aggregate fixed-scale items into subscales measuring latent constructs, with composites usually retained as per-subscale vectors.UWES-17 illustrates this structure through vigour, dedication, and absorption subscales.Source: H Psychometrics primer for… · p. 43
• Correct item means are insufficient because composites, α, ω, mediation, and Cohen’s d depend on item correlations, covariances, path coefficients, and within-stratum variability.This motivates decomposing PSS into six dimensions rather than reporting only an aggregate.Source: H Psychometrics primer for… · pp. 43–44
• α and ω assess internal consistency, while Tucker’s φ compares aligned factor loadings; conventional φ thresholds can be inflated by chance, motivating permutation nulls.The paper reports both α and ω, with the LLM–human gap essentially identical across them; α, ω, and φ retain their printed markers.Source: H Psychometrics primer for… · pp. 44–45
• Mediation separates direct c′ and indirect ab paths, HTMT >0.85 indicates poor discriminant validity, and invariance tests configural, metric, and scalar comparability.LLM samples average 7.7/12 subscale pairs above HTMT 0.85 versus 4/12 in humans; bifactor ECV and ωh summarize general-factor dominance and reliability.Source: H Psychometrics primer for… · p. 45

Headline grid

• 18,500 headline-grid calls generated ∼105 million tokens across 37 models, five conditions, 100 respondents, and one repeat, costing approximately $136 and ∼6 hours.The grid used an average of 4,500 prompt tokens and 1,200 completion tokens per call, with provider-specific concurrency limits.Source: I Compute and cost breakdown · p. 45

Ablation and probe costs

• The five extensions added approximately $121 and 12 wall-clock hours across 21,730 calls, comprising counterfactual swaps, reasoning-effort, cross-language, presentation-mode, and memorisation runs.The components contributed approximately $65 and 4 hours, $18 and 2 hours, $15 and 1 hour, $15 and 1 hour, and $8 and 4 hours, respectively.Source: I Compute and cost breakdown · pp. 45–46

Test–retest stability

• ∼20,400 calls in the test–retest stability run cost ∼$150 and required ∼6–8 wall-clock hours, licensing the R=1 headline design.The run covered 30 respondents and 20 repeats per model on C3, yielding ∼17,500 completed cells across the lineup.Source: I Compute and cost breakdown · p. 46

Total cost

• Approximately $390 of API spend and ∼25 wall-clock hours covered the full experiments, dominated by the headline grid and test–retest stability run.The headline grid cost approximately $136 and took ∼6 hours; the stability run cost ∼$150 and took ∼6–8 hours.Source: I Compute and cost breakdown · pp. 45–46

Analysis-side compute

• 74,000 Tucker factor analyses across 500 permutations, 37 models, and four instruments took ∼40 minutes, while PSS bootstrap confidence intervals took ∼15 minutes on one CPU node.Figure generation took < 5 minutes, and the per-call disk cache made reruns of completed cells cost nothing.Source: I Compute and cost breakdown · p. 46

gpt-5.4-mini

• gpt-5.4-mini captures Persona A’s higher mean and Persona B’s wider response distribution, although Persona A responses cluster near the human distribution’s high end.Persona A yields eight 4s and two 5s; Persona B yields a wider mix of 2–5 responses.Source: J Worked qualitative example: a single… · p. 46

claude-sonnet-4-6

• claude-sonnet-4-6 produces a roughly 2-point persona mean gap versus the human gap of roughly 0.5, exemplifying stereotype amplification from education, role, and sector.Its Persona B distribution is also narrower and shifted downward than the human Persona B distribution.Source: J Worked qualitative example: a single… · pp. 46–47

grok-4-1-fast-non-reasoning

• grok-4-1-fast-non-reasoning gives nearly identical distributions to substantially different personas, failing to condition meaningfully on the persona block.This range-restricted behavior is identified as a failure mode driving low PSS, possibly because the non-reasoning variant integrates persona information less effectively.Source: J Worked qualitative example: a single… · p. 47

Cross-model qualitative pattern

• Models at the top of the PSS leaderboard track both human mean and within-stratum variance shifts, whereas amplifying models exaggerate means and low-PSS models track neither.The qualitative example reproduces the quantitative ranking pattern across model families.Source: J Worked qualitative example: a single… · p. 47

Negative result 1: temperature

• Temperature effects remain within sampling noise, making temperature negligible relative to model identity, persona disclosure, and presentation mode.Run-to-run noise was 0.14 and 0.12 for fixed-temperature controls, versus 0.16, 0.09, and 0.07 for models varying temperature.Source: K Negative results · p. 47

Negative result 2: free-text persona

• Free-text personas reduce PSS by 0.04–0.07 units because over-narrated biographies pre-commit responses to a behavioural style.The structured C3 profile was retained instead of the richer narrative persona.Source: K Negative results · p. 47

Negative result 3: chain-of-thought reasoning trace as data

• Available chain-of-thought traces provide no measurable psychometric signal, consisting mainly of surface-level rationalisations of the persona.The traces did not yield a clean signal at the level of the headline grid.Source: K Negative results · p. 47

Negative result 4: prompt-based debiasing and few-shot conditioning

• Prompt debiasing and few-shot human exemplars leave overall PSS essentially unchanged, indicating that the gap is structural rather than a conditioning artifact.Debiasing changes response style but not PSS (0.570→0.574) or correlation (PSSρ 0.371→0.358), while few-shot conditioning yields PSS 0.570→0.581 with unchanged SD.Source: K Negative results · p. 48

L PSS weight sensitivity analysis

• 0.970 versus 0.731: restricting PSS to correlation and reliability leaves the Gaussian-copula baseline ahead of every LLM by at least 0.24.Under equal weights, the ranking correlates 0.99 with the default ranking; under LLM-favoured weights, the copula ties deepseek-v4-pro at 0.749.Source: L PSS weight sensitivity analysis · p. 48
• The Gaussian copula beats every LLM on sample-driven components under all tested weightings, while the best LLM remains essentially tied with it on the five-component PSS.The default weights give roughly equal voice to sample-driven and downstream components, and practitioners can override them via configs/pss_weights.yaml.Source: L PSS weight sensitivity analysis · p. 48

M Extended ethics statement; Collection process

• The human dataset was collected in March–April 2020 with informed consent covering academic re-use of de-identified aggregated data, and no new human data were collected for this benchmark.The original data collector oversees the de-identified record-level release.Source: M Extended ethics statement · p. 48

M Extended ethics statement; Composition

• De-identification removed free-text industry fields, rounded timestamps to year-month resolution, used coarse demographic variables, and ensured every non-empty joint demographic cell contained at least three respondents.The dataset contains no direct identifiers, and the full rationale is recorded in the Croissant metadata file.Source: M Extended ethics statement · pp. 48–49
• The benchmark uses synthetic LLM responses conditioned only on eight de-identified demographic fields and two ordinal change-context ratings; no human free text was shown.The dataset comprises anonymous adult Lithuanian employees completing a 68-item questionnaire, with n = 263 fully completed responses and 1.4% item-level missingness.Source: M Extended ethics statement · p. 49, Composition · pp. 49–50

Motivation; Uses; Collection process

• The dataset was created to study links among organisational-change attitudes, work engagement, and self-rated performance among Lithuanian-speaking employees, and supports Lithuanian-language validation and related survey research.It originated in a master’s thesis and was acquired through an online questionnaire with no compensation.Source: Motivation · p. 49, Collection process · p. 50, Uses · p. 50

Pre-processing / cleaning / labeling; Distribution; N Datasheet for the Lithuanian Organisational Psychology Survey (2020)

• Processing re-derived Likert integers, harmonised binary fields, and computed documented composite scores, with raw and processed files released for reproducibility under CC BY-NC 4.0.The release includes item-level, demographic, and composite frames alongside the raw CSV.Source: Pre-processing / cleaning / labeling · p. 50, Distribution · p. 50

Maintenance; M Extended ethics statement; Distribution

• The repository provides version-tagged releases, issue-based errata and alternative cleanings, and explicit safeguards against using the benchmark to legitimise human-sample replacement.The framework’s stated positive scope includes pilot studies, sensitivity-analysis lower bounds, and bias-audit instruments, while re-identification attempts are prohibited.Source: M Extended ethics statement · p. 49, Maintenance · p. 50