PaperlayerView exact revision on arXiv

Source-linked AI summary

Item-Mean Surrogates: Why Richer Persona Data Fail to Improve LLMs as Human Surrogates

Daehwan Ahn, Chengfeng Mao, Dokyun Lee

arXiv:2608.29455v1cs.CLcs.CYcs.LG

TL;DR

LLMs can match item averages, but whether that supports their use as surrogates for specific people remains an open question. The paper tests respondent-specific prediction after removing item means and finds that aggregate fit does not translate into individual-level fidelity.

  • Problem

    Whether item-mean tracking is sufficient for predicting respondents’ individual responses after removing average item responses remains an open question.

    Source: Results · p. 2

  • Method

    The paper tests respondent-specific prediction by subtracting each item’s human mean from human and LLM responses and comparing the remaining deviations.

    Source: Result 1: LLMs Track Item Means, Not… · p. 3

  • Results

    Across four datasets, apparent aggregate surrogate success reflects item-mean tracking rather than individual fidelity; de-meaned predictions explain 3.05% of remaining variation versus the 53.6% human test–retest ceiling.

    Source: Result 1: LLMs Track Item Means, Not… · p. 3, Conclusion · p. 6

  • Takeaways & Limitations

    Item-mean tracking may support rough aggregate expectations, but the evidence does not support treating LLM outputs as substitutes for individual respondents.

    Source: Scope and Limitations · p. 6

    The findings are an empirical pattern in the tested settings, and their robustness across broader tasks, populations, and model designs remains open.

    Source: Scope and Limitations · p. 6

Abstract

from arXiv · show

LLMs are increasingly used as human surrogates, often on the premise that richer persona data could make them substitutes or exploratory tools for specific individuals. We test this premise across four datasets covering more than 400,000 participants and more than 6,000 survey items and experimental outcomes. LLMs perform well at the aggregate level: their average responses closely align with average human responses to the same items. But this success largely reflects predicting each item's average human response. Once each item's human mean is removed, LLM predictions explain only 3.05% of the remaining respondent-specific variation, far below the 53.6% human test-retest benchmark. Richer personas, model variants, and fine-tuning do not close this gap. In variance analyses, once item means are removed, the reliable remaining signal is person-by-item. It captures how a respondent departs from the mean on a particular item and is about 8.9x larger than the stable person effect. Persona data encode the respondent, but not this item-specific deviation. LLM responses also compress human response distributions, using less spread, fewer response categories, and distorted distributional shapes. We call this pattern item-mean surrogacy. Current LLM surrogates can approximate item averages, but not the distributions or respondent-specific deviations needed to replace individual humans. We propose four empirical tests for LLM-based human-surrogate claims.

Results

LLMs track average item responses, but their response distributions are compressed and reshaped, and they recover little respondent-specific variation after item means are removed. The missing reliable signal is mainly person-by-item interaction, and richer personas, fine-tuning, and model choice do not close the individual-prediction gap.

Results

  • Median LLM-to-human item-level SD ratios were 65% in Megastudy, 50% in SocSci210, and 57% in the Survey.LLMs also used fewer effective response categories, with median ratios of 43%–73% across these datasets; shape accounted for 31%–40% of squared Wasserstein-2 distance in the same three datasets. ANES spread was closer to human spread, but category use remained lower.Source: Results · p. 2

Result 1: LLMs Track Item Means, Not Individual Deviations

  • POMP-scaled LLM item means correlated with human item means at Pearson r = 0.78 and Spearman ρ = 0.72.This item-level alignment does not establish respondent-specific accuracy after item means are removed.Source: Result 1: LLMs Track Item Means, Not… · p. 2
  • The LLM underperformed the leave-one-out human item-mean baseline: mean rLLM = 0.34 versus mean rhuman-LOO = 0.45 across 1,631 respondents.These per-respondent correlations reward item-mean tracking and should not be interpreted as respondent-specific accuracy; LLM itemwise RMSE also followed a human-item-mean-only predictor’s error profile (r = 0.73 across 133 items).Source: Result 1: LLMs Track Item Means, Not… · p. 3
  • Correct persona matching yielded pooled de-meaned R2 = 3.05%, versus a maximum of 0.028% across 10,000 shuffled-persona swaps.Fine-tuning did not close the gap: a fine-tuned GPT-4.1 Megastudy model reached 2.31%, below the strongest non-fine-tuned Megastudy prompt at 4.44%; Socrates-Qwen2.5-14B reached 7.57% on seen SocSci210 studies and 0.73% on held-out studies.Source: Result 1: LLMs Track Item Means, Not… · pp. 3–4
  • Person-by-item interaction accounts for 44.0% of total variance, about 8.9× the stable person main effect, while transient error accounts for 42.4%.Stable person main effects explain 4.9% of variance and item main effects 8.7%; the interaction is the item-specific respondent signal that a perfect surrogate could recover.Source: Result 1: LLMs Track Item Means, Not… · p. 3, Figure 2 · p. 4
  • Pooled de-meaned R2 was 3.05% in Megastudy, 5.78% in SocSci210, and 10.77% in ANES’s four-item primary row, all below the 53.6% reference.The 53.6% Twin-2K-500 test–retest benchmark is a common descriptive reference for SocSci210 and ANES, not a dataset-specific reliability ceiling.Source: Result 1: LLMs Track Item Means, Not… · pp. 4–5, Table 2 · p. 5

Why Personas Have Limited Leverage

The section explains why richer persona data do not recover respondent-specific deviations: after item means are removed, the reliable signal is item-specific and generalizes weakly to new items. It argues that aggregate fit alone is insufficient evidence of surrogate fidelity.

Why Personas Have Limited Leverage

  • Personas encode stable attributes, not respondent–item deviations; richer personas may improve aggregate fit, but deviations observed on past items generalize weakly to new items.The remaining reliable signal lies in respondent–item pairings, a larger component than stable person differences, within the standard psychometric structure (8, 11).Source: Why Personas Have Limited Leverage · p. 5

Item-Mean Tracking Connects Prior Findings

  • Item-mean surrogacy unifies weak persona effects (12), distributional compression (13–15), low simulation scores (16), and aggregate-but-not-individual fidelity (18, 19), including insufficient individuation (7).Fine-tuning can improve item-level calibration (10, 24), but does not recover respondent-specific deviations in these tests.Source: Item-Mean Tracking Connects Prior… · p. 5

Aggregate Fit Does Not Establish Surrogate Fidelity

  • Individual substitution requires outperforming an item-mean baseline, recovering respondent-specific deviations, and preserving response distributions; high aggregate fit alone shows item-level calibration.Validation should match the claim: average prediction requires item means, while distributional fidelity requires response spread and shape.Source: Aggregate Fit Does Not Establish… · p. 5

Conditions for Falsification

The paper proposes four tests for whether LLM surrogates support individual substitution and clarifies that the claim concerns out-of-sample prediction from persona information, not replaying observed responses.

Conditions for Falsification

  • No tested configuration meets the four proposed tests for overcoming item-mean surrogacy and supporting individual substitution.Future human-surrogate claims should be evaluated against these tests.Source: Conditions for Falsification · p. 6
  • A surrogate should recover respondent-specific deviations from item means and generalize that recovery to new items and tasks.Reproducing item means alone does not satisfy these criteria.Source: Conditions for Falsification · p. 6
  • Prediction of individual deviations should improve when personas are correctly matched to respondents rather than shuffled across them.If shuffling does not reduce prediction, the model may be recovering a generic conditional response pattern rather than simulating a specific individual.Source: Conditions for Falsification · p. 6
  • A surrogate should recover response distributions, not just item means.Source: Conditions for Falsification · p. 6
  • Memorizing or retrieving previously observed respondent–item responses would not falsify the claim, which concerns predicting responses for new items, tasks, or individuals from persona information.The distinction is whether predictions are out of sample rather than based on direct access to the target response.Source: Conditions for Falsification · p. 6

Scope and Limitations

Item-mean surrogacy is an empirical pattern in the tested settings, and its broader generality remains uncertain. The evidence supports limited aggregate uses, not substituting LLM outputs for individual respondents.

  • The pattern’s robustness beyond tested settings remains unknown, and ANES recovery varies by item, strongest for party identification and ideology and near zero for patriotism.Source: Scope and Limitations · p. 6
  • Analyses cover ordered, bounded responses on a 0–100 POMP scale and omit full-history retrieval and explicitly individual-trained systems; in-sample respondent–item reproduction is retrieval, not out-of-sample fidelity.Source: Scope and Limitations · p. 6
  • Megastudy items are likely novel, but exact training-data overlap is unknown; the repeated pattern spans novel Megastudy items and established Survey, SocSci210, and ANES instruments.The pattern comprises item-mean tracking, distributional collapse, and weak respondent-specific recovery.Source: Scope and Limitations · p. 6
  • Linear de-meaning and variance-decomposition analyses may miss nonlinear respondent-specific structure.Source: Scope and Limitations · p. 6
  • LLMs may help identify endpoint-saturating items, pilot gross design problems, and form rough aggregate expectations, but evidence does not support substituting them for individual respondents.Source: Scope and Limitations · p. 6

Conclusion

Across four datasets, LLMs’ aggregate success can reflect tracking item means rather than recovering response distributions or respondent-specific variation. Richer personas, model variants, and fine-tuning do not bridge the gap between mean-level accuracy and individual-level fidelity.

Conclusion

  • Across four datasets, richer personas, model variants, and fine-tuning did not close the gap between aggregate accuracy and individual-level fidelity.Apparent surrogate success can arise from tracking item means rather than recovering response distributions or respondent-specific variation.Source: Conclusion · p. 6
  • Aggregate fit is a starting point for surrogate validation, not evidence that an LLM can stand in for individual respondents.No tested approach bridged mean-level accuracy and individual-level fidelity.Source: Conclusion · p. 6

Materials and Methods

  • The methods define POMP-scaled human and model responses and raw and de-meaned prediction metrics, alongside REML variance decomposition and four-dataset analyses.The datasets include the Survey test–retest benchmark and ANES data underlying prior “silicon samples” claims; primary POMP analyses use eligible ordered, bounded outcomes, with broader comparisons reported in the supplement.Source: Materials and Methods · pp. 6–7

Research Ethics

The study reanalyzed four de-identified public datasets without collecting new human-subjects data, and the University of Georgia determined it was not human-subjects research. In the primary Megastudy analysis, person and item effects explained relatively small shares of absolute prediction-error variance, while the residual was partitioned using test–retest reliability.

Research Ethics

  • The analysis used four previously released, de-identified public datasets; no new human-subjects data were collected, and no participant was identifiable.The University of Georgia Human Research Protection Program determined that the project was not research involving human subjects under DHHS and FDA regulations (IRB ID PROJECT00014809, 18 June 2026).Source: Research Ethics · p. 7
  • 86.4% of absolute prediction-error variance was residual in the primary POMP-scaled Megastudy analysis, compared with 4.9% for person main effects and 8.7% for item main effects.The study estimated two-facet Person × Item Generalizability theory models by REML using lme4.Source: Research Ethics · p. 7
  • 44.0% of total variance was attributed to person-by-item interaction and 42.4% to transient error after partitioning the residual using within-item test–retest reliability, rtt = 0.536.The authors describe the interaction as an item-conditioned person-by-item component; Table S4 and the SI Appendix show how the split changes when reliability varies.Source: Research Ethics · p. 7

Item-Mean Benchmark

The benchmark separates agreement on item means from prediction of respondent-specific deviations after item-level signal is removed. It defines the primary de-meaned metric for comparison with human test–retest reliability.

Item-Mean Benchmark

  • Item-level agreement was assessed on raw and POMP-scaled means, with Spearman ρ measuring scale-specific rank agreement and Pearson r on POMP as the main linear association.Because POMP rescales each item separately, raw-scale and POMP-scale correlations address different questions.Source: Item-Mean Benchmark · p. 7
  • Primary de-meaning centered both human responses and model predictions by each item’s human POMP-scale mean, leaving deviations above or below that mean.This follows profile-similarity methodology (30–32), equivalent to group-mean centering in multilevel analysis (33).Source: Item-Mean Benchmark · p. 7
  • Pooled de-meaned R2 is the squared Pearson correlation across respondent–item pairs and the primary individual-prediction metric, comparable to the test–retest reliability ceiling.Own-mean centering instead centers each series by its own item mean and yields higher pooled de-meaned R2 by removing calibration differences; the two schemes can differ in per-person correlations.Source: Item-Mean Benchmark · p. 7

Persona Swap

The persona-swap test compared pooled de-meaned R2 under correct persona–respondent matching with 10,000 random reassignment permutations in the Megastudy only.

Persona Swap

  • The test used 10,000 within-study persona–respondent reassignments to compare pooled de-meaned R2 with correct matching, and was conducted only on the Megastudy.SocSci210 and ANES lack individual-level persona descriptions; the Survey is a single fully crossed study, not separate studies within which personas could be permuted.Source: Persona Swap · p. 7

Enrichment and Cross-Family Analysis

The analysis tested how persona enrichment, model family, prompting, and fine-tuning affected respondent-level prediction, evaluated primarily with pooled de-meaned R2. It used model specifications and outputs from the original Megastudy digital-twin work.

Enrichment and Cross-Family Analysis

  • The Megastudy evaluated 23 prompt and model specifications, with respondent-specific prediction assessed primarily by pooled de-meaned R2.The original dataset authors selected frontier models to optimize their digital-twin framework and supplied the outputs (7).Source: Enrichment and Cross-Family Analysis · p. 7
  • Persona enrichment varied from ∼50-token empty personas to ∼2,100-token full psychographic profiles, while GPT-4.1 variants also tested a chain-of-thought instruction.This was a roughly 42-fold increase in persona information; raw per-item R2 was tracked across the enrichment gradient.Source: Enrichment and Cross-Family Analysis · p. 7
  • Cross-family comparisons covered GPT-4.1, GPT-5, DeepSeek, Gemini 2.5 Flash, Gemini 3 Pro, Llama 3.1 70B, and Centaur (34).Pooled de-meaned R2 and de-meaned per-person r were reported in SI Appendix Tables S5 and S7; five models used identical prompts, while GPT-4.1 and Centaur used distinct formats.Source: Enrichment and Cross-Family Analysis · p. 7
  • Fine-tuning robustness analyses included respondent-split GPT-4.1 tuning and Socrates-Qwen2.5-14B variants evaluated on held-out studies under the original protocols (10).Survey analyses also used GPT-4.1-mini fine-tuned on the same participant pool and Gemini 2.5 Flash as base models.Source: Enrichment and Cross-Family Analysis · pp. 7–8

Ceiling Analysis

The 53.6% performance ceiling is an upper-bound benchmark for predicting de-meaned respondent deviations, and performance remains far below it even at the reliability estimate’s lower confidence bound.

Ceiling Analysis

  • The ceiling was derived from within-item test–retest reliability of 0.536 and bounds R2 for deterministic prediction of respondent deviations from item means.The estimate used 2,057 paired respondents and 126 Survey items, with a median retest interval of four weeks; temperature-zero decoding meets the deterministic-predictor condition.Source: Ceiling Analysis · p. 8
  • The 53.6% benchmark applies only to de-meaned targets; if Megastudy behavioral items are less reliable than Survey items, its true ceiling could be lower.Raw per-item R2 is descriptive, not benchmarked to this ceiling, because it retains item-level structure.Source: Ceiling Analysis · p. 8
  • At the lower CI-bound reliability, the ceiling was 50.6%, while de-meaned R2 remained 3.05% under human-mean centering and 3.87% under own-mean centering.The lower-bound analysis leaves the gap intact.Source: Ceiling Analysis · p. 8

Statistical Reporting

The section reports pooled de-meaned estimates with bootstrap confidence intervals where indicated and explains that no formal multiple-comparison corrections were applied. Analysis code and derived data tables are slated for public release upon publication.

Statistical Reporting

  • rtt = 0.536 (95% CI [0.506, 0.565]) across 126 items and 2,057 paired respondents.Bootstrap 95% confidence interval.Source: Statistical Reporting · p. 8
  • No formal multiple-comparison corrections were applied because conclusions relied on large contrasts, including 44.0% person-by-item interaction versus 4.9% person main effects.The passage also contrasts 3.05% pooled de-meaned R2 with the 53.6% empirical ceiling on the de-meaned target.Source: Statistical Reporting · p. 8

Data, Materials, and Software Availability

Supporting Information for: Item-Mean Surrogates: Why Richer Persona Data Fail to Improve LLMs as Human

The supporting information examines the limits of LLMs as individual-level human surrogates through supplementary prediction, response-distribution, and diagnostic analyses. It also considers fine-tuning and literature coding in relation to respondent-specific fidelity.

POMP Scaling

  • Primary POMP analyses scaled eligible ordered, bounded responses to 0–100 and centered human and LLM responses on each human item mean.POMP = 100 × (x − a)/(b − a); nominal or background variables, computed scores, objective-correctness measures, open numeric estimates without defined endpoints, and unresolved invalid codes were excluded from primary POMP rows. The supplementary comparison protocols included cross-family model rows and adaptations; the Socrates comparison used a study-level seen/held-out split of 106 seen and 22 held-out studies, but this part supplies no outcome for that comparison.Source: POMP Scaling · p. 11, Cross-Family and Adaptation Protocols · p. 13, Socrates Seen/Held-Out Study Comparison · p. 15

De-Meaning Procedure for Residual Analysis

Variance Decomposition Procedure

  • 42.4% of prediction-error variance was assigned to reliable person-by-item variation at rtt = 0.536, versus a 4.93% person main effect.Across the rtt 95% CI, the estimated person-by-item share ranged from 41.3% to 46.7% and exceeded the person main effect by at least 8.4×; across rtt ∈ [0.30, 0.70], the ratio remained at least 4.6×. The partition applies an external Survey reliability estimate to single-occasion Megastudy prediction errors, so its value depends on cross-domain comparability.Source: Variance Decomposition Procedure · p. 13

Item-Mean Tracking Metrics

  • Pearson r = 0.777 measured item-mean agreement for the primary full-persona prompt on the 133-item POMP-scaled Megastudy row.Spearman ρ was 0.843 on raw-scale Megastudy items with analyzable means and 0.721 on the POMP-scaled primary row; the retained item sets and item-specific rescaling differed.Source: Item-Mean Tracking Metrics · p. 14

Distributional Collapse Detail

  • Across all four datasets, LLM outputs showed variance compression, repertoire collapse, and elevated Wasserstein-1 distances relative to human response distributions.Median SD ratios were 0.50 in SocSci210, 0.65 in the Megastudy, 0.57 in the Survey, and 0.99 in ANES; median effective-category ratios were 0.43, 0.73, 0.43, and 0.85, respectively. ANES diagnostics use four ordered items; 368 SocSci210 outcomes elicited a constant LLM response.Source: Distributional Collapse Detail · p. 14

Shape Distortion Beyond Variance Compression

  • Compression-only nulls preserved item-level skewness more strongly than LLM outputs, with correlations 0.88 vs. 0.49 (Megastudy, N = 130), 0.58 vs. 0.31 (SocSci210, N = 3,554), and 0.69 vs. 0.48 (Survey, N = 35).Source: Shape Distortion Beyond Variance… · p. 14

Leave-One-Out Item-Mean Baseline

Persona Swap Control

  • Correct persona matching yielded de-meaned R2 = 3.05% under human-mean centering, versus a maximum within-study permutation value of 0.028%.The 10,000 within-study persona reassignments produced an empirical p = 0.0001; the control was conducted only on the Megastudy because the other datasets lacked suitable within-study persona descriptions or permutation structure.Source: Persona Swap Procedure · p. 12, Persona Swap Control · p. 15

Socrates Seen/Held-Out Study Comparison

  • Socrates supervised fine-tuning achieved R2 = 7.57% on seen studies but 0.73% on held-out studies, with 1,389,137 and 307,393 pairs, respectively.The 7.57% versus 0.73% descriptive gap indicates that gains were concentrated in studies marked as seen in study-level metadata, rather than closing the held-out individual-prediction gap.Source: Socrates Seen/Held-Out Study Comparison · p. 15

Model-Family and Adaptation Coverage

  • Fine-tuned GPT-4.1-mini raised Survey item-level r from 0.859 to 0.992, while centered r fell from 0.038 to −0.042 and pooled de-meaned R2 from 6.16% to 0.31%.Fine-tuned GPT-4.1 had centered r = 0.100; the mini-model comparison used the same 126-item set, and its fine-tuned row included 1,557 persons and 149,498 pairs.Source: Model-Family and Adaptation Coverage · p. 16

SI Figures

  • Median LLM/human SD ratios were 0.50 (SocSci210), 0.65 (Megastudy), 0.57 (Survey), and 0.99 (ANES), with compression in 95.5%, 92.5%, 83.7%, and 50.0%, respectively.SocSci210 also had 368 zero-variance outcomes; these are reported separately from the nonzero-LLM-SD denominators used for median ratios and compression shares.Source: Figure S1 · p. 18
  • LLMs used fewer effective response categories than humans on 100.0% of Megastudy items, 98.9% of SocSci210 outcomes, 93.9% of Survey items, and 75.0% of ANES items.Aggregate response frequencies also showed LLMs avoiding scale extremes in the Megastudy and SocSci210.Source: Figure S2 · p. 19
  • Across 133 POMP-scaled Megastudy items, the LLM’s itemwise individual-prediction RMSE profile correlated r = 0.726 with the error profile of an item-mean-only predictor.The item-mean-only predictor gives every respondent the human sample mean for each item, so its error is the human response dispersion around that mean.Source: Figure S3 · p. 20

Literature Classification

  • The 63-paper pool includes 44 empirical advocates (33 M, 7 D, 4 I), nine paradigm-level non-empirical advocates, and 10 critiques.Fidelity coding records advocates’ strongest validation check; individual-level coding requires per-respondent validation against that respondent’s actual responses. The pool covers papers proposing, deploying, or empirically testing LLMs as human surrogates, not the whole field’s views.Source: Literature Classification · pp. 16–17, Table S1 · p. 22

SI Tables

  • Person-by-item variance remained at least 8.4× the person main effect at rtt = 0.506 and at least 4.6× across rtt ∈[0.30, 0.70].The primary Megastudy decomposition reported Person = 4.93% and Residual = 86.41%; the sensitivity analysis varies the reliability calibration used to partition residual variance.Source: Table S4 · p. 29, SI Tables · p. 29
Loading 2608.29455v1…