Source-linked AI summary
Can Open-Weight Large Language Models (LLMs) Simulate Human Survey Populations? A Cross-Instrument Calibration Study
Grandee Lee, Wang Yue
TL;DR
Whether open-weight LLMs preserve real human statistical structure in simulated survey respondents, rather than surface plausibility, remains unresolved. The paper conditions personas on real respondents’ answers to one instrument and measures them on another across controlled construct distances. Simulated correlations track human correlations, but the capability varies across releases and is not uniformly precise.
Problem
Whether open-weight LLMs reproduce human cross-instrument statistical structure rather than plausible-looking survey responses remains unresolved, as prior evidence largely concerns proprietary models.
Source: §1 · p. 1Method
Three open-weight model families are conditioned on real respondents’ verbatim answers to one instrument and measured on another across a construct-distance-controlled grid.
Source: §1 · p. 1, §4.3 · p. 2Results
Takeaways & Limitations
Open-weight models show a replicable capability relevant to behavioral-simulation and digital-twin pipelines, but verification should be release-specific and distance-aware rather than one-time.
Source: §5 · p. 7The 139 pairs reuse respondents and instruments, so the reported statistics are descriptive; without a non-LLM baseline, the correlation cannot be judged against simpler methods.
Source: §6 · p. 7
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly used to generate synthetic survey respondents and digital twins of real people, but whether their output preserves real human statistical structure, rather than surface plausibility, remains unresolved, and most existing evidence comes from proprietary models rather than open-weight ones. We evaluate three open-weight LLM families on a cross-instrument calibration task: conditioning personas on real respondents' verbatim answers to one psychometric instrument and measuring them on a second, construct-distance-controlled instrument, checked against a 2,058-person human panel. Across a 139-pair grid, the simulated cross-instrument correlation tracks the real human correlation at r = 0.70 - 0.73 in every model, driven mainly by correct sign rather than precise magnitude and concentrated in pairs of moderate construct distance. A correlation of this magnitude, obtained from untuned open-weight models conditioned only on individual-level survey data, is a substantively encouraging result for LLM-based behavioral simulation and digital-twin applications: specific model families and releases already reproduce a meaningful share of real human cross-instrument structure without any fine-tuning. This capability does not, however, improve monotonically across model releases: on a matched panel, the newest of three tested Llama releases performs worst on two of three headline metrics, so realizing its promise in practice requires release-specific, distance-aware verification rather than a one-time benchmark.
1 Introduction
The paper asks whether open-weight LLMs reproduce human cross-instrument structure rather than merely plausible survey responses, and whether that capability persists across model releases. It tests this by conditioning personas on real respondents’ verbatim answers and comparing simulated with human correlations across instruments.
1 Introduction
- Prior work reports plausible means alongside collapsed response distributions, exaggerated group attitudes, or target scores reached by echoing prompt or scoring-key language.These findings motivate testing whether cross-instrument agreement reflects inference rather than repetition.Source: §1 · p. 1
- Personas are conditioned on one real respondent’s verbatim answers to a seed instrument, then measured on a different instrument with construct distance controlled.Agreement between simulated and human correlations is treated as evidence of inference from the seed.Source: §1 · p. 1
- Simulated cross-instrument correlations track human correlations at r≈0.70–0.73 across three independently trained open-weight model families.The study also reports that the newest of three tested Llama releases performs worst on two of three headline metrics.Source: §1 · p. 1
- The paper also reports that reverse-direction agreement is not guaranteed and that, on one model, a many-instrument seed generally out-calibrates a single instrument for a held-out trait.The supplied passage does not identify the model or quantify this comparison.Source: §1 · p. 2
2 Related Work
The study situates its contribution in prior work on simulated survey populations and identifies a design gap in evaluations of open-weight models.
2 Related Work
- Table 1 summarizes the lines of work the study builds on and how it relates to each.The supplied passage does not specify the individual works or comparisons in the table.Source: §2 · p. 2
- The study’s design combines individual-level survey conditioning, explicit construct-distance control, and multi-model and multi-release comparison, a combination the authors say prior open-weight evaluations have not addressed.The design separates whether agreement exists, whether surface repetition explains it, and whether it is stable across models and releases.Source: §2 · p. 2
- The authors contrast their individual-level persona prompting with Lutz et al., which concerns sociodemographic rather than individual-level prompting.The authors describe their broader design combination as, to their knowledge, absent from prior open-weight evaluations.Source: §2 · p. 2
3 Method
The method uses a human reference panel to condition model personas on one instrument and measure them on another. It controls instrument distance and evaluates calibration against human cross-instrument correlations.
3.1 Human reference panel
- The reference panel contains 2,058 respondents with complete psychometric-instrument responses, and fifteen instruments have independently verified item-level coverage for use as seeds or targets.A further candidate instrument was excluded because its ground-truth coverage was incomplete.Source: §3.1 · p. 2
3.2 Persona conditioning
- Each persona receives one real respondent’s verbatim item-level answers to a single instrument, then completes a different instrument in a fresh context using that instrument’s response scale.The procedure uses 40 respondents per seed–measure pair, drawn from respondents common to both instruments.Source: §3.2 · p. 2
3.3 Construct-distance control
- Instrument distance is scored using maximum item-level Jaccard token similarity and a four-tier construct-distance judgment, rather than assigned ad hoc.The authors call this a literature-grounded starting point, not an independently validated distance metric.Source: §3.3 · p. 2
3.4 Calibration criterion
- Calibration is assessed at population and respondent levels: simulated versus human correlations across pairs, and each persona’s simulated score versus that respondent’s real score.The passage points to Eqs. 1–4 for population-level agreement.Source: §3.4 · p. 2
3.5 Models and inference settings
- The main evaluation uses gemma4:31b, llama3.1:70b, and qwen3.8:27b at their documented default temperatures; a separate comparison adds llama3:70b and llama3.3:70b.The default temperatures are 1.0, 0.8, and 0.8, respectively, for the three main models.Source: §3.5 · p. 2
5 Discussion
Across the tested models, cross-instrument calibration is replicable, but relationship direction is more reliable than magnitude and results can differ by seed direction. The discussion frames this asymmetry as a possible consequence of the difficulty of inferring a profile from a single instrument, and calls for release-specific, distance-aware checks.
5 Discussion
- A proposed account is that projecting a rich profile onto an unseen instrument is easier than inferring that profile backward from one instrument, producing directional asymmetry.The authors present this as a unifying frame, not as a mechanism directly tested by the study.Source: §5 · p. 7
- Open-weight models offer a substantial foundation for behavioral-simulation and digital-twin pipelines, but calibration should be checked continually by release and construct distance.The authors characterize this verification overhead as modest compared with collecting new human data.Source: §5 · p. 7
6 Limitations and Future Directions
The study’s comparisons are descriptive and lack a non-LLM baseline, while several extensions and release-level findings are limited in scope. The authors recommend broader comparisons and verification before treating preliminary results as general.
6 Limitations and Future Directions
- The 139 pairs per model reuse 40 respondents and instruments, so the reported statistics are descriptive rather than supported by independence-assuming significance tests.Without a non-LLM baseline, the reported r≈0.70–0.73 cannot be judged against simpler methods.Source: §6 · p. 7
- The behavioral extension covers one model and one respondent, release comparisons cover one family on a small panel, and release tags lack weight checksums for exact reproducibility.The leave-one-out result is also a single-model study on llama3.1:70b, with only thirteen targets.Source: §6 · p. 8
- Future work should establish a non-LLM baseline, expand release comparisons, investigate weaker forward–reverse agreement for qwen3.8:27b, and re-verify the behavioral extension.Source: §6 · p. 8
7 Conclusion
The study evaluates three open-weight models on a construct-distance-controlled cross-instrument task and compares them with human correlational structure. Simulated correlations track human correlations, but release performance is not monotonic, supporting release-by-release verification.
7 Conclusion
- r≈0.70–0.73: simulated seed–measure correlations track human correlations across all three open-weight model families.Source: §7 · p. 8
- The newest of three tested Llama releases performs worst on two of three metrics, so capability does not improve monotonically across releases.Source: §7 · p. 8
- Open-weight models provide an encouraging foundation for behavioral-simulation and digital-twin applications without fine-tuning or proprietary infrastructure.llama3.1:70b and qwen3.8:27b were the most consistently calibrated in these tests.Source: §7 · p. 8
- Because performance varies across construct distance and releases, the authors recommend verifying models release by release rather than assuming a one-time validation persists.Source: §7 · p. 8
B Construct-Distance Tier Assignments
The appendix describes a deterministic procedure for assigning instrument pairs to construct-distance tiers and clarifies how those assignments relate to the evaluated grid. It also identifies the instrument table as listing the fifteen instruments and their sources.
B Construct-Distance Tier Assignments
- The procedure assigns all 105 unordered pairs among fifteen instruments to four tiers using family-based defaults and literature-backed overrides.The assignments are deterministic; the 139-pair evaluation grid uses a subset expanded into ordered runs, with BFI split into five measured facets.Source: B Construct-Distance Tier Assignments · pp. 8–9
- The fifteen-instrument table lists the instruments used and their sources.Source: Table 9 · p. 9