Total Simulated Survey Error: Designing and Diagnosing Survey Responses from Large Language Models

1 Introduction

• LLMs are used to simulate individuals or groups and answer survey questions on their behalf, but their training data and post-training can skew behavior.The underlying assumption is algorithmic fidelity: that LLMs can be steered to mimic diverse human subgroups.Source: §1 · p. 2
• High aggregate alignment with human responses can coexist with distortions in variance, subgroup heterogeneity, and downstream statistical relationships.Existing work documents failure modes including social bias, sycophancy, homogenization, and “alienness.”Source: §1 · p. 2
• TS2E enumerates measurement and representation errors across the LLM-generated survey lifecycle, distinguishing researcher design choices from inherent LLM limitations.It also identifies evaluation fallacies and is illustrated through theoretical examples and a multiverse case study of U.S. 2024 presidential vote choice.Source: §1 · pp. 2–3
• TS2E is intended to help designers diagnose error sources, document errors beyond their control, and reason about trade-offs among error components.The framework also aims to provide a common vocabulary and inform reporting standards and benchmarks.Source: §1 · p. 3

Before Fielding

• Before fielding, designers define the construct and target population, draw a human sample or map population units to simulation units, and construct personas.Simulation units can represent individuals or subgroups, so they need not correspond one-to-one with target-population units.Source: §2 · pp. 3–4
• Task articulation combines a persona with a question, output format, and specialization such as few-shot examples, background knowledge, or reasoning instructions.Simulation units can be grounded in auxiliary data, while personas are produced from those units through a persona-design function.Source: §2 · p. 4, Figure 2 · p. 4

Fielding

• During fielding, humans answer a questionnaire through a partly stochastic response process, while LLMs generate responses from a prompt containing a task and persona.LLM design choices include the backend and inference settings; the simulated response function maps task and persona to a predicted response.Source: §2 · p. 5
• Simulation errors can be traced to task articulation, persona construction, or LLM response generation and persona simulation.Human and simulated responses share a target population but use different question/task, respondent/persona, and response-generation representations.Source: §2 · p. 5

After Fielding

• After fielding, responses are processed and may be adjusted through reweighting before the survey statistic is analyzed; evaluation can compare responses with ground truth or theory.Individual-level metrics depend on mapping simulation units to reference individuals, whereas distribution-level metrics can be used regardless of simulation unit.Source: §2 · p. 5

3 A Quality Perspective on LLM-generated Survey Responses: Total Simulated Survey Error

• TS2E treats random or systematic gaps between true and obtained measures as measurement or representation errors, while evaluation fallacies concern how reference data are selected and compared.The framework organizes these errors across survey stages and artifacts.Source: Figure 3 · p. 6, §3 · p. 7

Questionnaire Design

• Specification error arises when survey questions fail to translate the construct and its facets into an adequate measure.Inaccessible questions about concepts such as primaries or caucuses can require political literacy and introduce this error.Source: Questionnaire Design · p. 7

Task Articulation

• Task articulation error arises when adapting a question into an LLM prompt or paraphrasing it changes the question’s meaning, answer options, or task format.The error is comparable to design-specific response error in the Total Survey Error framework.Source: Task Articulation · p. 7

Persona Construction

• Persona construction error reflects mismatches between the target population and designed personas, including effects of persona granularity, attributes, expression, and inherited sample errors.The appropriate persona attributes are not established; reused survey or census infrastructure can also carry coverage and sampling errors into the simulation.Source: Persona Construction · p. 8

LLM Selection

• LLM selection choices include model type and specialization; instruction-tuned models are easier to use but may reduce response diversity.Errors associated with model selection become tangible after responses are generated.Source: LLM Selection · p. 9

LLM Inference

• At inference, imperfect LLMs create response-generation measurement errors and persona-simulation representation errors; stochastic outputs also threaten test-retest reliability.Persona simulation can be shaped by interactions between prompt design, model choice, and inherent model factors, while proprietary models may change without notice.Source: LLM Inference · pp. 9–10

LLM Response Processing

• Response processing can introduce measurement error when converting generated answers into analysis-ready data, and stochastic processing can also affect test-retest reliability.For example, exact text matching may discard valid candidate responses expressed with a nickname or only a first or last name.Source: LLM Response Processing · pp. 10–11

LLM Respondent Reweighting

• Post-survey adjustment can address persona simulation error, but reweighting choices or variables can themselves introduce adjustment error.Using human-survey weights is contingent on humans and LLMs having the same representation errors, which may not hold.Source: LLM Respondent Reweighting · p. 11

Ground Truth Fallacy

• Human survey data may not be a reliable ground truth for real-world behavior, and contamination can inflate apparent alignment without predicting performance in unseen contexts.Contamination can occur when training data include the survey being simulated or when supposedly human responses were partly or fully LLM-generated.Source: §4 · pp. 11–12

Single Best Metric Fallacy

• No single metric establishes simulation performance across goals, and high alignment may not translate into faithful downstream analyses.Metric choice depends on the simulation’s type, context, and purpose; multivariate aims call for comparing downstream effect sizes and directions.Source: §4 · p. 12

Context Drift Fallacy

• Validation in one setup does not establish efficacy in a supposedly similar context, whose similarity may be difficult to define or quantify.Temporal drift is especially relevant in volatile settings, while undocumented changes to closed-source models can threaten reproducibility.Source: §4 · p. 12

Methods

• The ANES 2024 case study compares simulated vote choice with human responses, varying five design choices across 288 configurations and evaluating weighted F1 and TVD.The setup includes 144 pre-adjustment configurations run over five random seeds.Source: §5 · pp. 12–14

Results

• Across configurations, F1 and TVD rankings correlate (ρ = −0.84, p < 0.001), but the metrics select different best models; responses are generally stable across five seeds.Qwen-30B is best by F1, while OLMo-32B is best by TVD.Source: §5 · p. 14, Figure 4 · p. 14
• The best configuration reaches F1 = 0.686 and TVD = 0.097, versus F1 = 0.308 and TVD = 0.489 for the worst, showing high simulation sensitivity.The authors interpret the variation as evidence of LLM brittleness.Source: §5 · pp. 14–15
• Overall weighted average F1 is 0.52, with higher performance for liberal and conservative groups and lower performance for moderate and slightly conservative groups.Design choices explain less variance for moderate and conservative groups.Source: §5 · p. 15
• Task articulation has no statistically significant impact, while adding attitudinal variables such as party identification and ideology produces a large gain [Argyle et al., 2023].The authors identify this gain with persona construction error.Source: §5 · pp. 15–16
• Qwen-30B is not best for conservative and extremely conservative groups; free-text regex parsing and ANES weighting slightly reduce overall performance, while conservative groups favor JSON.Only the slightly liberal group benefits from adjustment.Source: Figure 5 · p. 15, §5 · p. 16

Limitations

• The evaluation aligns simulations with ANES rather than actual vote choice, and ANES itself carries survey error.The authors also cannot rule out some LLM contamination of survey responses.Source: §5 · p. 16
• Personas and target vote choice both come from ANES 2024, so the study does not test prediction when target responses are unavailable or across survey waves.The authors note that party identification may be unavailable in realistic settings and is strongly correlated with vote choice.Source: §5 · p. 16

6.1 Evaluation of Survey Data Quality in the Social Sciences

• Survey data quality concerns whether measures capture intended constructs and whether samples represent target populations; the Total Survey Error framework organizes these as measurement and representation errors.It supports retrospective evaluation and prospective design by linking survey operations to statistical error components.Source: §6.1 · pp. 16–17
• Total Survey Error treats survey design as balancing multiple error sources: reducing nonresponse can increase response error when reluctant respondents provide lower-quality answers [Olson, 2006, Fricker and Tourangeau, 2010].This makes design an optimization across total error components rather than a single indicator such as response rate.Source: §6.1 · p. 17

6.2 Evaluation of LLM-powered Surveys

• Benchmarking finds modest LLM simulation fidelity and sharply degraded performance for specific demographic groups, while validation studies often rely on human-response comparisons or subjective believability [Hu et al., 2025; Castricato et al., 2025; Desai et al., 2026; Larooij and Törnberg, 2025].The literature also documents social and cognitive biases, sycophancy, alienness, and compressed response variance.Source: §6.2 · p. 17
• Existing work quantifies overall error more often than it explains its pipeline origin, separates researcher design choices from inherent LLM limits, or distinguishes measurement from representation errors.The paper develops a lifecycle framework to map these errors to their sources.Source: §6.2 · p. 17

7 Discussion

• TS2E covers representation and measurement errors across the simulation lifecycle, including designer choices, current LLM limitations, their interplay, and LLM-specific logical fallacies.The fallacies raise additional challenges when synthetic responses are evaluated against human responses.Source: §7 · pp. 17–18
• The framework can guide preregistration, benchmarks that assess nuanced errors beyond one score, and targeted interventions such as improving robustness to persona formats.Tracing the origins of response-generation and persona-simulation errors requires interpretability methods, while training data and curation decisions are rarely public.Source: §7 · p. 18
• Assistive post-training may reduce simulation of fringe groups, creating a potential trade-off between realism and assistiveness and an open dual-use challenge for specialized models.The authors describe designing specialized social-simulation models as an interdisciplinary engineering challenge.Source: §7 · p. 18
• Designers should document simulation contexts, choices, and model characteristics, and assess validity, generalization, reliability, and sensitivity to inconsequential changes.The authors urge caution with proprietary models for past-survey simulations when training-data provenance cannot be guaranteed, because this prevents safeguards against contamination.Source: §7 · p. 18
• Some TS2E errors, including persona construction and simulation, may apply beyond surveys, but other settings can introduce distinct errors such as feedback loops in multi-round GABMs.The authors identify specialized frameworks for other use cases as future work.Source: §7 · p. 18
• Public acceptance data for using LLMs to complete surveys is lacking, while simulating individuals raises sharper consent and predictive-privacy concerns than simulating broad groups.The authors also raise democratic legitimacy and the possibility that replacement of human respondents could leave no human-generated data for validation.Source: §7 · pp. 18–19
• The framework’s lifecycle stages can contain finer-grained error dimensions, and the empirical use cases illustrate TS2E rather than generalize to all LLM-powered surveys.The framework focuses on fully LLM-generated samples; mixed human–LLM designs may introduce additional errors.Source: §7 · p. 19

9.1 Description of the Total Survey Error Framework

• The TSE framework distinguishes measurement errors in defining and measuring constructs from representation errors in inferring from samples to target populations.The appendix describes the framework’s account of error and bias and its extensions to other forms of social data.Source: §9.1 · p. 25

9.2.1 Predicting Vote Choice of Different Ideological Groups

• Qwen-30B, the best overall model by F1, heavily underestimates non-voter rates, especially in moderate and conservative subgroups.Figure 7 contextualizes persona simulation error and the regression findings.Source: §9.2.1 · p. 27, Figure 7 · p. 27

9.2.2 Total Variation Distance and Interactions

• The best configuration depends on the evaluation metric: TVD uses OLMo-32B and free-text with an LLM judge as references, unlike F1.The appendix identifies this metric-dependent ranking as the Single Best Metric Fallacy.Source: §9.2.2 · p. 27
• Personas with attitudinal variables outperform demographics-only personas on both F1 and TVD across nearly all ideology subgroups.This is the case study’s finding that holds regardless of metric.Source: §9.2.2 · p. 27
• Adding LLM-by-design-choice interactions substantially boosts adjusted R2, while patterns from regressions without interactions still hold.The interaction analysis assesses whether LLMs and design choices systematically affect simulation performance.Source: §9.2.2 · p. 28

9.2.3 Simulation Prompts

• The appendix provides the case-study prompts for free-text and JSON response generation, alongside the stated response options Harris, Trump, and Non-voter.The prompts frame the task as predicting vote choice in the 2024 U.S. presidential election.Source: §9.2.3 · p. 31, System prompt for free-text response… · p. 31, System prompt for JSON response… · p. 31

User prompt for first-person personas and Task v1 (example)

• The first-person Task v1 example asks a persona to report whether they will vote and, if so, for whom.The example describes a Black 28-year-old woman from South Carolina with low political interest.Source: User prompt for first-person personas… · p. 31

User prompt for first-person personas and Task v2 (example)

• Task v2 changes the first-person question wording to “Please tell me if you will vote and if so for whom.”The persona attributes remain the same as in the Task v1 example.Source: User prompt for first-person personas… · p. 31

User prompt for interview personas and Task v1 (example)

• The interview Task v1 example presents the persona attributes and vote-choice question as interviewer–interviewee exchanges.The question asks whether the interviewee will vote and, if so, for whom.Source: User prompt for interview personas and… · p. 31

User prompt for interview personas and Task v2 (example)

• Interview Task v2 uses the same exchange format but changes the vote-choice question to “Please tell me if you will vote and if so for whom.”The answer options are Harris, Trump, and Non-voter.Source: User prompt for interview personas and… · p. 31

System prompt for the annotation of free-text responses with an LLM judge / User prompt for the annotation of free-text responses with an LLM judge

• The annotation system prompt casts the model as an expert labeling vote-choice statements and requires a JSON answer.The accompanying user prompt asks whether a statement expresses preference for exactly one listed option and, if so, which one.Source: System prompt for the annotation of… · p. 32, User prompt for the annotation of… · p. 32

9.3 Guideline and Checklist for Documenting LLM-powered Survey Responses

• The TS2E checklist covers population definition, persona construction, and evaluation choices, and can be combined with Guide-LLM fields D1, F1, and G1.It builds on AAPOR reporting guidelines and GuideLLM, with deeper attention to LLM survey simulations.Source: §9.3 · p. 33

9.3 Guideline and Checklist for Documenting LLM-powered Survey Responses / Q1.3 Question Variations / Q1.4 Answer Option Variations / Q1.5 Rationale for Questionnaire / S.1 Unit Description

• Table 2 presents lifecycle documentation items with explanations and worked examples; the checklist is illustrated with a study predicting the 2024 European Parliament elections.The examples include question variations, answer-option variations, questionnaire rationale, and simulation-unit items.Source: Table 2 · p. 33, §9.3 · pp. 33–34

S.2 Sample Size / S.1 Unit Description

• The checklist includes simulation sample size and the unit of observation, with the example identifying individual survey respondents.The sample-size item asks how many entities were simulated and how that number was set.Source: §9.3 · p. 34

S.3 Sampling Procedure

• The example uses 25,916 Eurobarometer survey respondents, with about 1,000 per country and smaller or larger samples for specified countries.It identifies the source as the Eurobarometer EB 99.4 sample.Source: §S.3 · p. 34
• The sampling-procedure item asks how units were selected from the frame, including stratification and random seed, and whether selection was probabilistic.The example describes stratified, multi-stage probability samples.Source: §S.3 · p. 34

PC.2 Persona Characteristics

• The checklist defines the persona as a prompt-based persona used to steer LLMs during inference and asks which attributes compose it and where their values come from.The example uses a second-person declarative persona and lists country, age, gender, education, class, employment, and urbanity, with a variation adding political attributes.Source: §S.3 · p. 34, PC.2 Persona Characteristics · p. 34

PC.3 Rationale for Persona Content and Format

• The example reports no evidence that first-person declaratives outperform second-person declaratives and names voting-behavior research as the rationale for the selected attributes.The listed sources are Braun & Schäfer (2022), Ford & Jennings (2020), and Giebler & Wagner (2015).Source: PC.3 Rationale for Persona Content and… · p. 34
• The checklist item asks which content goes in the system prompt versus the user prompt; the example places the persona and question in the user prompt without a system prompt.The item also asks about task-specific prompt augmentations and whether the model is prompted to reason.Source: PC.3 Rationale for Persona Content and… · pp. 34–35
• The example adds temporal and political context, randomizes the competing-party list, and requests a short response to optimize matching and avoid long or cut-off answers.The instruction asks the model to make its best prediction from the information given.Source: PC.3 Rationale for Persona Content and… · p. 35

Data collection

• The example reports one run per persona and experimental setting and requests a precise model/version pointer and selection rationale.It identifies GPT-4-Turbo (version 2024-04-09), Llama-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.3.Source: Data collection · p. 35
• The checklist asks for model knowledge cut-off and training-data availability; the example gives dates and notes limited or unavailable training-data information.It lists December 2022 for Mistral and December 2023 for GPT and Llama; GPT training data is undisclosed, while open-weight model information is limited.Source: Data collection · pp. 35–36
• The checklist records whether a model is open or proprietary and instruction-tuned or base, as well as its language capabilities and temperature.The example uses instruction-tuned models and temperature 0.9 to retain response variance.Source: Data collection · p. 36
• The checklist documents access mode, retrieval augmentation, and custom fine-tuning or steering; the example accesses GPT by REST API and hosts Llama and Mistral locally.The example does not provide corresponding values for retrieval augmentation or customization.Source: Data collection · p. 36

RG.1 Output Mode / RG.2 Decoding Parameters

• The output-mode item gives possible response forms (free text, structured JSON, a forced token, or log-probabilities over options); decoding parameters cover settings beyond temperature, including max_tokens, top_p, penalties, stop sequences, and seed.Source: RG.1 Output Mode · p. 36, RG.2 Decoding Parameters · p. 36

RP. Response Processing

• Response-processing documentation records whether an LLM interprets or classifies raw outputs; this setup uses no LLM for that stage.Source: RP. Response Processing · p. 36

RP.2 Other Automatic Classifier

• Non-LLM classifiers can use regex, matching, dictionaries, or trained classifiers; this example matches party names, acronyms, synonyms, and invalid-vote keywords.The framework also calls for documenting failure modes of non-LLM processing.Source: RP.2 Other Automatic Classifier · p. 36

RW.1 Source of Adjustment Weights

• Weighting documentation records weight source and approach: reused Eurobarometer poststratification weights (W87: EU28 minus UK) feed per-country marginal and intercellular weighting across sex, age, region, and locality.Source: RW.1 Source of Adjustment Weights · pp. 36–37

• The reference is official per-country EU election turnout and party vote shares.Source: §E.2 · p. 37
• Aggregate estimates are evaluated using average absolute differences in party vote shares per country and turnout.Source: §E.2 · p. 37
• Winning-party matches and party rank orderings provide additional, more lenient evaluation measures.Source: §E.2 · p. 37

• Absolute differences suit the point-estimation goal, but variance and confidence intervals are omitted because error may also arise from biases in LLM data generation.Source: §E.3 · p. 37
• Winning-party matches and rank orderings are included as more lenient alternatives to absolute differences.Source: §E.3 · p. 37
• Individual-level intended or reported vote-choice responses were unavailable, and party vote shares are specified at the country-party level.Source: §E.3 · p. 37
• Metric and reference choices should be justified against the estimand and stakes, including what they fail to capture.Source: §E.3 · p. 37
• Inter-run variability is identified as a separate question of whether it is reported.Source: §E.3 · p. 37