Artificial Societies Benchmark: A Validation Framework for Synthetic Research

1 Introduction

• A correct overall support level does not establish who supports a proposal, why they do so, what could change their views, or whether they would act.The data-centre example distinguishes average support from group comparisons, relationships with concerns, responses to changed benefits and costs, and petition signing.Source: §1 · p. 1
• Excess consistency can misrepresent human retest variability, while plausible item answers can still alter relationships among items or make distinct traits interchangeable.A scale that changes meaning in the synthetic sample can mislead comparisons despite plausible individual answers.Source: §1 · p. 1
• The benchmark organizes eleven human-referenced tests across internal, construct, and external validity to connect research uses with the evidence they require.It brings response processes, psychological measurement, and population and experimental fidelity into a common framework.Source: §1 · p. 2
• The demonstration evaluates twenty human sources and nine language models, varying respondent information and simulation approaches while including statistical controls.Comparisons use the same respondents, instruments, and experimental assignments wherever the data allow.Source: §1 · p. 2
• The paper contributes a scorecard that retains estimates, uncertainty, and failures, alongside human references and statistical population controls.Its five elements also include tests tied to research uses and comparisons of profiles, prior answers, and personal descriptions.Source: §1 · p. 2

2 Background

• Synthetic survey methods vary in what respondent information models receive and whether answers are generated as population proportions or individual responses.Inputs can include demographics, personal descriptions, or earlier answers; output formats include probabilities across options or forced choices.Source: §2 · p. 2
• Existing benchmarks examine distributions, individual predictions, demographic differences, and respondent groupings, with improved distributional agreement sometimes coexisting with weak individual prediction.These approaches include held-out answers from the same people and comparisons across groups, topics, countries, or time.Source: §2 · p. 3
• Other studies assess population-pattern properties and experimental effects; strong correlation with human effects can coexist with systematic overestimation.Experimental validation therefore examines both whether predicted magnitudes track human effects and whether their sizes are accurate.Source: §2 · p. 3
• Psychometric studies test scale reliability, factor structure, relationships between measures, and invariance, finding that consistency can coexist with distorted structure [10].Prior work also distinguishes directional agreement from accurate magnitudes and asks whether results transfer across populations, times, and settings.Source: §2 · p. 3
• The benchmark extends complementary validation approaches by organizing response-process, measurement, and population comparisons around the evidence a proposed research use requires.This helps identify both performable checks and claims the available data cannot yet support.Source: §2 · p. 3

3 The Artificial Societies Benchmark

• The framework groups eleven tests into internal validity for response processes, construct validity for psychological measurement, and external validity for population and experimental fidelity.A claim about actual charging additionally needs comparison with observed behaviour.Source: §3 · pp. 3–4, Table 1 · p. 4

3.1 Internal validity

• Internal-validity tests compare human and synthetic retest agreement, logical-constraint violations, paraphrase transitions, and sensitivity to survey wording, order, or labels.Excessive stability can itself depart from human wording sensitivity, and errors in survey shifts can alter synthetic conclusions differently from human conclusions.Source: §3.1 · pp. 4–5

3.2 Construct validity

• Construct-validity tests compare item associations and scale reliability, links to independent criteria, and separation between distinct traits against human relationships.Unusually high reliability does not by itself establish validity, and criterion evidence supports claims only about the characteristic or outcome actually observed.Source: §3.2 · p. 5

3.3 External validity

• External-validity tests assess answer distributions, group and individual errors, and treatment-effect differences, while latent-structure tests compare loadings, factor correlations, and ordinal invariance.Close population distributions do not establish correct person-level answers or relationships; unused response categories can prevent latent-structure estimation.Source: §3.3 · pp. 5–6
• Researchers should select relevant tests and read estimates alongside human references, uncertainty, and coverage rather than applying a universal pass threshold.Formal equivalence claims require prespecified, use-specific tolerances and uncertainty in the human–synthetic comparison.Source: §3.3 · p. 6, Figure 1 · p. 6

4 Methods

• The benchmark draws on twenty human sources with surveys, panels, inventories, and experiments, supplying references for distributions, linked-person predictions, measurement, and treatment effects.Figure 2 distinguishes available estimates from attempted tests without estimates; a filled square indicates evidence, not good performance.Source: §4 · p. 7, Figure 2 · p. 7
• Nine proprietary and open-weight language models are evaluated alongside statistical controls that sample independent item marginals or approximate relationships between items.The controls help distinguish matching answer frequencies from preserving joint response patterns.Source: §4 · pp. 7–8, Table 2 · p. 8
• The adapted simulation protocol varies respondent information and answer format, including questions alone, demographic information, personal descriptions, and earlier survey histories where available.The GSS comparison changes answer format as well as information, while Twin compares six conditions on the same people and target questions.Source: §4 · p. 8
• Twin’s distilled profiles organize personal descriptions into seven fields and are checked for faithful representation before comparisons on the same 430 retained people.The checks do not test subsequent prediction accuracy; the paired-information analysis compares only questions with valid answers in both conditions.Source: §4 · pp. 8–9
• Most estimates require 90 percent valid model answers and human-target coverage plus at least 30 people, while exploratory structural comparisons require at least 100 complete matched participants.These operational thresholds limit sparse-output reporting; they are not universal validity requirements or guarantees.Source: §4 · p. 8
• Uncertainty intervals resample people while preserving their answers and experimental assignments, so they are conditional on generated answers rather than fresh simulation runs.Unanswered questions are also assessed using the most and least favourable full-question-set differences their unknown errors permit.Source: §4 · p. 9

5 What the benchmark reveals

• Eleven tests reveal distinct failures across models and sources; Figure 3 profiles strengths and weaknesses rather than producing an overall ranking.Compare models within each diagnostic column; shading across columns does not imply equal error or importance.Source: Figure 3 · p. 9, §5 · p. 9

5.1 Reliable-looking answers can hide measurement problems

• Seven models exceed human duplicate-item agreement, while eight models agree across paraphrases 100% of the time versus 85% for humans.Scout repeats answers about 93% of the time yet contradicts opposing personality items about 21% of the time; human contradictions average about 8%.Source: §5.1 · pp. 9–10
• Mistral’s option-order shift is −10.6 points versus humans’ −4.4 points; every model also omits ESS response categories.The benchmark does not merge unused categories to force ordinal fits, and merely using every category would not establish equivalent measurement.Source: §5.1 · p. 10
• Five models’ SAPA facet-loading errors range from 0.28 to 0.46; independent item draws also miss trait relationships despite close item-frequency fits.Four models lack usable loading estimates, and these exploratory results do not establish measurement invariance.Source: §5.1 · pp. 10–11

5.2 The strongest model depends on the research question

• Qwen’s first-generation SAPA error is 0.38 total variation distance, yet its Bainbridge contradiction rate is 0.24 versus humans’ 0.08.Cint has only one eligible item–group combination out of 400 for subgroup checks, and Bovitz has none.Source: §5.2 · p. 12
• Across FSD3587, Grok has the smallest mean absolute effect error at 0.23 rating points; for an Allais item, OpenAI’s numeracy association is 0.11 versus humans’ 0.28.Averaging effect errors can conceal whether individual contrasts are weakened, exaggerated, or reversed; small average correlation error does not establish a substantively useful relationship.Source: §5.2 · pp. 12–13

5.3 More personal information helps selectively

• Distilled profiles reduce error for OpenAI and Gemini but increase it for Mistral; Scout’s point estimates improve with distilled rather than original self-descriptions.Missing-answer sensitivity supports the main improvements but weakens the evidence for Mistral’s worsening.Source: §5.3 · p. 14
• Gemma fails to rate “Tell the truth” in 478 of 512 requests, illustrating how item misinterpretation can limit measurement evidence.DeepSeek declines 39 times; Gemma’s facets cells consequently have no estimate.Source: §5.3 · p. 15

6 Discussion

• Qwen’s relatively close SAPA distributions coexist with frequent Bainbridge contradictions, while all nine models omit ESS response categories used by humans.The benchmark therefore connects each research claim to the evidence needed to support it rather than declaring one model generally valid.Source: §6 · p. 15
• More consistency is not necessarily more human-like: duplicate-item and paraphrase findings show that synthetic answers can miss human response variability.These tests assess observable response patterns, not whether models use human cognitive mechanisms.Source: §6 · pp. 15–16
• Close item frequencies can coexist with weak trait relationships, and ESS category collapse prevents the planned ordinal comparison without yielding an invariance estimate.Claims about relationships between constructs need evidence beyond agreement on average scores; the size of the substantive effect remains unidentified.Source: §6 · p. 16
• The benefit of richer profiles depends on the model and representation: OpenAI and Gemini improve with distilled text, while Scout responds differently to original and distilled descriptions.Sensitivity analysis supports the main improvements but makes Mistral’s worsening uncertain; using OpenAI for distillation and Claude Opus 5 for auditing may favor those providers.Source: §6 · p. 16
• Self-report predictions do not establish consequential behaviour, and experimental rating comparisons support claims about those ratings rather than later behaviour.Profile-level counterfactual claims would additionally require repeated draws under the same condition and evidence that between-condition differences correspond to human effects for comparable profiles.Source: §6 · p. 16
• Population averages can conceal subgroup gaps: sparse item-level samples leave most Cint and Bovitz demographic comparisons unresolved.Human–model measurement comparability and comparability between demographic groups require separate tests.Source: §6 · p. 16
• Twin text comparisons cover 84% of profiles that pass without corrections, and public human sources leave model-training overlap unknown, so unseen-study generalization is not established.Model comparisons also describe evaluated configurations rather than isolating architecture; researchers should interpret results within the relevant dataset and endpoint, accounting for uncertainty and coverage.Source: §6 · pp. 16–17

7 Conclusion

• The scorecard helps researchers select relevant checks, compare synthetic results with human evidence, and identify additional data their claims require.Behavioural claims require evidence about what people do, while claims about a particular group require validation that reaches that group.Source: §7 · p. 17

A Metric definitions

• Shared notation aligns matched human and synthetic responses, repeated generations, item scales, and normalized losses for the benchmark’s metrics.Human and synthetic statistics are reported as signed and absolute differences; direct comparisons use matched participants.Source: A Metric definitions · p. 21

A.1 Internal validity

• Repeated-question, linked-question, paraphrase, and survey-perturbation tests compare synthetic response consistency and shifts with human references.The tests include agreement and kappa, logical-constraint violations, paraphrase transitions, and response changes across survey versions or graded stimuli.Source: §A.1 · p. 21

A.2 Construct validity

• Construct tests assess item relationships, independent-criterion associations, and separation between distinct traits.Correlations follow source-specific estimators, and summaries can be unavailable when required correlations are undefined.Source: §A.2 · p. 22

A.3 External validity

• External tests compare human and synthetic response distributions, group and individual outcomes, experimental effects, and latent structures.The latent-structure diagnostics compare loadings and factor correlations; ordinal invariance fits use joint human and synthetic data.Source: §A.3 · pp. 22–23

A.4 Common-cohort information comparisons

• Common-cohort information comparisons estimate paired changes in loss for the same accepted people, reporting item coverage and uncertainty.Comparisons require at least 30 paired people, may use different available item sets, and negative changes favour the second condition.Source: §A.4 · p. 23

A.5 Independent Twin criteria

• Independent Twin criteria compare human and model correlations with unprompted behavioural or personality measures, retaining failed and non-estimable pairs.The criteria include cognitive reflection, numeracy, Need for Cognition, and BFI-44 domains; constant model answers yield undefined correlations recorded as collapse.Source: §A.5 · p. 24

A.6 Missing answers, uncertainty, and interpretation

• Missing-answer bounds and explicit coverage, failure, and weighting rules qualify what each diagnostic estimate can establish.Aggregation keeps units separate and reports incomplete endpoint support rather than treating a diagnostic block as proof of a valid test.Source: §A.6 · p. 24

B Design details

• The design tables catalogue human sources, study families, and persona-information conditions, with information labels adapted from Gordon et al. [3].The appendix distinguishes those model-input labels from the framework’s diagnostic levels.Source: Table 4 · p. 24, B Design details · p. 24, Table 5 · p. 26

B.2 Reading the illustrated scorecard

• Figure 3 presents selected endpoints for all nine models, while its relative heatmap shading supports within-endpoint comparisons, not cross-test ranking or acceptance thresholds.The examples use the same source and endpoint within each model column; unavailable estimates remain visible, and complete scorecards retain intervals, coverage, and failures.Source: §B.2 · p. 26

C Source-specific administration and scoring

• Source-specific administration records cohorts, instruments, eligibility rules, and planned requests, with a practical 512-participant cap for most tasks.Nine proprietary and open-weight systems were selected to represent state-of-the-art systems.Source: C Source-specific administration and… · p. 28, §C.1 · p. 28

C.1 Cohorts, eligibility, and reuse

• Cohort eligibility and reuse vary by source; GFS benchmark rosters are sampled from adults observed in both waves across three countries.Public records lack respondent-level language, so these GFS cohorts are English-majority rather than language-matched at the respondent level.Source: §C.1 · p. 28

C.2 Prompt reconstruction and recorded personas

• Prompt reconstruction uses released instrument text and recorded personas, while omitting unavailable fields and preserving source-specific wording where feasible.Exact live screens and interviewer probes are not reproduced, and a fixed order replaces ValueConsistency’s unrecorded per-person option shuffle.Source: §C.2 · p. 28

C.3 Twin free text, distillation, and information comparisons

• Twin compares six information conditions using open-ended self-descriptions, closed-ended history, and distilled profiles on a shared accepted cohort.The common comparison uses 430 accepted profiles and 32 behavioural item variants; distillation retains 430 of 512 unedited profiles (84.0 percent).Source: §C.3 · p. 29

C.4 ANES 2020 verbatims and ANES 2024 order

• ANES tasks compare persona-information conditions and response-order effects, with later ANES problem statements treated as reconstructed conditions.The ANES 2020 thermometer analysis retains 1,205 observations after excluding potentially disclosed targets from every condition.Source: §C.4 · pp. 29–30

C.5 Repeatability, retest, and logical coherence (IV-1, IV-2)

• Repeatability and coherence tests compare human retest variation with model repetition and assess logical constraints and contradictory answers.Weather coherence uses 27 linear probability constraints with a one-percentage-point tolerance; keyed-opposite pairs and graded consensus supply additional checks.Source: §C.5 · p. 30

C.6 Paraphrases and method effects (IV-3, IV-4)

• Paraphrase and method-effect tests use screened question pairs and source-specific response transitions or contrasts.ValueConsistency’s primary scoring retains 89 of 108 pairs after rule-based exclusions, not an independent human equivalence rating; ESS and FSD3587 add method and source-identity contrasts.Source: §C.6 · p. 30

C.7 Experimental contrasts (EV-3, IV-4)

• Experimental contrasts compare matched human and synthetic treatment effects, while subgroup support is too sparse in the two acquiescence sources for broad EV-2 evidence.No Bovitz item-group and only one of 400 Cint item-group combinations meet the required minimum support.Source: §C.7 · pp. 30–31

C.8 Distributions, relationships, and subgroups (EV-1, EV-2, CV-1)

• Distribution and relationship diagnostics compare marginals, item associations, and subgroup distributions, but unsupported intersections remain unresolved.The analysis does not provide every diagnostic within every intersection or a general within-group calibration procedure.Source: §C.8 · p. 31

C.9 Traits, criteria, and factor structure (CV-1 to CV-3, EV-4)

• Trait and factor diagnostics compare scale properties, external criteria, loadings, and ordinal invariance, with interpretation bounded by criterion type and category coverage.Trait-criterion correlations are associations rather than person-level predictive calibration; the ESS models omit response categories, limiting the full-category joint fit.Source: §C.9 · p. 32

C.10 2026 cross-sections (HTOPS)

• HTOPS uses three 2026 cross-sections, sampling 512 benchmark people per wave and administering life-satisfaction items on the displayed 0 to 10 scale.The sections use recorded unallocated demographics and broad geography, omitting imputed fields and age boundary codes.Source: §C.10 · p. 33

C.11 Baselines and missing outputs

• GSS controls include empirical-distribution, demographic-regression, nearest-donor, and Gaussian rank-copula baselines, with missing outputs reported alongside conditional scores.The copula approximates dependence, while the independent-marginal control samples from the same marginals; bounded-loss worst-case bounds are also reported.Source: §C.11 · p. 33

• The benchmark used internal frozen analysis plans but was not formally preregistered.Source: §C.12 · p. 33
• Several diagnostics were specified after model answers were available and are interpreted descriptively with pointwise uncertainty intervals.The authors also classify several measures by validity test and treat ESS method pairs as method comparisons because they change multiple administration features.Source: §C.12 · p. 33
• The parser maps responses to unique permitted options using documented rules, but longer replies can remain ambiguous and invalid or missing answers remain in coverage counts.It does not use the human target answer; raw outputs are preserved and each normalization is recorded.Source: §C.12 · p. 33
• The additional option-mapping rules recovered 2,179 previously invalid answers in the batch rescore, including 2,119 for Mistral and none for five models.The counts exclude other formatting rules and sequence recovery.Source: §C.12 · p. 33
• The shared re-ask policy allowed one repeat for specified empty or invalid outputs and connection drops, while collection-route effects on sampling variability were not estimated.The Mistral ESS request exceeding the image limit failed through both routes.Source: §C.12 · p. 33

• Scout, Qwen3.8-27B, and Gemma 4 31B each returned all 24,080 distilled answers for the 430-person common cohort under the shared content and model-specific configurations.Source: §C.13 · p. 34
• After shared parsing, Scout had one truncated answer, Qwen had three invalid options, and Gemma had none.Source: §C.13 · p. 34
• All 30 common-cohort ladder comparisons per local model and all 36 local Twin criterion cells had estimated summaries, with failed item-criterion pairs retained.Source: §C.13 · p. 34

D Detailed results and completion

• The scorecard contains 7,128 task, model, condition, repetition, and test cells, with diagnostics in 3,186; only three family-level combinations lack estimates.These are ESS under CV-1 and EV-4 and Bovitz acquiescence under EV-2; individual tasks and endpoints may also lack estimates.Source: D Detailed results and completion · p. 34
• Seven models exceeded human duplicate agreement, while all nine had zero consensus sequences with decreasing belief compared with about 48 percent of human sequences.ValueConsistency agreement was 1.00 for eight models, versus about 0.85 for humans; supported March HTOPS construct-separation gaps ranged from −0.09 to 0.07 across four models, against about 0.20 for humans.Source: D Detailed results and completion · p. 34
• Across 108 Twin model-condition-repetition cells, mean absolute criterion-correlation error was about 0.08 to 0.10, while signed error ranged from about −0.03 to +0.01.The largest human association was about 0.28; OpenAI’s recorded-condition Allais-choice association with numeracy was about 0.11.Source: D Detailed results and completion · p. 34
• First-generation SAPA Big Five mean item total-variation distance ranged from 0.38 to 0.70 across models; mean absolute FSD3587 messenger-effect errors ranged from 0.23 to 1.10 on a 0-to-10 scale.Human effects averaged 0.52 points in absolute size; rank-copula and independent-marginal controls had between-trait correlation errors of 0.040 and 0.170.Source: D Detailed results and completion · p. 34

D.1 Paired information comparisons

• The paired information comparison retained 430 of 512 profiles (84.0 percent), yielding 216,720 distilled answers across nine models; acceptance and prediction accuracy measure different properties.The excluded participants’ mean loss differed from the included cohort by approximately −0.01 to 0.03 across available conditions, with intervals often including zero.Source: §D.1 · p. 34
• All block-order contrasts were estimable for every model, while sequence completion varied; Scout had 9,603 valid pages out of 9,709, or 0.989 coverage.ESS completion was 512 for seven models, 501 for Mistral, and 504 for Qwen; block-order completion ranged from 499 for Scout to 512 for five models.Source: §D.1 · p. 34
• Table 8 reports paired changes in Twin prediction loss from demographic profiles, with repetitions 1 and 2 in each cell and negative values favoring added information.The local models used the shared distilled profiles and matching question, profile, and response-format content through model-specific vLLM configurations.Source: Table 8 · p. 35, §D.1 · p. 35

D.2 Twin missing-answer sensitivity analysis

• The bounds hold observed errors fixed, give each person equal weight, and allow each missing loss to range from zero to one without imputing answers.They address missing responses within the accepted cohort, not the 82 people whose profiles do not qualify.Source: §D.2 · p. 35
• The pointwise sensitivity envelope resamples the 430 people jointly across both bound endpoints 500 times; it is not a simultaneous confidence band.All 270 comparisons retain 430 people, and minimum item coverage is 187 of 216 (86.6 percent).Source: §D.2 · p. 35
• Sample bounds exclude zero for 256 of 270 comparisons, but bootstrap envelopes do so for 102; the envelopes preserve principal improvement examples while weakening some directional conclusions.Fifteen primary intervals that excluded zero had sensitivity envelopes including zero; Mistral’s worsening is among the conclusions weakened.Source: §D.2 · p. 35, Table 9 · p. 36

E Declarations

• The study analyzed existing human datasets and model responses without recruiting new human participants or obtaining institutional ethics approval; code and specifications are available through the repository.Original human datasets remain subject to the providers’ access conditions.Source: E Declarations · p. 36