PaperlayerView exact revision on arXiv

Source-linked AI summary

Out of One, Many: Using Language Models to Simulate Human Samples

Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua Gubler, Christopher Rytting, David Wingate

arXiv:2209.06899v1cs.LGcs.CL

TL;DR

Using language models as proxies for human subpopulations requires evidence of their fidelity. The authors condition GPT-3 on demographic backstories and compare simulated responses with human responses across three studies. They find that GPT-3 mirrors partisan descriptions and relationships among demographics, attitudes, and reported behaviors in U.S. public opinion.

  • Problem

    The extent to which language models faithfully reflect attitudes across human subpopulations is unexplored, despite its importance for their use in social science.

    Source: §1 · p. 3

  • Method

    The authors condition GPT-3 on first-person demographic backstories to generate silicon samples and compare simulated responses with human counterparts across three studies.

    Source: §1 · p. 3

  • Results

    Across three studies of U.S. public opinion, GPT-3 demonstrates algorithmic fidelity, reproducing human partisan descriptions and relationships among demographics, attitudes, and reported behaviors.

    Source: §1 · p. 3, §9 · p. 15

  • Takeaways & Limitations

    The findings support using GPT-3 as an aggregate proxy for studying human attitudes in U.S. politics and public opinion.

    Source: §1 · p. 3, §9 · p. 15

    Demographic conditioning alone does not guarantee faithful subgroup simulation; algorithmic fidelity must first be assessed for the domain being studied.

    Source: §4 · p. 5

Abstract

from arXiv · show

We propose and explore the possibility that language models can be studied as effective proxies for specific human sub-populations in social science research. Practical and research applications of artificial intelligence tools have sometimes been limited by problematic biases (such as racism or sexism), which are often treated as uniform properties of the models. We show that the "algorithmic bias" within one such tool -- the GPT-3 language model -- is instead both fine-grained and demographically correlated, meaning that proper conditioning will cause it to accurately emulate response distributions from a wide variety of human subgroups. We term this property "algorithmic fidelity" and explore its extent in GPT-3. We create "silicon samples" by conditioning the model on thousands of socio-demographic backstories from real human participants in multiple large surveys conducted in the United States. We then compare the silicon and human samples to demonstrate that the information contained in GPT-3 goes far beyond surface similarity. It is nuanced, multifaceted, and reflects the complex interplay between ideas, attitudes, and socio-cultural context that characterize human attitudes. We suggest that language models with sufficient algorithmic fidelity thus constitute a novel and powerful tool to advance understanding of humans and society across a variety of disciplines.

1 Introduction

The paper introduces algorithmic fidelity: a language model’s capacity to reproduce human subpopulation response distributions. Three studies provide evidence of this property in GPT-3 [9] within U.S. politics and public opinion.

1 Introduction

  • Algorithmic fidelity measures how accurately a model reproduces human subpopulation response distributions when conditioned on targeted identity and personality profiles.The authors reinterpret algorithmic bias as multiple, demographically correlated patterns rather than a single model-wide property. High fidelity enables studying attitudes across demographic groups and their intersections.Source: §1 · pp. 2–3
  • Three studies find GPT-3 [9] mirrors human partisan descriptions and relationships among demographics, attitudes, and reported behaviors in U.S. politics.The studies provide evidence that GPT-3 satisfies four proposed criteria of algorithmic fidelity.Source: §1 · p. 3
  • Language models are proposed as human-respondent surrogates for social science, supporting theory generation and testing before costly human studies.The demonstrated application is hypothesis exploration in U.S. politics, conditional on establishing algorithmic fidelity in the relevant domain.Source: §1 · pp. 2–3
  • Silicon sampling generates virtual respondent populations using first-person demographic backstories, correcting skewed marginal statistics in training data.The studies generate one silicon subject per human participant using the 2012, 2016, and 2020 ANES waves [16] and Rothschild et al.’s “Pigeonholing Partisans” data. Simulated subjects complete the same tasks as humans.Source: §1 · p. 3

2 The GPT-3 Language Model

GPT-3 generates text using token probabilities that depend on an initial conditioning context. Its probabilistic generation allows many possible continuations from the same context.

2 The GPT-3 Language Model

  • 175 billion parameters and more than 45 terabytes of training text characterize GPT-3, whose generated texts are difficult to distinguish from human compositions.The section presents these capabilities as promising for social science research.Source: §2 · p. 3
  • Conditioning supplies GPT-3 with an initial text context that increases the probabilities of some outputs and decreases those of others.The context consists of input tokens provided before text generation begins.Source: §2 · p. 4
  • GPT-3 repeatedly selects tokens probabilistically, appends them to its context, and continues until a specified token count or external stopping condition.The model defines a conditional probability distribution over a fixed vocabulary; probabilistic selection permits multiple continuations for the same context.Source: §2 · pp. 3–4

3 Algorithmic Fidelity

Algorithmic fidelity describes how accurately a model reproduces relationships among ideas, attitudes, and socio-cultural contexts across human subpopulations. The authors propose four complementary criteria for assessing its suitability as a source of surrogate human responses.

3 Algorithmic Fidelity

  • Algorithmic fidelity requires models to mirror human subpopulations’ underlying relationships among concepts, ideas, and attitudes, rather than merely produce superficially human-like outputs.The premise is that structured conditioning selects among multiple response distributions. Such fidelity supports generalizing findings from models to humans with matching backgrounds.Source: §3 · p. 4
  • Demographic conditioning can produce survey-response distributions strongly correlated with human distributions, without implying simulation of specific individuals or consistently coherent responses.Known shortcomings and inaccuracies of large language models still apply [19].Source: §3 · p. 4
  • The Social Science Turing Test requires human-indistinguishable responses; Backward Continuity requires consistency with conditioning attitudes and demographics that humans can infer from responses.The first criterion compares generated responses with parallel human texts; the second tests whether readers can recover key elements of the input.Source: §3 · p. 4
  • Forward Continuity requires responses to follow conditioning naturally in form, tone, and content; Pattern Correspondence requires human-like relationships among ideas, demographics, and behavior.Pattern Correspondence is assessed against comparable human-produced data.Source: §3 · p. 5
  • Confidence requires repeated support for all four criteria across data sources, measures, and groups, rather than fixed numerical thresholds.Failure on any dimension reduces confidence, with failures on multiple dimensions reducing it further; appropriate statistics depend on data structures and disciplinary standards.Source: §3 · p. 5
  • The subsequent studies examine GPT-3’s fidelity in U.S. politics and public opinion using expected relationships established in political science research.Established relationships among political ideas, concepts, and attitudes provide the comparison basis.Source: §3 · p. 5

4 Silicon Sampling: Correcting Skewed Marginals

Silicon sampling corrects skewed model marginals by conditioning GPT-3 on backstories drawn from a representative sample. Its usefulness depends on accurately modeling conditional responses and establishing algorithmic fidelity for the domain and demographic groups studied.

4 Silicon Sampling: Correcting Skewed Marginals

  • Silicon sampling corrects skewed marginal estimates by conditioning GPT-3 on backstories drawn from a representative sample, such as the ANES.For voting estimates, the method combines GPT-3’s conditional voting probabilities with the sampled population’s demographic distribution rather than the model’s own backstory distribution.Source: §4 · p. 5
  • Uncorrected marginals can be skewed because GPT-3’s backstory distribution differs from target populations, while internet training data are demographically unbalanced and fixed in time.The training population is not necessarily representative of the populations social scientists want to study.Source: §4 · p. 5
  • Population-level inference requires accurate conditional responses; demographic conditioning alone does not establish algorithmic fidelity for the domain and groups studied.The authors therefore require examining fidelity before treating sampled component distributions as faithful representations of human subpopulations.Source: §4 · p. 5
  • Demographically conditioned opinions, biases, and voting patterns can differ drastically from generic GPT-3 completions, analogous to Simpson’s Paradox.Silicon sampling exposes identified subpopulations whose patterns may differ from those apparent in the combined distribution.Source: §4 · p. 5

5 Study 1: Free-form Partisan Text

GPT-3’s partisan word lists were judged similarly to human lists in apparent authorship, content, and tone, with similar patterns across ideological subgroups. Both sources conveyed writers’ partisanship above chance, although human lists enabled more accurate identification.

5 Study 1: Free-form Partisan Text

  • First-person participant backstories generated GPT-3 word lists, which evaluators rated and researchers compared using demographic controls and evaluator fixed effects.The study recruited 2,873 evaluators to assess 7,675 human and GPT-3 texts, with three evaluations per text.Source: §5 · pp. 6, 8
  • GPT-3 and humans used similar partisan vocabularies, including a shared set of Democrat descriptors rarely applied to Republicans.Source: §5 · p. 6, Figure 2 · p. 7
  • 61.7% of human lists and 61.2% of GPT-3 lists were judged human-generated, with no significant source difference (p=0.44).The study reports consistent support for Turing Test and Backward Continuity criteria, with initial evidence for Pattern Correspondence across population subsets.Source: §5 · pp. 8, 10
  • 72.3% of human lists and 66.5% of GPT-3 lists mentioned personality traits, with similar evaluations across all five content and tone characteristics.Evaluators classified 39.8% of human lists and 41.0% of GPT-3 lists as extreme; traits were the most prevalent characteristic in both sources.Source: §5 · p. 8, Figure 3 · p. 9
  • 60.1% versus 52.8%: evaluators correctly identified human versus GPT-3 writers’ partisanship, respectively, both above 33% chance but significantly different (p < .001).Both sources contained sentiment cues that enabled evaluators to infer the writer’s partisanship.Source: §5 · pp. 8–10, Figure 3 · p. 9
  • GPT-3 mirrored human patterns in positive and extreme word use across writers’ ideological subgroups, providing evidence for Pattern Correspondence.Source: §5 · p. 8, Figure 3 · p. 9

6 Study 2: Vote Prediction

Conditioned GPT-3 samples closely matched ANES presidential vote patterns in 2012, 2016, and 2020, despite modest aggregate differences and weaker correspondence for pure independents. The 2020 results suggest possible generalization beyond GPT-3’s training period.

6 Study 2: Vote Prediction

  • Researchers conditioned GPT-3 on ANES backstories, recorded Republican-versus-Democratic candidate completion probabilities, and dichotomized predictions at 0.50 to compare with human votes.Conditioning covered race/ethnicity, gender, age, ideology, party identification, political interest, church attendance, political discussion, flag-associated patriotism, and state of residence. Patriotism and state were unavailable for 2020.Source: §6 · p. 10
  • 0.391 versus 0.404 (2012), 0.432 versus 0.477 (2016), and 0.472 versus 0.412 (2020) were GPT-3 versus ANES Republican-vote probabilities.These relatively small aggregate differences indicated mild GPT-3 bias against Romney in 2012, Trump in 2016, and Biden in 2020.Source: §6 · p. 10
  • 0.90 (2012), 0.92 (2016), and 0.94 (2020) were the overall tetrachoric correlations between GPT-3 and ANES binary presidential vote choices.The study interprets these results as supporting Criteria 3 (Forward Continuity) and 4 (Pattern Correspondence).Source: Table 1 · p. 11, §6 · pp. 11–12
  • More than half of subgroup tetrachoric correlations reached 0.90 or higher in each year, but vote-choice correspondence was poor for pure independents, especially in 2020.All other correspondence measures exceeded 0.65, generally approaching 0.8 or 0.9; raw agreement was also high across the three elections.Source: §6 · p. 11, Table 1 · p. 11
  • The 2020 results suggest that appropriate conditioning may extend GPT-3’s fidelity to people and groups outside its original training corpus.GPT-3’s training corpus ended in 2019 [9], making the 2020 election a test beyond that temporal boundary.Source: §6 · pp. 10, 12

7 Study 3: Closed-ended Questions and Complex Correlations in Human Data

Using 2016 ANES data, Study 3 finds that GPT-3’s closed-ended survey responses closely reproduce human patterns of association among demographics, attitudes, and behaviors. The evaluation concerns distribution-level correspondence, not individual response matching.

Study 3: Closed-ended Questions and Complex Correlations in Human Data

  • An interview-style template leverages zero-shot learning [9] to elicit answers from supplied options, conditioning GPT-3 on eleven human survey responses to predict the twelfth.Restricted to the 2016 ANES, this was the paper’s most rigorous evaluation of Criterion 4 (Pattern Correspondence).Source: §7 · p. 12
  • Cramer’s V compares human survey-item associations with associations between human conditioning values and GPT-3 answers, accounting for variation in base rates.The evaluation tests relationships among attitudes, demographics, and reported behaviors rather than only high-level conditional probabilities.Source: §7 · p. 12
  • Stochastic token sampling motivates evaluating overall response distributions in sufficiently large silicon samples rather than expecting exact matches to individual human responses.Different silicon-sample draws can vary; the appendix examines pattern correspondence under different GPT-3 sampling parameters.Source: §7 · p. 13
  • -0.026 was the mean difference between human and GPT-3 Cramer’s V values, with GPT-3 mirroring both stronger and weaker survey-item associations.Correspondence extended beyond aggregate toplines to inter-correlations among personal behaviors, demographic characteristics, and complex attitudes, supporting Criterion 4 (Pattern Correspondence) and algorithmic fidelity.Source: §7 · pp. 12–13, Figure 4 · p. 14

8 Where do we go from here?

Once algorithmic fidelity is established for a topic or domain, silicon samples can inform research before or without human data. The studies illustrate applications in theory development, study design, and analysis planning.

8 Where do we go from here?

  • $29 was Study 1’s GPT-3 cost, illustrating how silicon samples could guide survey questions, experimental treatments, and codebooks with fewer resources than parallel human data collection.GPT-3 alone revealed distinct Republican and Democratic stereotypes, with affect and extremity systematically tied to political beliefs and identity. Stereotypes included issue, group, and predominantly trait references; others could guess individuals’ partisanship from their stereotypes.Source: §8 · p. 15
  • Establishing algorithmic fidelity within a topic or domain enables researchers to use silicon samples before, or without, human data.Researchers can pilot question wording, prioritize measures, identify relationships for closer evaluation, and develop analysis plans before collecting human responses.Source: §8 · pp. 14–15
  • Study 3 could help researchers identify relationships for experimental or observational follow-up and potential confounds for pre-analysis plans in models with causal aspirations.These insights would be available from GPT-3 results without the human baselines used in the paper.Source: §8 · p. 15
  • Study 2’s ablation analysis suggests which variables researchers should include when studying Americans’ voting behavior.The analysis appears in Appendix 3.Source: §8 · p. 15

9 Discussion

Across three studies, GPT-3 shows algorithmic fidelity for demographically varied U.S. public-opinion groups. These findings support potential social science applications while highlighting misuse risks, the need for ethical standards, and the importance of broader validation.

9 Discussion

  • Algorithmic fidelity provides a framework for evaluating language models as aggregate proxies for human cognition, using four criteria and empirical assessment methods.The framework also positions these models as general-purpose windows into human thinking.Source: §9 · p. 15
  • GPT-3 consistently reproduces viewpoints of demographically varied U.S. subpopulations, with evidence supporting all four algorithmic fidelity criteria across data sources, measures, and time points.Study 1 supports the social science Turing Test, Backward Continuity, and Pattern Correspondence; Studies 2 and 3 support Forward Continuity and more granular Pattern Correspondence. Careful conditioning helps address temporality and replicability.Source: §9 · p. 15
  • Language-model methods offer additional possibilities for social science research, with cost identified as a strength and other comparative strengths and weaknesses still anticipated.The studies illustrate possible applications rather than establishing a comprehensive comparison with traditional social science methods.Source: §9 · p. 15
  • High-fidelity models could enable targeted misinformation, manipulation, and fraud [9], motivating clear ethical standards for research and deployment.The authors advocate transparent, research-based, community-accountable exploration to recognize and prevent abuse by private actors.Source: §9 · pp. 15–16
  • The methodology is general purpose, but its demonstrated scope is domain-specific; broader scientific participation is needed to establish algorithmic fidelity’s extent and limitations.The authors invite researchers across social science fields to undertake validation beyond what one research team can accomplish.Source: §9 · p. 16

A General details on GPT-3 usage

The authors use GPT-3’s Davinci model through OpenAI’s API, generating text at temperature 0.7 and estimating probabilities by aggregating equivalent tokens and normalizing.

A General details on GPT-3 usage

  • 175 billion parameters comprise Davinci, the GPT-3 model accessed through OpenAI’s remote API to obtain text completions and corresponding log-probabilities.API inputs include the text prompt, model specification, and temperature.Source: A General details on GPT-3 usage · p. 16
  • Probability estimation uses log-sum-exp conversion, sums probabilities across tokens expressing the same idea, and normalizes the collapsed token sets to sum to 1.For example, differently capitalized tokens for Donald or Trump are combined when estimating the probability of a Trump vote.Source: A General details on GPT-3 usage · p. 16
  • In text-generation Studies 1 and 3, the authors used Completion API defaults except for an untuned temperature of 0.7, encouraging more deterministic answers at the expense of response diversity.Source: A General details on GPT-3 usage · p. 16

B Details on Study 1 … Results screen

Study 1 matched human respondents with demographically conditioned GPT-3 counterparts, then used Lucid evaluators to judge their partisan descriptions and distinguish human from computer authorship. GPT-3 supplied more usable lists, but its responses were longer on average.

B.1 Details on Human and GPT-3 samples

  • 2,107 human respondents with complete background characteristics were matched to 2,107 GPT-3 respondents conditioned on individual backstories.Backstories used fixed-order ideology, partisanship, race, gender, income, and age fragments, with age and income mapped to verbal categories. Each respondent described both parties; GPT-3 generated 128 tokens, followed by regular-expression extraction and light manual correction, retaining the first four entries and leaving missing entries blank.Source: §B.1 · p. 17
  • 4,083 GPT-3 lists and 3,592 human lists survived exclusions; GPT-3 was more compliant but produced longer responses on average.The 7,675 retained lists included descriptive phrases; mean text lengths were 7.78 words for GPT-3 and 4.54 for humans. Both samples’ modal response length was four words.Source: §B.1 · p. 19, Figure 7 · p. 19

B.2 Lucid survey design

  • 2,873 Lucid evaluators each assessed eight randomly assigned lists for content and eight different lists for human-versus-computer authorship.Each list was intended to receive approximately three evaluations per task; 120 lists received only two and seven received four. Excluding these unevenly coded lists did not change the results.Source: §B.2 · p. 19

Welcome Screen

  • The welcome screen warned that lists might contain offensive or nonsensical language, and participants completed a CAPTCHA intended to deter bots.It also previewed judgments about author party, positivity, and extremity.Source: Welcome Screen · p. 20

List Evaluation Task, shown consecutively for 8 different lists

  • Content evaluations measured inferred author party, positivity, extremity, and mentions of personality traits, policy issues, or social groups.The inferred-party choices were Republican, Independent, or Democrat.Source: List Evaluation Task, shown… · pp. 20–21

Turing task introduction screen

  • Turing-task instructions introduced eight further lists, explaining that human and computer responses could appear in varying proportions.Evaluators were asked to guess each response’s origin.Source: Turing task introduction screen · p. 21

Turing Evaluation Task, shown consecutively for 8 different lists; Comments screen; Results screen

B.3 Lucid results analysis · C Details on Study 2 · C.1 Data generation

Lucid analyses used regression models to examine text characteristics and inferred writer partisanship, including assessments stratified by writer ideology. Study 2 conditioned GPT-3 on first-person ANES backstories and estimated candidate voting probabilities without sampling completions.

B.3 Lucid results analysis

  • Lucid analyses used OLS with evaluator fixed effects and standard errors clustered by evaluator and list.Binary outcomes yielded linear probability models, whose results did not significantly differ under logit; estimation used the fixest R package. Each evaluator assessed eight lists, and each list was evaluated three times.Source: §B.3 · p. 22
  • All Lucid models controlled for text source and original writers’ gender, ethnicity, income, age, and party identification.Ethnicity used separate Hispanic/non-Hispanic and White/Other indicators; income was rescaled from an 11-point measure to 0–1.Source: §B.3 · pp. 22–23
  • Predicted values fixed demographics to Female, Not Hispanic, White, and Democrat, with income and age at their means.The “Percent correctly predicted” model additionally controlled for list word count, fixed at its mean for predictions.Source: §B.3 · p. 23
  • Ideology-stratified models assessed positive and extreme ratings for texts describing Republicans and Democrats.Tables 4–7 underlie Figure 4A and subset list writers using the seven-point ideological scale.Source: §B.3 · p. 23, Table 4 · p. 26, Table 5 · p. 27
  • Lucid regression tables cover five evaluated text characteristics and participants’ accuracy in guessing writers’ partisanship.Tables 2 and 3 provide the full model results underlying the corresponding predictions in Figure 4B.Source: §B.3 · p. 23, Table 2 · p. 24, Table 3 · p. 25

C.1 Data generation

  • Study 2 constructed silicon samples from the 2012, 2016, and 2020 ANES datasets using respondents’ demographic, political, religious, and social characteristics.Conditioning included race/ethnicity, gender, age, ideology, party, political interest, church attendance, political discussion, flag patriotism, and state; flag patriotism was not asked and state was unreleased for 2020. ANES self-reported votes supplied the voting measure.Source: §C.1 · p. 30
  • Backstory templates translated ANES responses into closely matched first-person declarations, omitting fragments whenever a respondent’s corresponding variable was missing.Age and state values were inserted directly; other responses mapped to short strings before all available fragments were concatenated.Source: §C.1 · p. 30
  • Candidate voting probabilities came from successor-token probabilities rather than sampled completions, making temperature and sampling strategy irrelevant.Year-specific token sets grouped candidate-name and political-label variants for Romney/Obama (2012), Trump/Clinton (2016), and Trump/Biden (2020), including lexical variations. Other tokens were ignored; sets targeted common cases and were not optimized to improve results.Source: §C.1 · p. 30

C.2 Data analysis … D Details on Study 3

Vote-prediction analyses compare GPT-3 with human ANES responses using agreement measures that can become unreliable when vote choices barely vary. Backstory ablations show contributions from multiple attributes, while model comparisons show 6B-parameter GPT-Neo rivaling 175B-parameter GPT-3.

C.2 Data analysis

  • 0.50 is the cutoff used to convert GPT-3 Republican-vote probabilities into binary choices for comparison with ANES respondents’ reported votes.Probabilities above the cutoff indicate Republican votes; those below indicate Democratic votes. No predicted probability equaled 0.50.Source: §C.2 · pp. 30–31
  • Agreement is evaluated using tetrachoric correlation, Cohen’s Kappa, intraclass correlation coefficient, and proportion agreement, extending the main text’s two-metric presentation.Tetrachoric correlation assumes underlying continuous normal variables, while Cohen’s Kappa adjusts for chance agreement. The reported ICC is the lowest of three averaged variants and changes little when GPT-3 probabilities replace binary votes.Source: §C.2 · p. 31
  • Near-complete agreement among Strong Democrats can coexist with low correlations and Cohen’s Kappa because almost no vote variation makes these measures unreliable.The authors include descriptive, chance-unadjusted proportion agreement because other measures perform poorly when more than 95 percent of observations occupy one frequency-table quadrant.Source: §C.2 · pp. 31–32

C.3 Ablation analysis

  • 8 demographic backstory elements together outperform any single element after Party and Ideology are removed, and no single element accounts for all predictive power.The ANES 2016 ablation systematically removed one or two elements from the 10-element template and tested single-element backstories.Source: §C.3 · p. 35, Figure 8 · p. 35
  • Party predicts vote choice better than Ideology, while adding State or Political Interest mildly hurts performance.GPT-3 can use either Party or Ideology to predict vote choice.Source: §C.3 · p. 35, Figure 8 · p. 35
  • The backstory template and its 10 selected elements were an unoptimized first attempt, leaving their optimization as a potential route to improved results.The authors suggest optimizing templates and backstory elements before future experiments.Source: §C.3 · p. 35

C.4 Model comparison

  • 6B-parameter GPT-Neo rivals the much larger, costlier 175B-parameter GPT-3 on vote prediction.The comparison tested five model families. Performance depends strongly on training-corpus details as well as parameter count, with the largest models generally performing well.Source: §C.4 · pp. 35–36, Figure 9 · p. 36

D.1 Data generation … E Cost Analysis

Study 3 compared human and GPT-3 response distributions using interview-conditioned samples and Cramer’s V. Entirely synthetic calculations preserved similar relationship patterns, while temperature 0.7 minimized discrepancies among tested settings.

D.1 Data generation

  • More than 50,000 conditioning texts across 12 variables and 4,270 ANES respondents elicited held-out interview answers from GPT-3.Each interview supplied a real respondent’s other answers, with the unanswered target placed last; GPT-3 sampled 5 tokens at temperature 0.7. Lowercased, whitespace-trimmed completions were matched exactly to allowable ANES answers, with unmatched responses coded as missing.Source: §D.1 · pp. 36, 38

D.2 Data analysis

  • Distributional correspondence, rather than individual prediction accuracy, was evaluated using Cramer’s V across all pairs of 22 human and synthetic variables.Turnout and vote choice were combined, yielding eleven variables per data source. Cramer’s V accommodates categorical data and ranges from 0 to 1, with higher values indicating stronger relationships.Source: §D.1 · p. 36, §D.2 · p. 43

D.2.2 Descriptive Statistics

  • Descriptive statistics for Study 3 variables were reported separately for ANES humans and the GPT-3 silicon sample.The supplied Table 16 caption identifies this comparison but provides no numerical values.Source: §D.2.2 · p. 43, Table 16 · p. 48

D.2.1 Missing Data

  • For each question item considered separately, GPT-3 produced valid and compliant answers in more than three-quarters of cases, with compliance rates above 99 percent on three items.Missing-data rates varied substantially for both humans and GPT-3. Analyses retained 1,782 of 4,270 cases with valid responses for all ANES and GPT-3 variables.Source: §D.2.1 · p. 43

D.3.2 GPT-3 Temperature Variation

  • 0.001 increased error, overstated relationships, and classified every respondent as white, preventing race/ethnicity Cramer’s V calculations.Race/ethnicity was therefore excluded from calculations at this temperature. Lower temperatures reduced invalid completions, while higher temperatures admitted more invalid responses; mid-range temperatures appeared to balance validity and completeness.Source: §D.3.2 · pp. 47, 49, 50

D.3.1 Completely Synthetic Data

  • Entirely synthetic Cramer’s V calculations retained relationship patterns highly similar to those using ANES inputs, despite additional estimation noise.The main-text comparison instead paired ANES inputs with GPT-3 outputs, keeping one side of the relationship calculation identical across human and model comparisons.Source: §D.3.1 · pp. 43, 47, Figure 11 · p. 49

D.3.2 GPT-3 Temperature Variation

  • 0.7 yielded the lowest human–GPT-3 Cramer’s V differences among tested temperatures, with minimal differences from 1.0.The tested temperatures were 0.001, 0.7, and 1.0. Each temperature query was run once, without selecting best-fitting results from repeated runs.Source: §D.3.2 · p. 47, Table 17 · p. 49

E Cost Analysis

  • $29, $75, and $1,428 were the reported final-run costs for Studies 1, 2, and 3, respectively.Study 3 used prompts averaging 458 tokens and generated at most 5 tokens per query, costing $119 per experiment across 12 experiments. These totals exclude additional experimental runs.Source: E Cost Analysis · p. 50

Related papers

Loading 2209.06899v1…