Source-linked AI summary

LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals

Joon Sung Park, Carolyn Q. Zou, Jonne Kamphorst, Niles Egan, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Percy Liang, Robb Willer, Michael S. Bernstein

arXiv:2411.10109v3cs.AIcs.HCcs.LG

TL;DR

Individual simulations have often relied on demographic or persona descriptions; this study tests LLM agents grounded in people’s self-reports. On GSS items, self-report agents reached 82–86% of participants’ two-week test-retest consistency and also predicted personality and experimental effects.

  • Problem

    Individual simulations have often relied on demographic or persona descriptions, leaving the accuracy of self-report-grounded agents across outcomes an open question.

    Source: Discussion · p. 10
  • Method

    The authors built LLM agents from interviews, surveys, or both, and evaluated how well each reproduced the attitudes and behaviors of its source individual.

    Source: Using Individuals’ Self-Reports to… · p. 5, Discussion · p. 10
  • Results

    82–86%: on GSS items, self-report agents reached this share of participants’ two-week test-retest consistency and also predicted personality and tracked effects across five experimental replications.

    Source: Discussion · p. 10
  • Takeaways & Limitations

    The findings indicate that accurate individual simulation depends more on the depth and reliability of source data than on model scale or synthetic persona engineering.

    Source: Discussion · p. 11

    The evaluation establishes individual-level accuracy, but not whether agents reproduce population-level correlations among attitudes.

    Source: Discussion · p. 11

Abstract

from arXiv · show

Machine learning can predict human behavior well when substantial structured data are available for well-defined outcomes. Such models are typically outcome-specific, however, requiring training data for each target outcome, limiting their applicability to new domains. We test whether large language models (LLMs) can relax these requirements by using self-report data to build attitudinal and behavioral simulations, or "generative agents," that can predict responses across outcomes without outcome-specific training data. Using data from a diverse national sample of 1,052 Americans, we built agents from (i) two-hour, semi-structured interviews elicited using the American Voices Project interview schedule, (ii) structured surveys including General Social Survey items and the Big Five personality inventory, or (iii) both sources combined. On held-out General Social Survey items, interview-only, survey-only, and combined agents achieved accuracies equal to 83%, 82%, and 86% of participants' own two-week test-retest consistency benchmark, respectively, compared with 74% for demographics-only agents. Combining interviews and surveys produced the highest accuracy, though gains over either source alone were modest, suggesting that predictive benefits from data begin to asymptote once the model has observed sufficient evidence within a domain. We find that these agents also predict personality traits, economic-game behavior, and experimental responses, while reducing accuracy disparities across racial and ideological groups relative to demographics-only agents. Together, these results show that LLM agents grounded in qualitative or quantitative self-reports can support general-purpose simulation of individuals across outcomes, without requiring task-specific training data.

Using Individuals’ Self-Reports to Create Generative Agents … Replication materials

This section evaluates agents built from interviews, surveys, or both across consistency and other outcomes, and documents the project’s contributions and replication materials.

Using Individuals’ Self-Reports to Create Generative Agents

  • The study recruited 1,052 participants for two-hour AI-led interviews and surveys, then prompted GPT-4o agents with interview transcripts, survey responses, or both.The AI interviewer used a broad American Voices Project protocol with adaptive follow-ups; agents also used expert-generated reflections. The study provides aggregated survey responses and open-source agent and interviewer code.Source: Using Individuals’ Self-Reports to… · pp. 3–5, Materials and Methods Summary · p. 12

Materials and Methods Summary

Predicting Individuals’ Attitudes and Behaviors

  • On GSS items, normalized accuracy was 0.83 for interview agents, 0.82 for survey agents, and 0.86 for combined agents, versus 0.74 for demographic agents.All three self-report agent types significantly outperformed demographic and persona agents; combined agents also significantly outperformed interview-only and survey-only agents.Source: Predicting Individuals’ Attitudes and… · pp. 6–7, Figure 2 · p. 6
  • For Big Five prediction, interview agents achieved normalized correlation 0.80 and combined agents 0.77; economic-game differences were not significant.Survey-only agents had normalized correlation 0.65 on Big Five, while interview agents achieved 0.66 on economic games; no economic-game pairwise contrast was significant.Source: Figure 2 · p. 6, Predicting Individuals’ Attitudes and… · p. 7

Predicting Experimental Replications

  • Across five experimental replications, agent-estimated effect sizes were larger than and highly correlated with participants’ estimates (r = 0.91 to r = 0.99), but the design was underpowered.The study used five experiments; its limited number of studies constrains strong conclusions about average treatment-effect prediction.Source: Predicting Experimental Replications · p. 8

Richer Background Data Reduces Bias in Generative Agent Accuracy

  • Self-report agents generally reduced ideological and racial DPD relative to demographic agents, while gender DPD remained low across agent types and tasks.For ideological GSS DPD, demographic agents had 13.75% versus 6.22% for survey agents; for racial Big Five DPD, values were 0.173 versus 0.081 for demographic and combined agents, respectively.Source: Richer Background Data Reduces Bias in… · pp. 9–10, Figure 3 · p. 9

Discussion

  • The authors identify direct retrieval and inference as processes behind interview-agent predictions, and suggest that combining sources offers diminishing returns once self-reports describe a person sufficiently.Interview-only agents performed nearly as well as combined agents; the authors note that evaluation focused on individual accuracy, not sample-level correlations, and that replication findings were based on five studies.Source: Discussion · pp. 10–11
  • The authors argue that simulation quality depends on the depth and reliability of the underlying self-report data, rather than only on model scale or persona design.They caution that accurate individual predictions do not establish recovery of population-level attitude correlations, and that transfer to other model families and linguistic or cultural contexts remains untested.Source: Discussion · p. 11

Contribution Statement

  • The authors divided work across project conception, technology and methodology development, data collection and analysis, writing, manuscript review, and research supervision.J.S.P. led the investigation; J.S.P., M.S.B., and P.L. conceptualized the project and developed the technology.Source: Contribution Statement · p. 13

Replication materials

  • Replication files are available in the project’s OSF repository.The passage provides the repository URL: https://osf.io/t6g7k/files/osfstorage.Source: Replication materials · p. 13

Supplementary Materials for … 2. Creating the AI Interviewer Agent

The supplement details how the researchers built a 1,052-person U.S. agent bank from interviews, surveys, and behavioral experiments, and how they recruited, studied, and informed participants. It also describes an AI interviewer designed to conduct consistent semi-structured interviews at scale.

Supplementary Materials for

  • The supplement covers agent-bank construction, interviewer design, agent architecture, surveys and experimental constructs, evaluation, supplementary results, model variants, and tables.It also lists sections on GSS pre-analysis-plan deviations and why interview-based generative agents work.Source: Supplementary Materials for · p. 19

1. Constructing the Agent Bank

  • 1,052 participants’ interviews, surveys, and behavioral experiments formed a knowledge base for conditioning agents and assessing their fidelity.The researchers recruited a stratified U.S. sample and used interview transcripts and survey responses to condition agent behavior.Source: §1 · p. 20

Data Collection Procedure

  • Participants completed a two-hour AI interview, then the GSS, BFI-44, five economic games, and five behavioral experiments; question and study orders were randomized where specified.The GSS followed its documented sequence; the BFI-44 questions and subcomponents of the games and replication studies were randomized for counterbalancing.Source: Data Collection Procedure · p. 21
  • Two weeks later, participants repeated the surveys and experiments, but not the interview, to measure response consistency over time.The study platform made the self-consistency module available two weeks after participants completed the earlier modules.Source: Figure 1 · p. 20, Data Collection Procedure · p. 21

Recruitment and Demographics

  • The researchers recruited 1,300 people anticipating about 20% retake attrition, then retained 1,052 participants after follow-up participation exceeded expectations.The target was 1,000 participants completing all study components, with sample size chosen to provide statistical power for the five behavioral experiments.Source: Recruitment and Demographics · p. 21
  • Recruitment aimed for national representation across age, race, gender, region, education, and political ideology; participants had to be U.S. residents aged 18 or older.Participants were recruited through Bovitz; the mean age was 47.55 years, and 593 identified as female, 459 as male, and 833 as white/Caucasian, with ethnicity responses allowing multiple selections.Source: Recruitment and Demographics · pp. 21–22
  • Consent procedures disclosed that interview data could remain identifiable, agents might be shared with academic researchers, and participants could withdraw and request data removal.Participants were informed of potential privacy risks from future model capabilities; removal requests would be honored for 25 years after study completion to the best of the researchers’ ability.Source: Recruitment and Demographics · p. 22

2. Creating the AI Interviewer Agent

  • The AI interviewer used a generative-agent architecture to standardize semi-structured interviews and scale data collection to over 1,000 participants.It used participants’ utterances and the interview script to generate follow-up questions or advance to the next module, with reflection summarizing the interview and inferring insights.Source: §2 · p. 22, Figure 2 · p. 23

AI Interviewer Agent Architecture

The AI interviewer combines a researcher-controlled script with language-model decisions about whether to ask a follow-up or move on. Reflection notes preserve relevant context, while an abbreviated American Voices Project script supports two-hour interviews.

  • The agent asks each scripted question verbatim, then uses recent conversation and the question’s time limit to choose a relevant follow-up or advance to the next question.The script is an ordered list of questions with a time allocation for each; the language model decides whether a response needs elaboration or is complete.Source: AI Interviewer Agent Architecture · p. 23
  • To avoid losing relevant context in long prompts, a reflection module summarizes the conversation into participant notes, which are combined with the transcript’s most recent 5,000 characters.The notes capture inferences such as birthplace and whether a participant is outdoorsy; the design addresses language models’ difficulty considering all information in very long prompts.Source: AI Interviewer Agent Architecture · p. 24
  • Researchers aimed to use a script designed independently of the evaluation metrics, reducing the risk of tailoring interview content to the study’s specific surveys and experiments.The other stated script-design goal was to elicit meaningful answers that provide a nuanced portrait of each modeled individual.Source: AI Interviewer Agent Architecture · p. 24
  • The study used an abbreviated American Voices Project script covering life experiences and social, political, and value-related topics, omitting overly specific questions to fit two-hour interviews.The script was selected for broad coverage of lived experiences, while detailed spending questions and COVID-era life-pattern questions were among the omitted material.Source: AI Interviewer Agent Architecture · pp. 24–25

Implementation

The implementation combined a voice-to-voice AI interviewer with scripted, dynamically generated questions and reflections, producing two-hour interviews with substantial participant responses.

Implementation

  • A web application enabled voice-to-voice interviews, disclosed the interviewer was AI, and calibrated audio before the interview.The interface used an interviewer avatar and progress indicator, with responses generally returned within 4 seconds to maintain interview flow.Source: Implementation · pp. 25–26
  • An ordered JSON script with manually set question time limits guided interviews toward a two-hour duration; Audio (69) spoke prompts, Whisper (70) transcribed responses, and GPT-4o (71) used transcripts to guide conversational moves.Source: Implementation · p. 26
  • GPT-4o generated concise reflections and follow-up questions using participant notes, the current transcript, and the interview objective.The prompt assessed progress toward the objective, proposed on-topic follow-ups, and instructed the interviewer not to press participants who explicitly refused to answer for privacy reasons.Source: Implementation · pp. 26–27
  • The agent spoke 5,372.59 words (std=2,406.12), asked 81.71 follow-up questions from 99 scripted questions, and participants responded with 6,491.19 words (std=2,540.56).Source: Implementation · p. 27

Development Process and Evaluation

Pilot testing found that generative agents built from AI-interviewer transcripts matched those built from human-interviewer transcripts when predicting GSS responses. The research team also judged the AI interviewer’s transcripts comparable in quality to expert human transcripts.

  • The research team judged AI-interviewer transcripts comparable in quality to transcripts produced by expert human interviewers.Team members trained in the social sciences assessed transcripts as training data for agents on the same attitudinal and behavioral tasks used in the main results.Source: Development Process and Evaluation · p. 27
  • The team iteratively refined and assessed the AI interviewer using three waves of three pilot interviews conducted from August to November 2023.Across the pilots, 10 interviews were conducted by humans and 25 by the AI interviewer.Source: Development Process and Evaluation · p. 27
  • Generative agents built from AI-interviewer transcripts matched agents built from human-interviewer transcripts when predicting GSS responses.The AI interviewer could sustain smooth conversation, ask relevant follow-up questions after terse responses, and respond sympathetically to difficult experiences.Source: Development Process and Evaluation · p. 28

3. Generative Agent Architecture

The agents use participants’ self-reports as memory for an LLM that predicts how each participant might respond. Expert reflections add domain-specific inferences about information embedded in those reports.

3. Generative Agent Architecture

  • Agents combine participants’ self-report data and expert reflections in memory, then use a language model to predict how the source participant might respond to a question.Unlike agents with manually specified behavior for particular scenarios, these agents use the memories to generate responses across circumstances.Source: Figure 4 · p. 29, §3 · p. 29
  • For every participant, GPT-4o was prompted with self-report data to generate four expert-specific sets of up to 20 observations or reflections each, adopting psychologist, behavioral economist, political scientist, and demographer personas.Source: §3 · pp. 29–30
  • For each question, the model selects the most relevant expert, retrieves that expert’s reflections, and uses them with the participant’s self-report data to generate a prediction.The expert persona guides reflections intended to surface insights that may not be explicitly stated in the self-reports.Source: §3 · pp. 29–30

Generating a Prediction With Generative Agents

Generative agents used chain-of-thought prompts to predict survey responses from self-reports. Prompts structured reasoning differently for categorical and numerical answers, and experimental stimuli and prior actions were appended when context needed to be maintained.

Generating a Prediction With Generative Agents

  • Categorical-response prompts used four steps: interpret each option, consider why the participant might choose it, reason about the best prediction, and predict the response.The prompt asked agents to base their prediction on the interview and their reasoning.Source: Generating a Prediction With… · p. 31
  • Numerical-response prompts interpreted each range endpoint and reasoned about which option best predicted the participant’s response.For experimental tasks requiring context, agents received the stimuli and prior actions appended in natural language to their self-report data and reflections.Source: Generating a Prediction With… · pp. 31–32

4. Surveys and Experimental Constructs

The section describes constructs used to assess whether generative agents predict participants’ attitudes and behaviors across surveys and experimental settings. For constructs included in agent inputs, the predicted GSS question or the full Big Five question block was excluded from the input.

Evaluation procedure

  • For constructs also used as agent inputs, the predicted GSS question or all questions in the same Big Five block were removed from the input.This prevented the specific test question or its block from being present in the agents’ input.Source: §4 · p. 32

The General Social Survey

  • The GSS evaluation included 177 categorical and 6 numerical core questions, after excluding conditional, free-form, and over-25-option questions.Questions were administered online through a custom-built Qualtrics survey.Source: §4 · p. 32

Big Five Personality Traits

  • The study assessed Big Five traits using the 44-item BFI-44 and calculated trait scores using the original work’s aggregation methods.The five dimensions are openness, conscientiousness, extraversion, agreeableness, and neuroticism.Source: §4 · p. 33

Behavioral Economic Games

The study used five incentivized economic games to assess social and behavioral preferences, with participants’ actual Dictator Game earnings determining their bonus.

Behavioral Economic Games

  • The study administered five prominent economic games designed to reveal altruism, trust, reciprocity, cooperation, and competition under real financial incentives (74).Participants received a bonus based on their choices in the games.Source: Behavioral Economic Games · p. 33
  • The Dictator Game measured altruism through a $5 split, while the Trust Game measured trust and reciprocity through sending and returning tripled money (29, 30).In the Trust Game, the First Mover chose how much of a $3 endowment to send; the Second Mover chose how much of the tripled amount to return.Source: Behavioral Economic Games · p. 33
  • The Public Goods Game measured cooperation through contributions from a $4 endowment to a doubled, equally shared pool, while the Prisoner’s Dilemma tested cooperation versus defection (31, 32).In the Prisoner’s Dilemma, mutual cooperation paid $6 each; unilateral defection paid $8 to the defector and $2 to the cooperator, while mutual defection paid $4 each.Source: Behavioral Economic Games · p. 34
  • Participants were told one game would be randomly selected for bonuses of up to $8 to $10, and actual Dictator Game earnings determined payment (29).The Dictator Game had been selected for bonus calculation before the study; participants were randomly paired afterward.Source: Behavioral Economic Games · p. 34

Replication Studies of Experimental Treatment Effects

This section tests whether generative agents reproduce aggregate human responses to interventions by applying them to five previously replicated experiments selected for natural-language presentation and feasible sample sizes.

Replication study sample

  • The sample drew on Camerer et al.’s replicated studies and retained five experiments that could be presented to language models and whose effects were expected to be observable with 1,000 or fewer participants.The selection was intended to avoid favoring studies likely to support agent predictions and to use effects independently replicated in preregistered studies.Source: Replication Studies of Experimental… · pp. 34–35
  • In Ames & Fiske, 2015 (33), participants exposed to intentional rather than unintentional medication-switching were more likely to choose tasks involving blame and punishment.The vignette concerned a nursing-home employee who switched patients’ medications.Source: Replication Studies of Experimental… · p. 35
  • In Cooney et al., 2016 (34), participants expected to feel less upset about a withheld bonus when its allocation was decided fairly by coin flip rather than personal choice.The study used a modified dictator game in which participants imagined themselves as receivers.Source: Replication Studies of Experimental… · p. 35
  • In Halevy & Halali, 2015 (35), participants recalling intervening in a friends’ conflict rated intervention as more beneficial and less costly than those recalling not intervening.Source: Replication Studies of Experimental… · p. 35
  • In Rai et al., 2017 (36), participants were more willing to harm a stranger for money when the person was described in dehumanized rather than humanized terms.Source: Replication Studies of Experimental… · p. 35
  • In Schilke et al., 2015 (37), participants in the high-power condition were less willing than those in the low-power condition to offer a potential client a free service sample.Source: Replication Studies of Experimental… · p. 35

5. Evaluation Methods

The evaluation compares agent predictions with participants’ responses and two-week self-consistency, using response-appropriate metrics and normalized accuracy to assess performance across constructs. The main analysis aggregates accuracy across individuals, with construct-level analyses also reported.

Predicting Individuals’ Attitudes and Behaviors

  • Normalized accuracy benchmarks agents’ prediction accuracy against participants’ own two-week internal consistency, with 1.0 indicating equal accuracy.The evaluation uses first-phase responses to score agent predictions and second-phase responses to measure participants’ internal consistency.Source: §5 · p. 36
  • The evaluation reports accuracy for categorical responses, MAE for numerical responses, and Pearson correlation across constructs, primarily averaging individual-level results.Construct-level analyses average predictive accuracy by question, dimension, or game; correlations are averaged using Fisher’s z-transformation and its inverse.Source: §5 · p. 37

The General Social Survey

  • The GSS subset contains 183 questions—177 categorical or ordinal and 6 numerical—with accuracy and correlation reported for the former and MAE and correlation for the latter.Categorical-ordinal accuracy requires exact response matches; for correlation, categorical answers are binary-coded with adjusted weights, while ordinal answers are evenly scaled from 0 to 1.Source: §5 · pp. 37–38
  • Numerical GSS responses are normalized to 0–1 using historical question ranges before MAE calculation; normalized accuracy is unavailable for MAE when internal consistency is zero.When first-wave GSS items are inputs for second-wave predictions, the target question is excluded from agent input.Source: §5 · p. 38

BFI-44

  • BFI-44 dimension scores average reverse-coded responses to 44 items, and the five scales are evaluated using MAE and correlation.Normalized accuracy is reported only for correlation; when predicting second-wave BFI responses, agents exclude the target question and other questions in its personality grouping from inputs.Source: §5 · p. 38

Economic games

  • For the five economic games, responses are scaled from 0 to 1 using each game’s offered minimum and maximum, then evaluated with MAE and correlation.Normalized accuracy is reported only for correlation.Source: §5 · p. 38

Replicating Experiments With Generative Agents

The section tests whether generative agents reproduce sample average treatment effects by comparing simulated predictions with a direct replication run using this study’s human participants.

Replicating Experiments With Generative Agents

  • Rerunning five experiments with this study’s human participants helps distinguish agent failures from failures of the original studies to replicate in this sample.The authors use this direct replication rather than published results because their sample may differ and some studies may not replicate.Source: Replicating Experiments With… · pp. 38–39
  • Evaluation asks whether effect direction and significance replicate and whether simulated and human effect sizes correlate, with effects expressed as Cohen’s d.The authors use each original study’s statistical method to derive p-values and calculate Pearson correlations between human and simulated effect sizes.Source: Replicating Experiments With… · p. 39
  • Ames & Fiske (2015) (33) tested blame-task choices by χ2; Cooney et al. (2016) (34) tested fairness feelings × allocation procedure by F-test; Halevy & Halali (2015) (35) compared intervention benefits by t-test.The tests were a chi-square test of equal proportions, an Outcome × Procedure interaction F-test in a 2x2 ANOVA, and an independent-samples t-test, respectively.Source: Replicating Experiments With… · p. 39
  • Rai et al. (2017) (36) used an independent-samples t-test on willingness to harm a humanized versus dehumanized stranger under instrumental motives; Schilke et al. (2015) (37) used χ2 to compare trust by structural power.Schilke et al. compared participants with high versus low structural power.Source: Replicating Experiments With… · p. 39

Interviews Improve Agents’ Prediction Accuracy

This section describes comparisons of self-report-based agents with demographic and persona baselines, then outlines exploratory analyses of data sources, interview content, and prediction disparities across demographic groups. It specifies evaluation designs but reports no comparative results in the supplied passages.

Main analysis

  • The main analysis compared interview-, survey-, and combined-data agents with demographic descriptors modeled on prior work and participant-written personas.Demographic descriptors reconstructed GSS responses as first-person accounts of ideology, race, gender, and age, following an approach similar to Argyle et al. (34).Source: Interviews Improve Agents’ Prediction… · p. 40
  • Performance differences among interview, demographic, persona, and survey agents were assessed using ANOVA with post hoc Tukey tests.Personas were short self-descriptions participants wrote for a stranger, covering personal background, personality, and demographics.Source: Interviews Improve Agents’ Prediction… · p. 40

Exploratory robustness analysis

  • Survey agents used wave 1 GSS and Big Five responses to predict wave 2 answers, with GSS outcomes held out individually or by module and Big Five traits by block.Holding out only an outcome left related GSS questions available; holding out its full module reduced access to similar questions.Source: Interviews Improve Agents’ Prediction… · p. 41
  • Exploratory variants added interviews or economic-game data to surveys, converted interviews into factual summaries, or randomly removed 0–80% of interview question-response pairs.These comparisons tested added information from interviews and games, the role of linguistic features, and how much interview content was needed.Source: Interviews Improve Agents’ Prediction… · pp. 41–42

Demographic Bias in Agent Predictions

  • The demographic-bias analysis compared prediction-rate gaps across GSS, Big Five, and economic-game outcomes for five agent conditions using Demographic Parity Difference.Lower DPD indicates more equitable predictions; separate regressions also tested how demographic variables related to predictive performance.Source: Interviews Improve Agents’ Prediction… · p. 42

6. Supplementary Results

Supplementary results show strong numerical GSS prediction and robustness across agent inputs. Regression analyses find demographic performance discrepancies were relatively rare, with notable exceptions for several political and sexual-orientation groups.

Numerical GSS Prediction Results

  • Numerical GSS predictions averaged correlation r = 0.97 (std = 0.82) and mean absolute error (MAE) of 0.14 (std = 0.15).Source: §6 · p. 43

Exploratory Robustness Analysis Results

  • Maximal agents achieved normalized accuracy of 0.87 (std = 0.13) on the GSS, slightly exceeding interview-based agents.Source: §6 · p. 43
  • Survey agents reached normalized accuracy of 0.82 (std = 0.18) excluding same-category pairs, versus 0.77 (std = 0.12) excluding the full GSS module.Source: §6 · p. 43
  • Adding interview transcripts to survey agents with only the GSS outcome question removed raised normalized accuracy to 0.85 (std = 0.19).Source: §6 · p. 43
  • Interview-agent normalized accuracy fell from 0.82 (std = 0.13) with no utterances removed to 0.79 (std = 0.12) after excluding 80%, yet short interviews still outperformed survey-and-experiment agents.Source: §6 · pp. 43–44
  • Summary agents scored normalized accuracy of 0.81 (std = 0.13), slightly below interview agents at 0.82 (std = 0.13).The passage suggests that much interview-agent performance came from interview information rather than low-level language cues.Source: §6 · p. 43

Demographic Bias in Agent Predictions

  • Significant demographic performance discrepancies were relatively rare, but predictions were stronger for strong liberals, strong Democrats, and non-heterosexual/straight participants than comparison groups.Source: §6 · p. 44

7. Deviations from the Pre-Analysis-Plan for the GSS Outcome Questions

The authors removed GSS outcome questions that overlapped with AVP interview questions to reduce the risk of inflated accuracy from direct answer retrieval. Including those questions raised accuracy by 0.02 points; the revised 27-question exclusion set was used in the main paper.

7. Deviations from the Pre-Analysis-Plan for the GSS Outcome Questions

  • To limit answer-retrieval inflation, GPT-4.1 classified 54,694 GSS–interview question pairs for synonymy, and researchers manually confirmed the model-flagged overlaps.Source: §7 · pp. 44–45
  • The final exclusion set contained 27 questions: 15 about respondents’ demographics, five about parental education or employment, and the rest about topics including spouses and partisanship.Source: §7 · p. 45
  • Manual review and a coder’s assessment of 500 pairs found three additional missed questions, while the estimated false-negative rate’s extreme upper bound remained below 1%.The coder confirmed 21 of the 24 model-flagged questions; researchers also manually checked the flagged set for false positives.Source: §7 · p. 45
  • Accuracy was 0.02 points higher when including GSS questions identical to AVP questions; Figure 5 reports the pre-registered models using the full outcome set.Source: §7 · p. 45, Figure 5 · p. 46

8. Why do interview-based generative agents work?

Interview-based agents draw accuracy gains from retrieving answers directly from interview transcripts and inferring answers from other interview content. Their advantage persists after questions susceptible to both mechanisms are excluded, suggesting further mechanisms remain unidentified.

Mechanism 1: Direct retrieval of answers

  • After 80 likely-retrievable questions were removed, interview agents retained 0.79 accuracy versus 0.71 for demographic agents; removing such questions progressively reduced only interview-agent accuracy.The removed questions were ranked from most to least likely to have a verbatim answer in the transcript.Source: Figure 7 · p. 50, §8 · p. 51

Mechanism 2: Reasoning based on other answers

  • Inference flagged approximately three times as many questions as direct retrieval, and interview-agent accuracy progressively fell as likely inferable questions were removed, unlike demographic and persona agents.The inference test used the same 59 agents and GSS questions as the direct-retrieval test.Source: §8 · p. 51, Figure 8 · p. 52
  • After removing 30 direct-retrieval questions, interview agents led demographic agents by 0.095 points; removing 30 additional inference questions narrowed the gap to 0.080.This comparison isolates inference beyond its overlap with direct retrieval.Source: §8 · p. 53

Overall interpretation

  • Interview agents likely benefit from combining rich interview narratives with LLM retrieval and inference, yet they still outperform alternatives after questions susceptible to both mechanisms are excluded.Interviews capture reasoning behind attitudes, long-tail experiences, and narratives of identity and history; additional mechanisms remain to be identified.Source: §8 · p. 53

9. Results using different models

On the GSS task, more advanced GPT models yielded small accuracy gains, cheaper variants traded some performance for lower cost, and a newer GPT-4o instance showed minimal drift. Fine-tuning did not improve performance.

Results using different models

  • GPT-5, GPT-4.1, GPT-o1, and GPT-o3 each scored 0.67 versus GPT-4o’s 0.66; GPT-o3-mini and GPT-o4-mini each scored 0.64.The advanced-model advantage was small and statistically indistinguishable with 50 agents. GPT-4o-mini scored 0.60 and GPT-o1-mini 0.62.Source: Figure 9 · p. 54, §9 · pp. 54–55
  • GPT-4o’s 2025 run scored 0.64 versus 0.66 for the 2024 instance, a 0.02-point decrease.The paper describes this as minimal model drift.Source: §9 · p. 55

Fine-tuning

  • Fine-tuning achieved normalized accuracy of 0.79 (std = 0.07) and normalized correlation of 0.81 (std = 0.13), below both non-fine-tuned comparisons.With the main-paper prompt, non-fine-tuned GPT-4o achieved normalized accuracy of 0.84 (std = 0.11) and normalized correlation of 0.87 (std = 0.15); with the fine-tuning prompt, it achieved 0.82 (std = 0.08) and 0.85 (std = 0.13), respectively.Source: §9 · p. 55

10. Supplementary Tables

The supplementary tables document the sample and interview materials and report agents’ predictive performance across General Social Survey items, Big Five traits, economic games, and experimental outcomes. They also provide construct- and item-level analyses, comparisons across agent specifications and demographic groups, regression notes, and robustness checks.

Demographic Distribution

  • The 1,052 participants collectively represent a stratified U.S. demographic sample across age, gender, race, region, education, and political identity.Participants could select more than one ethnicity.Source: Table 1 · p. 56

General Social Survey, Big Five Personality Traits, and Economic Games: Agent Architecture Comparison

  • For GSS accuracy, interview-, survey-, and Composite agents outperformed both demographic- and persona-based agents; for Big Five correlations, interview and Composite agents outperformed both baselines, but survey agents did not.Economic-game correlations showed no significant differences among specifications (F(4, 5255) = 1.63, p = 0.16).Source: Table 2 · p. 58

General Social Survey: Construct-Level Analysis

  • For natheal/y, the cited entry reports four item-specific values: 0.7, 0.91, 0.36, and 0.73. These are the values shown for that item in the supplied passage.The other supplied rows show natspac/y values of 0.33, 0.45, 0.17, and 0.25, and natenvir/y values of 0.6, 0.74, 0.51, and 0.69.Source: §10 · p. 59

Supplementary Tables

  • Item-level entries include shorthand labels such as librac/y, spkcom/y, and colcom/y, each followed by four numeric values.Source: §10 · p. 60
  • Table 3 covers construct-level accuracy, correlation, and replication-ratio metrics across the General Social Survey, Big Five traits, and economic games.Its caption describes this as a complement to the paper’s individual-level analysis.Source: Table 3 · p. 63
  • Six regression notes report R² values from 0.015 to 0.215, with p < 0.001 for each reported model.The reported R² values are 0.045, 0.022, 0.215, 0.155, 0.034, and 0.015.Source: §10 · pp. 64–65
  • Regression analyses found significant predictive-performance discrepancies for political ideology, party affiliation, and sexual orientation.Separate models examined demographic factors across the General Social Survey, Big Five personality traits, and economic games.Source: Table 4 · p. 71
  • Interview-based agents consistently reduced demographic parity differences relative to demographic-based agents across political ideology, race, and gender.Table 5 summarizes demographic parity differences across the General Social Survey, Big Five, and economic-game tasks.Source: Table 5 · p. 75
  • Interview Acc.=64.52 (std =7.85) and Nrm. Acc.=0.82 (std = 0.13).Source: §10 · p. 76

Robustness analysis

  • On GSS outcomes, interview-only agents achieved normalized accuracy 0.82 (std=0.13), maximal agents 0.87 (std=0.13), and summary agents 0.81 (std=0.13).The robustness analysis used a random subsample of 100 agents.Source: Table 6 · p. 77
  • Random-lesion interview agents’ GSS accuracy declined from 0.82 to 0.79 as 80% of utterances were excluded.The table describes this pattern as evidence that shortened interviews retain valuable insights.Source: Table 6 · p. 77

Experimental robustness analysis

  • The continued robustness table evaluates predicted treatment effects and sample-level means across five social science experiments, alongside human replication rates between waves 1 and 2.It calculates correlations and accuracy across study outcomes; the supplied passage gives no numerical results.Source: Table 6 · p. 78

Interview schedule

  • The interview schedule specifies a roughly two-hour semi-structured interview about participants’ life experiences and views on social topics, with responses shared in a research dataset.The schedule says the interview may include repetitive questions and is intended to be completed in one sitting.Source: §10 · p. 79

Supplementary Tables

  • Table 7 describes a two-hour interview guide adapted from the American Voices Project, covering life stories and social, political, and personal values, with selected portions abbreviated.The supplied script includes questions about work and finances as well as personal background and identity.Source: §10 · pp. 81–82, Table 7 · p. 83
  • Table 8 reports that agents predict behavior and attitudes well relative to participants’ internal consistency, and that interview-informed agents improve prediction on GSS and Big Five constructs.It reports GSS accuracy, Big Five and economic-game mean absolute error, and correlations across constructs.Source: Table 8 · p. 84
  • Table 9 compares experimental-study sample percentages with vendor-supplied quota targets derived from the 2023 American Community Survey.For example, the reported female sample percentage is 56.4% against a 50.9% quota target; Northeast percentages are 19.5% and 17.4%, respectively.Source: §10 · p. 85, Table 9 · p. 86
Loading 2411.10109v3…