PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals

Joon Sung Park, Carolyn Q. Zou, Jonne Kamphorst, Niles Egan, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Percy Liang, Robb Willer, Michael S. Bernstein

arXiv:2411.10109v3cs.AIcs.HCcs.LG

TL;DR

Prior models often require outcome-specific training data, so this paper tests whether LLM agents grounded in self-reports can simulate individuals across outcomes. Using interviews, surveys, or both, the study finds strong individual-level prediction across surveys, traits, games, and experiments, with self-report agents outperforming demographic and persona baselines.

  • Problem

    Outcome-specific behavioral models require training data for each target outcome, motivating tests of whether self-report-grounded LLM agents can generalize across outcomes.

  • Method

    The study built individual agents from semi-structured interviews, structured surveys, or both, and evaluated them across survey responses, personality traits, economic games, and replicated experiments.

  • Results

    Self-report agents recovered 82 to 86 percent of participants’ own two-week test-retest consistency on the GSS and outperformed demographic and persona baselines across additional behavioral outcomes.

  • Takeaways & Limitations

    Simulation quality depends more on the depth and reliability of an individual’s grounding data than on model scale or synthetic persona engineering.

  • Takeaways & Limitations

    The evaluation centers on individual-level accuracy and does not establish whether agents reproduce correlations among attitudes across a population.

Abstract

from arXiv · show

Machine learning can predict human behavior well when substantial structured data are available for well-defined outcomes. Such models are typically outcome-specific, however, requiring training data for each target outcome, limiting their applicability to new domains. We test whether large language models (LLMs) can relax these requirements by using self-report data to build attitudinal and behavioral simulations, or "generative agents," that can predict responses across outcomes without outcome-specific training data. Using data from a diverse national sample of 1,052 Americans, we built agents from (i) two-hour, semi-structured interviews elicited using the American Voices Project interview schedule, (ii) structured surveys including General Social Survey items and the Big Five personality inventory, or (iii) both sources combined. On held-out General Social Survey items, interview-only, survey-only, and combined agents achieved accuracies equal to 83%, 82%, and 86% of participants' own two-week test-retest consistency benchmark, respectively, compared with 74% for demographics-only agents. Combining interviews and surveys produced the highest accuracy, though gains over either source alone were modest, suggesting that predictive benefits from data begin to asymptote once the model has observed sufficient evidence within a domain. We find that these agents also predict personality traits, economic-game behavior, and experimental responses, while reducing accuracy disparities across racial and ideological groups relative to demographics-only agents. Together, these results show that LLM agents grounded in qualitative or quantitative self-reports can support general-purpose simulation of individuals across outcomes, without requiring task-specific training data.

Using Individuals’ Self-Reports to Create Generative Agents

The study builds generative agents from structured surveys, semi-structured interviews, or both, and evaluates whether they can reproduce individuals’ responses across social-science outcomes. Self-report-grounded agents generally outperform demographic and persona baselines, while combining sources yields modest additional gains and reduces some subgroup disparities.

  • Data and agent construction: 1,052 participants completed structured surveys and approximately two-hour interviews, which were used separately or jointly to create individual generative agents.Agents predicted held-out responses, personality traits, economic-game behavior, and experimental responses.
  • Predictive performance: 0.83, 0.82, and 0.86 normalized GSS accuracy was achieved by interview-only, survey-only, and combined agents, respectively.These values compare agent accuracy with participants’ two-week self-consistency benchmark.
  • Predictive performance: 0.74 normalized GSS accuracy was achieved by demographic agents, below interview-only, survey-only, and combined agents.Persona agents achieved 0.71, and combined agents significantly outperformed either single-source self-report agent.
  • Predictive performance: 0.80 normalized correlation was achieved by interview agents on Big Five personality traits, compared with 0.61 for demographic agents and 0.75 for persona agents.Survey-only agents did not significantly differ from either baseline, and combining sources did not improve on interview-only agents.
  • Predictive performance: 0.66 normalized correlation was achieved by interview agents in economic games, but differences across the five agent types were not significant.The games included Dictator, Trust, Public Goods, and Prisoner’s Dilemma tasks with incentivized decisions.
  • Robustness: Removing 80% of interview transcripts still produced 0.79 normalized GSS accuracy and 0.73 normalized Big Five correlation.Interview-summary agents retaining factual content but removing original linguistic features achieved 0.81 GSS accuracy and 0.70 Big Five correlation.
  • Bias and subgroup disparities: Interview- and survey-grounded agents generally reduced demographic parity differences relative to demographic and persona agents, especially across ideological and racial subgroups.Gender-based differences remained uniformly low across agent types and tasks.
  • Limitations: The evaluation establishes strong individual-level prediction but does not establish whether agents reproduce population-level correlations among attitudes.The authors also note that experimental-replication evidence is limited by the small number of studies and that only one model family was evaluated.

Supplementary Materials for

The study constructed a large bank of individual generative agents from interviews, surveys, and behavioral data collected through a staged custom platform. Participants completed consent, avatar creation, interviews, surveys and experiments, followed two weeks later by retesting for consistency measurement.

  • Constructing the Agent Bank: 1,052 participants were recruited for a stratified U.S. sample and completed two-hour AI-led interviews alongside surveys and behavioral experiments.The study aimed to represent U.S. adults across demographic and political characteristics.
  • Constructing the Agent Bank: The agent bank combined interview transcripts and survey responses to model real individuals.The interviews lasted about two hours, and the collected data included qualitative and quantitative self-reports.
  • Study Platform and Procedure: Participants completed study modules in sequence: consent, avatar creation, interview, surveys and experiments, then a self-consistency retake.The retake became available two weeks after the earlier modules.
  • Data Collection and Consent: The platform used an AI interviewer agent to collect interviews at scale, while emphasizing consent for potentially sensitive, difficult-to-anonymize data.Participants were informed about simulation uses, withdrawal rights, and continuing privacy risks.

AI Interviewer Agent Architecture

The AI interviewer followed a structured protocol while dynamically choosing scripted progression or context-sensitive follow-up questions. A reflection module condensed prior conversation into notes that supported relevant questioning without repeatedly supplying the full transcript.

  • Architecture: The interviewer agent used the interview protocol and recent participant utterances to choose between advancing the script and asking a follow-up question.Each scripted question had a time limit, while follow-ups were generated from the conversation so far.
  • Architecture: A reflection module synthesized the ongoing interview into concise notes containing facts and inferred participant characteristics.These notes could include information such as birthplace and inferred orientation toward outdoor activities.
  • Architecture: The architecture used reflection notes plus the most recent 5,000 transcript characters instead of the full interview transcript when generating actions.This design addressed the risk that very long prompts cause language models to overlook information.
  • Interview Script: The interview script was designed to elicit nuanced portraits while remaining independent of the evaluation metrics.The study used an abbreviated American Voices Project script to fit the interview within two hours.

Implementation

The interviewer was implemented as a voice-to-voice web application that used scripted prompts, transcript context, reflection notes, and language-model generation. Its interface supported audio interaction, progress tracking, subtitles, and pausing.

  • Implementation: The interviewer agent operated as a web application supporting low-latency voice-to-voice interaction through audio and microphone capabilities.The implementation was intended to support conversational rapport during interviews.
  • Interface: The interface represented speaking turns with animated avatars, showed participant progress toward a goal, and offered subtitles and pause controls.The interviewer avatar became a microphone emoji during the participant’s turn.
  • Prompting: The language model received recent conversation text and was prompted to summarize participant facts into concise reflection bullets.A separate prompt used reflection notes and the current scripted question to generate new interviewer questions.
  • Prompting: The interviewer prompt instructed the system to assess progress toward the interview objective, propose follow-ups, and author the next utterance.It also instructed the interviewer not to pressure participants who explicitly refused to answer for privacy reasons.
  • Interview Statistics: 5372.59 words were spoken by the AI interviewer on average, with 81.71 follow-up questions asked from 99 scripted questions.Participants responded with an average of 6491.19 words.

Development Process and Evaluation

The interviewer was iteratively refined through pilot interviews and assessed for transcript quality and downstream predictive usefulness. Pilot evidence indicated that AI-collected transcripts produced generative agents comparable to those based on human-conducted interviews for GSS prediction.

  • Development Process: 35 pilot interviews were conducted across three waves, including 10 human-led and 25 AI-led interviews.Research-team members trained in social sciences evaluated the resulting transcripts as training data for generative agents.
  • Development Process: AI-generated transcripts compared well with human-generated transcripts in the pilot-stage research-team assessment.The team evaluated their quality for constructing generative agents.
  • Evaluation: AI-interviewer transcripts produced generative agents whose GSS prediction performance matched that of agents built from human-interviewer transcripts.The pilot interviews also showed smooth conversation, relevant follow-ups for terse answers, and sympathetic navigation of difficult experiences.
  • Evaluation: The deployed AI interviewer generated follow-up questions dynamically because only the first question was scripted in the illustrated interview excerpt.Subsequent interviewer utterances were generated during the interaction.

3. Generative Agent Architecture

Generative agents simulate individuals by combining self-report data with expert-generated reflections stored in memory, then using a language model to predict responses. Expert personas specialize reflections for different domains and are selected when generating predictions.

  • Generative agents store self-report data and language-model-generated reflections in a text-based memory stream for later prediction.The reflection module synthesizes memories into higher-level insights about participants.
  • Expert reflections infer participant insights that may not be explicitly stated in the self-report data.The system uses personas such as psychologists and behavioral economists to guide these syntheses.
  • Example reflections separately capture autonomy, financial goals, mixed political views, and employment and household-income information.These examples illustrate how different expert personas extract distinct participant attributes from one interview.
  • When answering a question, the language model first selects the most relevant domain expert and retrieves that expert’s reflections alongside the self-report data.This routes question answering through domain-specific summaries of the participant.

Generating a Prediction With Generative Agents

The prediction procedure prompts a language model with a question, the participant’s interview transcript, and relevant expert reflections. For categorical responses, the prompt interprets options and reasons through the participant’s likely choice; numerical prompts adapt this process to response ranges.

  • The prediction prompt combines the target question with the participant’s interview transcript and relevant expert reflections.The model is instructed to predict how the participant would actually respond.
  • For categorical questions, the model interprets each option, reasons why the participant might choose it, selects the best prediction, and outputs the response.This uses the stages Option Interpretation, Option Choice, Reasoning, and Response.
  • For numerical questions, the model interprets the people associated with each end of the range, reasons over the likely value, and predicts the participant’s response.The numerical prompt replaces categorical option interpretation with range interpretation.
  • Experimental predictions append the received stimuli and prior actions to the self-report data and reflections to maintain experimental context.This supplies behavioral-history information when experiments require it.

4. Surveys and Experimental Constructs

The evaluation uses established survey and personality constructs to test whether agents predict individual responses across standardized social-science measures. The General Social Survey supplies structured attitude and behavior items, while the BFI-44 supplies five personality scores.

  • The evaluation framework assesses agent fidelity across four existing constructs used in social-science research.The stated goal is to measure prediction of attitudes and behaviors across disciplines and methods.
  • The General Social Survey: The General Social Survey is a biannual survey of representative U.S. adult cross sections covering demographics and viewpoints on social issues.Its items address topics including government spending, race relations, and beliefs about God.
  • The General Social Survey: The study focuses on the GSS Core’s structured categorical and numerical questions because they support quantitative assessment.Conditional and other excluded question types were not part of the primary evaluation.
  • Big Five Personality Traits: The Big Five framework measures openness, conscientiousness, extraversion, agreeableness, and neuroticism as broad personality dimensions.Individuals receive a score for each trait.
  • Big Five Personality Traits: Participants completed the 44-item BFI-44, and trait scores were calculated using the aggregation methods from the original work.The questionnaire was administered through a custom Qualtrics form.

Behavioral Economic Games

The study uses incentivized behavioral economic games to measure decision-making and social behavior. Five prominent games were administered through a custom survey platform, with participant choices determining possible bonuses.

  • Behavioral economic games measure decision-making and social behavior under real financial stakes.The games target constructs such as altruism, trust, cooperation, and competition.
  • Participants received bonus payments based on their game choices to encourage genuine preferences and strategies.Each game offered a maximum bonus of $8 to $10.
  • The study included five economic games selected for their prominence in the academic community.The listed games include the Dictator Game, Trust Game, and Prisoner’s Dilemma.
  • In the Dictator Game, participants divided $5 between themselves and another participant who could not affect the outcome.The game measures altruism through unilateral allocation decisions.
  • In the Trust Game and related group interaction tasks, participants decide how much money to keep or contribute to a common task whose contributions are doubled and shared.The group task involves four participants receiving $4 each.
  • All five games were administered through a custom Qualtrics form, and one randomly selected game determined participants’ bonuses after the study.Participants were not told in advance which game would be selected.

Replication Studies of Experimental Treatment Effects

The study replicates five social-science experiments using human participants and generative agents, selecting experiments that could be described in natural language and feasibly tested with about 1,000 participants.

  • Experimental design: The experimental designs used established treatment-versus-control comparisons, with random assignment supporting attribution of outcome differences to the intervention.
  • Study selection: The replication sample was drawn from Camerer et al.’s 41-study project and restricted to experiments describable to language models with sufficient power at 1,000 or fewer participants.The resulting sample included five studies.
  • Study content: The experiments covered perceived intent and harm, fairness and emotional responses, and other behavioral effects documented in the selected replication studies.
  • Experiment administration: Participants were randomly assigned to conditions for each of the five experiments through a custom-built Qualtrics form.

5. Evaluation Methods

The evaluation measures individual-level predictive accuracy across surveys, personality, and economic games, using response-type-specific metrics and normalization against participants’ two-week self-consistency.

  • Evaluation framework: Agents’ predictions use first-phase responses, participants’ second-phase responses measure internal consistency, and normalized accuracy compares the two.A normalized accuracy of 1.0 means agent performance matches participants’ own two-week replication accuracy.
  • Metrics: Categorical-ordinal outcomes use exact-match accuracy, numerical outcomes use mean absolute error, and Pearson correlation provides a metric comparable across constructs.
  • Evaluation framework: The analysis primarily computes accuracy for each participant and averages across participants, while also reporting construct-level performance across questions, dimensions, or games.
  • Correlation aggregation: Fisher’s z-transformation converts correlation coefficients before averaging, then applies the inverse transformation to obtain the pooled correlation.
  • GSS measures: GSS evaluation covers 183 questions, reporting accuracy and correlation for categorical-ordinal items and MAE and correlation for six numerical items.
  • Metric limitation: Normalized accuracy is not computed for MAE because zero internal-consistency values can make its denominator zero.
  • Prediction setup: Inputs from the first GSS wave exclude the question being predicted in the second wave.

Replicating Experiments With Generative Agents

The experiment-replication evaluation tests whether generative agents reproduce treatment effects observed in a direct replication by the study’s human participants, separating agent failures from failures of the original findings to replicate.

  • Replication design: The study directly reruns five sampled experiments with its human participants before testing whether agents reproduce their sample average treatment effects.
  • Replication design: This direct replication distinguishes failures caused by nonreplication in the current human sample from failures caused by the agents.
  • Evaluation outcomes: The evaluation asks whether effect direction and significance replicate and whether human and simulated effect sizes correlate.
  • Statistical analysis: The analyses use each original study’s statistical method for significance and express all effect sizes as Cohen’s d.
  • Statistical analysis: The study reports which experiments produced significant results for human participants and simulated predictions and computes Pearson correlations between their effect sizes.

Interviews Improve Agents’ Prediction Accuracy

The study compares interview-, survey-, and combined-data agents to determine how different self-report sources contribute to prediction, while also testing data reductions, baselines, and demographic performance gaps.

  • Research question: The central question is whether qualitative interview data improve agents’ predictive capabilities relative to other self-report sources.
  • Agent comparisons: The main comparison includes interview-based, survey-based, survey-plus-interview, demographic, and persona agents.
  • Robustness analyses: Exploratory robustness analyses compare alternative information sources and design spaces using a random sample of 100 agents.
  • Survey agents: Survey agents use wave 1 GSS and Big Five responses to predict wave 2 outcomes, with either the target item removed or the entire containing module held out.
  • Combined sources: Survey-plus-interview agents add interviews to survey agents, while maximal agents additionally include economic-game data to examine the performance ceiling from combining sources.
  • Interview information: Summary agents convert interview transcripts into factual response-pair dictionaries, separating transcript information from most linguistic features.
  • Interview information: Random-lesion agents shorten interviews by removing 0%, 20%, 40%, 60%, or 80% of question-response pairs to assess how much interview content is needed.
  • Demographic bias: The demographic-bias analysis tests whether individualized self-report-based models reduce performance gaps across demographic groups compared with demographic-only models.

6. Supplementary Results

Supplementary analyses report strong numerical GSS prediction, robustness results across agent information sources, and demographic performance disparities for some identity groups.

  • r = 0.97 average correlation and 0.14 average MAE characterized predictions for six numerical GSS questions.
  • 0.82 normalized accuracy was achieved by interview agents in the 100-agent robustness subsample.
  • 0.82 normalized accuracy was achieved by survey agents when excluding same-category question-answer pairs, while removing the whole GSS module reduced accuracy to 0.77.
  • 0.85 normalized accuracy was achieved when interview transcripts were added to survey agents with only the predicted GSS outcome question removed.
  • 0.87 normalized accuracy was achieved by maximal agents incorporating surveys, experiments, and interviews, slightly exceeding interview-based agents.
  • Predictive performance was stronger for strong liberals, strong Democrats, and non-heterosexual participants than for more conservative, Republican, or heterosexual participants on GSS responses.

7. Deviations from the Pre-Analysis-Plan for the GSS Outcome Questions

The authors deviated from the preregistered full-GSS evaluation by removing outcome questions synonymous with interview questions, reducing the risk that interview agents could retrieve answers directly.

  • 54,694 unique GSS–interview question pairs were classified for synonymy using GPT-4.1, with similarity scores and binary judgments.
  • The classification prompt requested a 0–100 similarity score and a Same or Different judgment based on whether both questions sought the same information.
  • 15 of the 24 characterized excluded questions concerned respondents’ demographics, while five concerned parental education or employment.
  • Manual review of unsure cases identified two additional demographic questions for removal, supplementing the model-based exclusions.
  • 27 GSS outcome questions were excluded after identifying synonymous or nearly identical American Voices Project interview questions.
  • Including GSS questions identical to American Voices Project questions increased accuracy by 0.02 points relative to the revised evaluation set.

8. Why do interview-based generative agents work?

Robustness analyses indicate that interview-agent performance draws on both direct retrieval and inference from interview content, although these mechanisms do not fully explain the advantage.

  • Direct retrieval was tested by classifying whether interview answers contained information needed to answer each GSS question across 59 agents.
  • 0.79 interview-agent accuracy versus 0.71 demographic-agent accuracy remained after dropping 80 GSS questions ranked by retrievability.
  • Removing questions most likely answerable through inference caused interview-agent accuracy to decline progressively, unlike demographic and persona agents.
  • The interview-agent advantage was 0.095 points after removing direct-retrieval questions and 0.080 points after additionally removing inference questions.
  • Nearly all direct-retrieval cases also permitted inference, complicating attempts to isolate the two mechanisms.
  • The analyses suggest that interview agents combine semi-structured interview depth with LLM retrieval and reasoning capabilities.

9. Results using different models

Across model variants, advanced GPT models slightly improved GSS accuracy over the GPT-4o baseline, while cheaper mini models generally traded accuracy for lower cost. A 2025 GPT-4o run closely reproduced the 2024 result, and fine-tuning was evaluated separately on GSS outcomes.

  • Evaluation design: The study replicated the GSS prediction task on 50 agents while holding prompt templates, reasoning instructions, and temperature settings constant across models.Only the model parameter changed, and the reported scores were non-normalized accuracy values.
  • Model comparisons: GPT-5, GPT-4.1, GPT-o1, and GPT-o3 each reached 0.67 accuracy, slightly exceeding the 2024 GPT-4o baseline of 0.66.With only 50 agents, the advantage was statistically indistinguishable.
  • Model comparisons: GPT-4o-mini (0.60) and GPT-o1-mini (0.62) trailed their full counterparts, whereas GPT-o3-mini (0.64) and GPT-o4-mini (0.64) nearly matched the GPT-4o baseline.The comparison illustrates a cost–performance trade-off among mini variants.
  • Model stability: A 2025 GPT-4o run produced 0.64 accuracy versus 0.66 in 2024, a 0.02-point decrease.The analysis describes this difference as minimal model drift.
  • Fine-tuning: The authors also fine-tuned GPT-4o for GSS outcome questions using 500 randomly selected agents and evaluated it on the remaining 552 agents.The fine-tuning setup used one epoch and removed reasoning and reflection components from the output prompt.

10. Supplementary Tables

Supplementary analyses document the study sample, agent-comparison tests, construct-level results, demographic-performance analyses, and interview materials. The tables show significant architecture differences for some outcomes but not for economic-game correlations, alongside demographic performance discrepancies concentrated in selected constructs.

  • Agent architecture comparisons: Table 2 compares five agent specifications across the GSS, Big Five traits, and economic games using one-way ANOVA with Tukey HSD tests and N = 1,052 per agent type.Composite denotes the survey-plus-interview agent, and mean differences are group 2 minus group 1.
  • Agent architecture comparisons: For GSS accuracy, interview-only, survey-only, and Composite agents significantly outperformed demographic- and persona-based agents, while Composite agents also exceeded both single-source agents.All reported comparisons had p < 0.001.
  • Agent architecture comparisons: For Big Five correlations, interview-only and Composite agents outperformed both baselines, survey-only agents did not differ significantly from either, and economic-game correlations showed no specification difference.The economic-game comparison was F(4, 5255) = 1.63, p = 0.16.
  • Sample and materials: The study materials included a semi-structured interview expected to take roughly two hours and covering life experiences and views on social topics.The interview content was intended for inclusion in a dataset shared with scientific researchers.
Loading 2411.10109v3…