Source-linked AI summary
Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies
Gati Aher, Rosa I. Arriaga, Adam Tauman Kalai
TL;DR
The paper addresses how to evaluate whether language models can simulate human behavior beyond a single arbitrary individual. It introduces and applies zero-shot Turing Experiments across four classic studies, replicating the first three and revealing hyper-accuracy distortion in Wisdom of Crowds. These results support using TEs to assess model strengths and distortions while recognizing ethical and bias-related limitations.
Problem
The Turing Test provides limited diagnostic value about which human behaviors and populations a language model can faithfully simulate.
Method
The authors define zero-shot Turing Experiments that use language models, prompts, participant details, and experimental conditions to generate records for comparison with human-study findings.
Results
TEs reproduced economic, psycholinguistic, and social-psychology findings in the first three experiments, while Wisdom of Crowds revealed hyper-accuracy distortion in larger and more aligned models.
Takeaways & Limitations
TEs provide a way to evaluate language-model simulation across diverse populations and to identify distortions relevant to downstream applications.
Takeaways & Limitations
TE simulations may reflect biases in model training data rather than behavior in the target human population, and some experiments may be unethical to perform.
Abstract
from arXiv · showhide
We introduce a new type of test, called a Turing Experiment (TE), for evaluating to what extent a given language model, such as GPT models, can simulate different aspects of human behavior. A TE can also reveal consistent distortions in a language model's simulation of a specific human behavior. Unlike the Turing Test, which involves simulating a single arbitrary individual, a TE requires simulating a representative sample of participants in human subject research. We carry out TEs that attempt to replicate well-established findings from prior studies. We design a methodology for simulating TEs and illustrate its use to compare how well different language models are able to reproduce classic economic, psycholinguistic, and social psychology experiments: Ultimatum Game, Garden Path Sentences, Milgram Shock Experiment, and Wisdom of Crowds. In the first three TEs, the existing findings were replicated using recent models, while the last TE reveals a "hyper-accuracy distortion" present in some language models (including ChatGPT and GPT-4), which could affect downstream applications in education and the arts.
1 Introduction
The paper introduces Turing Experiments (TEs) to evaluate whether language models can simulate representative samples of human behavior and to identify systematic distortions. It applies this methodology across four classic experiments, finding faithful replication in three and a hyper-accuracy distortion in Wisdom of Crowds.
- Motivation: Turing Experiments evaluate which aspects of human behavior a language model can faithfully simulate and which it systematically distorts.Unlike the Turing Test's single-individual simulation, TEs target behavior in specific human-subject experiments.
- Method: A TE uses participant details and optional experimental conditions to generate synthetic experiment records that can be compared with prior human-study outcomes.The procedure is zero-shot and computer-run, with prompts producing records or distributions over possible records.
- Experiments: The authors apply TEs to the Ultimatum Game, garden-path sentences, the Milgram Shock Experiment, and Wisdom of Crowds across 5–6 language models.The simulations vary surnames and gender titles, while experimental conditions and outcomes differ by task.
- Results: In the first three TEs, larger models produced more faithful simulations and the largest model replicated findings consistent with prior human studies.The Ultimatum Game simulations also showed outcomes varying consistently by gender and name.
- Results: In Wisdom of Crowds, larger models did not outperform smaller ones and some recent GPT models displayed hyper-accuracy by giving perfectly correct answers to obscure quantities.The paper links this distortion to possible alignment procedures and discusses consequences for education and the arts.
- Implications and limitations: TEs may inform downstream applications and the design of costly human experiments, but their use raises ethical and bias-related risks.The authors caution that some experiments may be unethical to perform and that model training data may reflect author biases rather than population behavior.
2 Running TEs Using LMs
TE simulators query language models to generate experiment records and outcomes, using validated completions and probability distributions to represent repeated simulations.
- Simulator design: TE simulators take experimental details as inputs and produce records or probability distributions over records containing outcomes of interest.Inputs can include participant demographics and experimental conditions; outputs may describe judgments or administered shocks.
- Simulator design: Language models are queried using next-token probabilities or sampled completions, with valid-choice probabilities normalized by the validity rate.For valid completions c_1,...,c_k, p_i = p(s c_i)/Z, where Z is the normalizing constant.
- Record construction: A sample Milgram record combines generated subject behavior with programmatically appended experiment details and reactions.The record can be halted when a separate termination judgment classifies a generation as the experiment endpoint.
- Validation: Table 1 reports valid percentage generation rates for five models across three TEs, with standard errors below 0.05%.The validity criterion is adherence to the required completion format.
- Prompt design: Prompt validation prioritizes clear wording that maximizes the validity rate before running large-sample simulations and testing hypotheses.Undesirable completions are reduced by specifying desired completions early and avoiding phrases that models tend to repeat.
3 Models and Datasets
The study compares multiple pretrained GPT language models using diverse name-based participant inputs and experimental conditions drawn from prior human studies.
- Models: The simulations query eight GPT models, labeled LM-1 through LM-8 in order of increasing price and claimed capability.Models include text-ada-001 through text-davinci-003, gpt-35-turbo, and gpt-4.
- Models: Running TEs across different models is challenging because many available models cannot handle the long prompts, particularly in the Milgram TE.The study uses natural temperature = 1 and top p = 1 parameters for completions.
- Participant inputs: Participant names use Mr. or Ms. titles and racially diverse surnames sourced from the U.S. 2010 Census Data.The surname lists include 100 names from each of five racial groups.
- Datasets: The four TEs use experimental conditions and comparison findings from prior literature on the Ultimatum Game, Garden Path sentences, Wisdom of Crowds, and Milgram.The study also makes its reproduction code publicly available.
4 The Ultimatum Game TE
The Ultimatum Game TE varies offers and participant names to compare simulated acceptance patterns with human findings and examine consistent name- and gender-linked differences.
- Inputs: The simulator varies an offer from $0 to $10 and the proposer’s and responder’s names across a balanced sample of 10,000 name pairs.Names combine Mr. or Ms. titles with surnames and support a 2 × 2 gender design.
- Simulation: Each simulation uses a 2-choice prompt whose valid completions are accept and reject, and the record concatenates the prompt with its completion.The initial endowment is fixed at $10.
- Human-pattern fidelity: LM-5 closely matches human decision trends: offers of 50-100% of the endowment are almost always accepted, whereas offers of 0-10% are rarely accepted.Smaller models produce nearly flat acceptance rates across fair and unfair offers.
- Name sensitivity: LM-5 shows strong within-name-pair consistency across offer sizes, with Pearson correlations above 0.9 for offers $1-$4 and $6-$9.The positive correlations indicate sensitivity to names that is consistent rather than purely random.
- Gender sensitivity: LM-5 predicts a 60% mean acceptance rate for a $2 offer proposed by a female to males, versus 20% when proposed by a male to females.The gender-linked distributions differ significantly, with p < 1e−16.
- Gender sensitivity: The simulated gender pattern is consistent with the chivalry hypothesis, although gender differences in human experiments are not uniformly consistent across time and culture.The paper presents the model’s title and pronoun sensitivity as a consistent simulation outcome.
5 Garden Path Sentences TE
The Garden Path TE tests whether language models reproduce the human difficulty of parsing garden-path sentences through grammaticality judgments.
- Result: LM-5 faithfully reproduces the human finding that garden-path sentences are harder to parse, whereas smaller models do not.The simulation asks whether each sentence appears grammatical.
6 Milgram Shock TE
The Milgram Shock TE simulates obedience across a sequential shock procedure, including the victim’s refusal at 300 volts and the subject’s response. Its outcomes are compared with Milgram’s human results and with a novel variation designed to differ from prior studies.
- Procedure: The simulation presents free-response and two-choice prompts across up to 30 shock levels, terminating earlier when the procedure indicates that the subject stops.A classifier converts open-ended responses into shock or did-not-shock actions, allowing the simulation to model both subject and experimenter behavior.
- Procedure: At 300 volts, the victim refuses to participate, and the experimenter instructs the subject to continue shocking.This occurs at the 20th shock, when the victim pounds on the walls and stops selecting an answer.
- Results: 75 out of 100 simulated participants followed the experimenter’s instructions until the end, compared with 26 out of 40 participants in Milgram’s Experiment 1.The figure compares the proportion remaining obedient through the complete shock series.
- Results: The study also created an artificial obedience scenario whose conditions differ from the original Milgram Shock setup.This variation addresses the possibility that model training data included descriptions of the original experiment.
7 Wisdom of Crowds TE
The Wisdom of Crowds TE examines whether language-model simulations reproduce human group-estimation patterns for general-knowledge questions. Larger and more aligned models instead show a hyper-accuracy distortion, producing answers that are inhumanly accurate.
- Design: The TE focuses on general-knowledge questions and compares simulated estimates with human results across ten questions.Five questions were selected from Moussaïd et al. (2013), and five additional questions were created by the authors.
- Results: The TE therefore reveals a hyper-accuracy distortion rather than the expected advantage of human-like group estimation.The distortion becomes increasingly present in the larger models examined.
- Results: As language models become larger and more aligned, they become more likely to complete the task with inhumanly accurate answers.Figure 8 normalizes estimates by dividing them by the correct answer; bars show medians and black lines show quartiles.
- Results: LM-6, ChatGPT, and GPT-4 simulations all had a median normalized estimate of 1.0 with a 0.0 IQR.For LM-6, a majority of simulated participants gave exactly correct answers to all ten questions.
8 Risks and Limitations
The paper identifies ethical, representational, and validity risks for Turing Experiments. Some simulations may be harmful or inappropriate, model outputs may reflect author biases, and prior exposure to classic experiments can affect evaluation.
- Ethical risks: Some TEs may be unethical or harmful because they simulate distressing procedures, include potentially offensive content, or should not be performed even when accurate.The paper specifically discusses the Milgram Shock Experiment and possible harms to authors and readers.
- Representational risks: Because language models are trained on data produced by biased authors, simulations may reflect those authors’ biases rather than population behavior.The paper proposes using TEs to examine whether models preserve distinctions between human findings and social stereotypes.
- Evaluation risks: Prior descriptions of the studied experiments may be present in model training data, complicating claims that simulations independently reproduce human findings.The authors created new garden-path sentences, a novel destructive-obedience scenario, and new general-knowledge questions to address this concern.
9 Conclusion
The paper presents Turing Experiments as a framework for evaluating how faithfully language models simulate diverse human populations and behaviors. Across several domains, the approach reproduces established experiments but also exposes hyper-accuracy in some models, motivating broader evaluation and careful consideration of costly or sensitive studies.
- Conclusion: Turing Experiments evaluate how faithfully language models simulate human behavior across diverse populations.The framework is intended to assess collective human-behavior simulation rather than treating an AI as a single monolithic intelligence.
- Conclusion: The authors show that TEs can reproduce economic, psycholinguistic, and social psychology experiments.The conclusion summarizes applications across the domains studied in the paper.
- Conclusion: The Wisdom of Crowds TE uncovered a hyper-accuracy distortion in which larger and more aligned models simulate subjects giving unhumanly accurate answers.The paper identifies this distortion as a central finding of the evaluation.
- Conclusion: Future work should conduct larger and more systematic simulations across additional language models and human behaviors.The authors describe the current work as an initial exploration of TEs.
- Conclusion: If future models are sufficiently faithful, LM-based simulations might help form and evaluate hypotheses when human experiments are costly or raise scale, selection-bias, monetary, legal, moral, or privacy concerns.The paper gives sensitive experiments, including research on what to say to a suicidal person, as an example of a costly human-study setting.
C Ultimatum Game TE
The Ultimatum Game TE evaluates whether simulated responders reproduce human-like sensitivity to monetary offers. Among LM-1 through LM-5, only LM-5 shows offer sensitivity aligned with expected human behavior.
- Results: Only LM-5 shows offer sensitivity that aligns with expectations of real human behavior.LM-1 and LM-2 tend to accept every offer, LM-3 tends to reject every offer, and LM-4 accepts 60% of $0 offers and all $10 offers while showing little overall sensitivity.
- Results: Figure 9 reports mean responder acceptance fractions for LM-1 through LM-5 across offers from $0 through $10.
D Garden Path TE
The Garden Path TE tests whether language models judge garden-path sentences as ungrammatical more often than comma-disambiguated controls. Across prior and author-written datasets, the general garden-path effect remains evident, with LM-5 showing the strongest separation in the author-written set.
- Phenomenon: Garden-path sentences are grammatical sentences whose likely initial reading appears ungrammatical because a word or phrase permits multiple interpretations.The TE hypothesizes that simulations will rate them ungrammatical more often than controls.
- Inputs: The TE uses 24 garden-path sentences, 24 comma-disambiguated controls, and 1,000 names, covering Optionally Transitive and Reflexive Absolute Transitive verbs.The controls add a disambiguating comma after the subordinate-clause verb.
- Simulation: The simulator presents grammatical and ungrammatical 2-choice prompts and concatenates each prompt with its language-model completion into an output record.This simplifies the original human tasks used in the cited studies.
- Results: Garden-path sentences were generally rated as more ungrammatical than corresponding control sentences across the simulations.For most sentences, Figure 10 shows higher average ungrammatical ratings for garden-path sentences; LM-3 and LM-4 also generally preserve this direction, while LM-5 shows a consistently high garden-path probability relative to controls.
- Author-written sentences: In the author-written dataset, the OT/RAT relative difficulty switched, but the garden-path effect persisted, with LM-5 showing the strongest control-versus-garden-path difference.LM-3 and LM-4 support the same conclusion with smaller differences, and Figure 13b shows similar trends to the Christianson et al. dataset.
E Wisdom of Crowds TE
The Wisdom of Crowds TE simulates estimates for general-knowledge questions across named participants and language models. The results reveal hyper-accuracy: some recent models produce unusually many perfectly correct estimates, and larger models do not improve the simulation in the expected way.
- Inputs: The TE uses 500 racially diverse surnames with Mr., Ms., and Mx. titles, yielding 1,500 simulated title-surname combinations.The authors note that human estimates annotated by gender were unavailable for comparison.
- Simulation: The larger models’ validity rates were close to 100%, and all answers in the reported results were integers.The simulator required integer completions because pilot runs often produced full sentences instead of numeric estimates.
- Results: An IQR rate of 0 means that all values agree from the first to third quartile, indicating that a majority of simulated responses were identical.
- Results: The reported table includes questions, answers, and estimates for the Wisdom of Crowds TE, including davinci, gpt-3.5-turbo, and gpt-4.The gpt-4 responses almost all round the speed of light to 3 × 10^8.
- Results: Figure 15 reports results across nine language models, with more recent or aligned models exhibiting greater hyper-accuracy distortion.
F Milgram Shock TE
The Milgram Shock TE simulates obedience and termination decisions through a multi-stage experiment modeled on Milgram’s procedure. The simulations reproduce obedience beyond victim nonresponse while also showing disobedience and early termination, although the synthetic and human experiments differ.
- Inputs: The input pool contains 100 uniquely named subjects formed from common surnames across racial groups and Mr. and Ms. titles.
- Simulation: The simulator uses free-response prompts for synthetic subject behavior and 2-choice prompts for classifying disobedience and termination.The algorithm interleaves generated subject behavior with preset learner actions and experimenter responses.
- Simulation: A pilot adjustment added a termination classifier after generated text repeatedly declared that subjects had ended the experiment.The classifier evaluates the first sentence of each free-form generation.
- Results: 25 of 100 simulations were stopped early, including 23 after Shock Level 20, when the victim stopped providing answers at 300 volts.The simulator compared the percentage remaining at each voltage with Milgram’s human observations.
- Results: Some simulated subjects continued shocking unresponsive victims, while others disobeyed or terminated the experiment.The alternate destructive-obedience scenario also produced both continued obedience and disobedience, but its results cannot be directly compared with human responses because the setups differ.
- Caveat: The simulation can produce conflicting text in which a subject appears to stop while canned experimenter text continues the experiment.Later generated text sometimes attempts to resolve this incoherence by declaring that the experiment was restarted.
- Results: Human participants and LM-5 simulations generally obeyed instructions until a spike in disobedience and terminations at 300 volts.Figure 22 compares break-off distributions for humans and simulated participants.
F.2 Novel Destructive Obedience Experiment Developed by the Authors.
The authors developed a novel destructive-obedience experiment that changes the Milgram setup while retaining malevolent authority and destructive punishment. LM-5 simulations showed both continued obedience and disobedience, with behavior shifting sharply when the victim became non-compliant, although direct comparison with human results is limited.
- Experimental design: The experiment changes Milgram’s wording and details while retaining malevolent authority and destructive punishment.Subjects evaluated voices for helping a sleepy driver, then monitored the driver and could submerge her when she drove dangerously.
- Simulation outcomes: Simulated subjects sometimes submerged victims beyond unresponsiveness, while others disobeyed commands or terminated the experiment.The simulation therefore produced variation in destructive obedience rather than uniform compliance.
- Comparison with Milgram: The simulated results resembled human results through a sharp increase in termination and disobedience when the victim began exhibiting non-compliant behavior.The authors caution that the results cannot be directly compared because the experimental setups differ.
- Illustrative record: An example LM-5 record shows Mr. Tran repeatedly submerging the driver before refusing after the driver became unresponsive and the experimenter issued another instruction.The record ends when Mr. Tran is classified as terminating the experiment.
- Simulation procedure: The simulation used staged prompts and two-choice classifications to determine whether subjects stopped and whether they obeyed.The experiment also incorporated experimenter prods, with the first prod used only on the first disobedience after the twentieth submersion.