PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated Social Science: Language Models as Scientist and Subjects

Benjamin S. Manning, Kehang Zhu, John J. Horton

arXiv:2404.11794v2econ.GNq-fin.EC

TL;DR

The paper addresses the limited automation of generating and testing social-science hypotheses. It uses SCMs to organize LLM-based agents and experiments, finding results consistent with theory while showing that LLM predictions improve when conditioned on fitted models.

  • Problem

    Automated generation and testing of social-science hypotheses remain limited, despite emerging machine-generated hypotheses and LLMs’ potential as simulated subjects.

    Source: §1 · p. 2

  • Method

    The approach uses SCMs to express hypotheses and blueprint LLM-agent construction, experimental design, and analysis across social scenarios.

    Source: §1 · p. 2

  • Results

    The system found scenario-specific effects, auction simulations matched theory, and conditioning LLM predictions on fitted path estimates cut mean squared error sixfold.

    Source: §1 · pp. 3–4

  • Takeaways & Limitations

    Simulations can elicit information from an LLM that direct elicitation does not provide, and their results were highly consistent with theoretical predictions.

    Source: §6 · p. 27

    Conversation stopping has no universal rule in the authors’ account; more naturalistic stopping based on real conversation data remains future work.

    Source: §6.4 · p. 30

Abstract

from arXiv · show

We present an approach for automatically generating and testing, in silico, social scientific hypotheses. This automation is made possible by recent advances in large language models (LLM), but the key feature of the approach is the use of structural causal models. Structural causal models provide a language to state hypotheses, a blueprint for constructing LLM-based agents, an experimental design, and a plan for data analysis. The fitted structural causal model becomes an object available for prediction or the planning of follow-on experiments. We demonstrate the approach with several scenarios: a negotiation, a bail hearing, a job interview, and an auction. In each case, causal relationships are both proposed and tested by the system, finding evidence for some and not others. We provide evidence that the insights from these simulations of social interactions are not available to the LLM purely through direct elicitation. When given its proposed structural causal model for each scenario, the LLM is good at predicting the signs of estimated effects, but it cannot reliably predict the magnitudes of those estimates. In the auction experiment, the in silico simulation results closely match the predictions of auction theory, but elicited predictions of the clearing prices from the LLM are inaccurate. However, the LLM's predictions are dramatically improved if the model can condition on the fitted structural causal model. In short, the LLM knows more than it can (immediately) tell.

1 Introduction

The paper uses structural causal models (SCMs) to automate hypothesis generation and in-silico testing with LLM-powered agents. Across social scenarios, the system finds effects consistent with some hypotheses, while experiments show that direct LLM predictions are limited and improve when conditioned on fitted models.

1 Introduction

  • SCMs organize the process by translating hypotheses into agent and scenario variation, experimental design, and a prespecified estimation plan.Each explanatory variable indicates a dimension to vary for identifying an effect (Haavelmo, 1943, 1944; Jöreskog, 1970).Source: §1 · pp. 2–3
  • The open-source system can generate hypotheses, design and run experiments, and analyze results, though researchers selected hypotheses and edited some agents in two scenarios.It tested four settings: bargaining over a mug, a bail hearing, a job interview, and an ascending-price auction.Source: §1 · pp. 3, 5
  • The system found that mug deals rose as seller attachment fell, remorseful defendants received lower bail, and bar passage—but not height or interviewer friendliness—predicted hiring.Extensive criminal history was associated with higher bail, while the judge’s prior case count did not affect the bail amount.Source: §1 · p. 3
  • Auction simulations matched theory: higher bidder reservation prices raised clearing prices, which stayed close to the second-highest reservation.The reported pattern also matches empirical observations and auction theory (Maskin and Riley, 1985).Source: §1 · p. 3
  • Directly elicited auction outcome predictions were wildly inaccurate, falling even farther from auction theory than the simulation results.Source: §1 · p. 4
  • Across four experiments, LLMs overestimated SCM path estimates by 13.2 times on average, but conditioning on fitted estimates cut prediction error sixfold.They overestimated 10 of 12 paths, generally predicting the correct direction; conditional predictions were closer to theory than direct predictions, but remained closer to simulations.Source: §1 · p. 4

2 Overview of the system

The system treats an SCM as a blueprint for an automated social-science workflow, from scenario and hypothesis generation through agent simulations and analysis. Its decisions can be edited, and fitted results can inform follow-on experiments.

2 Overview of the system

  • The system is designed to run autonomously, but its decisions are editable; researchers supplied hypotheses and edited agents in some demonstrations.A fitted SCM can guide another experiment, although the transition between experiments is not automated.Source: §2 · pp. 6, 7, 10
  • SCMs make causal claims explicit and determine which variables to manipulate and how to estimate effects, making later workflow steps mechanically specifiable.The authors use simple linear SCMs as a starting assumption, while noting that this assumption may be incorrect.Source: §2 · pp. 6, 10
  • The workflow moves from a social scenario through outcome and cause hypotheses, experimental design, LLM-agent simulations, outcome measurement, and analysis.The estimated relationships can be used to plan a follow-on experiment.Source: §2 · p. 7
  • The system proposes outcomes, potential causes, and operationalizations, then uses them to construct an SCM that guides agent instantiation, interaction, and estimation.For example, a mug deal is coded as a binary outcome and a buyer’s budget as willingness to pay in dollars.Source: §2 · pp. 7–8
  • Independent LLMs play agents whose attributes vary across experimental conditions; the system runs conditions in parallel and uses a selected turn-taking protocol.An external LLM decides whether conversations continue after each agent speaks, subject to a 20-statement limit.Source: §2 · pp. 8–9
  • After simulations, agents answer prespecified survey questions and the system estimates the linear SCM from the resulting data.Agents retain text memories of interactions that can be used to answer questions about what happened.Source: §2 · pp. 8–10

3 Results of experiments

The experiments test proposed causes across four social scenarios, with human input limited to the first two and the system accommodating human input in the latter two. Results identify effects in each scenario, while also finding a nonsignificant standalone cause and interaction-dependent bail estimates.

3.1 Bargaining over a mug

  • Bargaining simulations found significant effects for all three causes: buyer budget raised deal probability, while minimum price and seller attachment lowered it.A one-dollar increase in budget raised deal probability by 3.7 percentage points; a one-dollar increase in minimum price lowered it by 3.5 points, and one attachment-scale unit lowered it by 2.5 points.Source: §3.1 · p. 11

3.2 A bail hearing

  • In the bail model, each prior conviction increased bail by $521.53; with interactions, only the case-count-by-remorse interaction was significant.Without interactions, remorse had a small, borderline-significant estimate; in the interaction specification, neither criminal history nor other standalone causes were significant.Source: §3.2 · p. 14

3.3 Interviewing for a job as a lawyer

  • Passing the bar was the only clear cause of hiring, raising the applicant’s job probability by 75 percentage points; no tested interaction was significant.Its standardized effect was the largest across the four scenarios (β* = 0.78, p < 0.001).Source: §3.3 · p. 16

3.4 An auction for a piece of art

  • Auction bidders’ budgets each increased the final price: the estimated slopes were $0.35, $0.29, and $0.31 per dollar.All three effects were statistically significant (β* = 0.57, 0.47, and 0.5, respectively; each p < 0.001).Source: §3.4 · pp. 16, 18

4 LLM predictions for paths and points

The paper tests whether LLMs can predict experimental outcomes and fitted effects through direct elicitation, and whether fitted SCM information improves outcome predictions. Direct predictions are inaccurate, while model-conditioned predictions improve but remain less accurate than auction theory.

4.1 Predicting yi

  • Figure 6 compares simulated clearing prices, auction-theory predictions, and direct LLM predictions; the LLM’s MSE was 8628 versus theory’s 128.The plot’s rows and columns vary bidders 2 and 3’s reservation prices, while the axes show bidder 1’s reservation price and final bid price.Source: §4.1 · p. 19, Figure 6 · p. 20

4.2 Predicting ˆβ

  • LLM path estimates averaged 13.2 times the experimental estimates, with 10/12 overestimates despite 10/12 correct signs and significance classifications.Removing the largest overestimate still left an average magnitude ratio of 5.3.Source: §4.2 · p. 21

4.3 Predicting yi|ˆβ−i

  • With leave-one-out fitted SCM information, LLM outcome predictions improved to MSE 1505 from 8628, but remained less accurate than auction theory’s MSE 128.The predictions were out of sample with unique inputs; the paper notes that the fitted SCM omitted bidders’ second-highest reservation price, a potential cause that could have matched theory.Source: §4.3 · pp. 21–22

5 Identifying causal structure ex-ante

The section shows how assuming or searching for causal structure in observational data can produce misleading conclusions. It contrasts those risks with SCMs, which specify outcomes and exogenous variation before data generation and analysis.

5 Identifying causal structure ex-ante

  • Open-ended simulations can leave researchers combing through unstructured text and inferring causal structure afterward, whereas SCMs specify what to measure and identify ex ante.The authors note that identification is guaranteed within the SCM framework.Source: §5 · p. 22

5.1 Assuming causal structure from data

  • Figure 7 shows buyer-budget and seller-minimum-price paths significant in the correctly specified SCM but insignificant and closer to zero after misspecification.Controlling for whether a deal occurred also makes deal status appear significant, although the experiment identifies reservation prices as significant causes of conversation length.Source: Figure 7 · p. 24, §5.1 · p. 24
  • SCMs specify downstream outcomes and exogenous manipulations in advance, avoiding bad controls and the need to search observational data for causal structure.The framework can also estimate effects on a newly measured outcome using the already identified exogenous variation.Source: §5.1 · p. 25, §5.2 · p. 27

5.2 Searching for causal structure in data

  • Searching all possible DAGs becomes infeasible as variables grow: four nodes already yield 543 candidate DAGs.The number of possible DAGs grows exponentially with the number of nodes.Source: §5.2 · p. 25
  • For the tax-fraud scenario, GES identified only a criminal-history–bail link but could not determine its direction, despite the experiment showing more convictions increased bail.The algorithm used raw data without information about which variables were exogenously varied.Source: §5.2 · p. 26, Figure 8 · p. 26

6 Conclusion

The paper demonstrates an SCM-based approach to automated in silico hypothesis generation and testing, and discusses its potential for controlled experimentation, researcher interaction, replication, and further development.

6 Conclusion

  • The system demonstrates automated in silico hypothesis generation and testing using structural causal models.The authors report that simulations can elicit information not available through direct elicitation and produce results highly consistent with relevant economic theory.Source: §6 · p. 27

6.1 Controlled experimentation at scale

  • The system can generate controlled simulations at scale with prespecified data collection and analysis, supporting rapid, inexpensive, replicable exploration while leaving a fundamental jump from simulations to human subjects.The authors note that social-science outcomes can vary across populations and environments, while human studies are expensive and time-consuming.Source: §6.1 · p. 28

6.2 Interactivity

  • Researchers can inspect or change system decisions, supply their own choices, and customize variables, agent attributes, interactions, or statistical analyses.Different LLMs can serve different roles in the system.Source: §6.2 · p. 28

6.3 Replicability

  • The procedure and fitted SCM can be exported with decisions, explanations, and simulation transcripts for others to inspect, replicate, or extend.The authors contrast this with the difficulty of replicating human-subject experiments when exact procedures are often unclear.Source: §6.3 · p. 29

6.4 Future research

  • The authors identify agent attributes beyond SCM variables as an open design problem: demographic, personality, or other traits might improve fidelity, but how to optimize them is unclear.The system currently includes such traits only when they are part of the SCM.Source: §6.4 · p. 29
  • The interaction protocol and stopping rule remain open design problems; the authors suggest a real-conversation Markov model as one possible future approach to more naturalistic simulations.The paper also notes that better ways to determine speaking order may exist.Source: §6.4 · pp. 29–30
  • A next step is to use simulation outcomes to guide continuous follow-on experiments that explore a scenario’s parameter space, although optimizing that exploration remains an open problem.Source: §6.4 · p. 30
  • The presented system is one implementation of the SCM-based approach, reflects subjective design choices, and leaves room for alternative implementations and improvement.Source: §6.4 · p. 30

Bakker, Michiel, Martin Chadwick, Hannah Sheahan, Michael Tessler,

This section contains bibliography excerpts that identify related work, rather than discussing the paper’s methods or findings.

Bakker, Michiel, Martin Chadwick, Hannah Sheahan, Michael Tessler,

Camerer, Colin, Anna Dreber, Felix Holzmeister, Teck-Hua Ho, Jurgen

Jumper, John, Richard Evans, Alexander Pritzel, Tim Green, Michael

Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi,

  • The bibliography fragments end with Zheng et al.’s 2022 article on The AI Economist in Science Advances.The supplied text gives the subtitle “Taxation policy design via two-level deep multiagent reinforcement learning” and identifies the article as 8 (18), eabk2607.Source: Wei, Jason, Xuezhi Wang, Dale… · p. 38

A Implementation details

The implementation uses reusable prompts and structural causal models to organize variable construction, agent attributes, experiments, and analysis.

A Implementation details

  • The system uses scenario-neutral prompts with scenario-specific placeholders, and more than 50 prompts gather information for SCM construction, experiments, and analysis.The system checks the LLM’s returned agent list for validity.Source: A Implementation details · p. 39

A.1 Constructing variables and drawing causal paths

  • The system builds simple linear SCMs by proposing an endogenous outcome and possible exogenous causes, with one outcome and a researcher-chosen number of causes.It currently does little hypothesis optimization, though it can test interaction terms.Source: §A.1 · pp. 39–40

A.1.1 Endogenous outcomes

  • For each endogenous outcome, the system specifies its operationalization, type, units, levels, measurement questions, and how answers will be aggregated into analysis data.When multiple questions measure an outcome, the system can combine responses using a selected aggregation method.Source: §A.1.1 · pp. 40–41

A.1.2 Exogenous causes

  • The system varies exogenous causes by assigning proxy attributes and treatment values to agents or scenarios, with scope and public or private status determining who receives the values.Default variation values are drawn from the variable’s levels or selected within those levels.Source: §A.1.2 · pp. 42–43

A.2 Building hypothesis-driven agents

  • Agents are constructed from roles and attributes; the system takes a minimalist approach, giving them goals, constraints, names, and proxy attributes while leaving broader attribute selection unresolved.The authors note that it is unclear which additional attributes are optimal, sufficient, or necessary.Source: §A.2 · p. 43

A.2.1 Assigning agents attributes

The system assigns attributes to agents, uses explicit natural-language goals, and selects interaction protocols to manage simulated conversations.

A.2.1 Assigning agents attributes

  • The system determines attributes for agents independently and cross-checks proxy values for feasible overlap before simulation.Figure A.1 illustrates bargaining agents; four values for each agent’s varied attribute produce 16 treatments.Source: §A.2.1 · pp. 43–45, Figure A.1 · p. 44

A.2.2 The importance of agent goals

  • Agents’ goals and constraints are expressed in natural language, and the system requires explicit goals while allowing researchers to vary or supply them.For example, a seller’s goal may be to obtain the highest price possible, with a constraint against accepting less than a minimum price.Source: §A.2.2 · p. 45

A.3 Simulation design and execution

  • Because agents speak sequentially and conversational turn-taking has no simple universal formula, the system selects from a menu of six interaction protocols.The authors describe speaking order as dependent on participants, setting, social dynamics, and emotional state.Source: §A.3 · pp. 45–46

A.3.1 Turn-taking protocols

The system offers turn-taking protocols for ordinary and central-agent interactions, plus coordinator-mediated options that allow more flexible control of conversations.

A.3.1 Turn-taking protocols

  • The menu includes predetermined and random speaker order, central-agent turn-taking, and two coordinator-mediated protocols.In coordinator-before, the coordinator selects the next speaker; in coordinator-after, it selects which agent response becomes part of the transcript.Source: §A.3.1 · pp. 46–48, Figure A.2 · p. 47
  • Central-ordered and central-random protocols alternate a central agent with non-central agents, whose speaking order is predetermined or random.The system determines the order when a protocol uses an order or central agent.Source: §A.3.1 · pp. 47–48
  • A separate coordinator can inspect transcripts and make simulation decisions; agents are unaware of it, and its use is the only part requiring quasi-omniscient supervision.Coordinator-after lets every agent respond privately, but only the coordinator sees those responses and realizes one as the actual response.Source: §A.3.1 · p. 48

A.3.2 Executing the experimental simulations

Simulations follow the selected interaction protocol, but the protocols do not specify when to stop. The system uses a coordinator decision and a statement limit to end each conversation.

A.3.2 Executing the experimental simulations

  • The system runs simulations in parallel, randomly sampling a subset when exogenous-variable combinations are too numerous to sample fully.Agents receive the scenario, private attributes, other agents’ roles, public or scenario-level attributes, and the conversation transcript.Source: §A.3.2 · p. 48
  • The protocols do not define stopping rules, and the paper notes that conversation endings are difficult to predict.The authors describe it as unclear how to construct an optimal or even good stopping rule.Source: §A.3.2 · p. 48
  • The coordinator checks after each agent speaks whether the conversation should continue, while a 20-statement cap limits each simulation.Agents are told how many statements remain; coordinator statements do not count toward the cap.Source: §A.3.2 · p. 49

A.3.3 Post-simulation survey and data collection

After each simulation, the system records survey-based outcomes alongside exogenous-variable values, converts responses into numeric data, and estimates the proposed structural causal model.

A.3.3 Post-simulation survey and data collection

  • Survey answers are recorded as outcome observations alongside exogenous-variable values; conditional outcomes without a reasonable answer are recorded as NA.The system asks agents or the coordinator the survey questions specified during structural causal model construction.Source: §A.3.3 · p. 49
  • The system uses an LLM to map responses to numeric values, applies the selected aggregation method for multiple questions, and stores one numerical column per model variable.Counts and continuous variables become integers or floats; ordinal and binary variables are mapped to integer sequences.Source: §A.3.3 · pp. 49–50
  • The 20-statement limit partly reflects GPT-4’s 8,192-token limit at the time of the simulations, because each turn resupplies the conversation to the agent.The note also describes the limit as partly a convenience.Source: §A.3.3 · p. 49

A.4 Path estimation & model fit

  • With the complete dataset and proposed model, the system estimates paths in lavaan and can standardize estimates, test interactions and nonlinear terms, and assess model fit.It also supports likelihood-ratio, Wald, and Lagrange Multiplier tests.Source: §A.4 · p. 50

A.5 Follow-on experiments

The paper describes follow-on experiments as a possible, not yet automated, extension. It also uses structural causal models to make hypotheses explicit as causal relationships and functional forms.

A.5 Follow-on experiments

  • Follow-on experiments are not automated, but could drop insignificant variables and test newly proposed ones alongside previously significant variables.The proposed loop could continue repeatedly, with new exogenous variables suggested in light of already tested paths.Source: §A.5 · p. 50

B Hypotheses as structural causal models

  • Natural-language hypotheses can leave causal relationships ambiguous, including the direction of a proposed mediated relationship.The paper uses alternative graphical interpretations of one natural-language hypothesis to illustrate the ambiguity.Source: B Hypotheses as structural causal models · pp. 50, 52, Figure A.3 · p. 51
  • A directed acyclic graph makes stated causal directions explicit: arrows indicate direction, while absent arrows indicate no stated relationship.Variables omitted from the graph have no stated causal relationships in that representation.Source: B Hypotheses as structural causal models · p. 52
  • Structural causal models state causal relationships and their functional forms, whereas DAGs do not specify functional form.The paper presents SCMs as sets of equations representing hypotheses.Source: B Hypotheses as structural causal models · p. 52
  • Assuming linear relationships, the paper writes equation-based SCM representations for graphical hypotheses, including Figures A.3a and A.3c.The passage introduces the equations for those two graphical interpretations.Source: B Hypotheses as structural causal models · p. 52
  • SCMs can also include interaction terms or use other link functions while representing the corresponding DAGs.These are presented as valid alternative representations of the same causal graph.Source: B Hypotheses as structural causal models · pp. 52–53

C Additional figures and tables

The appendix adds fitted-SCM figures across the four scenarios, reports GPT-4 path-estimate prediction tables, and compares auction clearing-price predictions with auction theory.

C Additional figures and tables

  • Figures A.4–A.7 show fitted SCMs with interaction terms for bargaining, bail, a lawyer job interview, and an art auction.Figures A.8 and A.9 show auction SCM variants using reservation prices and the second-highest bid as exogenous variables.Source: Figure A.4 · p. 54, Figure A.5 · p. 55, Figure A.6 · p. 56
  • The fitted-SCM figures report 405 bargaining simulations, 245 bail simulations, 80 job-interview simulations, and 343 auction simulations.The reported agents are buyer and seller; judge, defendant, defense attorney, and prosecutor; job applicant and employer; and three bidders and an auctioneer, respectively.Source: C Additional figures and tables · pp. 54–57
  • Tables A.1 and A.2 report GPT-4 path-estimate predictions at temperatures 0 and 1, with Table A.2 averaging 100 prompts.The tables compare predictions with experimental estimates, including predicted signs and differences in magnitude.Source: Table A.1 · p. 58, C Additional figures and tables · pp. 58–59, Table A.2 · p. 59
  • MSE was 8628 for LLM clearing-price predictions without the fitted SCM and 1505 with it, versus 128 for auction theory.The figure also tracks observed clearing prices across reservation-price combinations.Source: C Additional figures and tables · p. 61
  • The appendix identifies the prompt for eliciting LLM path-estimate predictions and gives an example of the information generated for each SCM variable.For each experiment, the accompanying table and TikZ diagram are input to the LLM.Source: Figure A.11 · p. 62, C Additional figures and tables · p. 62, Table A.3 · p. 63

Related papers

Loading 2404.11794v2…