PaperlayerView exact revision on arXiv

Source-linked AI summary

The Challenge of Using LLMs to Simulate Human Behavior: A Causal Inference Perspective

George Gui, Olivier Toubia

arXiv:2312.15524v3cs.AIcs.IRecon.EMstat.AP

TL;DR

The paper asks how to simulate counterfactual human behavior when LLM prompts leave unspecified factors that should remain constant. It develops an unblinding approach based on clarifying experimental design, and reports improved simulation performance across model generations, with benefits alongside fine-tuning.

  • Problem

    In counterfactual simulations, blind prompts can let treatment systematically alter unspecified variables that should remain constant, challenging valid causal inference.

    Source: §1 · p. 2, §2 · p. 5

  • Method

    The paper develops a theoretical account of prompt ambiguity and evaluates unblinding by explicitly communicating experimental design in LLM simulations.

    Source: §6 · p. 18

  • Results

    Unblinded prompts improve simulation accuracy across model generations and are especially valuable with fine-tuning, where they help distinguish causal from correlational patterns in training data.

    Source: §6 · p. 18

  • Takeaways & Limitations

    Unambiguous prompting is a design principle that complements model-improvement techniques and can support more accurate counterfactual simulations.

    Source: §6 · p. 18

    Adding detailed context to control confounders can make explicitly mentioned variables artificially salient, creating a trade-off with ecological validity.

    Source: §6 · p. 18

Abstract

from arXiv · show

Large Language Models (LLMs) have shown impressive potential to simulate human behavior. We identify a fundamental challenge in using them to simulate experiments: when LLM-simulated subjects are blind to the experimental design (as is standard practice with human subjects), variations in treatment systematically affect unspecified variables that should remain constant, violating the unconfoundedness assumption. Using demand estimation as a context and an actual experiment with 40 different products as a benchmark, we show this can lead to implausible results. While confounding may in principle be addressed by controlling for covariates, this can compromise ecological validity in the context of LLM simulations: controlled covariates become artificially salient in the simulated decision process. We show formally that confoundness stems from ambiguous prompting strategies. Therefore, it can be addressed by developing unambiguous prompting strategies through unblinding, i.e., revealing the experiment design in LLM simulations. Our empirical results show that this strategy consistently enhances model performance across all tested models, including both out-of-box reasoning and non-reasoning models. We also show that it is a technique that complements fine-tuning: while fine-tuning can improve simulation performance, an unambiguous prompting strategy makes the predictions robust to the inclusion of irrelevant data in the fine-tuning process.

1 Introduction

The paper studies how LLMs can simulate counterfactual human behavior, where researchers vary one factor while holding others constant. It argues that blinded prompts can create confounding in these simulations and that unblinding offers a way to make prompts unambiguous.

1 Introduction

  • When simulated participants are blind to experimental conditions, treatment variations can systematically change unspecified variables that should remain constant, violating unconfoundedness.Unlike human subjects and contexts, LLM-generated subjects and contexts are created from prompts.Source: §1 · p. 3
  • Across 40 products, blinded GPT simulations produce an inverted-U demand curve, unlike the downward-sloping curve from human participants.The paper also reports that price variation changes unspecified variables, including purchase history and competing prices, which violates unconfoundedness.Source: §1 · p. 3
  • Adding covariates to prompts may reduce confounding but compromise ecological validity by making named variables artificially salient to the LLM.The paper describes this as a tension that typically does not arise when covariates can be measured and controlled after human data collection.Source: §1 · p. 3
  • The paper’s theoretical framework attributes confoundedness in blind simulations to ambiguous prompts and proposes revealing experimental design as an unambiguous alternative.The authors argue this issue is not specific to a particular model or training dataset.Source: §1 · pp. 2–3
  • The paper aims to establish LLM-specific experimental design principles because conflicting simulation results lack a clear basis for assigning weight without shared standards.It highlights trade-offs among blinding, unconfoundedness, and ecological validity.Source: §1 · p. 4

2 Unintended Confounding in Blind LLM Simulations

The section uses demand estimation to examine how blind LLM prompts can create unintended correlations between treatment and unspecified variables. It compares simulated demand with human results and explains why those correlations matter for causal estimation.

2 Unintended Confounding in Blind LLM Simulations

  • Blind prompts leave details unspecified, allowing LLM assumptions about those variables to vary systematically with treatment and introduce confounding.Human random assignment balances pre-treatment variables and keeps contextual variables constant; LLM-generated subjects and environments are dynamic prompt outputs.Source: §2 · p. 5
  • The demand-estimation experiment covers 40 top-selling products from CPG categories, testing 11 prices from zero to 200% of each regular price.For each prompt, the study generated 50 responses at temperature 1 using gpt-4o-mini-2024-07-18.Source: §2 · p. 5
  • The human comparison used a survey of 1,000 respondents, with one purchase-intention question per product and randomized prices and product order.Respondents were recruited from Prolific for a sample intended to represent the US population based on census data.Source: §2 · p. 6
  • Human participants answered all 40 questions, whereas GPT used an independent API call for each question, a procedural difference the authors acknowledge.The authors describe the independent-call approach as standard practice.Source: §2 · p. 6
  • Figure 1 shows positive correlations between relative focal-product price and past price, competing-product price, and expiration days.These associations may be plausible in observational data but are undesirable when estimating price changes while holding other factors constant.Source: §2 · p. 6, Figure 1 · p. 7
  • Human demand curves slope downward, while GPT’s curve has an inverted-U shape that persists after removing the zero-price observation.Figure 2 compares curves averaged over 40 products.Source: §2 · p. 7, Figure 2 · p. 8

3 Potential Solution: Controlling for Covariates

Controlling for covariates can reduce confounding in LLM simulations, but prompt-level controls may change what simulated consumers attend to and compromise ecological validity. Empirically, added covariates have non-monotonic effects on accuracy, and fixing competing prices can produce implausible demand curves.

3 Potential Solution: Controlling for Covariates

  • Controlling for demographic variables improves average results over prompts without details, but does not fully resolve the issue.The comparison concerns demand curves elicited by the LLM.Source: Figure 3 · p. 9, §3 · p. 9
  • For Coca-Cola, fixing the competing price produces a step-function purchase rule rather than a smooth demand curve.Simulated customers purchase if and only if the focal product's price is no higher than the competing price.Source: Figure 4 · p. 10, §3 · p. 10
  • Prompt-level covariates can change decision factors, unlike post-collection adjustment, which leaves the original data-generating process unchanged.Covariates in prompts enter at data generation and can affect what simulated individuals consider.Source: §3 · pp. 9–10
  • Adding covariates can trade improved unconfoundedness for ecological validity when specified variables receive undue attention.The authors contrast prompt-level control with standard experimental analysis, where adding covariates typically improves performance.Source: §3 · pp. 9, 11
  • A sequential procedure adds consumer traits across 12 stages, expanding controls from 14 demographic variables to 30 total variables.The procedure uses a publicly available dataset that includes a replication of the pricing study on the same consumer panel.Source: §3 · p. 11
  • MAE improves with initial covariate additions but worsens beyond a threshold, producing a non-monotonic pattern.The authors suggest that explicit covariates can induce focalism and reduce ecological validity.Source: §3 · p. 11, Figure 5 · p. 12

4 Core Issue: Ambiguous Prompting Strategy

The paper defines ambiguity as using the same prompt for distinct questions with different correct answers. It argues that no LLM can answer both correctly under such a strategy, and that adding prompt details can multiply possible interpretations.

4 Core Issue: Ambiguous Prompting Strategy

  • An ambiguous prompting strategy maps distinct questions with different correct answers to the same prompt.A prompting strategy maps questions to prompts.Source: §4 · p. 12
  • For any fixed ambiguous prompting strategy, no LLM can correctly answer all questions; answering one of two ambiguous questions correctly means answering its counterpart incorrectly.The result is stated as an impossibility theorem.Source: §4 · p. 12
  • The interventional question asks what happens when price is set, whereas the observational question asks what happens when price happens to take a value.The answers differ: the intervention is represented by P(Y | do(D)), while the observational answer is P(Y | D).Source: §4 · p. 13
  • Adding variables to a prompt can increase causal ambiguity by allowing more valid interpretations.The paper illustrates this possibility with a detailed prompt example.Source: §4 · p. 13

1. Price as Treatment: P(Y |do(price), health info, sugar content)

The examples distinguish three explicit interventions: setting price while holding health awareness fixed, exposing customers to health information while holding other factors fixed, and highlighting sugar content.

1. Price as Treatment: P(Y |do(price), health info, sugar content)

  • The price-treatment example asks about experimentally setting price while holding customers' health awareness fixed.The prompt gives $4.99 as the price example.Source: §1 · p. 13
  • The health-information example treats exposure to health articles while holding price and sugar content fixed.The example uses a price of $4.99.Source: §1 · p. 14
  • The sugar-content example treats highlighting sugar content while holding price and prior health awareness fixed.Source: §1 · p. 14

5 The value of unblinding

Unblinding communicates the randomized treatment and its distribution, making prompts less ambiguous. It improves predictions both in out-of-the-box models and alongside fine-tuning, including when irrelevant observational data is added.

5 The value of unblinding

  • Unblinded prompts specify both the randomized treatment and the distribution from which it is drawn, communicating the experiment’s data-generating process.The authors test this approach first with out-of-the-box models and then in fine-tuning experiments.Source: §5 · p. 14
  • Unblinding improved prediction performance by 1% to over 60% relative to baseline; MAE reductions persisted across all tested models, including reasoning models, and the advantage remained substantial as models advanced.Source: §5 · p. 15, Figure 6 · p. 15
  • Without experimental context, a fine-tuned model may learn price–purchase correlations without a basis for interpreting them as causal.The authors argue that stating price was randomized may help the model interpret the learned relationship causally and generalize.Source: §5 · p. 15
  • The fine-tuning evaluation used leave-one-category-out validation across four product groups, with MAE measured against human-survey purchase probabilities.Each fold trained on three groups and tested on the held-out group.Source: §5 · p. 16
  • 15% lower MAE (0.134 to 0.113) was achieved by fine-tuning and evaluating with unblinded prompts, compared with the best-performing blinded model.The authors report this as the most accurate combination in the survey-data experiment.Source: §5 · p. 16, Table 1 · p. 16
  • Unblinded prompts made fine-tuned models more robust when an irrelevant observational Amazon dataset was included, while blinded models degraded significantly.The added dataset recorded purchases but not non-purchases.Source: §5 · p. 17, Table 2 · p. 17

6 Conclusion

The conclusion argues that prompt-generated subjects and contexts make blinded LLM experiments vulnerable to confounding. It presents unblinding as a complementary design practice that improves simulation accuracy while noting the ecological-validity trade-off of explicitly controlling covariates.

6 Conclusion

  • Because LLM-simulated individuals and environments are generated from prompts, changing treatment can unintentionally alter unspecified variables meant to remain constant.The resulting confounding makes counterfactual human behavior difficult to simulate accurately.Source: §6 · pp. 17–18
  • Adding detailed covariate context can reduce confounding but may cause simulated individuals to overweight explicitly mentioned variables, compromising ecological validity.The conclusion frames this as a trade-off in designing detailed prompts.Source: §6 · p. 18
  • The authors’ framework attributes confounding to ambiguous prompting strategies rather than model limitations, and proposes unambiguous prompts as a complement to other model improvements.They present this as a design principle for LLM simulations.Source: §6 · p. 18
  • Unblinded prompts that communicate randomization and experimental context improved simulation accuracy across model generations and complemented fine-tuning.The authors report particular value when fine-tuning, where this context helps distinguish causal from correlational patterns.Source: §6 · p. 18
  • The authors caution that blinded evaluations may mistake prompt ambiguity for model limitations, and that ambiguous prompts can produce conflicting interpretations across research teams.They also advise avoiding ambiguous prompts during model development and fine-tuning.Source: §6 · p. 18
  • The paper calls for experimental practices and evaluation principles tailored to LLMs rather than simply transferring human-subject protocols.It describes LLM simulation as a nascent field in which best practices have yet to be established.Source: §6 · pp. 18–19

A Category, Product and Price Information

The appendix identifies the 40 products and their regular prices used in the demand-estimation study, and points to prompts documenting how correlations were elicited.

A Category, Product and Price Information

  • The product set covers 40 categories and top-selling products, with regular prices recorded from Walmart.com around April 2024.The appendix table documents the categories, products, and prices; actual prices may vary because retail prices change.Source: Table A.1 · p. 22, A Category, Product and Price… · p. 22

B Simple prompts for eliciting correlation

C Covariance of unspecified variables

The appendix reports that price covaries with product, contextual, and pre-treatment variables in the simulated data. It also describes a two-step procedure that holds generated demographic personas fixed while varying price to elicit purchase decisions.

C Covariance of unspecified variables

  • The covariance matrix shows focal price correlated with expiration days, competing prices, and purchase history, despite realistic correlations among some covariates such as income and budget.The authors note these variables would be held constant or orthogonal to treatment in a human experiment.Source: C Covariance of unspecified variables · p. 24, Figure C.1 · p. 24

D Simulation that controls for demographic variables

  • The demographic-control simulation first generated 500 personas per category–product pair, then varied price for each persona and aggregated their purchase probabilities.Demographics were generated with temperature 1 and purchase decisions elicited at temperature 0.Source: D Simulation that controls for… · p. 26

D.1 Simulation that also controls for competing price

The appendix illustrates how adding covariates and using ambiguous prompts can distort simulated experimental demand. It also formalizes why a single ambiguous prompt cannot correctly answer distinct questions with different answers.

D.1 Simulation that also controls for competing price

  • With competing price fixed, simulated Coca-Cola demand becomes step-wise and non-smooth, implausible when customers vary in attentiveness.The simulation fixed the competing product’s price at the focal product’s regular price when eliciting purchase decisions from 500 personas.Source: §D.1 · pp. 27–28

E Non-monotonic impact of controlling for covariates

F.1 Impossibility of Ambiguous Prompting

  • For an ambiguous prompting strategy, no LLM can correctly answer every question when distinct questions with different answers map to the same prompt.The proof shows that answering one question correctly forces the same output for its differently answered counterpart.Source: §F.1 · p. 31

F.2 Illustration of the underlying DGP that LLM simulation mimics if prompts are interpreted differently

  • Under a blinded ambiguous prompt, an LLM may mimic observational rather than experimental data patterns, reflecting confounded relationships instead of intended causal effects.The illustration contrasts possible data-generating processes under the ambiguous prompt.Source: §F.2 · p. 32, Figure F.1 · p. 32
  • Adding covariates may restore identification if Cov(U, D|X) = 0, but can shift decision mechanisms and reduce ecological validity.The resulting estimate may reflect mechanisms M′ rather than the realistic mechanisms M of interest.Source: §F.2 · p. 32

F.3 Implications for Model Development

The section explains why fine-tuning on experimental data with ambiguous prompts may not recover the causal relationship and may fail to generalize across scenarios. It presents fine-tuning and retrieval-augmented generation as complements to unambiguous prompts.

F.3 Implications for Model Development

  • With ambiguous prompts, fine-tuning on randomized experimental data cannot guarantee learning βM and may instead learn a biased regression coefficient.The stated coefficient is ˆb = βM + Cov(U, D)/Var(D), under a prior that Cov(U, D) ≠ 0.Source: §F.3 · pp. 32–33
  • A model that learns independence assumptions in one training scenario may incorrectly retain them in a new scenario where the variables are correlated with unobserved factors.The example contrasts randomized Coca-Cola prices with fixed Pepsi prices against varying Pepsi prices with fixed Coca-Cola prices.Source: §F.3 · p. 33
  • Fine-tuning and RAG can incorporate domain knowledge, but the authors characterize them as complements to, not substitutes for, unambiguous prompt designs.The figure labels adding X as potentially addressing confounding while also potentially sacrificing ecological validity.Source: §F.3 · p. 33, Figure F.2 · p. 33

G Fine-tuning with additional irrelevant data

The section describes a robustness test that adds Amazon data to fine-tuning.

G Fine-tuning with additional irrelevant data

  • The robustness test samples 10,000 Amazon observations and incorporates them into fine-tuning using Prompt.The passage describes the test setup but provides no outcome.Source: G Fine-tuning with additional… · p. 33

Related papers

Loading 2312.15524v3…