Behavioral Economics of AI: LLM Biases and Corrections

• The study tests four LLM families across older versus advanced versions and, for advanced models, large versus small parameter scales, using cognitive-psychology and experimental-economics questions.The experiments draw on preference and belief tasks designed to document human biases and adapt financial decision-making tasks for LLMs.Source: §1 · p. 2
• LLM families differ substantially: Gemini is more human-like on preferences, whereas Meta Llama is more human-like on beliefs; Claude and Gemini otherwise resemble ChatGPT on beliefs.Source: §1 · p. 4
• Smaller models perceive autoregressive processes as more persistent than true, while larger models’ perceived persistence is similar to true; larger models’ investment depends more strongly on visual salience.Source: §1 · p. 4
• Brief role priming that instructs models to act as rational investors using Expected Utility makes responses more rational and less human-like across both task types, but improvements are economically modest and other debiasing methods are ineffective.Role priming also changes assigned confidence and reported reasoning type, leaving debiasing a challenging task.Source: §1 · pp. 4–5

Research motivation

• The study systematically investigates LLM behavior rather than treating models as neutral tools, documenting biases and heterogeneity across preference-based and belief-based questions.This matters for assessing the reliability of LLM-based experiments and simulations.Source: §1 · p. 6

Contribution

• The introduction frames the work as among the first systematic studies of the behavioral economics of AI, treating GenAI agents as a novel class of economic agents.Prior studies suggest that LLM behavior can mirror human behavior while remaining sensitive to prompt framing, training data, and model architecture.Source: §1 · p. 6

Research scope

• The paper examines multiple prominent LLM families, including cross-sectional and time-series variation within each family.The authors note that longer data samples are needed to study time-series variation in LLM responses.Source: §1 · p. 6
• It studies behavioral biases in both preference-based and belief-based questions, using insights from cognitive psychology and experimental economics and covering financially important biases.This extends literatures that have documented behavioral biases in human participants.Source: §1 · pp. 5–6

Bias mitigation

• The paper investigates methods to mitigate LLM biases by comparing existing approaches and proposing new ones.Its broader contribution is to establish a foundation for documenting LLM biases and exploring debiasing methods.Source: §1 · p. 7

2.1. Selection of Experimental Questions

• The study uses established cognitive-psychology and experimental-economics tasks to classify LLM responses as rational, human-like, or non-human-like and compare them with human behavior.Questions cover prospect-theory preferences, overextrapolation, overconfidence, forecasting beliefs, and investment decisions.Source: §2.1 · pp. 8–9
• Belief experiments compare the true autoregressive coefficient ρ with the perceived coefficient implied by LLM forecasts across a wide range of ρ values, while investment experiments mimic real-world investment tasks.Source: §2.1 · p. 10

2.2. Selection of LLMs

• The sample contains twelve models from ChatGPT, Anthropic Claude, Google Gemini, and Meta Llama, with each family represented by a benchmark, smaller-scale model, and predecessor.The benchmark is the most recent and best-performing model available when written, enabling comparisons across families and model versions.Source: §2.2 · p. 10
• The selected models differ in training-data size, architecture, and reinforcement-learning algorithm, with newer models generally trained on larger datasets.For example, Llama 3 used over 15 trillion training tokens versus 1.8 trillion for Llama 2.Source: §2.2 · pp. 10–11
• Models vary across families and generations in context-window length and architecture: Claude 2 is estimated at 200 billion parameters with approximately 100,000 tokens, whereas Llama 2 has 70 billion and approximately 4,000 tokens.Architectures evolved significantly within families, including Gemini’s transition from a single-transformer to a multi-transformer mixture-of-experts architecture.Source: §2.2 · p. 11

2.3. Prompt Design

• Prompts are designed to produce standardized responses while preserving comparability with the original human experiments.Each prompt combines general scenario instructions, a JSON-formatting code block, and the original experimental questions, with additional instructions specifying the elicited responses.Source: §2.3 · p. 12
• Each experimental-scenario response contains four components—choice, confidence, explanation, and reasoning—with confidence measured on a score between 0 and 1.Reasoning includes type “A,” which corresponds to reasoning based more on intuition.Source: §2.3 · pp. 12–13
• The design treats each API call as an individual participant and combines multiple scenarios to implement within-subject comparisons while accounting for stochastic responses.Prompts instruct the model to treat scenarios as separate, and repeated calls can yield different responses from the same model.Source: §2.3 · p. 13
• Graphical experiments exclude platforms without image support, require concrete numerical answers, and preserve sequential forecast dependence by feeding the full conversation history into successive API calls.Six platforms are excluded because they do not support graphical inputs; Afrouzi et al. (2023) experiments use sequential calls with prior prompts and responses included.Source: §2.3 · pp. 13–14

3.1. Baseline Results

• The baseline analyzes four benchmark models, collecting 100 responses for each question and each model.Responses are categorized as rational, human-like, or other; human-like responses are irrational but align with the most common behavior observed in human participants.Source: §3.1 · pp. 14–15

3.2. Heterogeneity in LLM Responses

• The heterogeneity analysis expands to twelve models spanning four LLM families, model generations, and model scales, separating preference-based from belief-based questions.It compares advanced large-scale benchmark models, advanced smaller-scale models, and older large-scale models while controlling for LLM family fixed effects.Source: §3.2 · pp. 16–17

3.2.1. Heterogeneity across LLM families

• For preference-based questions, Gemini models are 22.9% less likely than GPT models to produce a rational response, with this effect significant at the 1% level.Graphical evidence shows that Gemini 1.5 Pro produces a lower share of rational responses than GPT-4.Source: §3.2.1 · p. 17
• Across families, models are 16.7% more likely than GPT models to produce human-like responses on preference-based questions, while Claude and Llama responses are statistically similar to GPT.The 16.7% difference is significant at the 5% level.Source: §3.2.1 · p. 18
• For belief-based questions, Llama models are 25.0% less likely than GPT models to produce rational responses but 21.0% more likely to produce human-like responses.Both effects are significant at the 5% level, while Claude and Gemini responses are statistically similar to GPT models.Source: §3.2.1 · p. 18

3.2.2. Heterogeneity across model generations and model scales

• The generation and scale comparisons match the graphical patterns after controlling for LLM family fixed effects, comparing advanced with older models and large-scale with smaller-scale models.The study uses probit regressions with rational or human-like response indicators as outcomes.Source: §3.2.2 · pp. 18, 20
• More advanced or larger LLMs become more human-like on preference-based questions but more rational on belief-based questions, according to Fig. 3 and Table 5.For preference-based questions, rational responses decrease and human-like responses increase; for belief-based questions, rational responses increase and human-like responses decrease. The increase in human-like preference responses with scale is insignificant, while the other reported patterns are generally statistically significant.Source: §3.2.2 · pp. 18, 20
• Advanced large-scale models generate largely rational forecasts, while smaller models produce human-like excess persistence, especially when true persistence is low.For advanced models, estimated persistence is close to true persistence; for smaller models, it is significantly higher, and the gap is larger at lower true persistence.Source: §3.2.2 · p. 22
• Longer-term forecasts from advanced large-scale models exhibit human-like persistence biases absent from short-term forecasts, with larger gaps at lower true persistence.Providing detailed information about the data-generating process is counterproductive: it induces more human-like biases in LLM responses, unlike human forecasts.Source: §3.2.2 · p. 22
• As an LLM increases in scale, its investment decisions depend more strongly on non-traditional preferences, with interaction terms positive and mostly statistically significant, reinforcing patterns documented for human behavior.Source: §3.2.2 · p. 24

Role priming

• Role priming as rational investors increases rational responses by 4.3% for preference-based questions and 3.3% for belief-based questions, whereas retail-investor priming reduces preference-based rational responses by 3.9% and does not significantly affect belief-based responses.The rational-investor effects are significant at the 5% and 10% levels, respectively; the retail-investor reduction is significant at the 5% level.Source: §4 · p. 25

Mediation analysis

• Role priming affects LLM responses through confidence and reasoning type: rational-investor priming increases high confidence by 6.6% and system 2 thinking by 11.4%, while retail-investor priming decreases them by 6.0% and 8.0%.Both rational-investor effects are significant at the 1% level.Source: §4 · p. 26
• System 2 thinking increases rational responses by 40% under rational-investor priming and 38% under retail-investor priming, while high confidence decreases human-like responses by 35% and 58%, respectively.Including confidence and reasoning type as controls significantly attenuates the estimated role-priming effects.Source: §4 · p. 26

Additional debiasing techniques

• The section introduces two additional debiasing techniques: combining rational-investor priming with a four-step expected-utility procedure, or with a summary of Kahneman and Tversky (1979) followed by an instruction to avoid prospect-theory mistakes.The expected-utility procedure lists wealth outcomes, computes utility using a globally concave function, probability-weights outcomes, sums expected utility, and selects the highest-utility action.Source: §4 · p. 26

4. Correcting LLM Biases

• The four-step procedure intended to guide rational behavior is ineffective at reducing biases.This analysis covers prospect-theory questions on diminishing sensitivity, loss aversion, and probability weighting.Source: §4 · p. 28
• Providing key findings from Kahneman and Tversky (1979) reduces rational responses by about 29% and increases human-like responses by about 20%.The two detailed debiasing techniques are therefore either ineffective or counterproductive.Source: §4 · p. 28
• Additional information, even when useful for decision making, can hinder rational responses through information overload.This result is consistent with Section 3.3, where more information about the data-generating process increased human-like biases in Afrouzi et al. (2023) experiments.Source: §4 · p. 28
• Role priming that instructs LLMs to act as rational investors is effective, although its improvement is economically modest.The section concludes that debiasing LLMs remains challenging.Source: §4 · p. 28

Scope and method

• Responses vary substantially across ChatGPT, Anthropic Claude, Google Gemini, and Meta Llama, which the study evaluates using established cognitive-psychology and experimental-economics designs.The analysis examines four prominent LLM families and documents systematic behavioral-bias patterns.Source: §5 · pp. 28–29

Core findings

• LLMs become increasingly irrational and human-like on preference questions as models become more advanced or larger, while advanced large-scale models remain largely rational on belief questions.For preference-based questions, irrational responses often exceed 50% at baseline, making a reported 3–4% reduction small relative to baseline.Source: §5 · pp. 28–29

Role priming and debiasing

• Prompting LLMs to act as rational investors using Expected Utility reduces biases, whereas retail-investor role prompts produce less rational responses.Role priming changes confidence and reasoning type, but its overall economic impact is modest.Source: §5 · p. 29

Information interventions

• Providing a rational decision procedure or Kahneman and Tversky’s (1979) bias summary does not reduce LLM biases and can be counterproductive.The tested information interventions consist of detailed Expected Utility guidance or findings describing human biases.Source: §5 · p. 29

Advanced large-scale models

• Figure 2 reports rational, human-like, and other response proportions for four advanced large-scale models across six preference-based and ten belief-based questions.The models are GPT-4, Claude 3 Opus, Gemini 1.5 Pro, and Llama 3 70B.Source: Figure 2 · p. 36, Figures and Tables · p. 36

Heterogeneity across generations and scales

• Advanced models differ from older and smaller models in the prevalence of rational and human-like responses across preference- and belief-based questions.Figure 3 compares predominantly rational and human-like responses across model generations and scales, while Table 5 reports the corresponding probit-regression marginal effects.Source: Figure 3 · p. 37, Figures and Tables · p. 37, Table 5

LLM forecasts

• Figures 4 and 5 compare perceived persistence ˆρ with true persistence ρ against the 45-degree FIRE benchmark across Afrouzi et al. (2023) forecasting experiments.Figure 4 covers Experiment 1 for advanced large- and small-scale models, while Figure 5 covers Experiments 2 and 3 for advanced large-scale models; vertical bars show 95% confidence intervals.Source: Figure 4 · p. 38, Figures and Tables · pp. 38–39, Figure 5 · p. 39

Experimental questions

• The experimental benchmark contains sixteen questions from cognitive psychology covering preference biases such as diminishing sensitivity, loss aversion, probability weighting, narrow framing, ambiguity aversion, and hyperbolic discounting.The questions also cover belief-related phenomena including sample-size neglect, anchoring, selection, general knowledge, and overestimation, drawing on the cited psychology and economics literature.Source: Figures and Tables · p. 40, Table 1

Model coverage

• The analysis examines twelve models from the ChatGPT, Anthropic Claude, Google Gemini, and Meta Llama families, grouping advanced large-scale models against smaller-scale versions and predecessors.The model table also records release year, estimated parameters and tokens, instruction method, context window, MMLU score, and vision support.Source: Table 2 · p. 40, Figures and Tables · p. 41

Investment and role priming

• The investment and role-priming tables specify analyses of stock-investment decisions and prompts that prime LLMs as rational or real-world retail investors.The investment regressions relate the fraction of 1,000 endowed monetary units invested to visual-salience-weighted returns and their interaction with model scale; the supplied role-priming passages describe treatment specifications and mediation mechanisms but not their estimates.Source: Table 6 · p. 45, Figures and Tables · pp. 45–47, Table 7

Table 9: Prospect theory-related questions

• Role-priming increases rational classifications by 0.0375*** and 0.0401***, while reducing human-like classifications by 0.0225** and 0.0267**.The estimates come from models without and with model fixed effects, respectively.Source: Figures and Tables · p. 48
• Instruction-based prompting yields rational coefficients of –0.0987 and –0.0985 and human-like coefficients of 0.0836 and 0.0874.These estimates compare the instruction-based prompt with rational-investor role priming, with and without model fixed effects.Source: Figures and Tables · p. 48
• In probit regressions, knowledge-enrichment prompting yields rational-response marginal effects of –0.291*** and –0.287*** and human-like-response marginal effects of 0.203* and 0.204*.The prompt combines rational-investor priming with a summary of key findings from Kahneman and Tversky (1979) describing systematic human biases.Source: Figures and Tables · p. 48

Advanced small-scale models

• For four advanced small-scale LLMs, the figure reports rational, human-like, and other response proportions across six preference-based and ten belief-based questions.The models are GPT-4o, Claude 3 Haiku, Gemini 1.5 Flash, and Llama 3 8B; preference-based questions appear in the left panel and belief-based questions in the right.Source: §I · p. 50, Figure IA.1 · p. 50

Older models

• For four older LLM versions, the figure reports rational, human-like, and other response proportions across six preference-based and ten belief-based questions.The models are GPT-3.5 Turbo, Claude 2, Gemini 1.0 Pro, and Llama 2 70B; preference-based questions appear in the left panel and belief-based questions in the right.Source: §I · p. 51, Figure IA.2 · p. 51

Prospect theory

• Prompts test diminishing sensitivity, loss aversion, and probability weighting as core elements of prospect theory, drawing on Kahneman and Tversky (1979) and Barberis and Thaler (2003).Each prompt elicits a structured choice, confidence, explanation, and reasoning response.Source: §II · pp. 52–54

Intertemporal and uncertainty preferences

• The design also measures narrow framing, hyperbolic discounting, and ambiguity aversion using established questions from Tversky and Kahneman (1981), Frederick, Loewenstein, and O’Donoghue (2002), and Ellsberg (1961).The narrow-framing question changes the original “20 minutes” scenarios to “5 minutes” to account for inflation between 1981 and 2024.Source: §II · pp. 55–57

Judgment biases

• Judgment prompts cover sample size neglect, base rate neglect, conjunction fallacy, and gambler’s fallacy, using questions designed or discussed in the cited literature.Sample-size prompts use Tversky and Kahneman (1974), Well, Pollatsek, and Boyce (1990), and Bar-Hillel (1979); other prompts use Kahneman and Tversky (1973), Tversky and Kahneman (1983), and Rabin (2002).Source: §II · pp. 58–63

Overconfidence

• The overprecision prompt adapts ten general-knowledge questions from Deaves, L¨uders, and Luo (2009) to assess overconfidence through interval responses and reasoning.The requested output includes lower and upper bounds for each question and a reasoning field.Source: §II · pp. 67–68
• A separate investment-knowledge prompt assesses overestimation by pairing ten Financial Industry Regulatory Authority quiz questions with LLM-estimated response accuracy, following Moore and Healy (2008).The output records each answer, reasoning, and an integer accuracy estimate.Source: §II · p. 69