PaperlayerView exact revision on arXiv

Source-linked AI summary

Cognitive Bias in Decision-Making with LLMs

Jessica Echterhoff, Yao Liu, Abeer Alessa, Julian McAuley, Zexue He

arXiv:2403.00811v3cs.AIcs.CL

TL;DR

LLMs can display societal and human-like cognitive-bias patterns that threaten fair, explainable decisions, especially in high-stakes settings. BiasBuster evaluates these patterns with a large prompt-based framework and tests mitigation strategies, including selfhelp, which debiases prompts automatically. The authors report that self-debiasing can remove bias-inducing prompt content and improve decision consistency, particularly in high-capacity models.

  • Problem

    LLMs used for decision support can exhibit societal and human-like cognitive-bias patterns that may undermine fair and explainable high-stakes decisions.

  • Method

    BiasBuster uses 13,465 science-informed prompts, quantitative bias metrics, and mitigation strategies, including selfhelp, which has models rewrite prompts to reduce cognitive-bias patterns.

  • Results

    Self-debiasing successfully removes bias-inducing prompt parts and enables more consistent decisions, with effectiveness specifically observed in high-capacity models.

  • Takeaways & Limitations

    BiasBuster provides a systematic way to audit and mitigate cognitive-bias patterns in LLMs used for decision-making.

  • Takeaways & Limitations

    The authors discourage using LLMs independently for high-stakes decisions such as student admissions.

Abstract

from arXiv · show

Large language models (LLMs) offer significant potential as tools to support an expanding range of decision-making tasks. Given their training on human (created) data, LLMs have been shown to inherit societal biases against protected groups, as well as be subject to bias functionally resembling cognitive bias. Human-like bias can impede fair and explainable decisions made with LLM assistance. Our work introduces BiasBuster, a framework designed to uncover, evaluate, and mitigate cognitive bias in LLMs, particularly in high-stakes decision-making tasks. Inspired by prior research in psychology and cognitive science, we develop a dataset containing 13,465 prompts to evaluate LLM decisions on different cognitive biases (e.g., prompt-induced, sequential, inherent). We test various bias mitigation strategies, while proposing a novel method utilizing LLMs to debias their own human-like cognitive bias within prompts. Our analysis provides a comprehensive picture of the presence and effects of cognitive bias across commercial and open-source models. We demonstrate that our selfhelp debiasing effectively mitigates model answers that display patterns akin to human cognitive bias without having to manually craft examples for each bias.

1 Introduction

LLMs are increasingly used for decisions but can exhibit societal and human-like cognitive bias that undermines reliable assistance. BiasBuster introduces a systematic framework for evaluating and mitigating these patterns, including automatic self-debiasing.

  • LLMs increasingly support everyday decisions, yet their algorithmic and human-like cognitive biases can mislead users and require auditing in high-stakes settings.
  • Cognitive bias is a systematic deviation from rational judgment that can produce inconsistent decisions, and LLMs may functionally resemble these patterns without possessing cognition.
  • BiasBuster evaluates cognitive-bias patterns with science-informed prompts categorized as biased or neutral, quantitative metrics, and comparisons across zero-shot and few-shot mitigation strategies.
  • Selfhelp asks an LLM to debias its own prompts, reducing manual prompt-example construction while providing an automatic mitigation procedure.
  • The framework uses a high-stakes student-admissions scenario with novel testing and debiased prompts to reduce cross-contamination and assess self-debiasing utility.

2 Related Work

Prior work documents many societal and reasoning-related biases in LLMs, but existing evaluations often focus on isolated biases, token-level measures, or one-off questions. BiasBuster addresses this gap with broader metrics for generative decision patterns resembling cognitive bias.

  • Existing LLM-bias research covers gender, religious, stereotype, occupational, sentiment, disability, and other social biases, but commonly evaluates one bias at a time.
  • Societal-bias metrics based on word embeddings do not directly capture cognitive-bias patterns expressed across current or preceding context.
  • Prior cognitive-bias studies propose metrics or test individual biases, but often rely on classification, dummy inputs, paraphrases, or one-off question-answer evaluations.

3 Testing for Patterns of Cognitive Bias in LLMs

BiasBuster tests cognitive-bias patterns in LLM decisions through sequential and prompt-based admissions experiments, using controlled perturbations, bias-specific comparisons, and a large prompt dataset. The framework operationalizes several bias definitions with measurable decision-consistency or group-difference criteria.

  • The framework distinguishes prompt-based bias from inherent training-data bias and sequential bias induced by previous model answers.
  • Sequential Bias: Sequential-bias experiments vary the order of identical student sets and include prior students and model decisions in context to test whether decisions change.
  • Sequential Bias: Sequential confidence compares overall and per-student admission probabilities using normalized Euclidean distance, where higher values indicate higher confidence.
  • Prompt-Based Cognitive Bias: Status quo bias is tested by comparing neutral and default-option admissions prompts, with bias indicated when the default option is selected more often than the uniform expectation.
  • Prompt-Based Cognitive Bias: Framing bias compares admissions decisions under positive and negative presentations, while group attribution bias tests whether changing only gender alters mathematical-ability assessments.
  • BIASBUSTER Prompt Dataset: The dataset contains 13,465 baseline prompts, with scalable cognitive-science-based templates and bias-specific variations for status quo, framing, and group-attribution conditions.

4 Mitigating Cognitive Bias in LLMs

The section compares zero-shot, few-shot, and self-mitigation strategies for reducing LLM decision patterns resembling cognitive bias. It motivates selfhelp as an unsupervised, generalized alternative to manually crafted examples, while noting added computation and sequential-bias constraints.

  • Mitigation Strategies: Zero-shot methods provide information about possible cognitive bias without examples, whereas few-shot methods provide examples and self-mitigation uses the model to debias itself.These approaches are grouped as distinct mitigation families.
  • Few-Shot-Mitigation: Few-shot mitigation relies on tailored examples of desired or undesired behavior, making representative examples difficult to craft without introducing new biases.The challenge depends on the bias, use-case setup, and number of biases involved.
  • Few-Shot-Mitigation: Contrastive mitigation presents one correct or incorrect example tailored to the bias’s main failure case, while counterfactual mitigation presents both kinds of behavior.The paper provides such examples for individual biases, including group attribution, framing, status quo, and anchoring.
  • Self-Help: Can LLMs debias their own prompts?: Selfhelp rewrites prompts to mitigate cognitive bias without manual adaptation, offering a generalized and scalable alternative to bias-specific examples.The method can be applied iteratively to address multiple biases, although the study focuses on one bias at a time.
  • Self-Help: Can LLMs debias their own prompts?: Selfhelp requires an additional forward pass per sample, and anchoring prompts cannot be directly debiased because the bias is induced by previous decisions.For anchoring, the model instead revises its decisions using the last prompt containing student profiles and prior decisions.

5 Results

Across four language models, BiasBuster reveals framing, group-attribution, anchoring, and primacy patterns, while mitigation effectiveness varies by strategy and model capacity. Selfhelp generally improves balance or removes bias-inducing prompt elements, but can produce severe decision inconsistency for anchoring and failures for some alternatives.

  • 5.1 LLMs Display Patterns Analogous to Human Cognitive Bias: Sequential anchoring evaluations show small decision confidence in the random-order setup, potentially reflecting influence from previous decisions and unawareness of bias.The GPT evaluation for some biases used 400 prompts per experiment, with results assumed to generalize because those biases were considered order-insensitive.
  • 5.1 LLMs Display Patterns Analogous to Human Cognitive Bias: GPT-4 admitted 40.5% more students under reject framing, while Llama 2 7B classified 32.1% fewer females as good at math, demonstrating model-specific framing and group-attribution patterns.Table 3 compares admission-rate differences across framing and group-attribution conditions and notes that mitigation methods mostly improve over the biased baseline, although some cause instruction-following failures.
  • 5.3 Few-Shot Debiasing Can Lead to Failures: Primacy bias favors early options A or B over later options C or D, and selfhelp produces a more balanced distribution than awareness prompting or failure-prone counterfactual methods.Awareness prompting helps Llama 2 and GPT-4 only to a small extent, while GPT-3.5-turbo has less capacity to debias itself.
  • 5.3 Few-Shot Debiasing Can Lead to Failures: Few-shot and counterfactual mitigation can cause extreme outputs or disregard answer options, including admission or rejection probabilities driven to zero or one.These failures undermine instruction following and can occur across status quo, anchoring, framing, and group-attribution tests.
  • 5.4 Models Can Remove Bias Patterns: Llama models change 40–52% of anchoring decisions when given all information and asked to debias post-hoc, indicating severe inconsistency between sequential and selfhelp setups.The authors conclude that anchoring selfhelp should be performed by high-capacity models or used to debias lower-capacity models cautiously.
  • 5.4 Models Can Remove Bias Patterns: Selfhelp removes bias-inducing prompt elements most successfully in higher-capacity models, reducing group-attribution and framing bias to 0% in some cases.For Llama 2 7B and 13B, remaining framing-bias prompts were 1.4% and 0%, while remaining status-quo-bias prompts were 6% and 0%, respectively; GPT-4 reduced group-attribution elements to 0%.

6 Conclusion

BIASBUSTER evaluates and mitigates LLM decision patterns resembling human cognitive bias, with self-debiasing aimed at more consistent decisions across biases.

  • The 13,465-prompt dataset tests inherent, prompt-based, and sequential cognitive-bias patterns in LLM decisions.
  • Self-debiasing lets models autonomously rewrite prompts to remove bias-inducing parts and produce more consistent decisions.The technique does not require manually developed debiasing examples and applies across a variety of biases.
  • Self-debiasing is specifically successful in high-capacity models.

7 Limitations and Future Work

The paper frames its data as a basis for continuous bias testing while acknowledging synthetic-data limitations and warning against independent LLM use in high-stakes decisions.

  • The dataset supports automated, large-scale testing of LLM decision inconsistencies in high-stakes tasks.The authors publish the data under a CC-BY NC license and intend it to facilitate mitigation of inconsistent decisions.
  • The study uses synthetic data to analyze high-stakes decisions without privacy concerns, while acknowledging its limitations.
  • The authors discourage using LLMs independently for high-stakes decisions such as student admissions.They aim to analyze model reasoning processes further to assess impacts on humans in human-AI collaboration.
  • The framework measures whether individual biases are mitigated, while interactions among multiple cognitive biases remain underexplored.The methodology can accommodate multiple biases through repeated mitigation or prompt modifications.
  • Experiments: Experiments use NVIDIA RTX A6000 hardware for open-source models or official APIs with a fixed random seed.

A.1.1 Status Quo

The status quo setup adapts a car-choice experiment to admissions by presenting an applicant’s current affiliation as potentially influencing selection.

  • The status quo scenario asks participants to choose a next car after revealing that they currently own a Honda Civic.The Honda Civic is one of three purchase options, alongside the Hyundai Venue and BMW X1.
  • The setup introduces status quo bias by potentially steering decisions toward maintaining the existing situation.

A.1.2 Primacy Bias

The appendix describes cognitive-bias scenarios and prompt-based mitigation examples, including order effects, framing, group attribution, and anchoring in admissions decisions.

  • A.1.2 Primacy Bias: Primacy-bias experiments vary whether positive or negative traits appear first, with initial positive information producing more favorable impressions.The result illustrates how presentation order can influence judgments.
  • A.1.2 Primacy Bias: Anchoring bias can arise from prior decisions in sequential admissions, where repeated rejections may affect subsequent judgments.The anchor is information encountered earlier that decision-makers rely on excessively.
  • A.1.2 Primacy Bias: Framing experiments show that mathematically equivalent survival and mortality descriptions can produce different choices.
  • A.1.2 Primacy Bias: Admissions prompts instruct models to decide admit or reject from applicant information, with self-help variants removing explicit gender cues.
  • A.1.2 Primacy Bias: Self-help prompt examples remove or rephrase bias-relevant words, while counterfactual and contrastive prompts provide alternative mitigation formats.GPT-4 examples document self-help debiasing, and separate examples cover counterfactual and contrastive mitigations.
  • A.1.2 Primacy Bias: Primacy and sequential-bias examples contrast incorrect and correct decisions after changing the ordering or contextual information.
Loading 2403.00811v3…