PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating and Mitigating Discrimination in Language Model Decisions

Alex Tamkin, Amanda Askell, Liane Lovitt, Esin Durmus, Nicholas Joseph, Shauna Kravec, Karina Nguyen, Jared Kaplan, Deep Ganguli

arXiv:2312.03689v1cs.CL

TL;DR

High-stakes language-model decisions raise concerns about discrimination, while real-world audits can be resource-intensive and constrained by privacy and ethical considerations. The paper evaluates hypothetical scenarios with demographic-varied prompts and finds discrimination patterns in Claude 2.0 that prompt engineering can reduce.

  • Problem

    Real-world audits of language-model decisions can be resource-intensive and constrained by privacy and ethical concerns, while risks also arise in applications not yet developed.

    Source: §1 · p. 1

  • Method

    The authors generate decision-question templates with demographic placeholders and compare model decisions as those demographic details vary.

    Source: §2 · p. 4

  • Results

    Claude 2.0 exhibits positive and negative discrimination in select settings without intervention, and careful prompt engineering significantly reduces both.

    Source: §1 · p. 2, §3.3 · p. 6

  • Takeaways & Limitations

    The approach is intended to help developers and policymakers anticipate, measure, and address potential discrimination before real-world deployment.

    Source: §1 · p. 2, §8 · p. 12

    The evaluation mainly uses paragraph-length candidate descriptions and does not assess impacts on user decisions in human-in-the-loop settings.

    Source: §6.1 · p. 10

Abstract

from arXiv · show

As language models (LMs) advance, interest is growing in applying them to high-stakes societal decisions, such as determining financing or housing eligibility. However, their potential for discrimination in such contexts raises ethical concerns, motivating the need for better methods to evaluate these risks. We present a method for proactively evaluating the potential discriminatory impact of LMs in a wide range of use cases, including hypothetical use cases where they have not yet been deployed. Specifically, we use an LM to generate a wide array of potential prompts that decision-makers may input into an LM, spanning 70 diverse decision scenarios across society, and systematically vary the demographic information in each prompt. Applying this methodology reveals patterns of both positive and negative discrimination in the Claude 2.0 model in select settings when no interventions are applied. While we do not endorse or permit the use of language models to make automated decisions for the high-risk use cases we study, we demonstrate techniques to significantly decrease both positive and negative discrimination through careful prompt engineering, providing pathways toward safer deployment in use cases where they may be appropriate. Our work enables developers and policymakers to anticipate, measure, and address discrimination as language model capabilities and applications continue to expand. We release our dataset and prompts at https://huggingface.co/datasets/Anthropic/discrim-eval

1. Introduction

The introduction motivates proactive evaluation of discrimination in language-model decisions, especially for high-stakes and not-yet-deployed applications. It presents a prompt-based approach and reports discrimination patterns in Claude 2.0 that can be reduced through prompt engineering.

1. Introduction

  • Real-world audits are a gold standard for studying discrimination, but their expense and privacy and ethical constraints make broad studies difficult.The authors also seek to anticipate risks in applications that have not yet been developed.Source: §1 · p. 1
  • The authors generate prompts across 70 societal decision scenarios, vary only demographic information, and use human validation to check scenario plausibility.Source: §1 · pp. 1–2
  • Without intervention, Claude 2.0 shows positive and negative discrimination in select settings; careful prompt engineering significantly reduces both.Positive outcomes were more likely for women, non-binary people, and non-white people, and less likely for older people.Source: §1 · p. 2
  • The approach is intended to help developers and policymakers measure and address potential harms before real-world deployment.Source: §1 · p. 2

2. Generating language model prompts and decisions

The authors construct hypothetical binary decision prompts from generated societal scenarios, changing demographic details to compare model decisions. They retain 70 reviewed templates and assess their quality through human ratings.

2. Generating language model prompts and decisions

  • The evaluation considers 70 decision questions; Table 1 gives their high-level descriptions and marks cases where multiple generated questions share a scenario but differ in precise details.Source: Table 1 · p. 3, §2.1 · p. 5
  • The authors generated 96 decision topics spanning areas such as healthcare, relationships, and entertainment.Source: §2 · p. 4
  • Each template frames a third-party yes-or-no decision using age, race, and gender placeholders, with “yes” defined as a positive outcome for the person.Source: §2 · p. 4
  • Manual review removed malformed or ambiguous templates, leaving 70 of the 96 generated templates.Source: §2 · p. 4
  • The explicit approach inserts age, race, and gender values; the study uses nine ages, three genders, and five racial or ethnic categories, while noting the method can extend beyond them.Source: §2 · p. 4
  • The implicit approach conveys race and gender through names associated with those demographics, while specifying age.Source: §2 · p. 4
  • For each filled template, the model provides a yes-or-no decision; the authors record the predictive probability of “yes” from the next token.They checked that p(yes) + p(no) exceeded 0.99 on average across queries.Source: §2 · pp. 4–5

2.1. Analysis of the generated questions

  • Human raters assessed generated questions, which averaged 4.76 out of 5, with 98% of templates rated 4 or 5.Source: §2.1 · p. 5

3. Analysis and Results

The analysis models demographic associations with favorable decisions and finds both positive and negative discrimination in select Claude 2 settings without intervention. It also reports that prompt-based interventions can eliminate most of these differences.

3.1. Mixed Effects Model

  • The mixed-effects regression predicts logit-transformed normalized yes probabilities using demographic fixed effects and decision-question random effects, including demographic-by-question interactions.The response is logit[p(yes)/(p(yes)+p(no))]; question effects account for variation across decision types and their interactions with demographics.Source: §3.1 · p. 5
  • Differences in average logit scores between each demographic and the baseline provide a more accessible fixed-effect estimate, with standard errors yielding error bars.The authors report that this approach gives very similar estimates and error bars to the mixed-effects model.Source: §3.1 · p. 5

3.2. Discrimination Score

  • A discrimination score is a demographic coefficient measuring the change in logit yes probability relative to a 60-year-old white male baseline.Positive and negative scores indicate higher and lower favorable-decision scores, respectively; the age baseline was chosen for statistical convenience.Source: §3.2 · pp. 5–6

3.3. Results of evaluations for Claude 2

  • Claude 2 shows positive discrimination favoring non-male genders and non-white races, and negative discrimination against people over 60; explicit attributes generally have larger effects than names.Across decision questions, scores are almost always positive for Black subjects and neutral or negative for older subjects relative to younger ones.Source: §3.3 · p. 6
  • The logit transformation mitigates floor and ceiling effects that could artificially reduce measured discrimination when yes probabilities approach zero or one.The authors say it avoids filtering decision questions according to their average yes probability.Source: §3.3 · p. 6
  • Prompt-based interventions are reported to eliminate the vast majority of the observed differences.The paper presents and evaluates prompt-based methods intended to give users and policymakers additional control.Source: §3.3 · p. 6

4. Analysis of prompt sensitivity

The authors test whether changing prompt style and format affects discrimination patterns. The direction remains broadly stable across variations, although the magnitude can differ.

4. Analysis of prompt sensitivity

  • To test robustness to input sensitivity, the authors rewrite decision templates into alternate formats and styles.A language model generates the rewrites from the original templates.Source: §4 · p. 6, §4.1 · p. 6

4.1. Variations in the question style and format

  • Variations include first-person phrasing and formal, detached bullet lists of factual statements.Source: §4.1 · p. 6
  • Other rewrites use pros-and-cons lists, emotional language, or typos, lowercase text, and omitted words.Source: §4.1 · p. 6
  • The authors also add coded demographic language to test sensitivity to subtle indications of discriminatory preferences.One example is “clean-cut all-American type.”Source: §4.1 · p. 6

4.2. Results

  • Figure 2 contrasts demographic attributes and explicit versus name-based fills; discrimination is higher for explicit attributes than for names.It reports positive discrimination by race and gender relative to a white male baseline, and negative discrimination for groups over 60 relative to those under 60.Source: Figure 2 · p. 7
  • Across decision questions, explicit prompts typically show positive discrimination for Black candidates and neutral or negative discrimination by age.The figure compares age and Black candidates with a white 60-year-old baseline.Source: Figure 3 · p. 7
  • Prompt style does not change discrimination direction across templates, but emotional phrasing generally produces larger score magnitudes.The overall patterns hold across variations; a formal bulleted list can have a smaller effect.Source: §4.2 · p. 6, Figure 4 · p. 8

5. Mitigation Strategies to Reduce Discrimination

The section evaluates prompt-based interventions intended to give users and policymakers additional control over discrimination. Several sharply reduce discrimination, while selected interventions retain high correlation with the model’s original decisions.

5.1. Appending statements to prompts

  • Appended prompt statements tell the model not to use demographics, reject affirmative action, or treat demographics as a technical quirk or legally unusable.Some variants repeat “really” in the instruction not to discriminate.Source: §5.1 · p. 8

5.2. Requesting the model verbalize its reasoning process to avoid discrimination

  • Other interventions ask the model to verbalize reasoning while avoiding bias and stereotyping, with a variant addressing positive preference and a control asking only for an unbiased decision.The full intervention prompts are provided in Appendix B.Source: §5.2 · p. 8

5.3. Results

  • Several prompt interventions significantly reduce discrimination, often approaching zero; some can remove positive and negative discrimination in the questions studied.Illegal to discriminate, Ignore demographics, and Illegal + Ignore are especially effective.Source: §5.3 · p. 8, Figure 5 · p. 9

5.4. Do the interventions distort the model’s decisions?

  • Illegal to discriminate and Ignore demographics pair a discrimination score of ≈0.15 with 92% correlation to the original decisions.The study measures correlation between decisions before and after an intervention.Source: §5.4 · p. 9

6. Discussion

The discussion highlights both the breadth of the evaluation and limitations that may constrain how its findings generalize. It cautions against treating evaluation performance as sufficient for high-stakes deployment, leaves normative questions about positive discrimination open, and offers only speculative explanations for observed behavior.

6.1. Limitations with our evaluation

  • The evaluation seeks external validity through 70 applications, varied question phrasings, and human validation, but primarily tests paragraph-long candidate descriptions.Real-world inputs may include longer documents such as resumes or medical records, or interactive dialogues, which could affect conclusions.Source: §6.1 · pp. 9–10
  • ≈0.15 discrimination and ≈92% correlation with Default predictions mark a favorable tradeoff for the Illegal to discriminate and Ignore demographics interventions.The discrimination score averages results across seven attributes; the intervention variation is comparable to variation across decision-question styles.Source: Figure 6 · p. 10
  • The analysis covers race, gender, and age, while omitting other characteristics and leaving open the possibility that LM-generated evaluations bias which applications are considered.The omitted characteristics include veteran status, income, health status, and religion.Source: §6.1 · p. 10
  • Demographic names are difficult to select reliably, and the evaluation does not capture proxy discrimination or how model decisions affect human decision-makers.The authors also analyze individual demographic relationships rather than intersectional effects, and prompt-construction variations could alter conclusions.Source: §6.1 · p. 10

6.2. Should models be used for the applications we study?

  • Good performance on these evaluations is not sufficient grounds for high-risk use; the authors call for sociotechnical safeguards and naturalistic task evaluation.A human advisory role alone is not considered a sufficient guardrail.Source: §6.2 · pp. 10–11

6.3. How should positive discrimination be addressed?

  • The paper offers stakeholders tools to measure and control discrimination through prompting, while leaving debates about correcting positive discrimination unresolved.Those debates encompass differing policies and attitudes across the world.Source: §6.3 · p. 11

6.4. Where does this behavior come from?

  • Potential explanations for the observed behavior—including rater preferences and reinforcement-learning overgeneralization—are presented as speculation, not established causes.The authors describe these questions as active areas for further research.Source: §6.4 · p. 11

7. Related Work

Related work spans social-science audits, algorithmic discrimination research, and studies of bias and interventions in language models. This paper extends model-generated evaluation work by testing discrimination across diverse applications and prompt styles with human checks.

Measuring discrimination in the social sciences

  • Social-science audit and correspondence studies compare otherwise identical profiles that differ in protected characteristics, using acceptance-rate differences as evidence of discrimination.The paper adapts this approach to language models across hypothetical scenarios.Source: §7 · p. 11

Algorithmic discrimination

  • Algorithmic discrimination studies examine direct demographic effects and proxy effects across settings including advertising, lending, hiring, and medical treatment.Related responses include theoretical fairness frameworks and investigative practices.Source: §7 · pp. 11–12

Discrimination in language models

  • Prior LM studies found that prompting can reduce bias, while broader work identified discrimination in admissions and examined interventions [1?].The present study contributes a 70-application investigation of both positive and negative discrimination and prompting-based reductions.Source: §7 · p. 12

Model-generated evaluations

  • Prior work used LMs to generate evaluations and red-teaming attacks; this paper extends that approach with diverse use cases, prompt rewrites, demographic variation, and human evaluation.The human check addresses the possibility that LM outputs can be flawed.Source: §7 · p. 12

8. Conclusions

The conclusions describe a proactive approach to assessing and mitigating language-model discrimination through model-generated evaluations and human validation, alongside prompt templates for varying question phrasing, adding non-discrimination interventions, and eliciting yes/no decisions with bias-avoidance instructions. They note that variants of the method could support evaluation of additional characteristics and help anticipate societal risks.

8. Conclusions

  • The study combines model-generated evaluations with human validation to assess and mitigate language-model discrimination, with methods intended to interest policymakers and third-party stakeholders.The authors frame these methods as supporting better societal-impact evaluations and risk mitigation before harms occur.Source: §8 · p. 12
  • The authors anticipate adapting the technique to evaluate other demographic attributes, writing styles, language-use patterns, and mentions of topics or issues.They describe these adaptations as helping developers and policymakers anticipate and mitigate risks as language models advance.Source: §8 · p. 12

B. Prompts

  • The prompts used to generate evaluations and model decisions are available on the project’s Hugging Face page.Source: B. Prompts · p. 17

B.1. Prompts for generating default decision questions

  • Default-question prompts generate single-person yes/no decisions with favorable and countervailing factors, demographic placeholders, and no explicit demographic cues in the question.An example template asks about bail while specifying both mitigating circumstances and reasons for concern.Source: §B.1 · p. 18

B.2. Prompts for generating variations of the decision questions for prompt sensitivity

  • Prompt-sensitivity variations rewrite decision templates in first-person, formal bullet-point, or error-filled styles while retaining demographic tags.Source: §B.2 · pp. 20–22
  • Decision-question variations include emotional phrasing and coded-language cues while retaining demographic tags and avoiding new candidate information.The coded-language prompt asks for traits of an ideal candidate, with examples such as “clean-cut” and “professional.”Source: §B.2 · pp. 23–24

B.3. Prompts for Interventions

  • Intervention fragments appended to decision questions instruct models not to use demographics, with variants invoking legal risk and decisions based on redacted profiles.The prompts explicitly address both positive and negative discrimination.Source: §B.3 · pp. 24–25

B.4. Prompts and mitigations in the decision-making step

  • Decision prompts request yes-or-no answers, with variants directing models to avoid bias or stereotyping and stay grounded in question details.Other variants ask the model to verbalize its reasoning, including a version that flags potential bias favoring historically discriminated groups.Source: §B.4 · pp. 25–26

C. Filling templates with names

The authors construct implicit demographic cues by adding generated names to decision-question templates, then test whether raters associate those names with the intended groups.

C. Filling templates with names

  • The name set targets a subset of names associated with demographic groups rather than a representative sample of each group.The authors describe this as a narrower version of the name-selection problem.Source: C. Filling templates with names · p. 26
  • Names combine sampled surnames with model-generated first names for each race-and-gender pair.The surname list included 25 names per category; Native American surnames came from a separate source because common surnames overlapped with other groups.Source: C. Filling templates with names · p. 26
  • In a human study of N = 198 filled templates, raters largely identified race and gender from names, with lower accuracy for non-binary and some Black names.The caption identifies non-binary names such as “Ocean” and some Black names such as “Jada Williams” as more challenging or ambiguous.Source: C. Filling templates with names · pp. 26–27, Table 2 · p. 27
  • Lower accuracy for non-binary and Black names may partly explain their relatively lower Discrimination Scores, although the lower-discrimination pattern across racial groups persists with implicit names.The authors suggest future work use more representative names and broader methods for studying proxy discrimination.Source: C. Filling templates with names · p. 27

D.1. Validating the valence of each question

The authors validate that a “yes” answer represents a more positive outcome for the decision subject, reporting an average rating of 4.83.

D.1. Validating the valence of each question

  • The validation criterion was that a “Yes” response be more positive for the decision subject than a “No” response.Raters assessed agreement with this statement on a five-point scale.Source: §D.1 · p. 27
  • For each question, raters evaluated agreement that a “Yes” response was a more positive outcome for the decision subject than a “No”; the average score was 4.83.Ratings used a 1-to-5 scale; 0.975% of scores were either 4 or 5.Source: §D.1 · p. 27
  • 0.975% of scores were either 4 or 5, as reported by the authors.The passage presents this alongside the 4.83 average rating.Source: §D.1 · p. 27

D.2. Additional Methodological details

The Default decision question ratings covered both explicit demographic descriptions and implicit name-based descriptions.

D.2. Additional Methodological details

  • 405 ratings were collected for the Default decision question type across Explicit and Implicit fill types.The ratings were gathered from questions filled using demographics or names.Source: §D.2 · p. 27
  • The 405 ratings covered 135 filled questions drawn from 29 decision-question templates, with three raters per question..Source: §D.2 · p. 27
  • Raters were contracted through Surge and paid at least a California minimum wage..Source: §D.2 · p. 27

E. Fitting the mixed effects model

The analysis fits a mixed-effects model in R and extracts fixed-effect estimates with confidence intervals.

E. Fitting the mixed effects model

Loading 2312.03689v1…