PaperlayerView exact revision on arXiv

Source-linked AI summary

Robustly Improving LLM Fairness in Realistic Settings via Interpretability

Adam Karvonen, Samuel Marks

arXiv:2506.10922v1cs.LGcs.AIcs.CL

TL;DR

In high-stakes hiring, prior evidence for anti-bias prompts came from controlled evaluations, leaving their effectiveness with realistic contextual details uncertain. This paper tests internal representation interventions and finds that they reduce bias across realistic settings where prompting can fail.

  • Problem

    Whether anti-bias prompts that work in controlled hiring evaluations remain effective amid realistic contextual details is uncertain, despite the high-stakes consequences of automated hiring.

    Source: §1 · pp. 1–2

  • Method

    The paper identifies race- and gender-correlated directions using synthetic data and intervenes on model activations at inference time.

    Source: §1 · p. 2

  • Results

    Up to 12% differences in interview rates emerged with realistic context, while internal intervention reduced measurable bias to typically under 1% and below 2.4% in all tested cases.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The authors recommend more realistic bias evaluations and consideration of internal mitigation rather than relying solely on prompts for current models.

    Source: §5 · p. 10, §6 · p. 10

    The findings concern current models, for which prompting alone is an unreliable strategy despite model-specific variations in bias.

    Source: §5 · p. 10

Abstract

from arXiv · show

Large language models (LLMs) are increasingly deployed in high-stakes hiring applications, making decisions that directly impact people's careers and livelihoods. While prior studies suggest simple anti-bias prompts can eliminate demographic biases in controlled evaluations, we find these mitigations fail when realistic contextual details are introduced. We address these failures through internal bias mitigation: by identifying and neutralizing sensitive attribute directions within model activations, we achieve robust bias reduction across all tested scenarios. Across leading commercial (GPT-4o, Claude 4 Sonnet, Gemini 2.5 Flash) and open-source models (Gemma-2 27B, Gemma-3, Mistral-24B), we find that adding realistic context such as company names, culture descriptions from public careers pages, and selective hiring constraints (e.g.,``only accept candidates in the top 10\%") induces significant racial and gender biases (up to 12\% differences in interview rates). When these biases emerge, they consistently favor Black over White candidates and female over male candidates across all tested models and scenarios. Moreover, models can infer demographics and become biased from subtle cues like college affiliations, with these biases remaining invisible even when inspecting the model's chain-of-thought reasoning. To address these limitations, our internal bias mitigation identifies race and gender-correlated directions and applies affine concept editing at inference time. Despite using directions from a simple synthetic dataset, the intervention generalizes robustly, consistently reducing bias to very low levels (typically under 1\%, always below 2.5\%) while largely maintaining model performance. Our findings suggest that practitioners deploying LLMs for hiring should adopt more realistic evaluation methodologies and consider internal mitigation strategies for equitable outcomes.

1 INTRODUCTION

In hiring, simple anti-bias prompts that work in controlled evaluations can fail when realistic company details and hiring constraints are added. The paper tests internal interventions to reduce bias in these settings and reports low residual bias across its evaluations.

1 INTRODUCTION

  • Prior controlled evaluations found that simple anti-bias prompts could mitigate or eliminate bias, but whether that success holds with realistic context remains an open question.The paper identifies college affiliation as one contextual cue from which models can infer demographics and become biased.Source: §1 · pp. 1, 3
  • Realistic company details and selective-hiring constraints induced race- and gender-related interview-rate differences of up to 12% in previously unbiased model-prompt combinations.The tested models included GPT-4o, Claude 4 Sonnet, Gemini 2.5 Flash, Gemma-3, and Mistral-24B.Source: §1 · p. 2
  • The authors identify race- and gender-correlated directions in model activations using synthetic data, then intervene on those representations at inference time.They investigate this internal approach as an alternative to relying on prompts to tell models not to use demographic information.Source: §1 · p. 2
  • Internal intervention reduced measurable bias to typically under 1% and below 2.4% in all tested models and prompt combinations, including contextualized settings.It also worked when demographic attributes were inferred from contextual clues rather than stated explicitly.Source: §1 · p. 3
  • The intervention had limited measured performance effects: MMLU degradation was under 0.5% for Gemma-2 and Mistral-24B and 1–3.7% for Gemma-3.In unbiased settings, mean acceptance rates changed by at most 1.8%.Source: §1 · p. 3
  • Monitoring chain-of-thought reasoning did not detect demographic bias: models gave neutral-sounding rationales for decisions that were demonstrably biased.Source: §1 · p. 3

2 RELATED WORK

Prior hiring-bias evaluations often used controlled resume comparisons and found that anti-bias prompts could mitigate bias. This paper situates its work among efforts to test richer contexts and intervene on internal representations.

Evaluating Bias in Language Models

  • Controlled hiring evaluations using counterfactual resumes found that anti-bias prompts mitigated race and gender bias (Tamkin et al., 2023; Iso et al., 2025).Source: §2 · p. 3
  • Prior work had tested individual contextual elements, but the stability of bias patterns with multiple interacting details remained an open question.The paper reports that realistic company culture or selective-hiring constraints can reintroduce race and gender bias of up to 12% in models that appear fair in simpler evaluations.Source: §2 · p. 4

Linear Representation Interventions

  • Concept-erasure methods aim to remove protected attributes from representations, but earlier approaches relied on dense labels and summary vectors, limiting direct applicability to decoder-only LLMs.LEACE achieves linear erasure by making class-conditional representation means identical.Source: §2 · p. 4
  • Modern interventions identify concepts as directions in activation space, then use them to steer behavior, erase a concept, or shift its projection to a target value (Turner et al., 2024; Marshall et al., 2025).Source: §2 · p. 4
  • Earlier sensitive-attribute interventions reported moderate bias reductions and poor generalization across prompt templates, or targeted ambiguous pronoun resolution rather than hiring decisions (Nguyen & Tan, 2025; Zou et al., 2025).Source: §2 · p. 4

3 METHODS

The methods adapt controlled resume and scenario evaluations, systematically vary prompts and realistic hiring context, test demographic inference from college affiliations, and apply affine concept editing to model activations.

3.1 EVALUATIONS

  • The evaluation adapts scenario templates and real resumes from prior controlled hiring-bias frameworks, using contrastive candidate comparisons signaled by names, pronouns, or explicit demographic attributes.The scenario framework uses 116 LLM-generated templates; the resume framework uses 120 IT resumes after filtering nine outliers.Source: §3.1 · p. 5

3.2 PROMPT DESIGN AND SCENARIO ENHANCEMENTS

  • Prompt experiments vary response format and four anti-bias instructions, then add company names, locations, culture descriptions, or a top-10% hiring constraint to create realistic contexts.Results aggregate each model-context combination across the four anti-bias instructions.Source: §3.2 · p. 5

3.3 EVALUATING BIAS FROM INFERRED DEMOGRAPHICS VIA COLLEGE AFFILIATION

  • To test demographic inference, anonymized resumes receive college affiliations that signal Black or White candidates, using institutions of broadly comparable academic standing and region.The authors caution that affiliation also signals prestige, geography, and academic focus, so racial inference cannot be isolated as the sole mechanism.Source: §3.3 · pp. 5–6

3.4 INTERNAL MITIGATION

  • Affine concept editing targets race- and gender-related activation directions, using synthetic-data-derived directions to test generalization to real-world resumes.The approach shifts demographic projections toward a neutral target; zero-ablation damaged some models, and whitening helped reduce MMLU degradation on Gemma-3.Source: §3.4 · p. 6

3. Compute Bias Term for Each Direction: We compute a bias term b(l)

The bias term defines the neutral target projection used when the intervention shifts activations along demographic directions.

3. Compute Bias Term for Each Direction: We compute a bias term b(l)

  • The neutral target is the midpoint between the two group centroids’ projections onto the demographic direction, and the intervention shifts incoming activations to that point at inference.It is applied at every token position across all model layers.Source: §3 · pp. 6–7

4 RESULTS

Realistic hiring context can revive bias despite anti-bias prompting, while internal affine concept editing reduces bias across challenging settings. The intervention also generalizes to inferred demographic cues, with model-performance effects varying by model.

4.1 EXAMINING EVALUATION SETTINGS

  • In frontier models, Meta context induces 6–9% racial bias in binary evaluations; chain-of-thought removes measurable bias until a high-selectivity constraint is added.Claude 3.5 Sonnet was used for the selective condition because Claude 4 Sonnet rejected nearly all candidates there.Source: Figure 2 · p. 7, §4.1 · p. 7
  • Anti-bias prompts reduce bias to near zero in simplified evaluations, but company-specific context produces substantial bias, including an increase from 2% to 11% for Mistral Small 24B.Bias differences in interview rates reach 15% across model, context, and anti-bias-instruction combinations.Source: §4.1 · p. 7
  • Chain-of-thought justifications omit demographic references even when interview rates are biased, including in Claude 4 Sonnet’s counterfactual cases.The analysis of 63 resumes with flipped decisions found no explicit reasoning that acknowledged demographic factors; the authors caution that CoT faithfulness may vary by task domain.Source: §4.1 · p. 8
  • When realistic context induces bias, it consistently favors Black over White candidates and female over male candidates across tested models and companies.The reported pattern includes Meta, General Motors, and Palantir.Source: §4.1 · p. 8

4.2 INTERNAL MITIGATION

  • Internal affine concept editing reduces realistic-context bias to near zero across four open-source models and challenging settings, including selective hiring.Prompting showed induced bias up to 7.6% in the Figure 3 scenarios, while editing reduced it to typically below 2.5%.Source: §4.2 · p. 8, Figure 3 · p. 9
  • Affine concept editing mitigates race bias inferred from college affiliation, despite using demographic directions identified from explicit name-based signals.The tested implicit cues included affiliation with HBCUs in the Meta company-context setting.Source: §4.2 · p. 8
  • Capability effects vary by model: Gemma-2 27B and Mistral Small 24B show under 0.5% MMLU degradation, while Gemma-3 models decrease by 1.1% and 3.7%.In unbiased settings, mean acceptance rates change by a maximum of 1.8%; zero-ablation severely impaired Gemma-3 models, for which ACE had comparatively larger MMLU impact.Source: §4.2 · pp. 8–9, Table 1 · p. 9

5 DISCUSSION AND LIMITATIONS

Realistic context can expose biases in models that appear unbiased in controlled settings, underscoring gaps in current evaluations. The paper argues internal interventions may be more robust than prompting, while noting limits in both the evaluation setting and demographic scope.

5 DISCUSSION AND LIMITATIONS

  • Models that appear unbiased in controlled settings can exhibit significant biases when realistic contextual details are introduced.This fragility motivates more realistic evaluations, particularly for high-stakes deployment settings.Source: §5 · pp. 9–10
  • Internal interventions such as affine concept editing provide more robust bias mitigation across challenging, realistic conditions than prompt engineering alone.The authors argue that the complexity and variety of real-world contexts make consistent fairness through prompt engineering alone difficult.Source: §5 · p. 10
  • The study targets outcome-based hiring bias, neutralizing demographic influence on decisions without aiming to erase all race or gender concepts.Underlying conceptual associations, such as links between nursing and caregiving, may remain.Source: §5 · p. 10
  • The evaluation adapts two existing frameworks but omits parts of actual hiring pipelines, including detailed job descriptions and social-media review.The authors call for more comprehensive evaluation suites.Source: §5 · p. 10
  • The study covers only binary Black/White race and Male/Female gender categories, leaving other protected characteristics and intersectional biases unexamined.The authors recommend extending research to a broader range of characteristics and intersectional biases.Source: §5 · p. 10

6 CONCLUSION

The authors recommend realistic evaluation and internal-representation mitigation for high-stakes LLM decision-making. The paper also identifies the models evaluated in the study.

6 CONCLUSION

  • Practitioners should adopt more realistic evaluations and consider internal-representation mitigation rather than relying solely on prompts.The conclusion frames this as an urgent concern because such systems are already deployed at scale in high-stakes decisions.Source: §6 · p. 10

A MODEL DETAILS

  • Table 2 lists the large language models evaluated in the study.The supplied passage identifies the table's scope but does not provide its model entries.Source: Table 2 · p. 14, A MODEL DETAILS · p. 14

B EXPLORATORY WHITE-BOX ANALYSIS FOR BIAS PREDICTION

Exploratory white-box analyses suggest internal features may relate to later bias and that demographic signals can be surfaced from later layers in chain-of-thought settings. The adjacent acceptance-rate analysis reports small changes in simple contexts and changes primarily reflecting bias correction in realistic contexts.

B EXPLORATORY WHITE-BOX ANALYSIS FOR BIAS PREDICTION

  • SAE-feature prevalence showed a suggestive correlation with eventual realistic-evaluation bias magnitude, based on attribution patching around 25% model depth.The analysis ranked SAE features by indirect-effect magnitude.Source: B EXPLORATORY WHITE-BOX ANALYSIS FOR… · p. 14
  • Later-layer attribution patching at approximately 70% depth surfaced demographic-related features more effectively than backward passes from output logits in chain-of-thought settings.The authors suggest demographic reasoning may be present in internal layers but suppressed in chain-of-thought outputs.Source: B EXPLORATORY WHITE-BOX ANALYSIS FOR… · p. 14
  • The exploratory analysis could not support strong conclusions because of limited model sample size, analytical flexibility, and inconsistent prediction of anti-bias instruction failures.The authors describe this direction as exploratory.Source: B EXPLORATORY WHITE-BOX ANALYSIS FOR… · p. 14

C CHANGE TO MEAN ACCEPTANCE RATE

  • In simple contexts, the maximum absolute change in average acceptance rates after intervention was 0.018; realistic-context changes primarily reflected bias correction.Table 3 reports raw acceptance rates for simple, unbiased-baseline and realistic, biased contexts.Source: Table 3 · p. 15, C CHANGE TO MEAN ACCEPTANCE RATE · p. 15

D DIVERSITY FILTERING RESULTS

Figure 4 examines frontier-model bias after diversity-related phrases are removed from Meta context and in a Palantir Texas-hiring context; both settings show bias favoring Black and female applicants.

D DIVERSITY FILTERING RESULTS

  • Both the filtered Meta context and the Palantir Texas-hiring context show consistent bias favoring Black and female applicants.Figure 4 covers removal of diversity-related phrases from Meta culture context and use of Palantir culture information for a Texas role.Source: Figure 4 · p. 15

E GENDER BIAS RESULTS

Across the four models, gender bias is minimal in Simple Context but becomes substantial and favors female candidates when realistic context is added; internal mitigation reduces it to low levels. Binary evaluations and chain-of-thought reasoning show different sensitivity to contextual elements.

E GENDER BIAS RESULTS

  • All four models show minimal gender bias in Simple Context, but realistic context induces substantial bias favoring female candidates that internal mitigation reduces to low levels.The appendix evaluates the same models and contexts as the main text, changing the demographic attribute from race to gender.Source: Figure 5 · p. 16, E GENDER BIAS RESULTS · p. 16, Figure 7 · p. 17
  • Adding Meta company context induces gender bias in binary evaluations, while chain-of-thought reasoning shows different sensitivity to contextual elements.Source: Figure 6 · p. 16

F STATISTICAL METHODOLOGY

The analysis uses paired counterfactual resumes to measure interview-rate differences between demographic groups and test paired outcomes. Summary plots average bias across four prompts, with pooled variance used for 95% confidence intervals.

F STATISTICAL METHODOLOGY

  • Bias is the interview-recommendation-rate difference between demographic groups for the same resume; negative values favor Black or female candidates, and positive values favor White or male candidates.The experiments evaluate each resume with different demographic markers.Source: F STATISTICAL METHODOLOGY · p. 18
  • McNemar’s test assesses statistical significance for paired categorical data, and error bars represent 95% confidence intervals.Source: F STATISTICAL METHODOLOGY · p. 18
  • Main-body plots average the four anti-bias-prompt scores; pooled variance across runs sets the mean’s 95% interval at ±1.96 × SEavg.SEi is the standard error from each individual run.Source: F STATISTICAL METHODOLOGY · p. 18

G UNFAITHFUL CHAIN OF THOUGHT EXAMPLE

This section presents a Claude 4 Sonnet example in which changing only a resume’s name and pronoun led to opposite interview decisions with contradictory reasoning, and documents company-context materials and experiment tables reporting acceptance rates by gender and race. The tables include chain-of-thought and internal-mitigation evaluations.

G UNFAITHFUL CHAIN OF THOUGHT EXAMPLE

  • The example used a less-selective constraint that raised Claude 4 Sonnet’s mean acceptance rate to 39%, with 20 interview slots for 200+ applicants.The paper says its typical selectivity constraint led Claude 4 Sonnet to reject almost all candidates.Source: G UNFAITHFUL CHAIN OF THOUGHT EXAMPLE · p. 18
  • The identical resume was accepted for the Black female candidate and rejected for the White male candidate, with the model giving each decision a plausible-sounding rationale.The accepted candidate was described as interview-worthy despite presentation concerns, while the rejected candidate’s government/military background and lack of modern tech experience were treated as reasons he might not rank among the top applicants.Source: G UNFAITHFUL CHAIN OF THOUGHT EXAMPLE · pp. 18–20

H.1 BASE PROMPT TEMPLATES

  • The base prompt offered either a Yes/No-only answer or a short chain-of-thought followed by a Yes/No decision.Both templates supplied job and company-culture information, a candidate resume, and a selectivity prompt.Source: §H.1 · p. 20

H.2 HIGH SELECTIVITY HIRING INSTRUCTIONS

  • The high-selectivity instructions asked the model to interview only candidates in approximately the top 10% of IT professionals.The template specified 10 interview slots from 200+ applications and noted that most solid IT professionals would not make the cut.Source: §H.2 · p. 20

H.3 ANTI BIAS INSTRUCTIONS

  • The anti-bias instructions prohibited demographic influence, urged checks for hidden assumptions, and emphasized evaluating candidates on job-relevant skills and experience.They explicitly cautioned against both positive and negative discrimination based on race or other demographic characteristics.Source: §H.3 · pp. 20–21

H.4 COMPANY CONTEXTS

  • The Meta context used mid-level IT roles in three locations and culture information sourced from the company careers page in May 2025.The prompt included Meta’s culture and values; diversity-related content was removed by deleting material beginning with “ACCESSIBILITY AND ENGAGEMENT AT META”.Source: §H.4 · p. 21
  • GM and Palantir culture materials provide realistic hiring context, including company values, inclusion statements, and employee benefits.The GM prompt concerns mid-level IT roles; the Palantir prompt concerns mid-level IT roles in Texas.Source: §H.4 · pp. 24–25

I ALL EXPERIMENT DATA

  • Tables 4, 5, 7, and 8 report male/female and White/Black acceptance rates for simple evaluations and realistic Meta evaluations.The captions identify the evaluation setting and demographic order but do not provide the table values in these passages.Source: Table 4 · p. 26, Table 5 · p. 26, Table 7 · p. 28
  • Tables 9–12 cover realistic Meta evaluations with chain of thought, selectivity, or college affiliation, including internal-mitigation variants.Their captions specify male/female and White/Black acceptance-rate reporting; the supplied passages do not show the values.Source: Table 9 · p. 29, Table 10 · p. 29, Table 11 · p. 30
  • Tables 13 and 14 report male/female and White/Black acceptance rates for GM plus selectivity, with and without internal mitigation.The supplied captions identify the comparison but do not provide acceptance-rate values.Source: Table 13 · p. 31, Table 14 · p. 31

Related papers

Loading 2506.10922v1…