Latent space bias directions in LLMs capture confidence, not fairness

1 Introduction

• The classifier separates highest- versus lowest-probability choices (AUROC 0.94) better than biased versus anti-biased prompts (AUROC 0.57).The comparison is shown for Llama-3.1-8B-Instruct.Source: Figure 1 · p. 2, §1 · p. 2
• The debiasing direction is dominated by confidence: it tracks likelihood differences and points from high- to low-likelihood regions of activation space.The prompts are linearly separable in intermediate layers, but the separating factor is likelihood difference.Source: §1 · p. 2
• The authors evaluate steering on BBQ, CrowS-Pairs, and StereoSet bias benchmarks, plus MMLU and OpenBookQA general-knowledge QA benchmarks.They apply activation steering across a suite of LLMs.Source: §1 · p. 2
• For Llama-3.1-8B-Instruct on BBQ ambiguous prompts, activation steering lowers the bias score from 0.08 to 0.01, but this reduction reflects increased abstention rather than correction of the underlying stereotype preference.This occurs when the model is given the option to abstain from responding.Source: §1 · p. 2
• Steering along the direction lowers model confidence and has unintended effects on accuracy and entropy, so apparent bias reduction is not evidence of fairer models.The authors argue that reported steering-based debiasing results should be interpreted with care.Source: §1 · pp. 2–3

Activation steering

• Prior evaluations have questioned steering reliability and note that probabilistic outputs can obscure effects on model behavior.Related work calls for evaluations based on underlying token-probability distributions.Source: §2 · p. 3
• Related work links bias steering with refusal, while this paper reports that anti-bias steering triggers low-confidence abstention.The cited prior work reports that steering toward bias triggers refusal [19].Source: §2 · p. 3
• Prior activation-steering debiasing methods contrast biased and unbiased prompt activations, reduce bias on target benchmarks, but show limited transfer and can degrade unrelated capabilities.The cited approaches include FairSteer and steering-vector methods.Source: §2 · p. 3

Bias and likelihood

• Prior work uses likelihood differences to measure bias, while this paper connects those differences to activation-space geometry and finds the debiasing direction largely encodes confidence.The cited measures include relative log-likelihood on stereotyped and anti-stereotyped pairs and mean-perplexity differences.Source: §2 · p. 3

Datasets

• BBQ, StereoSet, and CrowS-Pairs measure societal bias, while MMLU and OpenBookQA probe general-knowledge performance.BBQ includes ambiguous contexts, where abstention is correct, and disambiguated contexts with a definite correct answer; the other two bias datasets contain ambiguous contexts.Source: §3 · p. 4

Models

• The study evaluates pretrained and instruction-tuned variants from four open-weight model families, ranging from 7B to 9B parameters.The families are Llama-3.1-8B, Falcon3-7B, Ministral-3-8B, and Qwen3.5-9B.Source: §3 · p. 4

Evaluation metrics

• Evaluation uses AUROC for classifier separability, bias scores for benchmark preferences, and accuracy and answer entropy to assess steering trade-offs.Accuracy is defined for BBQ disambiguated, MMLU, and OpenBookQA; the bias score differs between BBQ and the other bias datasets.Source: §3 · p. 4

4.1 Background: baseline bias in the tested models

• Across all eight models at baseline (α=0), both BBQ bias scores are positive (s_amb 0.07–0.18; s_dis 0.01–0.03), while CrowS-Pairs and StereoSet stereotype rates exceed 50% chance (69.0–75.8% and 62.7–72.3%).Source: §4.1 · p. 5

4.2 Methodology: constructing a bias classifier

• Linear separability peaks at intermediate layers for most models, while instruction-tuned variants generally achieve higher AUROC than pretrained counterparts.Qwen3.5-9B base instead peaks at the final layer; Ministral-3-8B variants have similar intermediate-layer separability, with the instruction-tuned model only slightly better at final layers; the highest AUROC is 0.75 for Llama-3.1-8B-Instruct at layer 15.Source: §4.2 · pp. 6–7

4.3 Results: debiasing direction captures confidence

• Across models, BBQ Spearman correlations rise from 0.01–0.45 in ambiguous contexts to 0.57–0.80 in disambiguated contexts, indicating the classifier tracks confidence rather than bias.When answers are more certain, they receive both higher probabilities and more extreme bias scores, whether the answer is biased or anti-biased.Source: §4.3 · p. 8, Table 1 · p. 8
• For StereoSet and CrowS-Pairs, Spearman correlations between classifier bias scores and answer probability are 0.19–0.64 and 0.28–0.52, respectively.The paper describes these intermediate correlations as consistent with those in ambiguous BBQ contexts, although slightly stronger.Source: §4.3 · p. 8
• For disambiguated BBQ prompts, the classifier achieves AUROC 0.57 for biased-versus-anti-biased labels but 0.94 for higher-versus-lower answer probability.The stronger separation distinguishes the more confident answer from the less confident one, a split the paper says has no relationship to societal bias.Source: §4.3 · p. 8
• On disambiguated BBQ, BBQ-ambiguous-trained classifiers achieve AUROC 0.79–0.95 across all eight models, while transfer from StereoSet- and CrowS-Pairs-trained classifiers is less consistent.Source: §4.3 · p. 8
• The results show that the bias linear classifier primarily isolates model confidence, not semantic bias.Source: §4.3 · p. 8
• On OpenBookQA, Llama-3.1-8B-Instruct reaches AUROC 0.88 for separating highest- from lowest-probability answers using BBQ-ambiguous- or CrowS-Pairs-trained classifiers.MMLU and OpenBookQA contain no societal-bias concept, so this transfer indicates the classifier is separating something other than societal bias.Source: §4.3 · p. 8, Table 2 · p. 9

5.2 Results: steering along the debiasing direction increases abstention or entropy

• Steering broadly reduces confidence rather than correcting bias, increasing abstention when available and equalizing answer probabilities otherwise.The same non-targeted pattern extends to general-knowledge datasets without societal-bias content; results are qualitatively similar across models, except Qwen3.5-9B-Base, whose best-AUROC layer is its last and where steering has negligible impact.Source: §5 · p. 9, §5.2 · p. 12

5.1 Methodology: constructing a debiasing steering vector

• The CAA vector is the mean hidden-state difference between biased and anti-biased answers; a separate confidence direction contrasts highest- and lowest-probability answers.The confidence direction is computed from MMLU and OpenBookQA at the same layer, while steering adds a scaled, normalized debiasing direction at the best layer.Source: §5.1 · p. 9

5.2 Results: steering along the debiasing direction increases abstention or entropy

• Mean cosine similarities between debiasing and confidence directions are 0.68 for BBQ, 0.67 for StereoSet, and 0.50 for CrowS-Pairs.Across models and datasets, the reported similarities range from 0.29 to 0.91.Source: Table 3 · p. 11, §5.2 · p. 11
• With abstention available, steering increases abstain-answer probability across datasets and models, with the largest change in instruction-tuned models.The probability of both biased and, in most cases, anti-biased answers falls, concentrating probability on abstention rather than shifting preference toward anti-biased answers.Source: §5.2 · pp. 11–12
• Without abstention, steering pushes answer probabilities toward equality, raising entropy and uncertainty rather than favoring anti-biased answers.This pattern is reported for CrowS-Pairs and StereoSet, where lower bias-answer percentages accompany higher entropy rather than a genuine preference shift.Source: §5.2 · p. 12
• On ambiguous BBQ, lower bias scores reflect increased abstention under the score’s accuracy scaling, while disambiguous BBQ bias scores stay roughly flat.Steering also systematically reduces accuracy on disambiguous BBQ, despite enough information for a substantiated choice, supporting reduced confidence rather than reduced bias.Source: §5.2 · p. 12
• Steering Qwen3.5-9B with the StereoSet direction at natural magnitude raises OpenBookQA entropy from 0.24 to 0.50 and MMLU entropy from 0.34 to 0.61.The MMLU increase is reported as 80%.Source: §5.2 · p. 13
• The largest observed accuracy cost is Llama-3.1-8B-Instruct on OpenBookQA under BBQ steering, falling from 0.76 to 0.70.The passages also report that steering slightly lowers task accuracy while raising entropy on MMLU and OpenBookQA, which contain no specific societal-bias notions to correct.Source: §5.2 · p. 13
• Accuracy is mostly unaffected because steering flattens answer probabilities toward a common midpoint without changing candidate rankings or usually changing the greedy-decoded answer.The distribution flattens and entropy rises.Source: §5.2 · p. 13
• Bias-metric reductions are consistently accompanied by increased abstention or model entropy, depending on QA dataset format.The passages attribute this effect to the debiasing direction encoding model confidence.Source: §5.2 · p. 13, Table 4 · p. 13

• A classifier trained to separate biased and anti-biased prompts reaches AUROC as high as 0.9 on high- versus low-probability MMLU prompts, despite MMLU encoding no demographic-bias concept.Source: §6 · p. 14
• The debiasing direction has cosine similarities of 0.3–0.9 with confidence vectors from MMLU and OpenBookQA, while prompt likelihood differences are linked to biased-sentence overrepresentation in pretraining.Source: §6 · p. 14
• With an abstain option, steering lowers confidence and increases abstention while reducing both biased and anti-biased responses.Low-confidence steering directly increases abstention on multiple-choice QA benchmarks.Source: §6 · p. 14
• Without an abstain option, steering shifts probability from the biased response toward more evenly likely options, slightly reducing accuracy and increasing entropy across bias and general-knowledge benchmarks.Source: §6 · p. 14
• Steering-based bias reductions may reflect confidence shifts rather than fairer preferences, so they should be interpreted with care.The authors propose studying whether confidence-independent debiasing directions exist and whether the confound extends beyond activation steering and other linear-representation steering vectors.Source: §6 · p. 14

Generative AI Usage

• Claude Opus 4.8 was used to improve fluency, edit grammar, and improve table formatting.The authors state that no text generation was performed and that they take full responsibility for the work.Source: Generative AI Usage · p. 15

A.1 Datasets

• BBQ probes societal bias, StereoSet tests stereotypical bias, and CrowS-Pairs comprises 1,508 sentence pairs; StereoSet’s intrasentence subset contains 2,106 unique questions.Source: §A.1 · pp. 16–17
• CrowS-Pairs pairs differ by one word and are converted into two-option questions with biased and anti-biased completions.BBQ includes demographic categories and intersectional biases; StereoSet includes stereotypical, anti-biased, and unrelated options, and this work uses its intrasentence subset.Source: §A.1 · pp. 16–17
• The bias datasets use 70/30 training and test splits, with BBQ ambiguous and disambiguated forms kept together and balanced by correct answer.Training uses ambiguous prompts to fit classifiers and steering directions; test data assesses classification in ambiguous and disambiguated contexts and steering effects.Source: §A.1 · p. 17
• For confidence classification, each question is reduced to its two most probability-extreme answers to match the classifier’s two-way training task.The test set includes disambiguated BBQ, MMLU, and OpenBookQA questions.Source: §A.1 · p. 18

A.2 Prompt formatting

• The same core prompt—context when applicable, question, and answer options—supports answer generation and extraction of last-token hidden states.Those hidden states are used to train the linear classifier and compute the CAA direction.Source: §A.2 · p. 18
• For instruction-tuned models, prompts explicitly request a letter-only answer and use the model’s chat template.For hidden-state extraction, each candidate answer’s letter is appended to the prompt; probabilities instead come from a prompt-only forward pass.Source: §A.2 · p. 18

A.3 Evaluation Metric Details

• AUROC measures the probability that a random positive example receives a higher classifier score than a random negative example.0.5 indicates chance-level discrimination and 1.0 indicates perfect separation.Source: §A.3 · pp. 18–19
• The example prompts shuffle answer options, so option letters do not consistently correspond to a fixed answer type.StereoSet includes a neutral option, unlike CrowS-Pairs; CrowS-Pairs also lacks BBQ’s ambiguous/disambiguated distinction.Source: Figure 6 · p. 19, Figure 7 · p. 19, Figure 8 · p. 20
• BBQ uses separate bias scores for ambiguous and disambiguated contexts, while StereoSet and CrowS-Pairs scores are the percentage of biased answers.The BBQ scores are defined in the cited equations from [26].Source: §A.3 · p. 19
• Accuracy is the fraction of prompts where the model gives highest probability to the objectively correct answer, defined only for datasets with ground truth.CrowS-Pairs and StereoSet are excluded because they have no correct answer.Source: §A.3 · p. 20
• Entropy measures how concentrated probability mass is across answer options: high entropy indicates uncertainty, while low entropy indicates confidence.The candidate answers form the set A in the entropy definition.Source: §A.3 · p. 20

B.1 Linear classifier

• StereoSet and CrowS-Pairs are generally easier to separate than BBQ, with best AUROC typically higher by 0.15–0.25.The text attributes this pattern to shorter average prompts in StereoSet and CrowS-Pairs, which leave the bias signal less obscured.Source: §B.1 · p. 20
• Qwen3.5-9B-Base peaks at layers 26–30, later than its instruction-tuned counterpart at layers 16–17, with lower AUROC values of 0.68/0.76/0.82 versus 0.74/0.94/0.94.The appendix identifies this as a base-model outlier also seen for BBQ.Source: §B.1 · p. 20, Table 6 · p. 21
• For every model except Qwen3.5-9B-Base, a linear bias direction emerges at broadly similar depth and separates classes with comparably high AUROC across all three datasets.Table 6 reports each model’s highest AUROC and corresponding layer for BBQ ambiguous, StereoSet, and CrowS-Pairs.Source: §B.1 · p. 21, Table 6 · p. 21

B.2 Activation steering

• The CAA vector has a natural magnitude that can be scaled to control steering strength; Table 7 reports magnitudes before renormalisation.For the reported steering metrics, the authors use the natural magnitude when it is at least 1, and otherwise use α = 1.Source: §B.2 · p. 21
• With abstention allowed, steering increases probability mass on the abstain option as its strength rises.Table 8 reports the neutral (abstain) answer's mean-probability ratio to the anti-biased answer by dataset and model.Source: §B.2 · p. 21, Table 8 · p. 22
• With abstention allowed, the biased-to-anti-biased probability ratio is unchanged or reduced, but the model may place probability mass on abstention even when context supports a definite answer.The ratio remains constant when baseline bias scores are low, including all models on BBQ (dis.) and all but the Llama family on BBQ (ambig.).Source: §B.2 · p. 21, Table 9 · p. 22
• On general-knowledge QA benchmarks, steering also increases uncertainty and entropy, so the confidence effect is not confined to bias prompts.The cited passage identifies Table 5 as showing this increase.Source: §B.2 · p. 22
• Without abstention, steering moves the biased-to-anti-biased probability ratio toward 1, making both answer types more equally likely.The passage attributes this to the model becoming less certain about its answer choice.Source: §B.2 · p. 22, Table 10 · p. 23