PaperlayerView exact revision on arXiv

Source-linked AI summary

BiasTrace: Linking Reasoning Behaviours to Biased Outputs in LLMs

Varsha Ramineni, Hossein A. Rahmani, Jerome Ramos, Karin Sevegnani, Emine Yilmaz

arXiv:2608.14161v1cs.AI

TL;DR

Bias in LLMs is poorly understood at the reasoning level, despite risks from inaccurate and discriminatory inferences. BiasTrace labels reasoning behaviours in model-generated traces and links them to biased outcomes, showing that subtle patterns beyond explicitly biased language provide strong predictive signal for bias and improve detection.

  • Problem

    The relationship between LLM reasoning and biased outputs remains poorly understood because existing approaches primarily assess final-answer correctness rather than reasoning processes.

  • Method

    BiasTrace is a fine-grained scheme that labels bias-specific and general reasoning behaviours in model-generated traces and links them to biased conclusions.

  • Results

    Reasoning behaviours beyond explicitly biased language, especially overthinking, provide strong predictive signal for biased outcomes and improve bias detection.

  • Takeaways & Limitations

    Bias evaluation should consider a broader range of reasoning behaviours because seemingly neutral patterns can contribute to biased outcomes.

  • Takeaways & Limitations

    BiasTrace was developed and validated on the structured BBQ task and may not transfer to settings with different reasoning behaviours or without visible reasoning traces.

Abstract

from arXiv · show

LLMs exhibit social biases that can produce inaccurate and discriminatory inferences, posing risks in high-stakes applications. While prior work has made progress in measuring and mitigating bias, it largely focuses on final outputs of models, with limited understanding of the mechanisms that produce biased outcomes. Recent advances in LLM reasoning offers a new lens for investigating bias, yet the link between reasoning and bias remains poorly understood. Existing approaches focus primarily on final answer correctness or explicitly biased language, overlooking different behaviours in reasoning that can drive biased outcomes. We introduce BiasTrace, an annotation scheme for labelling reasoning behaviours in model-generated traces and linking them to biased outcomes. BiasTrace captures bias-specific behaviours (e.g., unsupported demographic assumptions) as well as general reasoning patterns that may implicitly contribute to bias (e.g. overthinking). We apply BiasTrace to reasoning traces in bias-sensitive contexts, scaled using validated LLM-as-a-judge methods, producing a large annotated dataset. Our analysis shows that biased outputs often stem from subtle reasoning behaviours rather than explicitly biased language, and that reasoning-level annotations improve bias detection. We further show that BiasTrace behaviours can be exploited for inference-time mitigation. These findings underscore the importance of examining a broader range of reasoning patterns to better understand bias in LLMs.

1 Introduction

LLM social biases can produce inaccurate and discriminatory inferences, while the relationship between reasoning behaviours and biased outputs remains poorly understood. BiasTrace addresses this gap by annotating reasoning-trace behaviours, linking them to biased conclusions, and enabling scalable analysis of their predictive and evaluative value.

  • Motivation: LLM social biases can produce inaccurate and discriminatory inferences, especially in high-stakes domains such as healthcare, employment, and criminal justice.Such outputs may violate anti-discrimination law and erode public trust.
  • Research gap: Existing bias research has developed benchmarks, fairness metrics, and mitigation strategies, but largely focuses on biased model outputs rather than the mechanisms producing them.Reported strategies include prompt engineering, fine-tuning, and preference optimisation techniques.
  • Research gap: Reasoning traces offer a lens for investigating bias, yet reasoning may also reinforce spurious associations or propagate stereotypes through multi-step inference chains.The relationship between LLM reasoning and bias therefore remains poorly understood.
  • Contribution: BIASTRACE is a fine-grained annotation scheme that labels behaviours in model-generated reasoning traces and links them to biased conclusions.It was developed through iterative inspection of traces in bias-sensitive contexts and informed by prior cognitive-science work.
  • Study design: The study uses a validated LLM-as-a-judge to apply BIASTRACE at scale, annotates traces across models and prompts, and performs predictive analyses of behaviours contributing to bias.The research questions ask whether behaviours beyond explicitly biased language contribute to biased outputs and whether they can identify traces leading to biased outcomes more reliably.

2 BIASTRACE Reasoning Behaviour Annotation Scheme

BIASTRACE is a fine-grained scheme that links reasoning behaviours in bias-sensitive LLM traces to biased outputs. It supports scalable, reliability-checked annotation and prompt-based evaluation of reasoning that leads to bias.

  • Scheme purpose: BIASTRACE systematically links LLM reasoning behaviours to biased outputs and supports construction of an annotated reasoning-trace dataset.The scheme also informs a prompt-based evaluation method for reasoning that leads to bias.
  • Scheme development: The scheme was developed through iterative manual inspection of around n = 40−50 traces from Qwen3-8B and Qwen3-14B on BBQ.Key behaviours were identified through manual inspection and analysis using Claude Opus 4.
  • Trace generation: Qwen3-8B and Qwen3-14B generated step-by-step reasoning traces from BBQ inputs under two prompt templates designed to elicit different reasoning behaviours.The inputs included the BBQ context, question, and answer options.
  • Behaviour categories: BIASTRACE groups binary behaviour labels into four categories covering bias-specific reasoning, meta-cognition, out-of-context knowledge, and reasoning under uncertainty.Labels mark whether a behaviour is present or absent and include descriptions and examples to improve annotation reliability.
  • Annotation validation: Human validation of 100 traces yielded strong inter-annotator agreement, while two labels with consistently below-moderate agreement (κ < 0.3) were excluded.Definitions and examples were refined on a pilot set of n = 12, and a validation set of n = 21 was used to compare judge models.

3 Experimental Setup

The experiments use BiasTrace to connect reasoning behaviours with biased outputs, identify predictive behaviours, and evaluate targeted detection against baseline methods. They analyze traces from multiple open models and reasoning settings while using stricter outcome and judge-based evaluations.

  • Experimental design: BiasTrace supports predictive analyses of behaviours associated with biased outputs and targeted evaluation prompts for detecting biased reasoning.The setup addresses RQ1 through predictive analysis and RQ2 through a targeted prompt incorporating key BiasTrace behaviours.
  • Dataset: Reasoning traces come from Qwen3-8B, Qwen3-14B, and GPT-OSS-120B across multiple reasoning-effort levels.These configurations enable analysis across model scale, architecture, and inference-time reasoning controls.
  • Outcome measurement: The study uses a stricter biased-outcome metric because BBQ incorrect answers include both stereotype-reinforcing errors and generic failures.The passage distinguishes errors that reinforce social stereotypes from failures such as choosing “unknown.”
  • Baseline evaluation: Baseline reasoning-bias evaluations commonly use LLM judges with 0–5 ordinal scores per reasoning step or binarised 0/1 bias labels.The setup compares targeted BiasTrace-based evaluation with established LLM-as-a-judge approaches.

4 Predicting Biased Outcomes Using BIASTRACE

BIASTRACE reasoning behaviours strongly predict biased outcomes, particularly in ambiguous contexts where stereotype-aligned errors are concentrated. Overthinking is the strongest predictor, while guided prompting selectively suppresses some bias pathways and amplifies others.

  • Outcome patterns: Among incorrect answers, stereotype alignment reaches 48.4%–82.7% in ambiguous contexts, peaking at 100% in some cases.Overall error rates remain low at 0.16%–2.28%; guided prompting reduces ambiguous errors, including Qwen3-14B from 1.30% simple to 0.16% guided.
  • Outcome patterns: Disambiguated contexts have higher overall error rates of 6.98%–29.15%, but stereotype-aligned errors fall to 0.4%–4.4%.Most disambiguated errors are neutral “unknown” responses, while guided prompts can increase errors, such as Qwen3-14B from 11.06% simple to 29.15% full.
  • Predictive modelling: A logistic regression predicts biased outcomes with N = 125,487, pseudo-R2 = 0.51, and held-out PR-AUC = 0.44.The model controls for prompt type, question type, BBQ category, and model in a highly imbalanced setting where biased outputs comprise 0.72% of outputs.
  • Reasoning predictors: Overthinking is the strongest reasoning-behaviour predictor, with its effect amplified in ambiguous contexts (OR = 2.34, p < 0.01).The passage links overthinking to excessive doubt or revisiting answers, while noting its correlation with reasoning verbosity.
  • Reasoning predictors: Group assumption is not significant as an explicit-bias-language measure, but its effect is larger for ambiguous questions (OR = 4.88, p < .001).This supports inferring unstated group attributes as a pathway to biased outputs in ambiguous contexts.
  • Prompt effects: Guided prompting reduces biased-output odds overall, amplifies overthinking (OR = 10.03, p < .001), and reduces group-assumption effects (OR = 0.38, p = .021).The results indicate selective suppression of some bias pathways rather than uniform mitigation.
  • Reasoning combinations: Overthinking appears in every high-bias label combination, with combinations involving group assumption or outside demographic knowledge reaching a bias rate of 0.30 and lift above 40.The analysis evaluates all two- and three-way combinations using bias rate and lift relative to the baseline bias rate.

5 Improving Bias Evaluation Using BIASTRACE Behaviours

BiasTrace improves bias evaluation by prompting an LLM-as-a-Judge to identify reasoning behaviours likely to produce biased outcomes. This reasoning-based signal generalizes across models and captures fairness-critical samples more effectively than baseline measures in most evaluated settings.

  • BIASTRACE evaluation prompt: The BIASTRACE prompt defines high-risk reasoning behaviours and combinations, then asks an LLM-as-a-Judge to output a 0/1 prediction of biased outcomes.DeepSeek-V3.2 evaluates both the BIASTRACE and baseline prompts using reasoning bias scores.
  • Generalization to GPT-OSS-120B: Across prompt types and reasoning levels, BIASTRACE correlations with biased outputs range from r = 0.08 to r = 0.41, versus r = 0.01–0.19 for baselines.The same reasoning-based signals generalize from Qwen3-8B and Qwen3-14B to GPT-OSS-120B despite differing reasoning patterns.
  • COMPAS fairness evaluation: On COMPAS, GPT-OSS-120B exhibits an EO gap of 0.28 and EOpp gap of 0.14 across 500 samples comparing African American and Caucasian groups.The benchmark concerns binary two-year recidivism classification from demographic and criminal-history features.
  • Fairness-critical sample identification: BIASTRACE scores achieve the highest correlation with per-sample fairness contributions at r = 0.38 and capture 52–62% of the top 10–20% fairness-critical samples.Baseline 0-5 slightly outperforms BIASTRACE on the top 5% slice.

6 Bias Mitigation at Inference-Time Using BIASTRACE Behaviours

BIASTRACE behaviours can be used at inference time to filter reasoning traces likely to produce biased outcomes before answer aggregation. Across four models on BBQ, this intervention improves accuracy and bias rate over single-trace selection and majority voting over all samples without compromising accuracy.

  • Method: For each BBQ question, the method generates N = 8 reasoning-answer samples and filters traces likely to lead to biased outcomes before answer aggregation.It compares Single, which selects the first sampled trace, with Maj-All and Maj-BIASTRACE, which applies majority voting over BIASTRACE-filtered unbiased samples.
  • Results: Across Qwen3-1.7B, Qwen3-4B, Llama-3.2-3B-Instruct, and Llama-3-8B-Instruct, Maj-BIASTRACE improves accuracy and bias rate over Single and Maj-All.The evaluation spans two Qwen3 models not used in prior analysis and two Llama models from a separate family.
  • Results: BIASTRACE reasoning behaviours can therefore be operationalised as an effective inference-time intervention that reduces biased outputs without compromising accuracy.The paper directs readers to Appendix H for sampling parameters, the majority-voting method, and BBQ sampling details.

7 Related Work

Prior work links LLM reasoning to bias amplification and evaluates reasoning traces primarily through LLM-as-a-Judge scores. BiasTrace instead decomposes reasoning into implicit and explicit behaviours that may produce biased conclusions.

  • Relationship between LLM Reasoning and Bias: Recent studies suggest that LLM reasoning can amplify bias, motivating methods that track reasoning-step probability shifts, fine-tune models for fairness, or re-weight reasoning trajectories.Examples include Bias Proxy, ReGiFT, and a process-level Fairness Reward Model.
  • Evaluating LLM Reasoning for Bias: Existing evaluation frameworks commonly use LLM-as-a-Judge to assign 0–5 ordinal bias scores to individual reasoning steps or complete chains of thought.Some approaches further binarise these ordinal scores using a fixed threshold.
  • BiasTrace: Unlike prior work, BiasTrace does not assume overtly biased language is the primary driver of biased conclusions.It decomposes reasoning into specific behaviours that may produce biased conclusions rather than treating bias as a single category.
  • BiasTrace: BiasTrace captures implicit behaviours and connects them with cognitive science research to study how seemingly neutral reasoning processes can produce bias.This provides a new lens for analysing the mechanisms underlying biased conclusions.

8 Conclusions

BiasTrace shows that biased LLM outcomes can arise from subtle reasoning behaviours beyond explicitly biased language. Reasoning-level signals improve bias evaluation and support mitigation, while motivating reasoning-aware training and broader generalisation studies.

  • Contributions: BiasTrace links diverse reasoning behaviours to biased outcomes, showing that final outputs and explicitly biased language alone do not fully explain LLM bias.The scheme is fine-grained and captures reasoning processes that may be subtle rather than explicitly biased.
  • Implications: Reasoning-level signals can improve bias evaluation and support mitigation by detecting seemingly neutral patterns that contribute to biased outcomes.The conclusions specifically identify overthinking among the reasoning behaviours providing strong predictive value, without reporting a quantitative result here.
  • Future directions: Future work includes reasoning-aware training objectives, smaller annotator models, and testing which reasoning–bias relationships generalise across models, languages, and free-form settings.The annotated dataset is intended to support these directions, including more efficient annotation models.

Limitations

The study’s findings are limited by imperfect access to reasoning processes, restricted evaluation coverage, manually operationalised labels, and validation primarily on constrained tasks. Broader generalisation across models, languages, reasoning settings, and fairness tasks remains unresolved.

  • Reasoning-process scope: The analysis examines associations between observable reasoning behaviours and biased outputs without claiming that chain-of-thought is faithful.The scheme may not transfer to models that reason differently or do not produce visible traces.
  • Evaluation coverage: Evaluation was limited to open-source model families and English, leaving unclear which behaviour–bias relationships generalise across models and languages.BiasTrace behaviours carried meaningful bias signals across the studied families, but the relationships were not identical.
  • Annotation methodology: Reasoning-behaviour labels were operationalised mainly through manual inspection, motivating quantitative alternatives and tests of different or continuous overthinking-revisiting measures.Such approaches could reveal stronger or more generalisable signals across model families and task types.
  • Task scope: BiasTrace categories were developed and validated primarily for BBQ, so less constrained free-form settings may require different or additional behavioural categories.The study also provides only an initial demonstration of linking reasoning behaviour to group fairness metrics for COMPAS.
  • Task scope: Robustly linking reasoning behaviours to fairness outcomes across diverse tasks remains an open challenge.The limitation extends beyond the initial COMPAS demonstration to broader task diversity.

Ethical Considerations

The findings should be interpreted within the study’s examined models, languages, and bias categories, because identifying biased outcomes involves value-laden experimental judgments. The authors also caution against misusing reasoning-behaviour findings to evade safety monitors or amplify bias.

  • The findings are scoped to the specific models, languages, and bias categories examined in the experiments.Their interpretation should not be generalized beyond this experimental scope.
  • Identifying biased outcomes involves value-laden judgments shaped by the experimental design.
  • The authors caution against using the findings to evade safety monitors or deliberately amplify biased behaviour in models.

Appendices … E.1 Ground Truth Labelling

The appendices document BiasTrace’s evaluation pipeline, from trace generation and biased-outcome labelling to behaviour taxonomy development and ground-truth annotation. They also provide extended related work, supplementary analyses, mitigation details, prompts, and implementation resources.

  • Appendices: The appendices cover extended bias-evaluation work, reasoning-trace generation, BBQ outcome labelling, BiasTrace development, annotation validation, predictive modelling, supplementary analyses, mitigation, prompts, and released code.The appendices are organised into sections A–I covering these topics, with code and annotated reasoning data available in the BiasTrace repository.
  • A Extended Related Work: BiasTrace extends bias evaluation from final outputs to reasoning behaviours that contribute to bias.This process-level focus addresses an under-explored area beyond benchmarks that primarily assess stereotypical associations, explicit bias, correctness, or sentiment.
  • B Reasoning Trace Generation: Reasoning traces are generated by prompting models to produce step-by-step reasoning in <think> tags before final answers.BBQ uses simple and guided prompts, while COMPAS uses a single prompt; the appendix also specifies sampling parameters and reasoning settings.
  • C Biased Outcomes on the BBQ Dataset: On BBQ, a biased outcome is an incorrect final prediction that aligns with a stereotype.Stereotype alignment is determined by matching the predicted answer to known stereotyped groups and interpreting that choice according to question polarity.
  • D Developing BIASTRACE: BiasTrace was developed through manual inspection of traces from Qwen3-8B and Qwen3-14B, followed by weighted sampling and analysis of 40 additional traces with Claude Opus 4.The development traces included correct and incorrect outcomes and were sampled according to correctness and stereotype alignment.
  • D.1 BIASTRACE Categories: The taxonomy includes bias-specific behaviours, meta-cognitive reflection, unsupported out-of-context knowledge, and uncertainty-related overthinking.These categories distinguish unsupported demographic assumptions and bias acknowledgement from evaluation-aware reasoning, injected background knowledge, and repeated reconsideration.
  • E.1 Ground Truth Labelling: Two annotators refined the labelling scheme on a 12-sample pilot before independently annotating 100 reasoning traces.Cohen’s κ was reported before and after refinement on the same pilot samples, with unavailable values marked when no positive instances existed.
  • E.1 Ground Truth Labelling: The ground-truth set contains weighted-sampled Qwen8B and Qwen14B traces from simple and guided prompts, with correctness and stereotype-alignment metadata.Sampling prioritised incorrect outputs while retaining a smaller proportion of correct answers; the set was used to evaluate the LLM-as-a-judge.

E.2 LLM-as-a-Judge Evaluation … I.1 BIASTRACE Prompt Structure Analysis

The paper validates BIASTRACE annotation with human labels, examines reasoning behaviours and length across models, and applies the resulting evaluation signal to bias mitigation and prompt analysis.

  • E.2 LLM-as-a-Judge Evaluation: E.2: DeepSeek-V3.2 was selected as judge after showing strong agreement with human annotations, with Cohen’s κ ranging from 0.64–0.84 for several behaviours and 0.30–0.46 for others.Validation used human labels with n = 100, a validation set of n = 21, and a hold-out test set of n = 86; two unreliable labels were excluded.
  • F Predictive Modelling Extra Details: F: Predictive modelling addressed quasi-complete separation because bias acknowledgement labels near-perfectly predicted outcomes in some observations.The analysis compared standard maximum-likelihood estimation with all labels, L1-regularised logistic regression, and standard estimation excluding bias acknowledgement.
  • F.1 Accounting for Reasoning Length: F.1: Overthinking traces averaged 885 tokens versus 378 without the label, while regression showed independent effects for overthinking and reasoning length.Overthinking remained significant (OR= 2.47, p = .018), as did length (OR= 4.09, p < .001), while the guided prompt x overthinking interaction remained large (OR= 8.99, p < .001).
  • G.1 Effect of Reasoning Effect Level for GPT-OSS-120B: G.1: GPT-OSS-120B reasoning effort increased average chain-of-thought length, with means of 86 and 24 tokens at medium and low effort, respectively.GPT-OSS-120B reasoning lengths were much lower on average than Qwen3’s.
  • G.2 Reasoning Behaviours for GPT-OSS-120B: G.2: GPT-OSS-120B showed no overthinking, while bias acknowledgement increased from 1.5% to 7.1% between low and medium reasoning effort.The evaluation correlations were lower for GPT-OSS-120B than Qwen3, where overthinking was the strongest predictor and informed the evaluation.
  • H Bias Mitigation Experiment Details: H: Bias mitigation sampled N = 8 reasoning chains per BBQ question and compared single-chain, all-candidate majority, and BIASTRACE-filtered majority strategies.Maj-BIASTRACE majority-voted among candidates labelled unlikely to lead to bias, falling back to all-candidate majority when none were labelled unbiased.
  • I Prompts for Reasoning Evaluation: I: The evaluation materials included simple and guided reasoning prompts, baseline bias-scoring prompts, and BIASTRACE prompts producing binary bias-likelihood scores.The BIASTRACE judge prompt instructed evaluators to assess reasoning traces against specified behaviours without solving the underlying problem.
  • I.1 BIASTRACE Prompt Structure Analysis: I.1: On the held-out test set (n = 87), a structure-only BIASTRACE prompt that removed behavioural descriptors already outperformed prior approaches focused on stereotypical language.The structure-only variant preserved the input/output specification while removing all behavioural descriptors, isolating prompt structure from rubric content.
Loading 2608.14161v1…