PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Reasoning Introduce Bias? A Study of Social Bias Evaluation and Mitigation in LLM Reasoning

Xuyang Wu, Jinming Nian, Ting-Ruen Wei, Zhiqiang Tao, Hsin-Tai Wu, Yi Fang

arXiv:2502.15361v3cs.CLcs.AI

TL;DR

Social fairness in reasoning-based LLMs remains underexamined, even as reasoning can reinforce stereotypes. This study evaluates bias in reasoning traces and outputs and proposes ADBP; results show that accuracy gains do not remove bias, which can intensify in ambiguous contexts.

  • Problem

    Prior evaluations focused largely on reasoning capabilities in math and coding, leaving social bias in reasoning-based LLMs and its interaction with demographic variables insufficiently examined.

    Source: §1 · p. 2

  • Method

    The study evaluates prediction accuracy and bias in reasoning traces and outputs, then proposes ADBP, which uses answer distributions as a bias proxy without external annotation.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Reasoning-based models improve accuracy but do not mitigate bias, often amplifying stereotypes in ambiguous contexts; stereotypical reasoning appears significantly more often in incorrect responses.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Reasoning can propagate social biases alongside its performance benefits, and ADBP is proposed to self-detect and address bias through answer-distribution patterns.

    Source: §7 · p. 9

    LLM-as-a-judge ratings were not verified through human labeling because manual annotation was prohibitively costly.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Recent advances in large language models (LLMs) have enabled automatic generation of chain-of-thought (CoT) reasoning, leading to strong performance on tasks such as math and code. However, when reasoning steps reflect social stereotypes (e.g., those related to gender, race or age), they can reinforce harmful associations and lead to misleading conclusions. We present the first systematic evaluation of social bias within LLM-generated reasoning, focusing on reasoning language models (e.g., DeepSeek-R1, OpenAI o1) that natively produce reasoning chains as part of their answers. Using the BBQ dataset, we analyze both prediction accuracy and reasoning bias across a broad spectrum of models, including instruction-tuned and CoT-augmented variants of DeepSeek-R1 (8B/32B), ChatGPT, and other open-source LLMs. We quantify how biased reasoning steps correlate with incorrect predictions and often lead to stereotype expression. To mitigate reasoning-induced bias, we propose Answer Distribution as Bias Proxy (ADBP), a lightweight mitigation method that detects bias by tracking how model predictions change across incremental reasoning steps. ADBP outperforms Stereotype-free Reasoning Pattern (SfRP) baseline in most cases, mitigating bias and improving the accuracy of LLM outputs. Evaluation and mitigation code is available at https://github.com/elviswxy/LLM_reasoning_bias.

1 Introduction

Reasoning models’ gains on logical tasks leave social fairness underexamined, while shifts in reasoning can reinforce stereotypes and skew answers. The study evaluates bias in reasoning traces and outputs, then proposes ADBP to detect and reduce it.

1 Introduction

  • Chain-of-thought prompting encourages multi-step problem-solving, and reasoning capabilities distilled into smaller models can improve their performance on math and coding tasks.Source: §1 · p. 1
  • Prior evaluations focused mainly on math and coding, leaving social fairness and interactions between reasoning patterns and demographic variables less examined.Source: §1 · p. 2
  • Reasoning-path shifts and superficial self-reflection can reinforce stereotypes in reasoning and lead to biased outputs, motivating analysis of demographic effects on processes and outcomes.Source: §1 · p. 2
  • ADBP uses answer distributions as a proxy to detect and reduce reasoning-induced bias without external annotation.Source: §1 · p. 2
  • Reasoning-based models improve accuracy but do not mitigate bias, often amplifying stereotypes in ambiguous contexts; stereotypical reasoning is significantly more frequent in incorrect responses.Stereotypical reasoning also appears in correct answers, but at a significantly lower frequency.Source: §1 · p. 2
  • The study extends social-bias assessment beyond final predictions to include reasoning steps themselves.Source: §1 · p. 2

2 Related Works

Prior work has examined social bias across multiple LLM applications and has begun studying bias in prompted reasoning. This paper distinguishes its focus on models that inherently generate reasoning steps and evaluates bias within those steps as well as final answers.

2 Related Works

  • Social-bias research has examined multiple-choice QA, open-ended generation, search and ranking, and retrieval-augmented generation, including persistent fairness concerns in RAG.Examples include BBQ mitigation work (Ma et al., 2024a), biased generated news (Fang et al., 2024a), and fairness issues in search and ranking (Wang et al., 2024).Source: §2 · p. 2
  • Prompted-reasoning studies report harmful or undesirable outputs in sensitive domains (Shaikh et al., 2023) and prompt-dependent gender bias (Kaneko et al., 2024).Source: §2 · p. 2
  • Prior work also uses LLMs as evaluators, while noting that evaluators may harbor cognitive biases (Wang et al., 2025a) and that implicit prejudices persist in state-of-the-art models.Source: §2 · p. 3
  • Unlike prior work on biased outputs or prompt-induced reasoning, this study examines models that inherently generate reasoning steps and assesses bias within intermediate steps as well as final answers.Source: §2 · p. 3

3 Evaluation Framework

The framework evaluates model outcomes through accuracy and bias measures, and assesses reasoning quality with an LLM judge. It also compares original reasoning with a version filtered for biased steps.

3 Evaluation Framework

  • The framework evaluates prediction accuracy and bias, while an LLM-as-a-judge assesses reasoning quality.Source: §3 · p. 3

3.1 Dataset

  • BBQ covers nine social-bias categories and two intersectional categories, pairing ambiguous contexts with disambiguated versions that differ by a clarifying sentence.Source: §3.1 · p. 3

3.2 Prompt and Reasoning Model Inference

  • Model-specific prompts support comparisons across open-source instruction-tuned, proprietary API-based, and reasoning-based models, using exact string matching against ground-truth labels.Source: §3.2 · pp. 3–4

3.3 Outcome Evaluation

  • Accuracy measures correct predictions, while Biasamb reflects non-stereotyping responses among ambiguous non-“unknown” cases and Biasdis ranges from -1 (against bias) to 1 (aligned with bias).Source: §3.3 · p. 4

3.4 Reasoning Step Evaluation

  • Reasoning tokens are split at newlines, and GPT-4o scores each step’s bias severity from 0 to 4 in five independent judgments resolved by majority vote.Source: §3.4 · p. 4

3.5 Stereotype-free Reasoning Pattern

  • SfRP removes every reasoning step assigned a nonzero bias score; the study compares original and filtered reasoning and uses the filtered version as a mitigation input.Source: §3.5 · p. 4

4 Experiments

The experiments compare model accuracy and bias across BBQ contexts, finding that reasoning-related accuracy gains do not ensure lower bias. Bias in reasoning often accompanies incorrect answers, and removing biased reasoning steps improves accuracy in the tested cases.

4.1 Experimental Settings

  • Models were evaluated zero-shot without fine-tuning on BBQ, comparing accuracy and bias across ambiguous and disambiguated questions and social-bias categories.The model set included DeepSeek-R1 distilled variants, their instruction-tuned counterparts, Marco-o1, and OpenAI model results.Source: §4.1 · pp. 4–5, Table 1 · p. 5

4.2 Outcome Evaluation

  • DeepSeek-8B led similarly sized models in accuracy across all 11 categories, yet had similar or worse bias in 9/11 ambiguous and 5/11 disambiguated categories.The table reports accuracy and bias across models, categories, and contextual conditions.Source: Table 1 · p. 5, §4.2 · p. 5
  • Reasoning models outperform base counterparts in disambiguated contexts, but their advantage diminishes under ambiguity, with some reasoning models underperforming GPT-4o.The reported gaps widen under ambiguity for o1, o1-mini, and o3-mini.Source: §4.2 · pp. 5–6

4.3 How Bias Arises in Reasoning?

  • Incorrect reasoning-model answers often show stronger bias that intensifies after initial steps, while correct answers usually contain little or isolated bias.The analysis finds this association does not establish perfect causality; some incorrect answers have unbiased reasoning.Source: §4.3 · pp. 6–7
  • 160 of 186 Figure 4b questions used non-negative polarity, which DeepSeek-8B often misinterpreted; DeepSeek-32B failures were more consistently associated with biased reasoning.Bias was sparser among incorrect answers in disambiguated contexts.Source: §4.3 · p. 7

4.4 Can Removing Bias Improve Predictions?

  • 0.517 and 0.717 accuracy gains in Cases 1 and 3, and 0.100 and 0.526 in Cases 2 and 4, came from SfRP versus original biased reasoning.These are the four reported cases comparing stereotype-free reasoning with original reasoning.Source: §4.4 · pp. 7–8, Figure 5 · p. 8

5 Robustness of Evaluation Setup

The evaluation trends were stable under two changes to the LLM judge prompts and rating scale, and similar bias patterns appeared in Religion and SES.

5.1 LLM-as-a-judge Prompt Variations

  • Relative bias-score differences between correct and incorrect predictions remained stable under revised judge instructions and a three-level scale, despite minor absolute-score shifts.The alternatives clarified stereotype definitions or reduced scale granularity from five levels to three.Source: §5.1 · p. 8

5.2 Evaluation for Additional Categories

  • Religion and SES showed the same pattern as Age: DeepSeek models had systematically higher bias scores on incorrect predictions.The additional-category analysis extends the earlier Age-category analysis.Source: §5.2 · p. 8

6 Mitigating Social Bias in LLM Reasoning

ADBP tracks answer changes as reasoning steps are added, using consistency or targeted comparison of conflicting answers to mitigate bias. It outperforms SfRP in many scenarios, though results vary across models and cases.

6 Mitigating Social Bias in LLM Reasoning

  • ADBP adds reasoning steps incrementally and tracks answer changes, selecting a consistent answer or comparing conflicting answers with their associated reasoning.When answers shift, the method isolates the steps linked to the final and most common alternative answers for targeted reconsideration.Source: §6 · p. 8

6.1 Mitigation Performance

  • ADBP performed better than SfRP in many scenarios, except when the base model was Qwen-32B, which may use filtered reasoning steps effectively.The comparison used the same experimental setup as SfRP-based mitigation.Source: §6.1 · p. 8
  • For initially incorrect DeepSeek-8B/32B cases, SfRP corrected 24∼44%, while ADBP raised this to 38∼60%; on Llama-8B, ADBP exceeded SfRP by +38%.The Llama-8B comparison covers cases where both the base and reasoning models were initially mistaken.Source: Table 2 · p. 9, §6.1 · p. 9
  • ADBP achieved the highest accuracy in all cases except Case 4 when compared with two prompt-only mitigation baselines.The baselines use self-debiasing via Explanation and Combined Debiasing Prompt.Source: §6.1 · p. 9

7 Conclusion and Future Work

The conclusion frames reasoning as both a performance aid and a potential channel for social bias, and presents ADBP as a way to detect and address bias through answer distributions. Future work aims to integrate mitigation into LLM training and inference.

7 Conclusion and Future Work

  • Chain-of-thought reasoning can improve performance on structured tasks such as math and coding while also propagating social biases.The study presents ADBP as a way to self-detect and resolve bias through answer distribution patterns.Source: §7 · p. 9
  • Future work aims to integrate social-bias mitigation more tightly into LLM training and inference pipelines to enhance robustness in reasoning.Source: §7 · p. 9

Limitations

The authors identify uncertainty in LLM-as-a-judge evaluation, exploratory design choices, and limits on model scale and refusal behavior as boundaries of the study.

Limitations

  • The LLM-as-a-judge results were not verified with human labels because manual annotation was too costly.The authors note that the judge method may still introduce uncertainty.Source: Limitations · p. 10
  • ADBP is exploratory and does not account for cost-efficiency or the logical flow of reasoning.Source: Limitations · p. 10
  • Computational constraints prevented experiments on full-size DeepSeek-R1, and refusal behavior was rare under the controlled BBQ questions and neutral framing.The authors note that prompt phrasing may affect refusal rates and merits further study.Source: Limitations · p. 10

A.1 Dataset Statistics

Table 3 reports the total number of examples in each BBQ category.

A.1 Dataset Statistics

  • Table 3 lists the total number of examples within each BBQ category.Source: Table 3 · p. 13

A.2 Prompts

The appendix presents model-evaluation and judging prompts, supplementary results, and examples of biased reasoning and ADBP use.

A.2.1 Outcome evaluation prompt for instruction tuned models and Marco-o1

  • Outcome-evaluation prompts cover instruction-tuned models and Marco-o1, as well as DeepSeek models; the described input includes context, question, and answer options.The prompt instructions specify JSON input containing those elements.Source: Figure 7 · p. 13, §A.2.1 · p. 13, Figure 8 · p. 14

A.2.4 Rewritten LLM-as-a-judge prompt

  • LLM-as-a-judge prompts include a rewritten version and a three-level version; the rewritten prompt supplies the reference answer and a reasoning segment for an impartial evaluator.The appendix also shows the general judge prompt.Source: Figure 9 · p. 14, §A.2.4 · p. 15, Figure 10 · p. 15

A.2.6 ADBP Prompt

A.3 Model-level Overall Evaluation Results

  • Reasoning-based models achieve higher accuracy but retain elevated bias across ambiguous and disambiguated BBQ contexts.Table 4 reports accuracy and bias scores for both context types.Source: Table 4 · p. 16

A.4 Evaluation for LLM-as-a-judge Prompt Variations; A.5 Evaluation for Additional Categories

  • The judge-assessed reasoning-step bias trend is robust across prompt variations, and additional severity results cover Age, Religion, and SES.Source: Table 5 · p. 16, Table 6 · p. 16

A.6 Biased Reasoning Examples

A.7 Non-negative Question Polarity Failure Example

  • One example is marked incorrect because the model assumes a positive-polarity question despite correct reasoning and answer meaning.Source: Figure 16 · p. 20, §A.7 · p. 20

A.8 ADBP Example

Loading 2502.15361v3…