PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Thinking Help Fairness? Reasoning Tokens Resolve Some Biases but Create More

Deng Pan, Joe Germino, Yihong Ma, Elizabeth Daly, Nuno Moniz, Ting Hua, Nitesh Chawla

arXiv:2609.30768v1cs.AI

TL;DR

Prior studies disagree about whether thinking mitigates or amplifies bias, and prior audits examine final predictions rather than bias dynamics within thinking. We compare thinking with within-model non-thinking inference and analyze trace depth and pair-state transitions. In all nine tested settings, thinking creates roughly five times more counterfactual flips than it resolves.

  • Problem

    Prior findings conflict on whether thinking mitigates or amplifies bias, while audits that measure only final predictions leave bias propagation during thinking unexamined.

    Source: §1 · p. 1

  • Method

    We compare thinking with within-model non-thinking inference and use CDPG to track bias with depth and BTM to track pair-state transitions.

    Source: §1 · pp. 1–2

  • Results

    In all 9/9 model–dataset settings, thinking creates roughly 5× more new counterfactual bias pairs than it resolves, with the asymmetry originating in pair-state transitions.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Thinking amplifies the direction of accumulated reasoning, sharpening either fair or biased outcomes depending on whether that reasoning is biased.

    Source: §5 · p. 8

    The study focuses on binary sex and race attributes, three Qwen-lineage models, tabular tasks, a demographic-parity contrast, and no mitigation interventions.

    Source: §6 · p. 9

Abstract

from arXiv · show

Thinking in reasoning language models (RLMs) has been subject to debate on whether it resolves or amplifies bias. Prior works have shown competing conclusions in both directions. Using a within-model thinking-vs.-non-thinking ablation across QwQ-32B, DeepSeek-R1-Distill-Qwen-32B, and Qwen3-32B on three high-stakes decision tasks (Adult, COMPAS, Credit), we show that thinking has an asymmetric dual effect on counterfactual fairness: it both resolves counterfactual flips produced by the non-thinking baseline and creates new flips at near-saturating model confidence. In all nine (model, dataset) combinations, the created flips outnumber the resolved flips by roughly 5 times. To explain the effect, we treat the thinking trace itself as a measurable site of fairness change and study it through two dynamic instruments: 1) We propose Counterfactual Depth Probability Gap (CDPG) to track bias evolution along thinking depth, and observe that bias propagates and amplifies with thinking. 2) We also formulate the Bias Transition Matrix (BTM) to show how predictions of counterfactual pairs change from non-thinking to thinking, and find that the asymmetric dual effect originates in the pair-state joint transition.

1 Introduction

Prior audits focus on final predictions, leaving the fairness dynamics inside reasoning traces unexamined. Across the tested settings, thinking both resolves and creates counterfactual flips, while depth- and pair-state analyses expose how these changes unfold.

1 Introduction

  • Mixed prior findings leave practitioners without a defensible fairness standard for RLM deployment, while outcome-only audits leave the thinking process unexamined.The paper therefore studies how bias propagates during thinking.Source: §1 · p. 1
  • In all 9/9 model–dataset settings, thinking creates roughly 5× more new counterfactual flips than it resolves.The flips it creates occur at high confidence.Source: §1 · pp. 1–2
  • CDPG tracks counterfactual bias along thinking depth, while BTM tracks how paired predictions transition from non-thinking to thinking.The analyses report that bias propagates and amplifies with depth, and that the asymmetry originates in pair-state joint transitions.Source: §1 · p. 2
  • Within a counterfactual pair, one side can maintain its initial prediction while the other drifts toward the opposite boundary, sharpening fair and biased outcomes in lockstep.Whether this resolves or creates a flip depends on whether the accumulated reasoning is itself biased.Source: §1 · p. 2
  • Group-level transitions are nearly identical across protected sides, but asymmetric pair-state transitions drive creation to outweigh resolution.The pair-level joint transition exposes asymmetry that group-level transitions do not.Source: §1 · p. 2

2 Related Work

Prior work distinguishes group-level audits from counterfactual comparisons, reports opposing effects of thinking on bias, and probes reasoning traces through truncation. This paper situates its trace-dynamics tools alongside those approaches.

2.1 Fairness audits of LLMs

  • Group-level audits use demographic parity and equalized odds, while counterfactual audits compare outcomes across protected-attribute values; the paper uses disagreement as its primary signal and reports group fairness alongside it.This makes it possible to examine where the two diagnostics agree or diverge.Source: §2.1 · p. 2

2.2 Competing reports on reasoning and fairness

2.3 Probing reasoning trace dynamics

  • Prefix-truncation probing truncates reasoning and asks the model to answer again; Lanham et al. (2023) use it to test faithfulness, and Wu et al. (2025) use answer shifts as a bias proxy.The paper adapts this construction to counterfactually paired inputs and complements it with pair-state transitions between non-thinking and thinking.Source: §2.3 · p. 3

3 Methods

The methods operationalize counterfactual and group fairness, then track fairness through thinking depth and changes in predictions from non-thinking to thinking.

3.1 Fairness Definitions

  • Counterfactual fairness is operationalized through protected-attribute substitution, hard flip rates, and soft per-pair probability gaps, alongside demographic-parity group disparity.The two fairness diagnostics capture different harms and can conflict.Source: §3.1 · p. 3

3.2 Within-pair Contingency Cells

  • Four contingency cells classify each pair by whether a counterfactual flip occurs under non-thinking and thinking, distinguishing resolved flips from created flips.Cell b represents flips resolved by thinking; cell c represents flips created by thinking.Source: §3.2 · p. 3, Table 1 · p. 4

3.3 Bias Propagation Tool: CDPGk

  • CDPG_k measures the per-pair decision gap after each thinking-trace prefix, re-fed to the model to estimate Yes/No probabilities and trace disparity across depth.Traces are divided into 10 equal-token segments; prefix re-feeding gives an interventional reading of the decision at that depth.Source: §3.3 · p. 4

3.4 Transition Matrix Tools

  • BTM compares group-level prediction transitions with pair-state transitions; the latter preserve joint counterfactual outcomes, mapping resolved and created flips to opposite off-diagonal blocks.Pair-state transitions capture dynamics marginalized out by group-level transitions.Source: §3.4 · pp. 4–5

4 Results

Across three tabular tasks and nine model–dataset settings, thinking consistently increased counterfactual disparity despite resolving some flips. Depth and pair-state analyses locate this dual effect in confidence amplification and asymmetric joint transitions that group-level audits can miss.

4.1 Experimental setup

  • The study compares three open-weight 32B RLMs on 5,000-record Adult, COMPAS, and Credit samples, pairing records by protected-attribute substitution.Only pairs parsing on both sides of both arms are retained; DeepSeek-R1-Distill-Qwen-32B on COMPAS has an effective N of 1,430 because the non-thinking arm often refuses.Source: §4.1 · p. 5

4.2 The phenomenon: asymmetric dual effect of thinking

  • Across nine settings, created flips exceeded resolved flips (3,508 vs. 620; ≈5.7×), while cell-c Ddisp rose from 0.024 under nothink to 0.835 under thinking (≈34×).Table 2 also reports that cell-b resolution is real but its per-pair disparity is approximately five times weaker than cell-c creation.Source: §4.2 · p. 5, Table 2 · p. 6
  • ∆Dcf was positive in all nine settings (+0.016 to +0.122), while ∆Dgroup amplified in four, mitigated in four, and was effectively unchanged in one.Group-disparity changes were small (|∆Dgroup| ≤0.035), and their direction varied by dataset.Source: §4.2 · p. 5, Figure 2 · p. 6

4.3 Bias propagation: the source of the dual effect

  • Across contingency cells, mean |CDPGk| rises toward saturation in c and d, where thinking ends with a flip, but stays near zero or decays in agreement cells a and b.The cell-c propagation slope’s sign and order of magnitude persist when segmentation granularity changes to K ∈ {5, 10, 20}.Source: §4.3 · p. 6, Figure 3 · p. 7
  • In cells b and c, one pair side anchors while the other moves: it converges with the anchor in b but moves toward the opposite boundary in c.The same confidence sharpening resolves or creates a flip depending on whether the accumulated evidence is biased.Source: §4.3 · p. 7

4.4 Pair-state asymmetry and group-level blindness

  • Protected-side group transition rates differ by at most 0.014, yet pair-state transitions yield |c| > |b|; within-pair correlation changes the magnitude, not the direction.Under a first-order conditional-independence assumption, empirical transition probabilities and the non-thinking starting population already predict |c| > |b| in every setting.Source: §4.4 · p. 7
  • The cell-c contribution to ∆Dgroup comes from its directional tilt; large created-flip counts can remain at the group-disparity noise floor when that tilt splits symmetrically.Cells a and b contribute zero to the thinking-arm marginal difference, while cell d is small (≤68 pairs pooled).Source: §4.4 · pp. 7–8

5 Conclusion

Thinking acts as a confidence amplifier that can both resolve and create counterfactual flips. The paper attributes the asymmetry to pair-state transitions that group-level audits can miss.

5 Conclusion

  • Thinking sharpens each pair side toward its current prediction boundary, resolving flips on some pairs and creating them on others depending on whether reasoning carries bias.The same dynamic amplifies fair and biased outcomes in lockstep.Source: §5 · p. 8
  • CDPG tracks bias through thinking depth, while BTM locates the asymmetric dual effect in joint pair-state transitions rather than near-matched group-level transitions.The cross-group disagreement driving the asymmetry is invisible to a single-group audit.Source: §5 · p. 8
  • The conclusion identifies pair-state transitions as the place fairness audits of reasoning models should examine for counterfactual bias under thinking.The paper states that group-level marginal transitions shift in near-lockstep across protected-attribute values.Source: §5 · p. 8

6 Limitations

The study is limited to counterfactual fairness audits on tabular binary tasks, with additional methodological constraints on measuring thinking depth. It diagnoses fairness dynamics rather than evaluating mitigation strategies.

Scope

  • The evaluation covers three tabular binary tasks with binary protected attributes (sex and race), not free-form, multiclass, or multilingual settings.The study also leaves broader group-fairness criteria and intervention-based mitigation for future work.Source: §6 · p. 9

Methodological assumptions

  • The CDPG sensitivity check varies K across 5, 10, and 20 for Qwen3-32B, but tests only one model family and a linear-slope summary.The reported check does not establish robustness of the full trajectory shape.Source: §6 · p. 9
  • The within-model non-thinking baseline uses an empty closed <think></think> block, and aggregates include only completions that respect that instruction.A small fraction may still emit reasoning tokens after the prefix; all inference uses 4-bit AWQ quantisation.Source: §6 · p. 9

Disposition

  • The study is diagnostic, not mitigating; evaluating concrete mitigation candidates against the propagation and transition-matrix objectives is left to future work.The datasets are used for measurement, not training or deployment.Source: §6 · p. 9

Ethics Statement

The paper uses public fairness datasets for measurement and cautions against treating visible deliberation as evidence of fairness. It does not release new model weights.

Ethics Statement

  • Adult, COMPAS, and Credit are public datasets used for measurement only, not for training or deployment.The reported concern is that thinking-token reasoning can construct high-confidence counterfactual disagreements at five times the per-pair magnitude of corresponding reductions.Source: Ethics Statement · p. 9
  • High-stakes deployment should not rely on the appearance of deliberation as fairness assurance.This caution follows from the reported high-confidence counterfactual disagreements created by thinking-token reasoning.Source: Ethics Statement · p. 9
  • The authors do not release new model weights.Source: Ethics Statement · p. 9

Use of AI Assistants

The appendix documents AI-assistant use, controlled profile generation, and supplementary analyses. Its checks report robustness to CDPG segmentation and persistence of the asymmetric effect beyond the Qwen lineage.

Use of AI Assistants

  • AI coding assistants helped develop and refactor experimental code, and tools were used for limited language polishing; authors reviewed code and retained responsibility for scientific claims.Source: Use of AI Assistants · pp. 9–10

A Natural-Language Profile Generation

  • Counterfactual profiles are generated deterministically from each row, with matched pairs differing only in the protected-attribute wording and dependent pronouns.Pronouns are gendered for Adult and Credit, and neutral for COMPAS.Source: A Natural-Language Profile Generation · p. 11

C Sensitivity of CDPG to the segmentation granularity K

  • The cell-c propagation slope remains positive and of the same order of magnitude at K=5, 10, and 20 across all three datasets.The check uses Qwen3-32B; the mild slope decrease with finer segmentation is attributed to saturation near the end of traces.Source: C Sensitivity of CDPG to the… · pp. 11–12, Table 6 · p. 11

D Generalization beyond the Qwen lineage

  • Created flips outnumber resolved flips by 4.9×–7.8× (pooled 7.5×) across powered non-Qwen settings, with the asymmetric effect also reproducing in confidence patterns.The replication uses gemma-2-27B and Llama-3.1-8B with prompt-elicited thinking traces; neither model has a native <think> channel.Source: D Generalization beyond the Qwen lineage · p. 12

E Inference Prompt Format

  • The question line varies by dataset: income for Adult, two-year recidivism for COMPAS, and next-payment default for Credit.Table 8 presents the prompt formats for the thinking and non-thinking arms on Adult.Source: E Inference Prompt Format · p. 12

F Absolute disparity values per (model, dataset) cell

G Group-level transition matrix: detailed breakdown

The detailed group-level transition analysis shows a strong overall shift toward positive predictions, but its group-stratified gaps are much smaller than the pair-state tilts. The latter depend on how both members of a counterfactual pair transition jointly.

G Group-level transition matrix: detailed breakdown

  • Group-stratified transition gaps are at most 0.014 for Qwen3-32B, 0.017 for QwQ-32B, and 0.012 for R1-Distill-Qwen-32B, far smaller than pair-state tilts such as 0.53.The pair-level tilt cannot be inherited from group-level bias; the joint transition of paired instances constrains whether they transition consistently.Source: G Group-level transition matrix:… · pp. 12–13
  • The pooled marginal transition matrix uses 84,428 per-instance observations retained when each instance’s own thinking and non-thinking arms both produce Yes/No.This per-side parseability filter is looser than the pair-level filter because one unparseable side does not remove the other.Source: G Group-level transition matrix:… · p. 13
  • Thinking moves predictions from N to Y 20,216 times versus 705 moves from Y to N, a 28.7× count ratio; transition probabilities are 0.048 and 0.289, respectively.Thinking moves predictions toward Y for both protected-attribute values and acts more aggressively on N baselines.Source: G Group-level transition matrix:… · p. 13

H Independence-bound check per (model, dataset) cell

The independence check compares predicted pair-state changes with empirical counts across model–dataset cells. The asymmetry |c| > |b| persists in both, while within-pair correlation dampens but does not reverse it.

Independence-bound check per (model, dataset) cell

  • The check estimates p and q from group-level transition matrices and uses pair-state row totals for the nothink populations.Those inputs produce independence-based predictions for |c| and |b| counts in each model–dataset cell.Source: H Independence-bound check per (model,… · p. 13
  • |c| > |b| holds under both independence predictions and empirical counts in every model–dataset cell.The directional asymmetry is present even when per-side flips are assumed independent, given the observed nothink-state population skew.Source: H Independence-bound check per (model,… · p. 13
  • Independence overpredicts |c| by 2–7× and underpredicts |b| by 2–10×; empirical |c|/|b| ratios are 5–100× smaller.Within-pair flip correlation dampens the base-count asymmetry but cannot reverse it.Source: H Independence-bound check per (model,… · p. 14

I Per-model pair-state transition matrices

Pair-state transitions show a shared asymmetry across models, with model differences concentrated in how transitions from NN divide between split and en-bloc changes. The reported CDPG trajectories are robust to the segmentation choice.

Per-model pair-state transition matrices

  • Across every model–dataset cell, NN dominates the starting distribution and NN→M is the dominant cell-c channel, while NN→YY rates vary by model.All three models exhibit the same |c| ≫ |b| mechanism, with heterogeneity in the relative weight of the two channels out of NN.Source: I Per-model pair-state transition… · p. 14
  • Qwen3-32B is most NN-inert, with NN-row stickiness of 89%/69%/98% across Adult/COMPAS/Credit; QwQ-32B reaches 48.4% NN→YY on COMPAS.DeepSeek-R1-Distill-Qwen-32B has the highest aggregated NN→M rate at Adult: 0.165; COMPAS has ∼72% refusal and 1,430 parseable pairs.Source: I Per-model pair-state transition… · p. 14, Table 13 · p. 15, Table 14 · p. 16
  • Across all nine cells, independence overpredicts |c| by 2–7× and underpredicts |b| by 2–10×, yet |c| > |b| holds under both prediction and data.Within-pair flip correlation makes empirical |c|/|b| ratios smaller than independence-bound ratios, without reversing the asymmetry.Source: Table 12 · p. 15
  • On COMPAS, QwQ-32B shifts NN→NY to 0.054 versus NN→YN at 0.016; on Credit, its NN→NY is 0.077 versus NN→YN at 0.050.On Credit, YY→NY is 0.037 versus YY→YN at 0.012, reinforcing the Female tilt.Source: Table 13 · p. 15
  • On COMPAS, R1-Distill shifts NN→NY to 0.086 versus NN→YN at 0.046; on Credit, the direction reverses, with NN→YN at 0.056 versus NN→NY at 0.042.The Credit split is Male-leaning and opposite to QwQ’s direction.Source: Table 14 · p. 16
  • Mean |CDPG_k| trajectories are reported as robust to segmentation choices K ∈ {5, 10, 20}.The figure pools a Qwen3-32B rerun subsample across the three datasets.Source: Figure 6 · p. 17

Related papers

Loading 2609.30768v1…