Source-linked AI summary
Beyond Accuracy: Measuring Bias Acknowledgment in Chain-of-Thought Reasoning for Responsible AI Evaluation
Xian Sun, Wei Gao, Yingshuo Wang, Lingdong Kong, Yanhang Li, Zhichao Fan, Zexin Zhuang, Wenlong Dong, Zhiyuan Zheng, Hrishikesh Paranjape, Abhishek Mandal, Johnny R. Zhang
TL;DR
Accuracy-only evaluation can treat responses with the same final answer as equivalent even when their reasoning traces differ in whether they flag injected bias. The paper introduces a two-axis diagnostic separating answer-level susceptibility from trace-level acknowledgment and applies it to biased GSM8K trials. GPT-4o and Claude Sonnet 4 are nearly matched in susceptibility but differ substantially in acknowledgment under the study’s rubric.
Problem
Accuracy-only evaluation misses differences in whether reasoning traces explicitly flag injected bias, despite such traces being reviewed in human-facing and higher-stakes settings.
Method
The paper evaluates 3,000 reproducible biased GSM8K trials with susceptibility for correct-to-wrong answer flips and acknowledgment for rubric-defined trace references to injected content.
Results
GPT-4o and Claude Sonnet 4 have near-identical susceptibility rates, 1.3% versus 1.2%, but strict acknowledgment rates of 13.0% versus 75.0%, respectively.
Takeaways & Limitations
Trace-level acknowledgment reveals model differences that final-answer-only evaluation aggregates over, while remaining a descriptive surface signal rather than evidence of internal awareness.
Takeaways & Limitations
Acknowledgment may reflect bias-specific flagging, general verbal caution, or both, limiting its mechanistic interpretation and motivating matched non-bias trials.
Abstract
from arXiv · showhide
Reasoning models are increasingly used in settings where the final answer is not the only object of review: educational tools may show students intermediate steps, decision-support systems may require human oversight, and audit workflows may inspect traces for misleading or biased input. In such settings, two responses can receive the same final-answer score while differing in whether the trace explicitly flags injected biasing content. Accuracy-only evaluation collapses these cases. We study this gap as a measurement blind spot for responsible evaluation and introduce a minimal trace-level diagnostic with two axes: \emph{susceptibility} (whether the bias breaks a previously correct answer) and \emph{acknowledgment} (whether the trace contains a rubric-defined surface reference to the injected content). Across thousands of biased GSM8K trials, GPT-4o and Claude Sonnet~4 have similar susceptibility rates ($1.3\%$ vs. $1.2\%$) but substantially different acknowledgment rates ($13.0\%$ vs. $75.0\%$) under the same rubric.
1. Introduction
Reasoning-model evaluation can miss important differences when it scores only final answers, because traces may expose whether injected bias was recognized. The paper therefore separates answer-level susceptibility from trace-level acknowledgment and applies both to biased GSM8K trials.
- Educational, decision-support, health, and other human-facing systems may require review of reasoning traces, not only final-answer correctness.
- Standard CoT evaluations under input bias score only the final answer, treating unflagged and explicitly flagged bias as equivalent.
- Susceptibility measures whether injected bias changes a previously correct answer, whereas acknowledgment measures whether the trace explicitly references the injected content.
- 3,000 biased GSM8K trials compare GPT-4o and Claude Sonnet 4 across irrelevant context, numerical anchoring, and misleading-hint biases, using strict and keyword acknowledgment labels.
- 13% versus 75% acknowledgment contrasts with near-indistinguishable susceptibility, revealing a trace-level difference that final-answer evaluation collapses.
2. Related Work
Prior work evaluates bias robustness mainly through final-answer outcomes, while related responsible-AI research increasingly measures behavior along multiple axes. This paper adds a deliberately surface-level acknowledgment measure rather than claiming mechanistic faithfulness.
- Bias-injection studies commonly perturb GSM8K or chain-of-thought inputs and score final-answer accuracy or output changes.
- The paper extends answer-level susceptibility evaluation with trace-level acknowledgment, distinguishing responses with the same final answer by what their reasoning exposes.
- Acknowledgment measures surface co-occurrence rather than causal influence, mechanistic faithfulness, or evidence that traces reflect the answer-producing process.
- Trustworthy-AI evaluation increasingly scores safety, security, human-in-the-loop, preference, retrieval, compositional-risk, multimodal-bias, and healthcare-LLM behaviors beyond one aggregate outcome.
3. Framework
The framework evaluates each biased trial with separate answer-level and trace-level binary indicators, then reports their empirical rates and a scalar joint-event summary. Susceptibility specifically captures correct-to-wrong failures caused by bias.
- Each biased trial yields binary susceptibility and acknowledgment indicators, reported as empirical rates over the evaluated sample.
- The framework also reports a scalar summarizing the joint event in which the two indicators fire in a specified direction.
- The setup records the biased answer, chain-of-thought trace, unbiased-run answer, and ground truth for each problem–bias pair.
- Susceptibility is a correct-to-wrong flip: the unbiased answer is correct and the biased answer is incorrect.
- Wrong-to-correct flips and unchanged-incorrect cases score zero, so the metric measures bias-induced failure rather than any bias-induced change.
4. Experimental Design
The experiment applies reproducible bias templates to a shared GSM8K test subset and evaluates two closed models under identical prompting and decoding conditions. Three injections vary in how overtly they present irrelevant or misleading information.
- 500 stratified GSM8K test problems are run unbiased and under three injections, producing 1,500 biased trials per model and 3,000 overall.
- Each bias is a deterministic fixed text template, enabling other groups to regenerate the same 1,500 trials per model.
- The example table contrasts B5’s anchor 72 and B6’s hint 69 with ground truth 71, while B4 remains topically unrelated to the arithmetic.
- B4 inserts topical distractors, B5 adds a salient irrelevant number near the true answer, and B6 prefixes a plausible incorrect answer.
- GPT-4o and Claude Sonnet 4 receive the same step-by-step system prompt, temperature setting, token limit, and one-sample decoding procedure.
5. Results
Across 1,500 biased trials per model, GPT-4o and Claude Sonnet 4 show nearly identical susceptibility but sharply different trace-level acknowledgment under the same rubric. The difference persists across bias types and evaluation rubrics, while conditional acknowledgment remains exploratory because failures are rare.
- 1.3% versus 1.2% correct-to-wrong rates show near-identical susceptibility for GPT-4o and Claude Sonnet 4.Biased accuracy differs by less than 1 percentage point.
- 75.0% versus 13.0% unconditional acknowledgment rates separate Claude Sonnet 4 from GPT-4o despite similar answer-level susceptibility.The primary acknowledgment denominator is 1,500 biased trials per model.
- 5.8× and 7.5× rubric-based separations show that Claude's higher acknowledgment persists under both the strict judge and looser keyword baseline.
- 55.6% versus 21.1% conditional acknowledgment favors Claude on bias-induced failures, but small denominators make this an exploratory direction rather than stable evidence.Failure cases number 18 for Claude Sonnet 4 and 19 for GPT-4o.
- 1.00% versus 0.53% composite silent-failure rates are reported for GPT-4o and Claude Sonnet 4, respectively.
- Answer-level metrics agree within 0.4 percentage points in every bias cell, while Claude's unconditional acknowledgment is higher across all three biases.
6. Discussion and Limitations
The study validates its trace-level judge while emphasizing that acknowledgment is a descriptive surface signal, not evidence of internal awareness or mechanistic faithfulness. Its conditional acknowledgment analysis is exploratory, and its findings are limited to a narrow evaluation setting.
- Validation: The LLM judge matched human labels on 49 of 50 traces (98%, κ=0.96), while the keyword baseline matched 45 of 50 (90%, κ=0.80).Every keyword-baseline disagreement was a keyword false positive.
- Interpretation: Acknowledgment measures a surface trace pattern and does not establish internal bias representation, causal influence on the answer, counterfactual robustness, or mechanistic faithfulness.The paper explicitly distinguishes this measure from faithfulness in the mechanistic sense.
- Interpretation: Acknowledgment may reflect bias-specific flagging, general verbal caution, or both, limiting its mechanistic interpretation.Disentangling these sources would require matched non-bias trials.
- Interpretation: The conditional A | SC2W analysis rests on only 18 and 19 failure cases per model, so its percentages indicate direction rather than stable point estimates.The study therefore uses unconditional acknowledgment with denominator 1,500 for primary comparisons.
- Scope: The reported separation is observed on single-turn responses from two closed-source models, one GSM8K task family, three bias types, temperature T=0, and one primary rubric.Whether it generalizes across models, domains, rubric choices, or sampling temperatures remains open.
7. Conclusion
Accuracy-only evaluation can hide differences in reasoning traces under input bias. The paper introduces susceptibility and acknowledgment as complementary measures and finds near-matched susceptibility but substantially different acknowledgment between GPT-4o and Claude Sonnet 4.
- Accuracy-only evaluation can assign the same score to responses with different reasoning traces under input bias.
- The diagnostic separates susceptibility, whether bias changes a previously correct answer, from acknowledgment, whether the trace references injected content under a rubric.
- Across 3,000 biased GSM8K trials, GPT-4o and Claude Sonnet 4 are nearly matched in susceptibility but differ substantially in acknowledgment.