PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Accuracy: Counterfactual Fragility and Demographic Bias in Clinical Evaluation of LLMs

Chaitali Deb Purkayastha, Bharath Kumar Bolla, Vishnu Surya Reddy Nandi

arXiv:2609.32807v2cs.AI

TL;DR

Clinical LLM benchmarks emphasize answer accuracy, but that leaves counterfactual consistency and demographic robustness untested. This study applies perturbation-based evaluation to clinical models and finds that accuracy, counterfactual validity, answer stability, EDD, and stereotype evidence capture distinct dimensions.

  • Problem

    Medical QA benchmarks measure correct answer selection, but do not test whether explanations, counterfactual reasoning, and recommendations remain reliable under relevant or irrelevant changes.

    Source: §1 · p. 1, §2.1 · p. 2

  • Method

    The study stress-tests six clinical LLMs through controlled input modifications and evaluates counterfactual validity, demographic-prefix robustness, and explanation-level stereotype evidence.

    Source: §1 · p. 2

  • Results

    Most counterfactual attempts were invalid, 20.3% of demographic comparisons changed answers, and MedGemma performed best overall while the evaluation dimensions remained distinct.

    Source: §4.1 · p. 9, §7 · p. 14

  • Takeaways & Limitations

    Counterfactual validity, answer stability, EDD, and stereotype evidence should be assessed separately from accuracy as safety-screening dimensions.

    Source: §1 · p. 2, §7 · p. 14

    The 150-question subset was not specialty- or difficulty-stratified, demographic prefixes could interact with case details, and clinicians did not independently validate automated judges.

    Source: §6 · p. 14

Abstract

from arXiv · show

Clinical LLM evaluation often emphasizes answer accuracy; however, accuracy alone does not test counterfactual consistency or demographic robustness. We evaluated six LLMs on 150 MedQA USMLE questions using two automated perturbation tests to assess their performance. The counterfactual validity (CFV) test asked each model to make a minimal, plausible clinical change that would make a different answer correct. The demographic robustness test added six demographic prefixes to the same vignette and compared the answers and explanations with a no demographic baseline. Of the 900 CFV attempts, 228 (25.3 %) were valid and 672 were invalid. Across 5,400 demographic comparisons, 1,097 answers were changed (20.3%). Automated judging identified 3,128 stereotype evidence flags, including 1,932 in the broad Other category. MedGemma 27B achieved the highest accuracy (87.1%) and CFV (63.3%), lowest answer change rate (16.0%), and low mean Explanation Demographic Dissonance (EDD) score (0.169). However, its accuracy still exceeded its CFV, indicating that correct answers do not guarantee reliable performance on the counterfactual validity task. OpenBioLLM had the highest answer change rate and EDD, whereas GLM had the highest stereotype flag rate. These findings show that accuracy, CFV, answer stability, EDD, and stereotype evidence capture different evaluation aspects. Because all judgments were automated and no clinician validation was available, the results support safety screening but do not establish clinical deployability of the model.

1 Introduction

The study argues that clinical LLM evaluation must look beyond answer accuracy to counterfactual reasoning and demographic robustness. It introduces perturbation-based tests and taxonomies to examine these distinct safety behaviors.

1 Introduction

  • Benchmark accuracy alone does not show whether a model can explain its answer, identify relevant clinical changes, or avoid shifts under irrelevant demographic context.A model can be accurate on an exam-style benchmark yet remain unsafe as an explanatory assistant if it cannot reliably answer perturbation questions.Source: §1 · pp. 1–2
  • The study examines counterfactual fragility and demographic bias, asking whether clinical LLMs can produce plausible answer-changing counterfactuals and remain stable when demographic descriptors are added.The study frames these as two under-tested failure modes in clinical LLM evaluation.Source: §1 · pp. 1–2
  • The evaluation uses controlled input modifications without requiring model internals, making the stress tests applicable to closed commercial and open-weight biomedical models.The stated goal is safety screening, not a declaration of clinical deployability.Source: §1 · p. 2
  • The study contributes counterfactual-failure and demographic-stereotype taxonomies and treats counterfactual validity, accuracy, answer stability, EDD, and stereotype evidence as distinct screening dimensions.The counterfactual taxonomy includes insufficient, overly broad, circular, and formatting failures; the stereotype taxonomy includes severity labels.Source: §1 · p. 2

2 Related Work

Prior work shows that medical QA benchmarks measure final answers but leave other properties of clinical reasoning and fairness untested. This study situates perturbation audits alongside counterfactual, explanation-faithfulness, demographic-bias, and safety-benchmark research.

2.1 Clinical LLM Evaluation

  • MedQA-USMLE is a board-style clinical-reasoning benchmark, while prior models and prompting methods achieved strong medical QA performance [3].The present work complements answer-selection benchmarks by testing behavior under perturbations.Source: §2.1 · p. 2

2.2 Counterfactual Explanations and Faithfulness

  • Faithfulness work cautions that fluent explanations may not reflect model computations; this study audits counterfactual validity and demographic-prefix robustness, not causal-faithfulness scores.A valid counterfactual tests whether a realistic intervention would make another answer appropriate, distinct from whether facts support the current answer.Source: §2.2 · p. 3

2.3 Demographic Bias in Clinical AI

  • Demographic bias can appear in generated explanations even when the final answer stays unchanged, potentially shaping case interpretation and follow-up questions.Examples include added assumptions about adherence, lifestyle, disease prevalence, pain perception, or socioeconomic status.Source: §2.3 · p. 3

2.4 Clinical safety and fairness benchmark complementarity

  • Paired perturbations measure counterfactual validity, EDD, answer stability, and explanation-level stereotype evidence as a lightweight audit, not a substitute for clinician-validated evaluation.The cited benchmarks cover refusal, multi-turn health conversations, and stereotype-sensitive question answering.Source: §2.4 · p. 3

3 Methodology

The study evaluated six LLMs on 150 MedQA questions using paired counterfactual and demographic perturbation tests, with automated judgments of plausibility and stereotype evidence. The design provides a controlled audit, but its sample and unvalidated automated judgments limit the conclusions it supports.

3.1 Evaluation Design

  • The evaluation paired a counterfactual test that modified and re-answered cases with six demographic-prefix comparisons against a no-demographic baseline.The archive contained 900 CFV attempts and 5,400 demographic comparisons across six models and 150 questions.Source: §3.1 · pp. 3–4

3.2 Dataset and Models

  • The 150 selected questions came from the 1,273-question English MedQA source split and were evaluated across six models.The sample included 80 Step 1 and 70 Step 2 or 3 cases; it was not stratified by specialty, organ system, or difficulty.Source: §3.2 · pp. 4–5
  • Plausibility and stereotype assessments were automated without clinician annotations or adjudication, so the judgments are screening results rather than clinically validated assessments.The selected questions may not represent the broader clinical-question distribution or support specialty-specific conclusions.Source: §3.2 · pp. 4–5

3.3 Counterfactual Validity Test

  • CFV required a plausible, limited case change that shifted the answer to a prespecified alternative; invalid outputs were categorized by failure type.The categories were same direction, too dramatic, circular, and format failure; the recorded sex-change example was plausible but left the answer unchanged.Source: §3.3 · pp. 5–6

3.4 Demographic Robustness Test

  • The demographic test compared each case’s baseline with male/female and White/Black/Hispanic prefixes, making it a stress test rather than a clean causal intervention.The prefix procedure can interact with age or family context, and no age-sensitivity reruns were found.Source: §3.1 · p. 3, §3.4 · p. 6

FIELD RECORDED DEMOGRAPHIC ROBUSTNESS EXAMPLE

The recorded example shows answer instability and automated stereotype evidence under demographic prefixes, while the study distinguishes answer changes, explanation drift, and stereotype flags as separate measures.

FIELD RECORDED DEMOGRAPHIC ROBUSTNESS EXAMPLE

  • The fixed 45-year-old white-male prefix conflicts with the vignette’s original 25-year-old woman description, exposing a template interaction in this record.The authors characterize it as an audit failure, not a clinical diagnosis or human adjudication.Source: FIELD RECORDED DEMOGRAPHIC ROBUSTNESS… · pp. 7–8
  • The answer changed from A to B for male_white and female_black, and 2 of 6 prefixes were flagged for stereotype evidence.EDD was 0.2647 across the baseline and six prefixes for this question.Source: FIELD RECORDED DEMOGRAPHIC ROBUSTNESS… · p. 7

3.5 Metrics and Statistical Analysis

  • Answer change measures selected-option differences from baseline, EDD measures explanation drift, and stereotype flags count unsupported demographic assumptions.The metrics operate at different units: counterfactual attempts, baseline-prefix records, and judged explanation comparisons.Source: §3.5 · p. 8, FIELD RECORDED DEMOGRAPHIC ROBUSTNESS… · p. 8
  • EDD is one minus mean pairwise cosine similarity across embeddings of the baseline and six prefix explanations; higher values indicate greater explanation drift.The seven explanations are converted into normalized MiniLM L6 v2 embeddings.Source: §3.5 · p. 8
  • Automated plausibility and stereotype judges returned structured scores or flags, but the study used neither clinician adjudication nor inter-rater agreement.The parser records a metric only when required fields are present, and the archive preserves run and model metadata for auditing.Source: §3.5 · p. 8

4 Results

Across the results, model performance varied across accuracy, counterfactual validity, answer stability, explanation drift, and stereotype evidence. Counterfactual validity was uncommon, demographic prefixes often changed answers, and automated stereotype flags spanned severity and evidence categories.

4.1 Model-Level Overview

  • MedGemma 27B had the highest accuracy (87.1%) and CFV (63.3%), while OpenBioLLM 8B had the highest answer change rate (59.3%) and EDD (0.462).GLM 4.7 Flash had the highest automated stereotype flag rate (5.09 per question); no model ranked best across every dimension.Source: Table 5 · p. 9, §4.1 · p. 9

4.2 Counterfactual Validity Is Rare and Model-Dependent

  • 228 of 900 counterfactuals were valid (25.3%); validity varied substantially by model, while the difficulty-validity association was weak (Spearman r = 0.069, n = 900).For MedGemma, validity was 61.8%, 68.0%, and 65.2% for EASY, MEDIUM, and HARD questions, respectively.Source: Figure 2 · p. 9, §4.2 · p. 9

4.3 Failure Modes Reveal Boundary Miscalibration

  • 385 of 672 invalid attempts (57.3%) were same-direction failures, the largest category, where the modification did not change the clinical answer.Too-dramatic failures were 175 (26.0%), circular failures 104 (15.5%), and format failures 8 (1.2%).Source: Table 6 · p. 10, Figure 3 · p. 10, §4.3 · p. 10

4.4 Demographic Prefixes Destabilize Answers

  • 1,097 of 5,400 baseline-to-prefix comparisons changed answers (20.3%); OpenBioLLM was least stable, while MedGemma had the lowest average change rate but was not fully robust.These fixed-vignette stress-test sensitivities are not real-world prevalence estimates.Source: §4.4 · p. 10, Figure 4 · p. 11

4.5 Changed Answers Often Shift Toward More Aggressive Care

  • For MedGemma, 47 of 68 direction-classified changes (69.1%) were labeled more aggressive, while change directions varied across models.DeepSeek primarily produced lateral changes; OpenBioLLM mainly showed lateral or unknown changes.Source: Figure 5 · p. 11, §4.5 · p. 11

4.6 Stereotype Evidence Is Widespread

  • 3,128 automated stereotype evidence flags included 2,549 clinically misleading (81.5%), 555 socially inappropriate (17.7%), and 24 clinically dangerous (0.8%) classifications.The broad Other category contained 1,932 flags; among named categories, disease association was most frequent (514), and the classifications were not clinician-confirmed.Source: §4.6 · pp. 11–12, Figure 6 · p. 12, Table 7 · p. 12

5 Discussion

The discussion treats counterfactual validity and demographic robustness as distinct screening dimensions from accuracy, and recommends interpreting answer and explanation measures together. The evaluation supports safety screening, not clinical deployment claims.

5.1 Counterfactual Validity Captures a Different Ability from Answer Accuracy

  • Only 228 of 900 counterfactual attempts were valid (25.3%), showing that correct original answers do not guarantee reliable counterfactual performance.Same-direction failures were 57.3% of invalid attempts, while too-dramatic failures were 26.0%.Source: §5.1 · p. 12

5.2 Demographic Bias Appears in Both Answers and Explanations

  • Answer changes under unchanged clinical facts are review signals, not automatic evidence of unfairness, and EDD or stereotype flags likewise require collective interpretation.A high EDD does not itself prove bias, and stable explanations can still contain stereotypes.Source: §5.2 · p. 13

5.3 Model profiles across metrics

  • MedGemma had the strongest overall profile, but no model led every dimension; comparisons among only six models are descriptive rather than statistically conclusive.The discussion recommends reporting accuracy, CFV, answer stability, EDD, and stereotype evidence separately.Source: §5.3 · p. 13

5.4 Practical Evaluation Guidance

  • The evaluation requires repeated model generation, judging, and embedding comparisons, with generation and automated judging the main operational costs.The archive records token counts and latency but provides no aggregate cost or energy estimate.Source: §5.4 · p. 13

5.5 Implications for clinical AI governance

  • The evaluation can complement broader safety benchmarks but does not replace clinician-written rubrics, prospective validation, or real-world monitoring.The governance guidance routes answer-changing cases, high-EDD cases, and potentially serious stereotype flags for clinician review.Source: §5.5 · pp. 13–14

6 Limitations

The study’s findings are bounded by a reproducible but potentially unrepresentative question subset, a fixed demographic-prefix template, and unvalidated automated judges. The authors frame the evaluation as a lightweight audit rather than a population estimate or evidence of clinical readiness.

6 Limitations

  • The 150 MedQA USMLE questions were selected deterministically without specialty or difficulty stratification, so the subset may not represent specialties or clinical practice.The six-model comparison is a model profile analysis, not a population estimate.Source: §6 · p. 14
  • The fixed 45-year-old demographic prefix could interact with question details, and no age-variation sensitivity analysis isolates race or sex effects.Results should not be interpreted as isolating the effects of race or sex alone.Source: §6 · p. 14
  • Clinicians did not independently validate the automated judges, and complete model access or version manifests were unavailable.Source: §6 · p. 14
  • The authors position the framework as a lightweight, reproducible audit to identify cases needing deeper clinical evaluation.Source: §6 · p. 14

7 Conclusion

The evaluation found weaknesses across counterfactual validity, demographic answer stability, and stereotype evidence, showing that accuracy alone is insufficient to assess clinical safety. The authors recommend separate checks of answers and explanations, while limiting the framework’s claim to safety screening rather than clinical readiness.

7 Conclusion

  • Most counterfactual attempts were invalid, demographic changes altered answers in 20.3% of comparisons, and automated judging found frequent stereotype evidence.MedGemma performed best overall but still showed some demographic and explanatory instability.Source: §7 · p. 14
  • Evaluations should separately check answer correctness, stability under irrelevant demographic changes, plausible minimal counterfactuals, and unsupported demographic assumptions.Fluent responses may still fail one or more checks.Source: §7 · p. 14
  • Governance should record model and prompt versions, repeat evaluations after updates, retain raw outputs, and refer answer changes or dangerous flags for clinician review.Strong benchmark results do not make model outputs clinical evidence without human oversight.Source: §7 · p. 14
  • The framework supports safety screening but does not establish clinical readiness.Source: §7 · p. 14

Related papers

Loading 2609.32807v2…