PaperlayerView exact revision on arXiv

Source-linked AI summary

Counterfactual Auditing of Bias in Open-Source Large Language Models for Clinical Triage

Manar Aljohani, Brandon Ho, Kenneth McKinley, Dennis Ren, Xuan Wang

arXiv:2610.01963v1cs.AIcs.CLcs.LG

TL;DR

Open-source LLMs may use demographic or contextual information in pediatric ED triage even when the clinical presentation is unchanged. This paper audits ten models with paired single-variable counterfactual vignettes and finds that sensitivity varies across models rather than reliably declining with size or medical pretraining; the fine-tuned Qwen2.5-7B is least sensitive overall.

  • Problem

    It is unclear how counterfactual bias varies across open-source model families, sizes, medical-domain models, and domain-adapted models used for clinical triage.

    Source: §I · p. 1

  • Method

    The study compares ten open-source LLMs on pediatric ESI prediction by changing one contextual variable in each vignette while holding the clinical presentation fixed.

    Source: §I · p. 2, D. Counterfactual Variable Injection · p. 3

  • Results

    5.27% any-shift rate and 0.0534 mean absolute shift for fine-tuned Qwen2.5-7B, versus 16.02% and 0.1706 for the base model; sensitivity did not consistently decline with model size or medical pretraining.

    Source: §I · p. 2, C. Proxy-Variable Injection Across All… · p. 5

  • Takeaways & Limitations

    Direction-aware analysis is needed because aggregate shift rates can conceal undertriage and overtriage patterns when variables with different clinical interpretations are averaged.

    Source: §V · p. 8

    The institutional evaluation set was filtered for substantive documentation and may not represent sparse or ambiguous real-world triage notes.

    Source: §V · p. 8

Abstract

from arXiv · show

Emergency department (ED) triage is a high-stakes prioritization task in which demographic, socioeconomic, and system-context information may improperly influence acuity assignment. Although open-source large language models (LLMs) are increasingly considered for local and privacy-preserving clinical decision support, it remains unclear how counterfactual bias varies across model families, sizes, medical-domain models, and domain-adapted models. We present a comparative counterfactual audit of ten open-source LLMs for pediatric Emergency Severity Index (ESI) prediction. Starting from real and handbook-style clinical vignettes, we construct paired counterfactual variants that change only one injected demographic, socioeconomic, healthcare-access, behavioral, social, or system-context variable while holding the clinical presentation fixed. Models include Qwen2.5-7B, Qwen2.5-14B-Instruct, a QLoRA fine-tuned Qwen2.5-7B, MedGemma variants, MedLLaMA2-7B, GPT-OSS-20B, and GPT-OSS-120B. We measure any counterfactual shift, undertriage, overtriage, shifts greater than one ESI level, mean shift, and mean absolute shift. Counterfactual sensitivity varied substantially and did not consistently decrease with larger model size or medical-domain pretraining. The fine-tuned Qwen2.5-7B showed the lowest overall sensitivity, with a 5.27% any-shift rate and mean absolute shift of 0.0534, versus 16.02% and 0.1706 for the base model. Several larger or medical-domain models showed more significant shifts. Stratified and correlation analyses further revealed clinically important directionality and shared failure patterns hidden by aggregate rates. These findings support counterfactual auditing as a lightweight, clinically interpretable framework for comparing fairness risks in open-source LLMs before clinical deployment.

I. INTRODUCTION

The paper audits ten open-source LLMs for pediatric ESI prediction using counterfactual changes to nonclinical context while holding clinical presentations fixed. It finds that larger size and medical-domain pretraining do not reliably reduce sensitivity, while the QLoRA fine-tuned Qwen2.5-7B showed the lowest sensitivity across most analyses.

I. INTRODUCTION

  • Disparities in ESI assignment have been reported across sex, race or ethnicity, language, and insurance, raising equity concerns in a widely used ED triage tool.Source: §I · p. 1
  • Clinical-text models may learn spurious associations between acuity and nonclinical descriptors that reflect historical bias rather than appropriate clinical evidence.Source: §I · p. 1
  • Prior studies examined narrower settings, while this audit compares ten open-source models on pediatric five-level ESI and a broader set of triage-relevant variables.Source: §I · pp. 1–2
  • The audit changes one contextual variable at a time while keeping the clinical vignette fixed, then distinguishes prediction changes toward undertriage from those toward overtriage.Source: §I · p. 2
  • Larger model size and medical-domain pretraining did not guarantee lower sensitivity; the QLoRA fine-tuned Qwen2.5-7B had the lowest sensitivity across most analyses.Source: §I · p. 2

II. RELATED WORK

The related work frames ESI disparities as an equity concern because triage decisions can shape subsequent care. It also motivates counterfactual fairness analysis and separate reporting of undertriage and overtriage.

A. Emergency Department Triage and Equity

  • ESI assignment varies across nurses, settings, and documentation quality, while reported disparities by sex, race, ethnicity, language, and insurance motivate auditing beyond overall accuracy.Early acuity assignments can influence waiting time, workup, pain treatment, disposition, and downstream outcomes.Source: A. Emergency Department Triage and… · p. 2

B. Algorithmic Bias in Healthcare

  • Healthcare proxies such as cost, insurance, prior ED use, and housing instability can encode unequal access, motivating counterfactual fairness analysis and separate undertriage and overtriage measures.The paper distinguishes these shift directions because their clinical consequences differ.Source: B. Algorithmic Bias in Healthcare · p. 2

C. LLM Bias Evaluation in Healthcare

Prior studies find that sociodemographic changes can alter LLM medical recommendations, while domain-adaptation methods make local fine-tuning feasible. This study compares paired baseline and counterfactual predictions on identical cases.

C. LLM Bias Evaluation in Healthcare

  • Prior LLM audits report medically unsupported recommendation differences and propose equity-focused evaluation, but this study focuses on open-source models for pediatric ESI and directional shifts.Source: C. LLM Bias Evaluation in Healthcare · p. 2

D. LLM Domain Adaptation

  • Open-source models can support local clinical deployment, and parameter-efficient methods such as LoRA and QLoRA make adaptation feasible with limited GPU resources.The study evaluates whether domain adaptation improves counterfactual stability as well as accuracy.Source: D. LLM Domain Adaptation · p. 2

A. Study Overview

  • Each vignette receives a baseline and a variant with one injected value; the clinical presentation and prompt remain the same, and predictions are compared within the same model and case.Source: A. Study Overview · pp. 2–3

B. Datasets and External Vignette Evaluation

The evaluation combines institutional pediatric ED cases with external handbook-style vignettes, compares ten open-source models, and constructs counterfactual inputs by adding one variable phrase to an unchanged clinical vignette.

B. Datasets and External Vignette Evaluation

  • The institutional evaluation set contains 352 filtered encounters, and the audit also uses 245 independent handbook-style vignettes as an external benchmark.The external analysis tests whether sensitivity persists across a noninstitutional source but does not replace validation on real EHR-derived cases.Source: B. Datasets and External Vignette… · p. 3

C. Models

  • The ten evaluated systems span Qwen, MedGemma, MedLLaMA2, and GPT-OSS families, including a QLoRA fine-tuned Qwen2.5-7B trained for ESI prediction.The fine-tuning data included expert-curated handbook vignettes and silver-standard clinical vignettes with nurse-assigned labels.Source: C. Models · p. 3, B. Datasets and External Vignette… · p. 3

D. Counterfactual Variable Injection

  • Each counterfactual input adds one variable-value phrase while leaving symptoms, vital signs, examination findings, mental status, and expected resource needs unchanged.Source: D. Counterfactual Variable Injection · p. 3

E. Prompt Template and Output Parsing

The audit compares each model’s baseline ESI prediction with predictions after adding one context phrase while keeping the vignette fixed. It summarizes the resulting shifts with directional and aggregate measures.

E. Prompt Template and Output Parsing

  • The injected variant differs from baseline only by an Additional Patient Context block, and deterministic decoding reduces random output variation.The prompt instructs models to use clinical severity and ignore non-clinical attributes.Source: E. Prompt Template and Output Parsing · pp. 3–4
  • Each vignette is evaluated without injected context, then with a single added variable phrase, and paired shifts are measured against that model’s baseline.The injected phrase may concern demographic, socioeconomic, healthcare-access, behavioral, social, or system context.Source: E. Prompt Template and Output Parsing · p. 3, Figure 1 · p. 4

F. Metrics

  • ΔESI is the injected-context ESI prediction minus the same case’s baseline prediction.A positive change means assignment to a lower-acuity level because ESI 1 is most urgent and ESI 5 least urgent.Source: F. Metrics · p. 4
  • A counterfactual shift is any nonzero ΔESI, with undertriage and overtriage reported separately as directional metrics.Positive ΔESI indicates lower acuity assignment.Source: F. Metrics · p. 4
  • Mean shift and mean absolute shift summarize average direction and magnitude; positive mean shift indicates undertriage tendency, while negative mean shift indicates overtriage tendency.Stratified analyses also report category-level shift, no-change, undertriage, and overtriage rates.Source: F. Metrics · p. 4
  • Stratified summaries report per-category shift rate, no-change rate, undertriage shift, and overtriage shift.These measures distinguish whether and in which direction an injected condition changes predictions.Source: F. Metrics · p. 4

G. Correlation Analysis of Shared Biased Outcomes

The analysis measures whether different injected category values tend to produce shared biased outcomes on the same cases. It scores value pairs encounter by encounter, then aggregates across values to variable pairs.

G. Correlation Analysis of Shared Biased Outcomes

  • For each pair of injected category values, the method scores +1 when both yield the same wrong ESI prediction, −1 when only one is wrong, and 0 when both preserve a correct or unchanged prediction.The example contrasts commercial insurance with private-vehicle arrival.Source: G. Correlation Analysis of Shared… · p. 4
  • Averaged pairwise scores indicate shared failure modes when positive, discordant or inconsistent errors when negative, and little systematic relationship near zero.Scores are averaged across encounters.Source: G. Correlation Analysis of Shared… · p. 4
  • Value-level pairwise associations are aggregated to variable-level scores by averaging all category-value pairs for each variable pair.Source: G. Correlation Analysis of Shared… · p. 4

H. Interpretation: Bias Versus Clinically Relevant Context

The paper distinguishes counterfactual sensitivity from potential bias, since some injected context may carry legitimate acuity information. It also excludes an unstable model from later bias analyses because output noise could confound sensitivity estimates.

H. Interpretation: Bias Versus Clinically Relevant Context

  • Demographic and socioeconomic identifiers should not change ESI unless they alter clinical severity or available history, whereas some contextual variables may be clinically meaningful.The paper reserves “potential bias” for shifts driven by non-clinical or clinically unjustified attributes and calls all shifts counterfactual sensitivity.Source: H. Interpretation: Bias Versus… · pp. 4–5
  • Undertriage shifts are generally more safety-critical; overtriage may be conservative escalation but can still create resource inequity.Source: H. Interpretation: Bias Versus… · p. 5

A. Repeated-Run Stability

  • MedGemma-4B-PT had 3.72% exact prediction consistency and a 1.52 mean pairwise absolute difference, so it was excluded from later bias analyses.The authors note instability could confound true counterfactual sensitivity with output noise; Table III covers five runs on n = 352.Source: A. Repeated-Run Stability · p. 5, Table III · p. 5

B. Aggregate Counterfactual Sensitivity in Non-Clinical Domains

Across proxy-variable injections, Fine-tuned Qwen2.5-7B had the lowest reported overall sensitivity, while larger size or medical-domain pretraining did not consistently correspond to lower sensitivity. Stratified results reveal differences in direction and category that aggregate rates can obscure.

B. Aggregate Counterfactual Sensitivity in Non-Clinical Domains

  • Fine-tuned Qwen2.5-7B preserved predictions in 95.37% of demographic and 94.89% of socioeconomic/living-condition injections, with no demographic significant shifts and 0.07% socioeconomic significant shifts.Source: B. Aggregate Counterfactual… · p. 5, Table IV · p. 5
  • Qwen2.5-7B had no-change rates of 80.80% for demographic and 83.17% for socioeconomic/living-condition variables, while MedGemma-27B-IT had no-change rates near 59% and significant-shift rates near 29%.Qwen2.5-14B-Instruct and GPT-OSS models showed larger sensitivity in some domains, particularly overtriage.Source: B. Aggregate Counterfactual… · p. 5, Table IV · p. 5

C. Proxy-Variable Injection Across All Domains

  • 5.27% any-shift rate and 0.0534 mean absolute shift were lowest for Fine-tuned Qwen2.5-7B, versus 16.02% and 0.1706, respectively, for Qwen2.5-7B.GPT-OSS-120B shifted in 40.75% of cases, and MedGemma-27B-IT shifted in 41.18% with mean absolute shift of 0.84.Source: C. Proxy-Variable Injection Across All… · p. 5, Table V · p. 6

D. Stratified Demographic and Socioeconomic Effects

  • 25.85% of non-English injections shifted Qwen2.5-7B predictions, versus 19.03% for English; overtriage was 17.05% versus 4.83%, while undertriage was 8.81% versus 14.20%.Fine-tuned Qwen2.5-7B showed no evidence of increased undertriage for non-English versus English cases.Source: D. Stratified Demographic and… · pp. 5–6
  • For Fine-tuned Qwen2.5-7B, commercial insurance had the largest socioeconomic shift rate at 8.52%, mostly undertriage shifts (7.67%); housing patterns also differed by model.Stable housing was associated with 18.18% undertriage for Qwen2.5-7B, while housing instability shifted toward overtriage (15.62%); these distinctions can be hidden by aggregate rates.Source: D. Stratified Demographic and… · p. 6

E. Healthcare-Access, Behavioral, Social, and System-Context Effects

Healthcare-access variables produced both undertriage and overtriage, while behavioral, social, and system-context variables had the highest fine-tuned sensitivity. The base model showed larger shifts for some of these contextual variables.

E. Healthcare-Access, Behavioral, Social, and System-Context Effects

  • 6.82% of Fine-tuned Qwen2.5-7B predictions shifted after recent hospitalization, with more overtriage (4.83%) than undertriage (1.99%); multiple recent ED visits produced 7.10% shifts, mostly undertriage (5.11%).Lack of primary-care access produced 3.41% undertriage and 1.42% overtriage, fewer shifts than established primary care.Source: E. Healthcare-Access, Behavioral,… · p. 6, Table VI · p. 6
  • 13.35% of Fine-tuned Qwen2.5-7B predictions shifted during an overnight/weekend high-volume surge, including 10.51% undertriage; EMS pre-notification produced a 10.51% shift rate.Private-vehicle arrival produced an 8.81% shift rate.Source: E. Healthcare-Access, Behavioral,… · p. 6, Table VI · p. 6
  • 34.66% of Qwen2.5-7B predictions shifted for domestic safety concerns, with 32.95% overtriage; EMS pre-notification produced 32.95% shifts.Domestic safety concerns produced no increase in shift rate for Fine-tuned Qwen2.5-7B relative to a safe home environment, but shifted direction from undertriage toward overtriage.Source: E. Healthcare-Access, Behavioral,… · p. 6, Table VI · p. 6

F. External Vignette Sensitivity

On external handbook-style vignettes, Fine-tuned Qwen2.5-7B remained among the most stable models. Correlation analyses found smaller and less structured shared biased-outcome associations for the fine-tuned model than for the base model.

F. External Vignette Sensitivity

  • The fine-tuned model preserved its baseline prediction in nearly 96% of external injected cases and produced nearly zero significant shifts across demographic and socioeconomic domains; MedGemma-27B-Text-IT showed high significant-shift rates.The fine-tuned model remained among the most stable models across these domains.Source: F. External Vignette Sensitivity · p. 7

G. Shared Biased-Outcome Correlation

  • Fine-tuned Qwen2.5-7B variable-level associations ranged from −0.0845 to 0.0249; the strongest positive pair was Arrival Mode–Arrival Time (0.0249), and the strongest negative pair was ED Crowding–Urban-Rural Context (−0.0845).The small magnitudes suggest limited systematic co-occurrence of biased outcomes.Source: G. Shared Biased-Outcome Correlation · p. 7
  • Qwen2.5-7B associations ranged from −0.2280 to 0.0369, including a −0.2280 association between Arrival Mode and Substance Use; value-level discordant pairs included Domestic Safety concerns with established PCP (−0.4290) and fully insured status (−0.4233).The base model showed stronger, more structured associations, while fine-tuning reduced cross-category clustering of biased outcomes.Source: G. Shared Biased-Outcome Correlation · p. 7
  • 0.0366 vs. 0.0849 was the average absolute off-diagonal correlation score for Fine-tuned Qwen2.5-7B versus Qwen2.5-7B, indicating fewer clustered failure patterns after domain adaptation.Fig. 2 heatmaps encode shared wrong outcomes as positive, discordant correctness as negative, and weak association near zero.Source: Figure 2 · p. 7, G. Shared Biased-Outcome Correlation · p. 7

H. Positive-Control Clinical Injection

The positive-control experiment found that Fine-tuned Qwen2.5-7B remained responsive to clinically relevant acuity information. The conclusion notes that the evaluation and perturbation design limit how broadly the results can be generalized.

H. Positive-Control Clinical Injection

  • 100.00% of active-seizure or cyanosis cases requiring bag-mask ventilation shifted toward higher acuity after injection; appropriate uptriage was also high for toxic appearance with poor perfusion (94.32%) and severe respiratory distress with hypoxia (92.61%).Severe pain produced 89.77% uptriage, while the resource-use control produced a 44.60% shift rate, consistent with affecting mainly lower-acuity ESI-4/5 cases.Source: Table VIII · p. 8, H. Positive-Control Clinical Injection · p. 8

V. CONCLUSION

  • The comparative audit found the fine-tuned model least sensitive in the clearest non-clinical bias domains, while several base, larger, and medical-domain models were more sensitive; direction-aware analysis exposed undertriage and overtriage patterns hidden by aggregate rates.Positive controls showed that lower non-clinical sensitivity was not simply due to ignoring added context.Source: §V · p. 8
  • The institutional evaluation set excluded sparse or ambiguous triage notes, nurse-assigned ESI labels were not perfect ground truth, injected phrases were templated, and most tests changed one variable at a time rather than testing intersections.The fine-tuned model may inherit label bias.Source: §V · p. 8
Loading 2610.01963v1…