PaperlayerView exact revision on arXiv

Source-linked AI summary

Sense and Sensitivity: Benchmarking LLM Clinical Triage Recommendations with Physician Experts

Abinitha Gourabathina, Haoran Zhang, Yuexing Hao, Walter Gerych, Marzyeh Ghassemi

arXiv:2609.38600v1cs.AI

TL;DR

Clinical chatbots are increasingly used, but clinical text varies in ways that may affect LLM triage reliability. This study benchmarks LLMs against physicians under gender and tone perturbations and finds more unnecessary care recommendations and greater model sensitivity to these changes. The results support evaluating clinical LLMs beyond aggregate accuracy, with robustness to realistic variation considered alongside expert behavior.

  • Problem

    Clinical LLMs can be brittle, while real clinical inputs vary widely in style, structure, and quality, making their response to realistic variation important to study before deployment.

    Source: §1 · p. 1

  • Method

    The authors build a physician-annotated triage benchmark with gender and tone perturbations and compare physician and LLM decisions across three triage tasks.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    LLMs are more likely than physicians to recommend unnecessary care and are more sensitive to clinically irrelevant gender and tone changes in patient presentation.

    Source: §1 · p. 2, §4 · p. 5

  • Takeaways & Limitations

    Aggregate accuracy alone is insufficient to assess deployment readiness; evaluation should also consider robustness to realistic variation and expert physician behavior.

    Source: §4 · p. 5

    The retrospective benchmark does not measure clinicians collaborating with LLMs in a real deployment environment.

    Source: Limitations · p. 5

Abstract

from arXiv · show

As large language models (LLMs) are increasingly used in clinical settings, it is critical to evaluate their reliability under realistic variation in clinical text. We study this question in clinical triage, comparing LLMs to practicing physicians under text perturbations that preserve the underlying clinical setting. We introduce a benchmark of over 6,000 clinical scenarios, 7,000 physician annotations, and 225,000 model responses. Using this benchmark, we make two key observations. First, LLMs are more likely than physicians to recommend unnecessary care at baseline, and this tendency increases under perturbed inputs. Further, we find that LLM recommendations are more sensitive to gender and tone perturbations than human recommendations. Together, these results demonstrate that LLMs can vary under clinically irrelevant textual changes, highlighting the need for deployment-oriented evaluations grounded in expert physician behavior.

1 Introduction

As patients and clinicians increasingly use chatbots for clinical recommendations and assistance, the paper examines whether triage recommendations remain reliable under realistic variation in clinical text. It compares LLMs with practicing physicians on care allocation and sensitivity to clinically irrelevant changes in presentation.

1 Introduction

  • Patients are increasingly turning to chatbots for clinical recommendations, and clinicians use them for tasks from administration to diagnostic assistance.The authors argue that studying responses to realistic textual variation is critical before deployment.Source: §1 · p. 1
  • Prior work describes clinical LLM brittleness, while real clinical inputs vary widely in style, structure, and quality.The paper notes an example in which changing a medical exam answer choice substantially reduced clinical LLM accuracy.Source: §1 · p. 1
  • The study asks whether LLM triage recommendations are robust to realistic text variation and whether their behavior matches practicing physicians.It focuses on care allocation when care may not be needed and sensitivity to clinically irrelevant patient-description changes.Source: §1 · p. 1
  • The authors introduce a physician-annotated benchmark of 6,232 scenarios, 7,356 physician annotations, and 226,191 LLM responses.They report that LLMs recommend unnecessary care more often than physicians and change recommendations more under gender and tone perturbations.Source: §1 · p. 2

2 Methodology

The study builds a clinical triage benchmark from four question-answering datasets, applies controlled gender and tone perturbations, and compares physician and model recommendations across three binary management decisions. Physician majority votes on baseline cases provide the reference labels.

2 Methodology

  • The benchmark draws on AskDocs, OncQA, Script Concordance Tests, and USMLE dermatology cases, including public and private cases.Tone perturbations are applied only to AskDocs and OncQA.Source: §2 · p. 2, Table 1 · p. 3
  • Each case can have up to five variants: an unperturbed baseline, gender swap or removal, and colorful or uncertain tone.GPT-4o-generated perturbations use few-shot prompts and curated examples intended to preserve clinical plausibility and semantic consistency.Source: §2 · p. 2
  • The three binary tasks assess whether patients should self-manage, receive an in-person visit, or receive a medical resource such as testing or referral.The decisions are evaluated independently.Source: §2 · p. 2
  • Ten MD physicians contributed annotations, and three independent physician reads on each baseline scenario determine its majority-vote gold label.Annotators do not see multiple variants of the same underlying scenario.Source: §2 · pp. 2–3
  • Five language models are evaluated, with each queried three times per scenario and task before majority-vote predictions are formed.The evaluated models are GPT-4o, DeepSeek-R1-32B, MedGemma-27B, Llama-3.3-70B, and Qwen2.5-32B.Source: §2 · p. 3

3 Results

Across baseline and perturbed clinical scenarios, LLMs often recommend more unnecessary care than physicians and show greater sensitivity to changes in patient presentation. The results also show that baseline accuracy and stability on physician-consensus cases do not guarantee stable recommendations under perturbations.

3 Results

  • Most LLMs perform comparably to clinicians on baseline cases; GPT-4o slightly exceeds clinicians on average, while Llama-3.3-70B leads on MANAGE.MedGemma-27B and DeepSeek-R1-32B underperform across all three tasks.Source: §3 · p. 3, Figure 2 · p. 3

3.1 RQ1: Do LLMs sensibly recommend treatment compared to human physicians?

  • Across the three tasks, physicians remain near the low-harm, low-overburden region, while most LLMs show higher overburden, often amplified under perturbations.The tradeoff visual compares harm and overburden rates; model behavior varies, including cases of higher harm or both high harm and overburden.Source: §3.1 · pp. 3–4, Figure 3 · p. 4

3.2 RQ2: Are LLMs more sensitive to realistic variations in clinical text than humans?

  • 10.11 percentage points: GPT-4o’s accuracy drop under colorful tone, despite 91.7% baseline accuracy versus clinicians’ 90.5%.Llama-3.3-70B and Qwen2.5-32B also degrade more under changes in symptom-description style than under explicit gender perturbations.Source: Figure 4 · p. 4, §3.2 · p. 4
  • DeepSeek-R1-32B loses more than 12 percentage points under every perturbation, while Qwen2.5-32B drops 15.91 points under colorful tone.MedGemma-27B has the lowest baseline accuracy but degrades by at most 3.62 points; thus, strong baseline accuracy does not ensure reliability under variation.Source: §3.2 · pp. 4–5

3.3 Sensitivity Persists When Physician Recommendations Are Stable

  • The sensitivity audit retains only cases where physician majority recommendations stay unchanged between baseline and perturbation.The authors frame this as a comparison of model changes with expert behavior, not proof that the rewrite is strictly clinically irrelevant.Source: §3.3 · p. 5
  • On clinician-stable cases, physicians’ performance changes by 0.8 percentage points on average, yet model degradation remains substantial.DeepSeek-R1-32B loses 12.5 and 19.5 points on VISIT and RESOURCE; Qwen2.5-32B loses 16.3 and 15.6 points, respectively.Source: §3.3 · p. 5, Table 2 · p. 5

4 Conclusion

The paper introduces a physician-annotated benchmark for evaluating clinical triage recommendations under realistic variation in patient text. LLMs can approach physician-level performance on unperturbed cases yet recommend unnecessary care more often and respond more sensitively to clinically irrelevant text changes.

4 Conclusion

  • The study introduces a physician-annotated perturbation benchmark for evaluating clinical triage recommendations under realistic variation in patient text.The benchmark is intended to compare model behavior under perturbations with physician behavior.Source: §4 · p. 5
  • Across three management tasks, LLMs can approach physician-level performance on unperturbed cases but more often recommend unnecessary care and show greater sensitivity to gender and tone changes.The authors conclude that aggregate accuracy alone is insufficient to assess deployment readiness; robustness to realistic variation is also an essential evaluation criterion.Source: §4 · p. 5

Limitations

The authors identify limits in both the benchmark’s scope and what its findings establish about clinical deployment.

Limitations

  • The retrospective benchmark does not measure clinician–LLM collaboration in real deployment, and covers only some realistic language variation using generated tone perturbations and binary triage decisions.Unchanged physician majority recommendations after a rewrite do not establish clinical equivalence; the authors also note that binary decisions simplify richer clinical reasoning and uncertainty.Source: Limitations · p. 5

A Potential Risks

  • Strong benchmark performance alone does not establish deployment readiness because real clinical use involves challenges not evaluated here.The authors state that clinical decision-making requires clinician oversight and clear safeguards.Source: A Potential Risks · p. 10

B Benchmark Curation Details

This appendix details benchmark curation, perturbation procedures, the model task prompt, and computational resources; Table 3 is identified as showing representative inputs from the source datasets.

B.1 Original Datasets

  • Table 3 presents representative baseline inputs from each source dataset used to construct the benchmark.The passage identifies the table’s purpose but does not enumerate the datasets or describe their inputs.Source: §B.1 · p. 10

B.2 Perturbation Procedure

  • GPT-4o generated up to four perturbations per case; gender rewrites covered all source components, while tone rewrites targeted AskDocs and OncQA patient text.For OncQA, tone changes leave the structured EHR context unchanged; answer options and gold labels remain unchanged across perturbations.Source: §B.2 · p. 10

B.2.1 Gender Case Filtering

  • Cases flagged for gender-specific clinical content across six term categories were excluded from both gender perturbations to avoid clinically incoherent vignettes.The categories include reproductive anatomy, gendered conditions, sex-specific cancers, hormonal markers, and reproductive health procedures.Source: §B.2.1 · p. 10

B.2.2 Gender Swap

  • Gender swap exchanges apparent gender indicators throughout the clinical context while keeping the remainder of the text verbatim.Dataset-specific formatting adaptations include Reddit shorthand and OncQA’s explicit Gender field.Source: §B.2.2 · p. 10

B.2.3 Gender Removal

  • Gender removal replaces gender markers with singular they/their/them and neutral alternatives, producing a gender-anonymous version of the case.The procedure also removes OncQA’s explicit gender field while preserving other medical content.Source: §B.2.3 · pp. 10–11

B.2.4 Colorful Tone / B.2.5 Uncertain Tone (uncertain)

  • Colorful tone adds mild symptom intensifiers, whereas uncertain tone inserts sparse hedges; both preserve the core question and avoid extreme language.For OncQA, these tone edits apply only to the patient message, not the EHR context.Source: §B.2.4 · pp. 11–12, §B.2.5 · p. 12

B.3 Task Prompt

  • Every model received the same prompt, with only the clinical context varying, and returned binary MANAGE, VISIT, and RESOURCE decisions plus a resource specification when applicable.DeepSeek-R1 used a concatenated single user turn, and its reasoning block was stripped before answer parsing.Source: §B.3 · p. 13

B.4 Compute Resources of Dataset Creation and Inference

  • Local inference for four models used approximately 250 A100 GPU-hours, with an estimated four-to-five-day runtime when runs were parallelized across GPU nodes.Perturbation rewrites were generated through GPT-4o via Azure OpenAI and required no local GPU compute.Source: §B.4 · p. 13

C.1 Supplemental Baseline Metrics

Baseline performance was near clinician level for some models, but balanced accuracy showed more variation across models.

C.1 Supplemental Baseline Metrics

  • Accuracy measures overall correctness, balanced accuracy averages sensitivity and specificity, and Cohen’s κ measures agreement beyond chance.Together, the metrics assess correctness, class balance, and alignment with clinician judgment.Source: §C.1 · pp. 13–14
  • GPT-4o reached 91.7% accuracy (κ = 0.72) versus clinicians’ 90.5% (κ = 0.69), while DeepSeek-R1-32B and MedGemma-27B scored 78.1% and 74.2%.Qwen2.5-32B scored 89.9% accuracy (κ = 0.69), and Llama-3.3-70B scored 89.7% (κ = 0.52).Source: §C.1 · p. 14
  • Balanced accuracy ranged from 0.664 for MedGemma-27B to 0.900 for GPT-4o, compared with clinician performance of 0.890.GPT-4o was the only model to exceed the clinician panel on this metric; most models remained above 0.70.Source: §C.1 · p. 14

C.2 Supplemental Perturbation Metrics

LLM accuracy shifts more than clinician accuracy under gender and tone rewrites, but the size and direction of changes vary by model and task. Balanced Accuracy reveals larger losses for some models and shows that apparent robustness can reflect a weak baseline.

C.2 Supplemental Perturbation Metrics

  • Clinician accuracy changed by at most 2.4 percentage points when gender cues changed, while tone-condition accuracy ranged from 78.5% to 90.7% versus 85.4%–94.0% at pooled baseline.Tone values cover AskDocs and OncQA only and are condition-level comparisons, not paired perturbation effects; RESOURCE accuracy showed the largest displayed difference, 92.1% →84.5% under colorful language.Source: §C.2 · p. 14, Table 5 · p. 15, Table 6 · p. 15
  • Qwen and DeepSeek had the largest average accuracy drops under gender rewrites (10.4 pp and 15.0 pp), while Llama and MedGemma were least affected (2.2 pp and 3.3 pp).Across models, tone differences exceeded gender drops for four of five models; MedGemma was the exception and had the weakest baseline accuracy, 70.1%–77.8%.Source: §C.2 · p. 14, Table 5 · p. 15, Table 6 · p. 15
  • Qwen’s average accuracy difference from baseline was 18.5 pp under tone, while DeepSeek’s was 15.4 pp; MedGemma’s was 2.4 pp, below its 3.3 pp gender drop.Qwen had the largest tone-associated difference: MANAGE accuracy fell from 84.4% at baseline to 65.6% under colorful language (18.8 pp). No perturbation dimension was uniformly most fragile across tasks.Source: §C.2 · p. 14, Table 5 · p. 15, Table 6 · p. 15
  • DeepSeek lost 0.194 in average Balanced Accuracy under gender rewrites and 0.298 on VISIT under gender removal (0.809 →0.511), while no model–condition cell fell below chance.GPT-4o and Qwen began near clinician-level Balanced Accuracy but averaged tone losses of 0.137 and 0.166; Llama and MedGemma began lower, and their smaller drops reflected weaker starting points.Source: §C.2 · pp. 14, 17, Table 8 · p. 16, Table 9 · p. 16

C.3 Ablation Across Datasets

Baseline performance and perturbation effects differ across dataset groups. Tone-related changes also vary between the two datasets where tone perturbations were tested.

C.3 Ablation Across Datasets

  • GPT-4o’s baseline macro accuracy (91.4%) matched the clinician panel (90.3%), while MedGemma and DeepSeek averaged 75.9% and 78.9%.AskDocs was hardest for every rater; clinician accuracy was 88.1%, and LLM averages ranged from 69.5% for MedGemma to 89.9% for GPT-4o.Source: §C.3 · p. 17
  • Under gender swap, GPT-4o fell 2.0 pp on AskDocs MANAGE but gained 8.0 pp on SCT, while DeepSeek lost 7.5–15.3 pp across corpora.DeepSeek lost more than 14 pp in three of four datasets, including standardized ones; its fragility was not limited to patient-facing corpora.Source: §C.3 · p. 17
  • Qwen lost 19.6 pp on OncQA versus 14.4 pp on AskDocs under colorful language, falling to 58.6% on OncQA MANAGE from an 82.8% baseline.Tone perturbations were available only for AskDocs and OncQA; the patterns indicate that robustness interacts with dataset difficulty and linguistic domain.Source: §C.3 · p. 17

C.4 Controlled Perturbations

After excluding rewrites that changed physician-majority judgments, LLM degradation remained, while paired comparisons showed little clinician movement. Physician agreement was high across variants, and one model showed a persistent female disadvantage on selected decisions.

C.4 Controlled Perturbations

  • On the stable subset, Qwen still lost 11.5 pp on MANAGE under colorful language and GPT-4o lost 9.4 pp on VISIT under uncertain tone.The subset retained cases where clinician-majority votes agreed between baseline and perturbed versions; paired baselines moved by at most 2.7 pp for clinicians, with a mean change of 0.9 pp. Residual LLM degradation was described as a conservative lower bound on surface-level sensitivity.Source: §C.4 · pp. 17–18

C.5 Inter-Annotator Agreement

  • Physician agreement was high and comparable across baseline and perturbed variants (α 0.81–0.87), and lowest for patient-authored AskDocs and OncQA text and the MANAGE decision.MANAGE marks the more contested boundary between self-care and intervention.Source: §C.5 · p. 18

C.6 Gender-Swap Direction

  • MedGemma-27B showed a consistent, significant female disadvantage on VISIT at baseline and after gender swap, and on RESOURCE after the swap.Most gender-gap estimates were not significant; the physician majority-vote gap was zero at baseline because those votes define the gold labels.Source: §C.6 · p. 18

C.7 Qualitative Examples

The examples show that tone rewrites can shift model recommendations in either direction while physician-majority judgments remain unchanged.

C.7 Qualitative Examples

  • For the stomach-pain case, all five models changed RESOURCE from NO at baseline to YES under the uncertain-tone rewrite, while physicians kept RESOURCE=NO.Physicians labeled both versions MANAGE=YES and VISIT=YES.Source: §C.7 · p. 19
  • For the swallowing-pain case, physicians kept VISIT=YES and RESOURCE=YES, but four models dropped both recommendations under the uncertain-tone rewrite.The models were GPT-4o, DeepSeek-R1-32B, MedGemma-27B, and Qwen2.5-32B.Source: §C.7 · p. 19

D Motivation for Perturbation Framework

The perturbation framework examines recommendation sensitivity to realistic changes in patient text, including gender and tone changes that preserve clinical content. Physician annotations were collected under a design intended to distribute scenario variants across raters.

D Motivation for Perturbation Framework

  • Tone perturbations change surface-level language without changing underlying clinical content, while gender perturbations target attributes that should not affect clinical decisions.The study frames these changes as realistic sources of variation and benchmarks recommendation sensitivity against physician performance.Source: D Motivation for Perturbation Framework · pp. 19, 22

E Physician Recruitment and Annotation Process

  • Annotators received task instructions and training examples drawn from baseline contexts excluded from the study’s actual scenarios.The study obtained annotator consent; compensation differed for US and non-US physicians.Source: E Physician Recruitment and Annotation… · p. 22
  • Each scenario variant received three independent physician reads, and no physician saw multiple variants of the same underlying case.Ten practicing MDs in generalist or acute-care specialties annotated approximately 732–736 scenarios each; they were not matched to cases by specialty.Source: E Physician Recruitment and Annotation… · p. 26
Loading 2609.38600v1…