PaperlayerView exact revision on arXiv

Source-linked AI summary

Auditing Escalation Reliability in Healthcare LLM Agents: A Perturbation-Controlled Benchmark of Workflow-State Failures

Zihao Zheng, Junyi Yao, Baichuan Li, Jiayu Long

arXiv:2610.21270v1cs.AIcs.CL

TL;DR

Healthcare agents must choose actions reliably when workflow evidence is missing, stale, conflicting, or late. This paper evaluates five open-weight configurations on synthetic perturbed episodes and tests confidence-based and workflow-aware routing. Workflow-state failures persist despite clean-condition performance, while a transparent controller improves in-benchmark cost-sensitive utility but is brittle to unseen lexical forms.

  • Problem

    Healthcare workflows change over time, so reliable action selection under missing, stale, conflicting, or late evidence is needed beyond accuracy on static inputs.

    Source: §I · p. 1

  • Method

    The study instantiates a workflow-aware protocol as 400 synthetic episodes with four actions, controlled variants, and a predeclared 5:1 missed-versus-unnecessary escalation cost.

    Source: §I · p. 1, A. Episode schema and task classes · p. 2, C. Statistics, splits, and cost model · p. 3

  • Results

    Workflow-conditioned action failures can coexist with strong clean performance; verbalized confidence does not reliably bridge model- and agent-level behavior, while the controller improves in-benchmark cost-sensitive utility over confidence-only routing.

    Source: §VII · p. 6

  • Takeaways & Limitations

    The results support explicit workflow-state evaluation and show that routing controls need their own generalization tests.

    Source: §VII · p. 6, §IX · p. 7

    Reference actions were authored by a single study author, so accuracy measures agreement with study policy rather than clinical correctness.

    Source: §VIII · p. 6

Abstract

from arXiv · show

Trustworthy biomedical AI requires reliable action selection when evidence is missing, stale, conflicting, or late, not only accurate answers on static inputs. We instantiate an episode-level evaluation protocol as 400 synthetic healthcare workflow episodes: 40 hand-authored semantic seeds across four task classes, two surface parameterizations, and five controlled variants, scored on ANSWER/ASK CLARIFICATION/ABSTAIN/ESCALATE under a pre-declared 5:1 missed-versus-unnecessary escalation cost. Reference actions are single-author study-policy labels, not clinician-adjudicated ground truth. Across five open-weight configurations (1.7B--9.7B), clean decision accuracy reaches 0.97, yet uncertainty-prompted models miss 32--97% of required escalations. The stronger 9.7B configuration narrows but does not remove the gap, falling from 0.97 clean accuracy to 0.78 under conflict. Verbalized confidence is inconsistent (ECE 0.10--0.54; failure AUROC 0.48--0.77), and a development-tuned confidence-only policy yields no utility gain. A transparent workflow-aware controller improves 5:1 utility for every model and reduces missed escalations to 0.12--0.24, at specificity 0.83--0.93. However, a 30-concept lexical stress test reduces each rule detector's recall from 1.00 on trigger phrases to 0.00 on unseen paraphrases, limiting the controller claim to this benchmark. Seed-clustered statistics, ablations, cross-task transfer, raw outputs, and manifests are packaged for audit. The results motivate explicit workflow-state evaluation while showing that apparently effective routing controls need their own generalization tests.

I. INTRODUCTION

Reliable healthcare agents must select actions as evidence changes, conflicts, or arrives late, rather than only answer accurately on static inputs. This paper experimentally instantiates an episode-level protocol to study workflow perturbations, verbalized confidence, and selective control.

I. INTRODUCTION

  • The benchmark extends an earlier episode-level protocol, which specified a schema and cost-sensitive escalation treatment but released no dataset or experiments.The earlier study identified a governed pilot with episodes, adjudication decisions, score distributions, and failure analyses as a next step.Source: §I · p. 1
  • The study tests whether missing, stale, conflicting, and late-arriving information degrade action reliability despite high clean-condition performance.It also asks whether verbalized confidence predicts failures and whether a workflow-aware controller improves cost-sensitive escalation decisions.Source: §I · p. 1
  • 400 episodes instantiate the published workflow-aware protocol across four task classes, paired clean–perturbed variants, and four allowed actions.The benchmark uses 40 hand-authored seeds, two surface parameterizations, and five variants.Source: §I · p. 1
  • The paper characterizes high clean accuracy alongside missed escalations, distinct model failure patterns, and better handling of salient stale summaries than contradictions.It also tests confidence calibration and failure discrimination, confidence-only deferral, controller ablations and transfer, and lexical out-of-vocabulary robustness.Source: §I · pp. 1–2
  • The benchmark measures decisions in simulated episodes and does not establish clinical outcomes, deployment readiness, or institutional generalization.Its claims remain within the stated boundary of the earlier protocol.Source: §I · p. 2

II. RELATED WORK

Prior work studies medical abstention, escalation, and multi-turn clinical agents, but does not jointly test controlled changes in evolving workflow state with cost-sensitive escalation scoring.

A. Abstention and escalation in medical LLMs

  • Static-question benchmarks assess medical abstention or triage, while multi-turn work examines clarification, abstention, or unsafe agreement drift across interaction rounds.The cited examples include Machcha et al., MedSafe-Dx, and MediQ.Source: A. Abstention and escalation in… · p. 2

B. Multi-turn and agentic clinical evaluation

  • Multi-turn clinical evaluations cover live EHR tasks, physician-rubric conversations, dialogue, longitudinal management, and sequential diagnosis.PatientAgentBench [17] is closest in deciding whether to gather information, act, or escalate.Source: B. Multi-turn and agentic clinical… · p. 2
  • PatientAgentBench [17] lacks controlled perturbation axes, an ABSTAIN action, paired clean–perturbed episodes, and cost-sensitive scoring.The benchmark here combines multi-turn workflow episodes, controlled paired state perturbations, and four actions with escalation as the primary cost-weighted outcome.Source: B. Multi-turn and agentic clinical… · p. 2

C. Selective prediction, calibration, and deferral

The paper applies established selective-prediction, calibration, uncertainty, and deferral methods to a workflow-conditioned setting, using a fixed episode schema and task-specific action semantics.

C. Selective prediction, calibration, and deferral

  • The measurement apparatus uses established selective-risk, calibration, uncertainty, learning-to-defer, cascade, clinical deferral, and conformal methods; workflow conditioning, not new selective-prediction theory, is the contribution.The cited machinery includes selective risk and risk–coverage analysis, and learning-to-defer with asymmetric costs,.Source: C. Selective prediction, calibration,… · p. 2

A. Episode schema and task classes

  • Each episode specifies an initial context, a state-changing event, allowed actions, a reference decision profile, and an author-provided rationale.Agents return one of four actions, confidence, and a short reason in a schema-enforced JSON object.Source: A. Episode schema and task classes · p. 2
  • The four task classes are documentation update, evidence retrieval, patient messaging, and triage handoff, each with class-specific action semantics.Ten seed scenarios were hand-authored for each class.Source: A. Episode schema and task classes · p. 2

B. Controlled variants

The benchmark uses deterministic paired variants to isolate workflow-state perturbations, with per-variant reference actions and seed-clustered evaluation under predeclared escalation costs.

B. Controlled variants

  • Each seed’s five variants are assembled deterministically so paired episodes differ only in the manipulated axis.The perturbations delete a critical fact, surface a superseded summary, append a contradiction or concerning-finding minimization, or deliver a red flag after the request.Source: B. Controlled variants · pp. 2–3
  • Reference actions vary by perturbation, and acceptable-action sets capture policy-permitted alternatives; labels are single-author operational targets, not clinician-adjudicated ground truth.The reference-policy manual records action definitions, variant logic, label inventory, and episode fields.Source: B. Controlled variants · p. 3

C. Statistics, splits, and cost model

  • 400 episodes comprise 40 seeds, two surface parameterizations, and five variants, with 100 episodes per task class and 130 reference ESCALATE actions.The five variants are clean, missing, stale, conflict, and late-risk.Source: B. Controlled variants · p. 2, C. Statistics, splits, and cost model · p. 3
  • Controller tuning uses 80 development episodes, while metrics use 320 test episodes, with uncertainty resampled across 32 test seeds.The primary escalation cost ratio is 5:1 for missed versus unnecessary escalation; a 10:1 missed-escalation cost is a robustness check.Source: C. Statistics, splits, and cost model · p. 3

IV. METHODS

The evaluation contrasts neutral action definitions with uncertainty prompting, then adds auditable text-based workflow-state detectors for conflict, missing information, and high-risk cues. A paraphrase diagnostic probes the detectors’ lexical dependence, not clinical validity.

A. Generation conditions

  • Uncertainty prompting adds a checklist for missing, stale, conflicting, and late-arriving risk information, emphasizes missed-escalation costs, and requests calibrated confidence.Both generation conditions use the same JSON output.Source: A. Generation conditions · p. 3

B. Workflow-state detectors

  • Three transparent regular-expression detectors derive conflict, missing-information, and high-risk indicators from episode text, after structural headers are stripped and without access to gold labels.In-benchmark precision/recall is 0.78/0.73 for C, 0.50/0.83 for M, and 0.75/0.83 for H.Source: B. Workflow-state detectors · p. 3
  • A 30-concept diagnostic pairs detector-triggering phrases with meaning-preserving unseen paraphrases; the authors characterize it as a mechanism audit, not clinical validation.The set has 10 pairs per detector.Source: B. Workflow-state detectors · p. 3

C. Workflow-aware selective controller

A post-hoc, training-free controller combines model uncertainty with workflow-state indicators and applies a two-threshold policy tuned on development data. The experimental setup specifies model runs, evaluation metrics, and audit records.

C. Workflow-aware selective controller

  • The controller computes a risk score from uncertainty and conflict, missing-information, and high-risk indicators, then overrides model actions using two thresholds.High risk triggers ESCALATE; an intermediate score changes ANSWER to ASK CLARIFICATION.Source: C. Workflow-aware selective controller · p. 3
  • Weights and thresholds are grid-searched on the development split to maximize 5:1 utility, with decision accuracy as a tiebreaker; the policy uses no training, gradients, or model internals.The described policy requires four multiplications and two comparisons per episode.Source: C. Workflow-aware selective controller · p. 3

V. EXPERIMENTAL SETUP

  • Five open-weight configurations are evaluated in two recorded runs, with deterministic decoding and 4,000 generations; the controller is computed post-hoc.The runs have a 100% parse rate and separate manifests for the first four models and Qwen3.5-9B.Source: §V · p. 3
  • Evaluation reports action accuracy, escalation metrics, cost-sensitive utility, confidence calibration and failure prediction, with seed-clustered bootstrap intervals and paired tests.The effective sample size is 32 independent test seeds; call outputs, latency, token counts, and run manifests are logged.Source: §V · p. 3
  • Tables I and II cover main test-split results and decision accuracy by variant, respectively; Table I defines its escalation and confidence metrics and notes controller rows lack calibrated probabilities.Table I covers 320 test episodes, including 100 escalation-required episodes.Source: Table I · p. 4, Table II · p. 4

VI. RESULTS

Clean-condition accuracy coexists with missed required escalations under uncertainty prompting, and perturbations expose substantial model-specific failures. Conflict and late-arriving risk cues distinguish models, while stale variants are often handled better than contradictions.

A. RQ1: Perturbations expose action-selection failures

  • Qwen3-1.7B and Qwen3-4B each reach 0.91 clean decision accuracy yet miss 97% and 46% of required escalations, while Qwen3.5-9B reaches 0.97 clean accuracy but falls to 0.69 on missing and 0.78 on conflict.Seven of 20 paired seed-level perturbation tests remain significant after Holm correction (p < 0.05).Source: A. RQ1: Perturbations expose… · p. 4
  • Conflict is the dominant failure axis for smaller answer-biased Qwen3 models, while adjacent actions often replace escalation on escalation-required episodes.Qwen3-1.7B answers 91% of these episodes; Llama-3.2-3B asks clarification 70% and escalates 10%; Qwen3.5-9B escalates 68% and asks 29%.Source: A. RQ1: Perturbations expose… · p. 4, Table III · p. 4
  • Late-risk sensitivity is 0.84, 0.78, and 0.75 for Qwen3.5-9B, Qwen3-4B, and Gemma-3-4B, compared with 0.16 and 0.05 for Llama-3.2-3B and Qwen3-1.7B.The figure omits the missing variant because it contains no escalation-required episodes by design.Source: A. RQ1: Perturbations expose… · p. 4, Figure 1 · p. 4

B. RQ2: Verbalized confidence is a weak control signal

The workflow-aware controller improves cost-sensitive routing in this benchmark, while verbalized confidence alone does not reliably control escalation. The findings are conditional on the tested workflow indicators and cost setting.

B. RQ2: Verbalized confidence is a weak control signal

C. RQ3: Workflow indicators improve in-benchmark cost-sensitive routing

  • At 5:1 costs, the controller improves utility for every model, with missed escalations at 0.12–0.24 and specificity at 0.83–0.93.Utility remains positive at 10:1, while every prompted baseline is negative; the gain is in-distribution and cost-conditional.Source: C. RQ3: Workflow indicators improve… · p. 5
  • Workflow-only rules match or exceed full-controller utility for every model, qualifying the added value of confidence in this experiment.Thresholds transferred across task classes retain nonnegative 5:1 utility, but evidence-retrieval missed rate reaches 0.50.Source: C. RQ3: Workflow indicators improve… · p. 5

D. Lexical OOV audit: the controller is not a safety guarantee

The detector stress test limits the controller’s lexical generalization, while uncertainty prompting shifts actions inconsistently and can reduce escalation. These findings bound the scope of the controller and prompt-level safety claims.

D. Lexical OOV audit: the controller is not a safety guarantee

  • Each detector’s recall fell from 1.00 on trigger phrases to 0.00 on unseen paraphrases, limiting controller gains to in-benchmark results.The study did not claim an end-to-end OOV score because model actions could change with the transcript.Source: D. Lexical OOV audit: the controller… · p. 5

E. Uncertainty prompting can backfire

  • Uncertainty prompting helped Qwen3-4B and Qwen3.5-9B but reduced Gemma-3-4B’s sensitivity from 0.64 to 0.52.Source: E. Uncertainty prompting can backfire · p. 5
  • For Gemma-3-4B, the safety checklist shifted actions from ESCALATE toward ASK CLARIFICATION, turning decisive escalations into hedged clarification requests.The study attributes the shift to prompt-level safety instructions redistributing probability among cautious actions rather than toward the correct action.Source: E. Uncertainty prompting can backfire · p. 5

F. Failure analysis

The failure examples show agents recognizing risk without consistently committing to escalation. The supplied figures frame confidence calibration, missed-escalation risk, and cost-sensitive utility as complementary views of this problem.

F. Failure analysis

  • When faced with a potassium-repletion request involving a newer 5.9 mmol/L result flagged “hemolyzed?” that contradicted the morning’s 3.1, Gemma-3-4B chose ABSTAIN at 0.95.Source: F. Failure analysis · p. 5
  • Fig. 3 plots stated confidence against decision correctness by bin; all models are overconfident, with varied calibration error.Source: Figure 3 · p. 6
  • Fig. 4 compares missed-escalation risk with the fraction of episodes handled by the agent; the controller has lower risk over most moderate-to-high coverage settings.Deferred episodes count as escalated.Source: Figure 4 · p. 6
  • Fig. 5 compares confidence-threshold and workflow-aware controller utility at 5:1 and 10:1 cost ratios.Source: Figure 5 · p. 6
  • Across examples, agents recognized risk but chose ABSTAIN, ASK CLARIFICATION, routine scheduling, or documentation rather than committing to the needed action.Examples include late-arriving thyrotoxicosis symptoms, unstable angina, and active suicidal ideation.Source: F. Failure analysis · p. 6

VII. DISCUSSION

Clean-condition strength can conceal workflow-state failures; confidence and safety prompts do not reliably address them, while workflow-aware routing improves in-benchmark utility. The authors limit clinical interpretation because labels are not clinician-validated and the controller lacks lexical generalization evidence.

VII. DISCUSSION

  • Clean-condition strength can conceal workflow-state failures; confidence is inconsistent, uncertainty prompts can backfire, and workflow-aware routing beats confidence-only control in-benchmark.The prompt effect was model-dependent in both magnitude and sign.Source: §VII · p. 6
  • The lexical stress test limits deployment claims: the auditable controller is brittle beyond the vocabulary used to build it.The conclusion reports failure on unseen lexical realizations.Source: §VII · p. 6, §IX · p. 7

VIII. LIMITATIONS

  • Reference actions are single-author study-policy labels, and 400 episodes are controlled variants of 40 semantic seeds, not 400 independent scenarios.The labels are not clinician-validated, and the seed pool and near-duplicate surfaces limit diversity.Source: §VIII · p. 6
  • The evaluation covers open-weight models only; 7–17% unnecessary escalations would burden clinicians, so governing clinical teams must set deployment costs.No proprietary frontier or clinically adapted model is included.Source: §VIII · p. 7

IX. CONCLUSION

  • Clinician re-annotation and independent external episodes remain prerequisites for clinical claims; the artifact packages episode data, raw outputs, manifests, tuned parameters, and code.The conclusion frames the results as policy-relative rather than clinical validation.Source: §IX · p. 7

Related papers

Loading 2610.21270v1…