PaperlayerView exact revision on arXiv

Source-linked AI summary

Trajectory-Derived Confidence for Reliable, Resource-Aware Clinical Text-to-SQL Agents

Mincheol Daniel Song, Joshua Ward, Jake Jung, Guang Cheng

arXiv:2610.04156v1cs.CLcs.AIcs.LG

TL;DR

Clinical text-to-SQL agents can return costly, untrustworthy answers and waste computation on trajectories that should be refused or discarded. Sentinel monitors an agent’s transcript and applies confidence-based stopping and withholding decisions. On the tested agents, it raises reliability at low stakes and can recognize when answers are too unreliable to deliver as stakes rise.

  • Problem

    Clinical agents may return wrong answers or answer questions the database cannot support, while their own reasoning and verbalized confidence are unreliable indicators of correctness.

    Source: §1 · pp. 1–2

  • Method

    Sentinel wraps an unmodified agent with trajectory-reading classifiers that estimate answerability and correctness to refuse, stop, or withhold at different stages.

    Source: §1 · p. 2

  • Results

    Sentinel raises the reliability of five tested text-to-SQL agents at low stakes and recognizes when their answers are not reliable enough to deliver as stakes rise.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Reliability monitoring can be added on top of an agent without modifying the agent itself.

    Source: §6 · p. 8

    Evaluation is limited to one benchmark and database, and the primary correctness label is an LLM judge that disagreed with human raters on 26% of its hardest cases.

    Source: §5 · p. 8

Abstract

from arXiv · show

LLM agents for clinical text-to-SQL applications reason autonomously over multiple steps but cannot assess whether their own reasoning or outputs can be trusted. In high leverage applications such as healthcare, this presents a critical risk where system mistakes can be costly. These reliability failures are also resource failures: an incorrect reasoning trajectory spends computation budget on outputs that must be discarded. We introduce Sentinel, a trajectory-derived, classifier-based confidence layer that analyzes an agent's reasoning, code and database outputs to decide at three points whether to stop: refusing unanswerable questions before the agent runs, halting doomed trajectories mid-run, and withholding untrustworthy answers at delivery. Here, utilizing Chow's rule, we optimize decisions under the EHRSQL shared task's Reliability Score, which penalizes incorrect answers given a utility weighting, and find on the benchmark EHRSQL that Sentinel raises this score from +0.08 to +0.24 when mistakes have a low utility weighting, well above the +0.03 earned by refusing every question, with delivered-answer accuracy rising from 54% to 69% as coverage falls from 84% to 55%. At higher stakes the agents we test rarely answer reliably enough to deliver, and Sentinel detects this on its own, abstaining to that same +0.03 where the unmonitored agent scores -3.02. The same stopping decisions cut computation: at low stakes, where the system still answers, Sentinel eliminates 13-28% of agent steps at little or no reliability cost.

1 Introduction

Autonomous clinical text-to-SQL agents can produce confident but untrustworthy answers, while their own confidence and reasoning are unreliable signals of correctness. Sentinel monitors externally grounded trajectory evidence to refuse, stop, or withhold answers, targeting both costly mistakes and wasted computation.

1 Introduction

  • Clinical mistakes can carry greater cost than declining, and unanswerable questions require abstention rather than an unsupported answer.Clinicians may not be positioned to audit the SQL behind a fluent answer.Source: §1 · p. 1
  • An incorrect trajectory wastes computation as well as risking a wrong answer; unanswerable questions can be refused at step zero, but vanilla GPT-4o averages 2.5 steps before failing on them.Fabricated answers average 3.35 steps, compared with 2.12 steps when GPT-4o correctly refuses.Source: §1 · p. 2
  • The agent’s trajectory exposes executed SQL, errors, empty results, and failed schema references that a separate monitor can use to estimate reliability without trusting its self-explanation or confidence.The paper notes that stated reasoning may not reflect the actual reasons behind an answer.Source: §1 · p. 2
  • Sentinel wraps an unmodified ReAct agent with answerability and correctness classifiers that can refuse before execution, stop mid-run, or withhold an unreliable answer at delivery.The system reduces delivered mistakes by abstaining rather than training the agent to be more correct.Source: §1 · p. 2
  • The paper frames clinical text-to-SQL reliability as a stakes-dependent decision problem in which an incorrect answer can cost more than abstaining.Source: §1 · p. 2

2 Related work

Prior work established abstention for unanswerable clinical text-to-SQL questions and selective prediction for model outputs. Sentinel applies confidence to an agent’s evolving trajectory, observing rather than steering its reasoning.

Text-to-SQL over EHRs and abstention

  • Sentinel derives confidence from the agent’s reasoning trajectory and uses it at multiple decision points, unlike prior EHRSQL and uncertainty-filtering approaches described here.Earlier EHRSQL work introduced unanswerable questions and the Reliability Score; cited systems split detection from SQL-error detection or use token log-probabilities.Source: §2 · p. 3

Selective prediction and agent confidence

  • Selective prediction studies the trade-off between coverage and selective risk, while Sentinel applies confidence both during execution and when deciding whether to deliver a completed answer.Source: §2 · p. 3
  • Unlike process-supervision methods that guide, rerank, or train reasoning, Sentinel observes the trajectory to predict delivered correctness and then stops or withholds under a reliability objective.Source: §2 · p. 3

3 Method

Sentinel treats clinical text-to-SQL reliability as a stakes-dependent decision over answerability, correctness, and compute. It scores each evolving agent transcript and uses calibrated confidence to exit trajectories or withhold answers.

3.1 Problem formulation

  • The method defines harm as answering an unanswerable question or delivering a wrong answer to an answerable one; an unanswerable question wastes all subsequent compute because abstention was available initially.Answerability is determined by whether any SQL query over the database can yield the correct answer.Source: §3.1 · pp. 3–4
  • The Reliability Score rewards correct answers and refusals, penalizes delivered mistakes according to stakes λ, and the paper evaluates λ ∈ {1, 4, 9}.Source: §3.1 · p. 4

3.2 Decisions

  • At any trajectory state Sentinel may EXIT with abstention, and at delivery it may WITHHOLD; either intervention avoids the penalty for a potentially costly mistake.Withholding is worth zero on an answerable question and a correct refusal is rewarded when it is unanswerable.Source: §3.2 · pp. 4–5
  • Chow’s rule delivers an answer only when its calibrated probability of a correct, answerable result meets the stakes threshold; higher stakes require a higher threshold.For λ values 1, 4, and 9, the stated thresholds are 0.5, 0.8, and 0.9; early-exit thresholds are instead selected empirically.Source: §3.2 · p. 5

3.3 Estimating confidence from trajectories

  • Two ModernBERT text-classifier heads read the question and the reasoning, SQL, and results in each transcript prefix to estimate answerability and eventual correctness.The answerability score at t = 0 uses the question alone, before the agent executes any step.Source: §3.3 · p. 5
  • Training expands each collected trajectory into labeled prefix snapshots, includes the final answer only in the terminal snapshot, and reports out-of-fold estimates grouped by question template.Source: §3.3 · p. 5

3.4 Inference procedure

  • Inference first exits when answerability falls below θ, with θ set to wrongly exit at most 0.5% of answerable calibration questions; resource-aware mode also exits when calibrated correctness falls below sλ.The early-stop threshold is selected to maximize Reliability Score, breaking ties in favor of saving more steps.Source: §3.4 · pp. 5–6
  • At delivery, Sentinel withholds answers whose calibrated joint correctness-and-answerability estimate is below τλ, and freezes thresholds and calibration using a calibration pool before one-time held-out evaluation.Source: §3.4 · p. 6

4 Experiments

Experiments compare Sentinel with unmonitored agents and component variants across stakes, then assess reliability and compute savings on held-out EHRSQL questions. At low stakes, Sentinel improves reliability over always abstaining while early stopping reduces agent work; at higher stakes, it largely abstains.

4.1 Setup

  • The study compares vanilla agents with the full layer, gate-only and Chow-only checks, and early EXIT at λ=1, 4, and 9.Evaluation uses held-out EHRSQL questions over MIMIC-III, five agent models, and semantic correctness labels based on execution-result equivalence.Source: §4.1 · p. 6

4.2 Reliability

  • At λ=1, Sentinel beats always-abstain for every agent (significantly for four of five); at λ=4 and 9 it matches abstention within noise.The table reports cross-agent reliability; GPT-4o delivers 55% of questions at 69% accuracy and gains +0.21 at λ=1.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 Resource savings

  • At λ=1, Early EXIT eliminates 13% of GPT-4o steps with no reliability loss; across agents it saves 13–28% at a cost of at most 0.01 RS(1).Table 2 reports agent steps, generated tokens, and wall-clock time never spent; the monitor’s two passes per step are not subtracted.Source: §4.3 · p. 7, Table 2 · p. 7

5 Discussion

The discussion finds that trajectory signals help distinguish correct from incorrect answers, but their usefulness depends on how legibly an agent fails. The evaluation and monitoring setup also has important scope limits.

5 Discussion

  • At delivery, GPT-4o correctness AUROC rises from 0.705 using the question alone to 0.762; one execution-evidence step lifts Qwen2.5-Coder-7B from 0.742 to 0.898.The calibrated estimate has held-out ECE 0.03–0.07 versus 0.25 raw, and only three held-out GPT-4o questions exceed 0.9 at delivery.Source: §5 · p. 7, Figure 3 · p. 8
  • Monitorability tracks failure legibility: the loudest failers score 0.920 and 0.921, while GPT-4o and Llama-3.1-8B score 0.762 and 0.707.The authors note that the signal may fade as agents improve and their failures become quieter.Source: §5 · p. 8
  • The evaluation uses one benchmark and database, one training seed, and no competing confidence method; the authors also flag disagreement on 26% of the judge’s hardest cases.The correctness labels come from an LLM judge, and a new agent requires its own trajectory collection for monitor training.Source: §5 · p. 8

6 Conclusion and Future Work

Sentinel adds reliability monitoring above an unmodified agent, while the conclusion identifies agent-directed use of confidence and cross-database testing as future work. The appendices document label, audit, and encoder decisions that qualify the evaluation.

6 Conclusion and Future Work

  • Sentinel raises reliability for five text-to-SQL agents at low stakes and abstains when their answers are unreliable at higher stakes, without modifying the agents.The authors conclude that reliability can be added on top of an agent.Source: §6 · p. 8
  • The monitor currently only abstains; proposed next steps include using it to guide fine-tuning or rerank resampled low-confidence trajectories, and testing transfer to eICU.The authors caution that training against a fixed monitor could teach an agent to appear confident rather than correct.Source: §6 · p. 8

A Leakage and artifact battery

  • The leakage audit reports that EHRSQL unanswerable questions use about 16 short topic-label templates, while answerable questions use about 100 full-question templates.Question-only TF–IDF reached 0.955 AUROC for answerability; the audit also reports shuffled-label controls near chance and no duplicate transcripts across CV folds.Source: A Leakage and artifact battery · p. 11

B Label decision record

  • The semantic correctness label uses an LLM judge, and a two-rater audit found disagreement on 26% of its hardest cases, with that error propagating to reported numbers.The raters agreed on 93 of 100 audited verdicts; among those 93, the judge matched the human verdict on 69.Source: B Label decision record · pp. 11–12

C Encoder decision record

  • ModernBERT-base replaced DistilBERT after meeting the adoption rule for both heads, while its flat per-step profiles showed the earlier late-step decline was a 512-token truncation artifact.The adoption rule required an AUROC improvement of at least 0.02; ModernBERT-base scored 0.889 / 0.986 for correctness / answerability.Source: C Encoder decision record · p. 12

D Confidence bins

Confidence scores become better aligned with delivered-answer correctness after isotonic calibration, but high-confidence answers are scarce. The accompanying operating-characteristics table compares answer coverage and delivered-answer accuracy across stakes levels.

D Confidence bins

  • After isotonic calibration, confidence-bin accuracy tracks stated confidence on the calibration pool and transfers to held-out dev_test, where only about 4% of answers score at least 0.8 under the head-alone calibration.On dev_test, the [0.5, 0.6], [0.7, 0.8], and [0.9, 1.0] bins are 51%, 77.7%, and 100% correct, respectively.Source: D Confidence bins · p. 12
  • The raw correctness scores are overconfident: 44% of delivered answers in the [0.9, 1.0] bin are correct only 71% of the time.The calibration pool also has 30% of delivered answers in the [0, 0.1] raw-score bin, correct 23% of the time.Source: D Confidence bins · p. 12

E Operating characteristics

  • Table 3 compares the fraction of questions answered and the accuracy of delivered answers by stakes level for GPT-4o-mini on dev_test.The caption says GPT-4o equivalents appear in the main text.Source: Table 3 · p. 12

F Always-abstain comparison and robustness

The section compares the full system against always abstaining and probes how its gains vary with benchmark composition, scoring, thresholds, and trajectory evidence. Results depend on the unanswerable-question share and on whether correctness uses semantic or strict scoring.

F Always-abstain comparison and robustness

  • Reweighting the held-out pool to 20% / 33% unanswerable questions leaves λ=1 gains over always abstaining of +0.16 / +0.12 for GPT-4o and +0.15 / +0.12 for GPT-4o-mini.At 33%, the gate's share of GPT-4o's λ=1 gain over the vanilla agent rises from 4% to about half.Source: F Always-abstain comparison and… · p. 12
  • The full system’s RS(λ) gain over always abstaining is reported in Table 4 with 95% template-cluster bootstrap intervals; significance is reported at λ=1 for four agents and is indistinguishable from zero elsewhere.The always-abstain policy scores +0.029 at every λ.Source: F Always-abstain comparison and… · p. 12, Table 4 · p. 13
  • Replacing τλ with the exact threshold (λ + ε)/(1 + λ) changes no full-system Table 1 entry by more than 0.01.Here ε is the calibrated estimate that a question is unanswerable.Source: F Always-abstain comparison and… · p. 13
  • Under strict execution-match scoring, λ=1 gains over always abstaining fall to +0.03 for GPT-4o and +0.05 for GPT-4o-mini, and both agents fall below that baseline at λ=4.The thresholds were calibrated to the semantic label, so the reported gain sizes depend on the judge.Source: F Always-abstain comparison and… · p. 13
  • The trajectory improves correctness AUROC by +0.04 to +0.18 over question-only baselines on every agent, while question-only input carries nearly all answerability signal.Table 5 reports held-out AUROC for each head on the question alone and at delivery.Source: F Always-abstain comparison and… · p. 13, Table 5 · p. 13

G Token-cost efficiency

Early stopping reduces token use as well as agent steps. The appendices specify the ReAct prompts and trajectory features used by the monitoring heads.

G Token-cost efficiency

  • Early stopping saves 16% / 47% / 47% of GPT-4o generated-token cost at λ=1 / 4 / 9, slightly exceeding step savings because eliminated steps skew late and long.The gate alone saves 1% of generated-token cost.Source: G Token-cost efficiency · p. 13

H Prompts

  • The agent prompt requires one action per turn, using exploratory SQL until ready to provide FINAL_SQL, and permits FINAL: null only when the database lacks the needed information.It warns that an empty result may indicate a mistaken query or time window rather than an unanswerable question.Source: H Prompts · p. 14

I Feature definitions

  • Trajectory features include execution error and empty-result rates, step count, and ungrounded SQL literals, with features computable from the first t steps and neutral defaults for unobserved quantities.The monitor also uses telemetry and schema checks that are not recoverable from the rendered transcript, while excluding purely linguistic signals it already reads.Source: I Feature definitions · pp. 14–15

J Feature analysis: the label-dependent screen

The feature screen’s value depends on the label: execution features add signal for semantic judgments, while text alone saturates the clean execution-match target. The hybrid was selected for delivery and its selection replicated across five agents.

J Feature analysis: the label-dependent screen

  • Under the semantic label, the hybrid improves on BERT-alone by +0.028 AUROC (CI [+0.011, +0.047]); failed-reference count survives screening, while removing execution costs −0.015.Uncertainty and schema families add little within the hybrid.Source: J Feature analysis: the… · p. 15
  • Under the strict label, no features survive screening: BERT-alone saturates the clean execution-match target, while observed execution events add signal for noisier semantic judgments.Source: J Feature analysis: the… · p. 15
  • On the calibration pool, the hybrid scores 0.780 vs. 0.746 for BERT-alone; selection replicates across five agents, with execution-family knockouts costing −0.009 to −0.038.The largest margin is for Llama-3.1-8B (0.748 vs. 0.691).Source: J Feature analysis: the… · p. 15
  • Table 6 reports per-agent feature survivors on the train pool, not held-out results; counts are highest for the two agents the head reads worst, but five agents cannot establish a trend.Source: Table 6 · p. 15

K Per-agent exhibits

Per-agent exhibits replicate the main-text analyses across five agents and describe system reliability and compute measures. One rendered transcript shows the answerability gate could refuse at step zero, avoiding seven steps before the agent’s own abstention.

K Per-agent exhibits

  • The main-text single-agent exhibits are replicated for all five agents, and released results include system ladders, risk–coverage sweeps, confidence bins, and per-step metrics.The per-agent tables describe system-component contributions, early-stop RS(λ) and skipped steps, and unanswerable-question step economics.Source: K Per-agent exhibits · p. 15, Table 7 · p. 16, Table 8 · p. 17
  • Four of five agents spend more mean steps fabricating than correctly refusing unanswerable questions; Phi-4-mini is the exception, with 1.75 vs. 1.70 steps.Its rare exploratory attempts reach the step cap, consistent with reflex fabrication rather than rationalization; its shallower transcripts are easiest to monitor.Source: Table 9 · p. 17, K Per-agent exhibits · p. 17
  • At λ=4, early stopping buys 64% of steps for 0.02 RS(4) on this agent; the stop threshold was tuned where calibration-pool coverage was thin.A timed serial rerun measures 2.2 seconds per step for this agent.Source: K Per-agent exhibits · p. 17

L Example rendered transcript

  • For an unanswerable question whose department scope is absent from the schema, the answerability estimate stays below 10^-3 from step zero; the gate would save seven steps before the agent’s own abstention.Source: L Example rendered transcript · pp. 18–19

M Reproducibility and cost

The reproducibility account specifies run choices, data coverage, software versions, and released artifacts, while distinguishing exact artifact reproduction from statistical retraining reproducibility.

M Reproducibility and cost

  • Seed 42 is the run of record; a three-seed GPT-4o-mini check found terminal-AUROC SD ±0.009, after which reruns were dropped as not worth the compute.Confidence intervals use 1,000 template-level cluster-bootstrap resamples, not seed reruns.Source: M Reproducibility and cost · p. 19
  • Collection gaps leave 2,099 held-out questions for GPT-4o and Qwen2.5-Coder-7B and 2,100 for GPT-4o-mini; each agent’s metrics use its completed questions.The pipeline pins transformers==4.56.2 and scikit-learn==1.6.1; the latter addresses version-dependent variation in hybrid fitting.Source: M Reproducibility and cost · p. 19
  • The GPT-4o-mini primary run is estimated at roughly $4 for collection and labeling, while a flagship GPT-4o run is estimated at ∼$110 from logged token telemetry.The estimates are not instrumented; exact artifacts are released, while retraining is expected to reproduce statistically.Source: M Reproducibility and cost · p. 19
Loading 2610.04156v1…