PaperlayerView exact revision on arXiv

Source-linked AI summary

KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy

Xueting Fang, Zehui Li, Yang Yang, Camilla Giovino, Shubh K. Patel, Shailly Prajapati, Vallijah Subasri, Caihua Shan

arXiv:2609.38480v2cs.CLcs.AI

TL;DR

Most clinical LM benchmarks test diagnosis from complete case descriptions, not whether agents can gather relevant information and complete an assessment. KlinikeBench evaluates clinician-authored encounters with simulated patients, tools, and explicit criteria; across models, diagnostic accuracy can be high while strict task success remains below 30%.

  • Problem

    Most clinical LM benchmarks test diagnosis from complete case descriptions, leaving untested whether agents can obtain needed information and complete an appropriate assessment.

    Source: §1 · pp. 1–2

  • Method

    KlinikeBench evaluates 333 clinician-authored tasks in which agents question simulated patients, use clinical tools, and meet case-specific criteria within a fixed interaction budget.

    Source: §1 · pp. 2–3

  • Results

    Across 31 models, diagnostic accuracy reaches 90.7%, while the best strict pass@1 is 29.4%.

    Source: §1 · p. 3, §4 · p. 7

  • Takeaways & Limitations

    Component scores make incomplete history-taking and clinical actions visible beyond diagnostic accuracy alone.

    Source: §1 · p. 3, §5 · p. 10

Abstract

from arXiv · show

Most clinical benchmarks evaluate language models (LMs) on diagnosis using complete case descriptions. In clinical practice, however, patients present information in different ways, and clinicians must obtain relevant history and determine which examinations are needed before reaching a diagnosis. Diagnostic accuracy alone therefore cannot establish whether an agent gathered essential information or conducted an appropriate clinical assessment. Furthermore, existing benchmarks lack professional clinicians' verification. To address this gap, we introduce KlinikeBench, a benchmark of 333 clinician-authored tasks, each providing an isolated sandbox environment with a virtual patient, clinical tools, and task-specific success criteria. More than 35 clinicians contributed to case authoring and benchmark evaluation. In an empirical study, clinicians gave simulated dialogues higher mean quality ratings than reference conversations, which is adapted from real conversation. In each task, an LM has a fixed budget of turns to communicate with the patient, ask about relevant history, request examinations, follow action constraints, and record a final diagnosis. We score these steps separately as well as together. Across 31 models and seven model families, the best-performing models (e.g., GPT-6-astra and Claude Opus 5) succeed on less than 30% of tasks, even though their diagnosis accuracy reaches 90.7%. Some models benefit from talking with the patient; others diagnose well from a complete chart but perform much worse in conversation. Overall, KlinikeBench provides a testbed for evaluating the full clinical encounter and reveals a substantial gap between diagnostic accuracy and performance in interactive clinical assessment. All the code and data is available on https://zehui127.github.io/klinikebench/

1 Introduction

Most clinical LM benchmarks test diagnosis from complete case descriptions, leaving open whether agents can gather information and conduct an appropriate assessment. KlinikeBench evaluates the full encounter, and its results show that diagnostic accuracy can substantially exceed complete-task success.

1 Introduction

  • Existing interactive benchmarks include dialogue, tools, or records, but interaction alone does not show whether agents completed appropriate assessments.Agents can reach the same diagnosis despite different questions and actions, so encounter-level questions and actions require explicit evaluation.Source: §1 · pp. 1–2
  • KlinikeBench comprises 333 clinician-authored tasks with simulated patients, clinical tools, hidden scoring criteria, and a fixed interaction budget.The verifier evaluates diagnosis, required tool use, action constraints, and must-ask coverage.Source: §1 · p. 2, Figure 1 · p. 2
  • Thirty clinicians rated virtual-patient conversations higher on average than MTS-Dialog, a dataset of realistic patient-clinician conversations.The separate dialogue-quality study is presented as evidence supporting the quality of authored tasks.Source: §1 · p. 3, Figure 2 · p. 3
  • Across 31 models, diagnostic accuracy reaches 90.7%, while the best strict pass@1 is only 29.4%, showing a substantial gap between diagnosis and complete encounter-level success.Strict pass@1 requires correct diagnosis, complete clinical actions, and full must-ask coverage.Source: §1 · p. 3
  • The project releases the benchmark, training and evaluation code, 13,962 agent trajectories, model checkpoints, and a consultation demo.The release also includes SFT data and documents task splits and environment setup.Source: §1 · p. 3

2 KlinikeBench

KlinikeBench packages clinician-authored clinical encounters as tasks in which agents question a simulated patient, use tools, and record a diagnosis. Its construction, fixed-budget workflow, multidimensional scoring, and disease coverage define the benchmark.

2.1 Benchmark construction

  • Clinicians select disease-linked findings, author explicit absent findings and case details, and specify tool requirements, action constraints, must-ask topics, and accepted alternatives.Missing graph edges do not imply absent symptoms.Source: §2.1 · p. 4
  • Clinicians write personas and requirements, then review live simulations and revise tasks; only approved tasks enter the released corpus.Thirty clinicians assessed 50 conversations for progressive history disclosure, emotional appropriateness, and human-likeness.Source: §2.1 · p. 4

2 KlinikeBench

  • Each task combines a clinical briefing, patient simulator, initial state, callable tools, and hidden criteria; agents have 32 turns, including tool calls, before recording a diagnosis.The corpus provides 140 tool names for retrieval, state updates, and case-specific study findings.Source: §2 · p. 3, §2.2 · p. 4

2.2 Task format and evaluation

  • The main evaluation uses all 333 tasks, with an additional split of 283 training and 50 held-out test tasks for evaluating training algorithms.The held-out split is provided for use in rare cases.Source: §2.2 · p. 4
  • Strict pass@1 requires a correct diagnosis, all required tools, action constraints, and complete must-ask coverage; separate gates expose component performance.Must-ask coverage is the fraction of required topics explicitly raised, assessed by a fixed LLM judge.Source: §2.2 · p. 5

2.3 Dataset statistics

  • The corpus contains 304 target-diagnosis labels, with most labels appearing in only one task; cases are assigned to descriptive groups for visualization.The 226 cases labeled Internal Medicine include cardiovascular, digestive or liver, endocrine or metabolic, and respiratory cases.Source: §2.3 · p. 5
  • The median case specifies four must-ask topics and five required tools.Figure 3 presents disease groups spanning 17 clinical subcategories.Source: §2.3 · p. 5, Figure 3 · p. 5

3 Experimental setup

KlinikeBench is evaluated in Harbor with a shared task and tool interface, using multiple model families, patient simulators, judges, and interaction budgets. The setup also includes Qwen3-4B training experiments and clinician ratings of virtual-patient conversations.

3.1 Implementation details

  • Harbor gives each model the same tasks and tool interface; Figure 4 ranks the 15 most frequent tools, excluding final diagnosis.Each model has one rollout per model–task pair.Source: §3.1 · p. 5, Figure 4 · p. 5

3.2 Models

  • The evaluation covers 31 models from seven families, all using medium reasoning effort.Source: §3.2 · p. 6

3.3 Patient simulator and evaluators

  • The main evaluation uses GPT-5.4-mini for patient simulation and must-ask judging, with diagnosis, tool use, action constraints, and history judged against hidden criteria.Patient simulation uses temperature 0.7; history judging uses temperature 0.Source: §3.3 · p. 6

3.4 Training setup

  • Qwen3-4B training uses 283 training tasks and 50 held-out tasks, comparing diagnosis-only and four-gate GRPO rewards from the same SFT checkpoint.SFT uses 1,235 successful frontier-model trajectories; the GRPO runs share data order and hyperparameters.Source: §3.4 · p. 6

3.5 Robustness and validation setup

  • Judge and patient sensitivity are tested on 50 tasks across eight clinician models, with four judge and four patient-simulator choices.Source: §3.5 · p. 6
  • Interaction-budget sensitivity compares default, 45-step, 60-step, and uncapped conditions; 30 clinicians also rate 50 conversations across three dimensions.The clinician assessment includes 35 virtual-patient and 15 MTS-Dialog conversations, rated on a five-point scale.Source: §3.5 · p. 6

4 Results

Across 31 models, diagnostic accuracy substantially exceeds full-encounter success, and longer interaction budgets do not consistently improve pass rates. Judge and patient substitutions broadly preserve rankings, clinicians rate virtual-patient conversations favorably, and Qwen3-4B results show SFT gains while early RL gains remain uncertain.

4 Results

  • 90.7% diagnosis accuracy for Claude Opus 5 contrasts with 29.4% strict pass@1; across 31 models, mean accuracy is 73.6% versus 13.4% strict success and 27.9% required-tool completion.GPT-5.6-sol leads must-ask coverage at 90.8%, while Gemini-3.8-flash leads required-tool completion at 53.8%.Source: §4 · pp. 6–7, Table 2 · p. 7
  • 28/34/32/36% for GPT-6-astra, 34/36/40/38% for Claude Opus 5, and 20/26/22/26% for Claude Sonnet 5 show non-monotonic pass@1 across default, 45-, 60-step, and uncapped budgets.The ten-patient-turn limit is held fixed; the authors report that longer conversation does not consistently increase success, despite a positive influence overall.Source: §4 · p. 7
  • Interactive diagnosis improves GPT-5.4, GPT-5.4-mini, and GPT-4o-mini by 12.9, 8.7, and 2.4 points, but DeepSeek-v4-pro and DeepSeek-v4-flash decline by 34.5 and 46.2 points.The single-turn variant supplies the full intake snapshot; DeepSeek-v4-pro and -flash fall from 63.9% and 64.2% to 29.4% and 18.0% in conversation.Source: §4 · p. 8

4.1 Ablations and patient-quality assessment

  • Judge substitutions broadly preserve model rankings but can change the leader; patient substitutions also broadly preserve rankings, with top-rank changes potentially reflecting rollout variability.Judge re-scoring changes only must-ask coverage; patient substitutions require fresh encounters.Source: §4.1 · p. 8, Figure 5 · p. 8
  • Virtual-patient mean ratings exceed MTS-Dialog references on history disclosure (3.69 vs. 3.43), emotional appropriateness (3.50 vs. 3.31), and human-likeness (3.76 vs. 3.45).Ratings of four or five account for 53.2–64.9% of virtual-patient ratings; emotional appropriateness is lowest for both groups.Source: §4.1 · p. 9, Table 3 · p. 9

4.2 Supervised fine-tuning and reinforcement learning

  • SFT raises strict pass@1 from 0% to 8%, diagnosis accuracy from 26% to 39%, required-tool completion from 3% to 33%, and must-ask coverage from 39.7% to 77.5%.The strict pass@1 gain is 8 percentage points, with a 95% confidence interval from 2 to 15 percentage points.Source: §4.2 · p. 9
  • At step 30, diagnosis-only and multi-gate RL reach 44% and 42% diagnosis accuracy and 10% and 6% strict pass@1, versus 39% and 8% after SFT; confidence intervals include gains and losses.Both RL runs reach 32% required-tool completion, and the early results establish neither improved full-encounter success nor a better reward.Source: §4.2 · p. 9, Figure 6 · p. 9

5 Qualitative analysis and Discussions

KlinikeBench performance shows a gap between diagnostic accuracy and complete encounter success, while a held-out case illustrates that passing required gates does not guarantee a coherent, precise, or appropriately confident dialogue. The paper also describes a shared clinician annotation workspace for constructing and revising cases.

5 Qualitative analysis and Discussions

  • 90.7% diagnostic accuracy contrasts with strict success below 30% across 31 models; component scores expose incomplete history-taking and clinical actions.Patient and judge substitutions have limited effects on rankings, clinician ratings provide descriptive dialogue-quality evidence, Qwen3-4B fine-tuning improves assessment, and early RL gains remain uncertain.Source: §5 · p. 10
  • Both models pass the diagnosis and required-tool gates and cover all five must-ask topics, yet GPT-6-astra’s dialogue is focused while Qwen3-4B’s becomes garbled and incoherent.GPT-6-astra records a specific diagnosis with low expressed confidence; Qwen3-4B records a broad diagnosis with high expressed confidence when forced by the step cap.Source: §5 · p. 10, Figure 7 · p. 10

Appendix contents

  • The appendix contents list further material on case authoring, clinical tools, training, additional results, and related work.The contents also identify released corpus statistics and tool execution and scoring details.Source: Appendix contents · p. 15

A Clinician annotation platform

  • Clinicians use a shared workspace to author personas and requirements, inspect simulated encounters, and revise cases before approval; the LLM assists with drafting and formatting.The platform brings disease selection, task editing, live simulation, and trajectory review together.Source: A Clinician annotation platform · p. 16

A.1 Case authoring and LLM assistance

The authoring platform guides clinicians from a target disease and associated findings to a reviewed patient presentation. Clinicians also test and revise simulated encounters, while the appendix points to corpus statistics and clinical-tool details.

A.1 Case authoring and LLM assistance

  • Clinicians start from a target disease, select symptoms and differentiating findings, then use LLM refinement and clinician review to shape a draft case.Clinicians author the patient’s background, history, communication style, disclosure instructions, diagnosis, required actions, and must-ask topics; the LLM helps refine and structure the content.Source: §A.1 · p. 16, Figure 8 · p. 16

A.2 Live simulation, inspection, and revision

  • Clinicians inspect simulated dialogues and tool traces, annotate individual turns, revise personas or criteria, and repeat simulation before approving a case.These development annotations are distinct from the benchmark scores computed using the evaluation metrics.Source: §A.2 · pp. 16–17, Figure 9 · p. 17

A.3 Released corpus statistics

B Clinical tools

  • Tasks expose between 3 and 24 tools, with a mean of 10.7, from a catalogue of 140 tool names.The catalogue includes 21 built-in tools, 12 aliases, six state-update actions, and 101 named diagnostic studies.Source: B Clinical tools · p. 18

B.1 Interface and task-specific availability

The clinical-tool interface exposes task-specific commands and records their calls and responses in the encounter trace. Tool execution can return stored or mocked information, so tool coverage is distinct from clinical evidence coverage.

B.1 Interface and task-specific availability

  • Agents discover task-specific tools and their JSON schemas; accepted calls and responses enter the transcript, while diagnosis and differential tools are included by default.Forbidden or unexposed tool names are excluded, and record_diagnosis stores the final diagnosis, confidence, and reasoning before ending the encounter.Source: §B.1 · p. 18, Table 5 · p. 18

B.2 State updates and returned findings

  • Write tools update the simulated encounter, but generic test orders acknowledge an order without generating a new measurement.Named studies can return precomputed task-specific findings, while missing information may instead yield not_available, mocked results, or fixed defaults.Source: §B.2 · p. 19

B.3 Execution traces and scoring

  • The required-tools gate checks whether required tool names appear in the trace, not whether returned data are clinically useful.Argument-level checks apply task-specific action constraints; final diagnosis and must-ask coverage are scored separately.Source: §B.3 · p. 20
  • 3,550 exposed task–tool pairs were audited: 3,003 used built-in handlers, 37 aliases, 378 state actions, and 132 cached study entries.No audited pair reached the generic echo fallback or raised a handler exception, but the audit did not validate every argument or the clinical accuracy of every response.Source: §B.3 · p. 20

C Training details

The training study fine-tunes Qwen3-4B on selected frontier-model trajectories, then applies GRPO and compares diagnosis-only with multi-gate rewards. The fixed 50-task test set is also used for ablations.

C Training details

  • The fixed random test sample contains 50 tasks, including 42 whose diagnoses do not occur in the 283 training tasks.The same test set is used for ablations.Source: C Training details · p. 21
  • 1,249 trajectories passed all four gates, and 1,235 remained after removing sequences longer than 20,480 tokens for supervised fine-tuning.Trajectories came from 27 frontier-model configurations and covered 187 training tasks; fine-tuning ran for three epochs.Source: C Training details · p. 21
  • GRPO starts from the SFT checkpoint, samples eight tasks and eight rollouts per task at each step, and caps rollouts at 32 agent turns and 15 minutes.The reported results cover the first 30 training steps; Claude Opus 4.6 serves as patient simulator and must-ask judge during RL.Source: C Training details · p. 21
  • The reward comparison tests diagnosis-only against weighted multi-gate reward, whose fractional must-ask coverage gives credit for partial history-taking.The runs share SFT initialization, data order, and hyperparameters; the planned length is 315 steps, while reported results cover the first 30.Source: C Training details · p. 21, Table 6 · p. 21

D.1 Complete model results

The complete-results table covers 31 models on 333 tasks, with strict success requiring correct diagnosis, complete clinical actions, and full must-ask coverage. Cost estimates show that higher clinician inference cost does not guarantee higher success.

D.1 Complete model results

  • Strict pass@1 requires correct diagnosis, complete clinical actions, and full must-ask coverage across results for 31 models on 333 tasks.The table caption identifies bold as the overall best and underlining as the best within multi-model families.Source: Table 7 · p. 22
  • Model citations point to verified reports or model cards, including the GLM-5 technical report and Kimi-K2.6’s version-specific model card.The GLM-5 report is the one recommended by its official model card.Source: §D.1 · p. 22

D.2 Cost and performance

  • 29.4% success at an estimated $0.30 per task makes Claude Opus 5 the highest-success priced model, while Opus 5.5 reaches 28.8% at $0.20.GPT-6-astra costs $1.09 per task with 24.3% success; estimates measure clinician inference cost, not total benchmark cost.Source: §D.2 · p. 23, Figure 10 · p. 23

D.4 Full sensitivity results

The sensitivity materials describe a five-clinician step-cap ablation and patient and judge substitutions, while noting unavailable substitution scores. They also specify the held-out Qwen3-4B evaluation setup.

D.4 Full sensitivity results

  • Table 9 covers strict pass@1 under all patient and judge substitutions, but dashes mean scores are unavailable rather than zero.Rank-agreement statistics use conditions with all eight scores; the patient-ranking figure also includes conditions with partially unavailable scores.Source: §D.4 · p. 23, Table 9 · p. 24
  • The step-cap ablation compares five clinicians on 50 matched tasks while keeping the ten-patient-turn limit fixed.Changes are reported against the default run in percentage points with paired-bootstrap 95% intervals.Source: Table 8 · p. 24
  • Table 10 evaluates Qwen3-4B on 50 held-out tasks, pooling two rollouts per task for base and SFT and one per RL checkpoint.RL rows report step 30, and scores are percentages.Source: Table 10 · p. 25

D.5 Qwen3-4B held-out results

The held-out results report uncertainty around SFT and step-30 RL changes in strict success, while an appendix figure describes extended gate trajectories. Related work positions KlinikeBench among interactive, process-aware, and workflow benchmarks.

D.5 Qwen3-4B held-out results

  • SFT improves strict pass@1 by 8 percentage points (95% CI: 2 to 15), while step-30 diagnosis-only and multi-gate RL changes have intervals spanning zero.Their changes versus SFT are +2 points (−7 to +12) and −2 points (−11 to +7), respectively; neither comparison establishes improvement over SFT.Source: §D.5 · p. 25

D.6 Extended GRPO gate trajectories

  • Figure 11 plots training and held-out trajectories for diagnosis, required-tool completion, must-ask coverage, and strict outcome through GRPO step 50.It contrasts diagnosis-only and multi-gate rewards and marks zero-shot and SFT held-out baselines.Source: Figure 11 · p. 26

E Related work

  • KlinikeBench links diagnosis to elicited history and actions, pairing clinician-authored requirements with executable tools and parameterized action constraints.Related work contrasts it with benchmarks focused on medical knowledge, interactive dialogue, consultation-cycle information acquisition, and clinical software workflows.Source: E Related work · p. 27
  • KlinikeBench focuses on gathering evidence and completing questions, examinations, and decisions within one assessment under an interaction budget, unlike work centered on longitudinal records or multi-session dialogue.It also tests how judge and patient substitutions affect scores and rankings.Source: E Related work · p. 27

Related papers

Loading 2609.38480v2…