Clinician use of language models diverges from how the models are evaluated

Main

• Evidence on LLMs in health care often comes from examination questions, while real-patient-care data were uncommon in published evaluations.A review of 519 evaluations through early 2024 found 5% used real patient-care data; a later review of 4,609 studies found 77.3% did not use real clinical data.Source: Main · p. 3
• Benchmarks predict deployment performance only to the extent that their items resemble real use, yet prior comparisons found only partial alignment.Prior work found real use diverged from tasks studied in the literature, and only half of a chart-connected assistant’s most common tasks appeared in MedHELM’s taxonomy.Source: Main · p. 3
• The study analyzes 127,833 queries from 6,342 clinicians and applies a clinician-validated framework to compare real use with 58 public benchmarks.The queries came from physicians, advanced practice providers, and nurses across 35 specialties during routine care.Source: Main · p. 3

RCQs from 6,342 clinicians across 35 specialties

• 127,833 conversations came from 6,342 clinicians across 35 specialties; 127,625 first-turn queries (99.8%) received valid annotations.A median clinician contributed four conversations, while the most active 10% generated 63.3% of conversations; confidence intervals accounted for clinician clustering.Source: RCQs from 6,342 clinicians across 35… · p. 4
• 44.1% of queries were instructions, 28.6% questions, and 27.2% keyword fragments; 52.9% used clinical abbreviations.The annotator matched a held-out clinician reference across 23 fields at 84.6%, compared with 83.0% agreement among clinicians.Source: RCQs from 6,342 clinicians across 35… · p. 4, Public benchmarks sample a different… · p. 9

Documentation and knowledge retrieval dominate clinical use, and diagnosis is rare

• Documentation and administration (36.2%) and knowledge retrieval (28.9%) together made up nearly two-thirds of use, while diagnosis accounted for 3.7%.Documentation requests were mostly patient-specific (68.9%) and asked for generated text (95.6%).Source: Documentation and knowledge retrieval… · pp. 4, 6, Figure 2 · p. 7

Clinical use differs by role and often crosses specialty boundaries

• Task mix differed by professional family (Cramér’s V = 0.14), and 43.9% of clinically anchored queries concerned conditions managed by another department.Residents had the largest shares of knowledge-retrieval (34.9%) and clinical-reasoning queries (19.7%); 56.0% of clinically anchored queries fell outside the asker’s division.Source: Clinical use differs by role and often… · p. 8

High-stakes RCQs are concentrated and partly identifiable from their text

• 13.2% of queries were rated high or critical risk, concentrated in treatment, procedural guidance, drug-information, and diagnosis tasks.Risk was highest for treatment and management (45.2%), procedural guidance (38.0%), drug information (36.2%), and diagnosis (31.3%); text-observable features distinguished high- or critical-risk queries with AUC 0.78, though capturing 90% would flag 60% of all queries.Source: High-stakes RCQs are concentrated and… · p. 8, Figure 3 · p. 10

More than a third of RCQs cannot be answered well as posed

• 36.1% of queries could not be answered well as posed, and 22.2% needed missing patient or local information.Another 12.6% required current evidence; 66.8% of high- or critical-risk queries could not be answered well as posed, compared with 31.4% of lower-risk queries.Source: More than a third of RCQs cannot be… · p. 8, Figure 4 · p. 11
• 61.2% of queries were safest to answer directly, while 26.8% called for clarification and 12.0% for retrieved authoritative sources.Among high- or critical-risk queries, 84.5% required something other than a direct answer.Source: More than a third of RCQs cannot be… · p. 9, Figure 4 · p. 11

Public benchmarks sample a different workload from real clinical use

• 31% median benchmark task-mix coverage (IQR 21–39%) fell below the 55% coverage of an even spread across ten task categories, and median documentation share was 0%.Suites intended to resemble clinical practice had median coverage of 35%, versus 40% for benchmarks used in frontier model reports.Source: Public benchmarks sample a different… · p. 9, Figure 5 · p. 12

Discussion

• Across 127,833 queries, documentation and knowledge retrieval made up nearly two-thirds of use, diagnosis 3.7%, and the average benchmark shared 31% of the real-use task mix.No single benchmark’s task distribution approached real use; examination benchmarks were dominated by diagnostic tasks.Source: Discussion · p. 13
• More than a third of queries could not be answered well as posed, and two-thirds of high- or critical-risk queries also could not.The paper cautions that benchmark scores, which usually assess correctness, do not capture this need for clarification or missing context.Source: Discussion · p. 13
• Documentation requests were common, often concerned identifiable patients, and could put protected health information in prompts or text that enters the medical record.The authors argue that documentation deserves particular attention in benchmark design.Source: Discussion · p. 13
• Clarification and grounding are likely safety priorities: missing patient or local information affected many queries, whereas escalation and abstention were rarely safest.The authors state that blanket refusal policies protect little and that achievable safety gains likely lie in clarification and grounding.Source: Discussion · pp. 13–14
• About one in eight queries was high or critical risk under model labeling, compared with about one in 27 under clinician consensus.Risk clustered in drug information, treatment and procedural guidance, and queries seeking to decide or verify something.Source: Discussion · p. 14
• The study covers one academic health system and assistant, did not observe other tools or assistant responses, and used public English text-only benchmarks.Clinician agreement was modest for potential harm, missing information and safest response, making their absolute prevalences less certain than task or group comparisons.Source: Discussion · p. 14

Study setting and data source

• The retrospective observational study covered a secure assistant available to staff at a large urban academic health system from 12 August 2025 to 3 April 2026, with a brief capture gap.Attending physicians, fellows, residents, APPs and RNs were included; podiatrists and dentists were excluded.Source: Study setting and data source · p. 15

Ethics, privacy and data release

• Investigators received de-identified label-level data, not query text or model outputs, and specialty labels were suppressed or pooled under a k = 10 threshold.The final corpus contained 127,833 conversations from 6,342 clinicians.Source: Ethics, privacy and data release · p. 15

Query extraction

• Each conversation’s first clinician turn was treated as the real clinical query; 127,625 queries received valid annotations after exclusions and retries.The valid annotation yield was 99.8%.Source: Query extraction · p. 15

Development of RCQ-Map

• RCQ-Map guidelines specified 24 forced-choice fields, then dropped two poorly applied properties and combined context fields, leaving 20 fields in six blocks.The guidelines included task and intent categories, boundary and tie-break rules, consistency rules, and worked examples.Source: Development of RCQ-Map · pp. 15–16

Assessment of the task and intent categories

• Clinicians applied task categories with Krippendorff’s α = 0.64 and intent categories with α = 0.45; categories also separated queries by potential harm and safest response.The authors identify this separation as necessary for using RCQ-Map to stratify evaluation.Source: Assessment of the task and intent… · p. 16

Correspondence of RCQ-Map to prior literature

• Task categories preserve established clinical question types while adding documentation, communication and administrative work associated with generative AI.Drug information remains separate from treatment and management; task categories are defined independently of profession.Source: Correspondence of RCQ-Map to prior… · p. 16
• Intent records the user’s motive behind a task, including verifying a plan, looking up a fact, making a decision, drafting text, or interpreting a result.Grouping 12 intents into five produced higher clinician agreement than single intents: α = 0.61 versus 0.45.Source: Correspondence of RCQ-Map to prior… · p. 17
• Risk measures potential harm if an incorrect answer is acted on, while answerability and the safest-response route distinguish direct answers, retrieval, clarification, escalation and abstention.The safest route is the least restrictive response that remains safe; clarification takes precedence over retrieval when patient information is missing.Source: Correspondence of RCQ-Map to prior… · pp. 17–18
• The framework also records literal query features, clinical ownership, AMA use case, patient specificity and missing patient, local or current-evidence context.Clinical departments are assigned through a precedence ladder so query patterns can be directed to services responsible for governance and safety review.Source: Correspondence of RCQ-Map to prior… · pp. 17–18

LLM annotation of the corpus

• GPT-5.6 sol annotated queries using the guidelines verbatim, each query as a single message, a strict JSON schema and validation with retries.The model received no chart data, user identity or conversation history.Source: LLM annotation of the corpus · p. 18

Clinician validation study

• Seven clinicians independently annotated a random 100-query cohort, with three annotations per query; the same cohort was annotated using the corpus model configuration.Clinicians used the full guidelines on a purpose-built platform and could not see other annotations or sender information.Source: Clinician validation study · p. 18

Five benchmarks examined in detail

• The five detailed benchmarks were purposively chosen to cover examination-style items and clinician-written questions or cases, with other MAST components excluded when held out or registration-restricted.The set included MedQA, HealthBench Professional, Real-POCQi, and two openly released MAST subsets.Source: Five benchmarks examined in detail · p. 18
• Benchmark items were rendered as models receive them, and distance from real use was measured by total variation distance between task, intent, harm, and safest-response distributions.The distance ranges from 0 for identical distributions to 1 for distributions with no overlap.Source: Five benchmarks examined in detail · p. 19
• The detailed benchmark annotations used the first user turn or benchmark prompt as the request unit, with the annotation configuration and rules unchanged from the corpus analysis.For HealthBench Professional, 115 conversations continued beyond the first user message.Source: Five benchmarks examined in detail · pp. 18–19

Clinical AI Benchmark Atlas

• 13 medical-LLM suites and frontier-model reports provided the Atlas sources; 23 documents covering 31 models reported results on 22 health benchmarks.The suite list was selected by the authors; model reports were identified from Epoch AI’s Notable AI Models database.Source: Clinical AI Benchmark Atlas · p. 19
• 58 of 175 identified benchmarks met the Atlas inclusion criteria after exclusions for access, language, modality, scope, and request format.The authors excluded 102 benchmarks before download and 15 after download.Source: Clinical AI Benchmark Atlas · p. 19
• Up to 200 items per benchmark were sampled uniformly, and RCQ-Map labeled 10,659 of 10,699 sampled items; agreement with real use was summarized as one minus task-distribution distance.The evenly spread distribution across ten task categories corresponds to 55% task-mix coverage.Source: Clinical AI Benchmark Atlas · pp. 19–20

Agreement statistics

• The held-out comparison used the other two clinicians’ shared label as the reference when they agreed, and the model was not treated as an additional rater in reliability calculations.Ordinal ratings were also compared using quadratic-weighted κ.Source: Agreement statistics · p. 20

Validation results

• Clinicians agreed less on needed context (α 0.25–0.32), safest response (0.40), and potential harm (α 0.36); unanimous harm grades occurred for only 12 of 100 queries.Grades were within one level for 61 queries.Source: Validation results · p. 20
• Grouping 12 intents into five raised agreement from α 0.45 to 0.61, while agreement on answerability was 0.57.Reliability was quantified across 23 judged fields using Krippendorff’s α and additional agreement measures.Source: Agreement statistics · p. 20, Validation results · p. 21
• The model matched the clinician reference in 84.6% of comparisons, versus 83.0% for held-out clinicians, and was significantly lower on none of 23 fields.The model’s agreement exceeded clinicians’ on 14 fields and significantly exceeded it on four.Source: Validation results · p. 21
• The model rated 24% of validation queries high or critical risk versus 6% by clinician majority, while calibrated corpus estimates put that share at 3.7%.The authors treated 13.2% as a cautious upper estimate of high-stakes use; 44.8% of RCQs were estimated to need more than a direct answer.Source: Validation results · p. 21

Statistical analysis of the corpus

• Prevalence intervals used clinician-resampled cluster bootstraps, with sensitivity analyses excluding the most active clinicians, non-clinical queries, or stratifying by professional group.Task mix by professional family was summarized with Cramér’s V; risk differences were tested with clinician-clustered logistic regression.Source: Statistical analysis of the corpus · p. 21
• Risk associations were modeled with clinician-clustered logistic regression, and text-only risk recognition was assessed by grouped five-fold cross-validation.The predictive model used 11 text-observable features and reported pooled out-of-fold AUC and the operating point capturing 90% of high- or critical-risk queries.Source: Statistical analysis of the corpus · p. 21
• Out-of-specialty status was assigned by mapping 31 discipline-defined specialties to their owning department or division, while excluding generalists and queries without an eligible clinical subject.Queries were out of department or division when their mapped clinical territory did not match the asker’s specialty or division.Source: Statistical analysis of the corpus · pp. 21–22
• Analyses used Python and listed statistical and machine-learning packages, with reporting following STROBE and TRIPOD-LLM where applicable.The reported packages included pandas, statsmodels, scikit-learn, and matplotlib.Source: Statistical analysis of the corpus · p. 22

Data availability

• Query text is unavailable for sharing because it may contain protected health information and is subject to institutional agreement.Label-level data without query text are available from the corresponding author on reasonable request, subject to institutional approval and a data use agreement.Source: Data availability · p. 22
• The Clinical AI Benchmark Atlas lists benchmark sources, versions, licenses, and labels for sampled items, with aggregate RCQ distributions included.CABA v1.0 is provided as Supplementary Data 1, archived and released as a Hugging Face dataset.Source: Data availability · p. 22
• Benchmark items are redistributed only where licenses permit; other items can be rebuilt from their original sources using repository scripts.Annotation guidelines are available in Supplementary Note 1 and the RCQ-Map repository.Source: Data availability · p. 22

• EKO reports equity in MarchAI and Artisight, spousal employment by Eikon Therapeutics, and consulting for Sofinnova Partners and Google.Source: Competing interests · p. 26
• KV reports consulting for ChartR Health; the remaining authors declare no competing interests.Source: Competing interests · p. 26

• EKO is supported by the National Cancer Institute’s Early-Stage Surgeon Scientist Program and the W.M. Keck Foundation.Source: Funding · p. 27
• The work was supported by IITP grants funded by the Republic of Korea’s Ministry of Science and ICT, including the KAIST Artificial Intelligence Graduate School Program and Global AI Frontier Lab.Source: Funding · p. 27

Extended Data

• Extended Data Fig. 1 shows clinician and conversation shares by professional family, conversation concentration, per-clinician counts, and usage across 35 specialties.Source: Figure 1 · p. 28
• Extended Data Fig. 2 compares headline estimates across all conversations, equal-clinician weighting, exclusions, and professional-family subsets, with gray bands showing the range across variants.Source: Figure 2 · p. 28
• Extended Data Fig. 3 reports queries by owning department and Department of Medicine division, while Fig. 4 presents specialty-level task mix, volume, patient specificity, risk, and response needs.Source: Figure 3 · p. 29, Figure 4 · p. 30
• Extended Data Fig. 5 compares specialty queries about conditions managed outside the asker’s department or division and identifies the departments most often queried from outside.Source: Figure 5 · p. 31
• Extended Data Fig. 6 covers inter-rater reliability and clinician-reference agreement across 23 judged fields, while Table 2 reports reliability and human–model concordance by field.Source: Figure 6 · p. 32, Table 2 · p. 36
• Extended Data Fig. 7 presents consensus-versus-model confusion matrices for safest response and answerability on 100 validation queries, with weighted κ values of 0.53 and 0.55, respectively.Source: Figure 7 · p. 33
• Extended Data Fig. 8 compares five public benchmarks with real queries across task, response, and risk characteristics, and reports total variation distance for task, intent, and safest response.Source: Figure 8 · p. 34
• Extended Data Table 1 lists RCQ-Map fields and allowed values; Table 3 covers logistic regression of high or critical potential harm, with models adding task category to surface features.Source: Table 1 · p. 35, Table 3 · p. 37, Extended Data · p. 37