PaperlayerView exact revision on arXiv

Source-linked AI summary

ALLMTS: Automated LLM-based Medical Triage System

Michał Remigiusz Janiszewski, Artur Stopa, Łukasz Lenkiewicz, Wojciech Achtelik, Bartosz Gonczarek

arXiv:2610.32469v1cs.AI

TL;DR

Telephone triage requires recommendations that follow clinical guidelines, but direct LLM decisions may not match required standards or provide an auditable link to them. ALLMTS makes an LLM execute guideline questions and applies a deterministic disposition rule. On 566 expert-validated scenarios, it achieves 0.95 accuracy versus 0.47 for a raw-LLM baseline.

  • Problem

    Direct LLM triage may not align with required standards or provide an auditable link from those standards to its recommendation, despite telephone triage’s asymmetric risks.

    Source: §1 · pp. 1–2

  • Method

    ALLMTS extracts intake information, selects a guideline, evaluates its triage questions, and derives disposition deterministically from the first positive answer.

    Source: §1 · p. 2

  • Results

    0.95 disposition accuracy versus 0.47 for the raw-LLM baseline on 566 expert-validated scenarios; ALLMTS reaches 0.98 on the curated 50-scenario subset.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The results support making the LLM execute an existing clinical standard rather than act as the clinical authority, with recommendations traceable to guideline questions and answers.

    Source: §7 · p. 10

    The closed-world assumption makes results dependent on sufficiently detailed intake notes because omitted findings are not distinguished from explicitly negative ones.

    Source: §6 · p. 9

Abstract

from arXiv · show

Telephone triage is a safety critical task: from a brief intake note, a nurse must decide whether a caller needs an ambulance, an emergency department, an appointment, or self care at home. Queried directly, large language models (LLMs) are unreliable: they over-escalate routine cases and cannot justify their recommendations against an auditable clinical standard. We present the Automated LLM-based Medical Triage System (ALLMTS), which grounds every disposition in the same commercially maintained telehealth guidelines nurses use. Rather than asking an LLM to triage, ALLMTS makes it execute the guideline: it selects the appropriate guideline and evaluates its triage assessment questions. The disposition follows deterministically from the first positive answer, using the same traversal rule a nurse follows. We evaluate ALLMTS on 566 synthetic, expert-validated scenarios spanning 14 dispositions and designed to test specific guideline questions. ALLMTS achieves 0.95 accuracy, compared with 0.47 for a raw-LLM baseline. On a curated 50-scenario subset released as an open benchmark, ALLMTS reaches 0.98 accuracy, on par with the 0.944 correct disposition rate that a five nurse expert panel recorded on those same scenarios. Critically, on that subset the baselines have over-triage rates of up to 0.50 (0.44 on the full dataset), whereas ALLMTS reduces the over-triage rate to at most 0.02 while maintaining low under-triage. These results support coupling LLM language understanding with executable clinical guidelines for triage from intake notes.

1 Introduction

Telephone triage requires nurses to apply ordered clinical guidelines amid asymmetric risks, while direct LLM recommendations may not follow required standards or provide an auditable rationale. ALLMTS instead executes guideline questions and derives dispositions deterministically; it outperforms a raw-LLM baseline on expert-validated scenarios.

1 Introduction

  • Telephone triage nurses select a guideline and assess ordered questions until a positive finding at the most acute level fixes the disposition, amid asymmetric risks of under- and over-triage.Under-triage can send a genuinely sick patient home; over-triage can crowd emergency departments with cases that could wait for an office visit.Source: §1 · p. 1
  • Direct LLM triage is unreliable because model knowledge may not match required standards, and free-form reasoning does not link recommendations audibly to those standards.Telephone triage guidelines specify ordered assessment questions and map answers to dispositions.Source: §1 · p. 2
  • ALLMTS selects a guideline, evaluates its triage assessment questions, and derives disposition from the first positive answer rather than generating it directly.Its pipeline combines retrieval-based selection, schema-locked question evaluation, deterministic traversal, and See More Appropriate Guideline re-routing.Source: §1 · p. 2
  • 0.95 disposition accuracy versus 0.47 for the raw LLM; ALLMTS also cuts over-triage from 0.44 to 0.02 across 566 expert-validated scenarios.The scenarios cover 14 disposition labels; ALLMTS reaches 0.98 accuracy on the curated 50-scenario open benchmark.Source: §1 · p. 2
  • The authors release a curated 50-scenario benchmark and compare ALLMTS with raw-LLM and web-search baselines across four label granularities and clinical error measures.The comparison also includes a guideline-context ablation and run-to-run stability.Source: §1 · p. 2

2 Related Works

Prior work grounds LLM decision support in retrieved evidence or explicit guidelines, while other triage systems generally use models to produce or refine dispositions. ALLMTS applies guideline execution to a broader maintained telehealth corpus and to existing intake notes.

2 Related Works

  • LLM medical applications use retrieval-augmented generation, adaptive questioning, uncertainty-aware retrieval, and tool-using agents, while evidence-synthesis systems show limits in evidence integration and claim-level citation support.The cited approaches include MEDIQ [25], UOT [13], and MedProbeBench [28].Source: §2 · pp. 2–3
  • Commercial virtual triage engines interview patients through curated knowledge bases, whereas this paper executes a published guideline against an existing intake note.A cited vignette comparison reported comparable care-referral accuracy and identical safety performance for one such engine and rules-based protocols.Source: §2 · p. 3
  • Guideline-grounded LLM systems include CPGPrompt, evaluated on three disease-specific guidelines, and oncology-focused MedGUIDE; ALLMTS operationalises 314 adult telehealth guidelines.The corpus spans a broad range of presenting complaints.Source: §2 · p. 3
  • TriageAgent, a KTAS-based system, C-PATH, and CLARITY rely on LLM parametric knowledge to produce or refine dispositions, unlike ALLMTS’s traceable guideline-question basis.Their mechanisms include multi-agent debate, role-play, fine-tuning, and a finite-state dialogue manager, respectively.Source: §2 · p. 3

3 Method

ALLMTS treats triage as execution of an existing guideline: it extracts information from an intake note, selects a guideline, evaluates its questions, and applies a fixed disposition rule. Its pipeline records the questions and answers that determine each recommendation.

3.1 Task setup

  • The system takes a free-text nurse intake note and returns a disposition from the selected guideline with the triage assessment question that settled the case.The note is organized as chief complaint, history, review of systems, and past medical history.Source: §3.1 · p. 3, §3 · p. 3

3.2 Pre-classification processing

  • Pre-classification extracts explicitly stated age and biological sex and identifies primary and secondary reasons for visit; absent demographics leave all guidelines in the candidate set.The primary reason is the most clinically urgent presenting complaint, defaulting to the chief complaint unless a clearly more urgent finding appears.Source: §3.2 · pp. 3–4

3 Method

  • Figure 1 distinguishes sequential core stages, pre-classification, concurrent question evaluation, and conditional guideline re-routing.Intermediate artifacts appear as arrow labels rather than separate boxes.Source: Figure 1 · p. 4

3.3 Guideline selection

  • Dense retrieval narrows the 314-guideline corpus to candidates, then a selection model chooses one using complaint scope, definition, category, gender, and patient fit.When multiple guidelines apply, the more specific anatomical guideline wins, and a life-threatening condition takes precedence.Source: §3.3 · p. 4

3.4 Triage assessment question evaluation

  • The pipeline evaluates every disposition level’s ordered triage assessment questions concurrently, with a fixed answer schema for each checklist.Answers use a closed-world assumption: unmentioned findings count as false; compound questions and exceptions are handled with boolean logic.Source: §3.4 · p. 4

3.5 Disposition determination and SMAG routing

  • Disposition follows the first positive question in descending acuity order; if none is positive, the pipeline recommends none.A positive question linked to an alternate guideline triggers re-evaluation there, subject to a depth limit and cycle prevention; the returned record includes the checklist and audit metadata.Source: §3.5 · pp. 4–5

4 Experimental setup

The evaluation uses a fixed disposition taxonomy and synthetic, expert-validated intake scenarios, with raw-LLM, web-search, and guideline-context comparisons. Scenario construction and evaluation choices are designed to test disposition prediction and guideline-question execution.

4.1 Models and inference settings

  • All methods use gpt-5.2 through the Azure OpenAI Responses API with low reasoning effort.Source: §4.1 · p. 5

4.2 Disposition label schema

  • Evaluation reports fine-grained dispositions and three separate grouped views: acuity, urgency, and urgency group.The 14 exercised dispositions span all five acuity bands, eight urgency ratings, and six urgency-group tiers; the views do not form one nested hierarchy.Source: §4.2 · p. 5

4.3 Dataset

  • The dataset contains 566 intake-note scenarios with reference dispositions, guidelines, primary reasons for visit, and first positive questions; five common classes account for most cases, while six rare classes have at most seven each.The curated 50-scenario subset with reference dispositions is released as an open benchmark.Source: §4.3 · p. 5

4.3.1 Clinical scenario development

  • LaGrange Medical Software’s clinical content team, authors of the guidelines, wrote and validated simulated cases through a structured 14-step process, each testing one question in one adult guideline.Cases vary patient-centered complaint wording and follow standardized clinical and demographic conventions.Source: §4.3.1 · p. 5

4.4 Baseline methods

  • The two baselines emit a disposition and an audit-only one-sentence reasoning string, and are scored like ALLMTS.Source: §4.4 · p. 6

4.4.1 Raw LLM

  • The raw-LLM baseline makes one structured call without guideline knowledge or retrieval, relying on pre-training under a closed-world assumption.Its prompt lists admissible labels, urgency definitions, and a clinical glossary.Source: §4.4.1 · p. 6

4.4.2 Web search

  • The web-search baseline must query for matching clinical triage guidelines and condition its disposition on the retrieved sources.Queries and source URLs are retained from the response trace for reproducibility.Source: §4.4.2 · p. 6

4.5 Guideline-context ablation

  • The guideline-context ablation selects a guideline, supplies its full text, and asks the model to choose a disposition directly, without explicit question answers or deterministic first-positive traversal.This isolates the execution stages from simply giving the model the selected guideline to read.Source: §4.5 · p. 6

5 Results

Across the 50-scenario subset and the full dataset, ALLMTS outperforms the baselines on disposition accuracy. The results also examine triage errors, run-to-run stability, prior methods, and external vignette sets.

5.1 ALLMTS versus baselines

  • 0.95 disposition accuracy on the full set versus 0.47 for Raw LLM and 0.50 for Web Search; ALLMTS reaches 0.98 accuracy on the 50-scenario set.The full set contains 566 scenarios; the curated subset contains 50.Source: Table 2 · p. 7, Table 1 · p. 7, §5.1 · p. 7
  • Raw LLM and Web Search over-triage 0.44 and 0.39 of full-set disposition cases, respectively.Baseline accuracy rises as labels become coarser; Raw LLM accuracy is 0.56 for ad-hoc urgency versus 0.49 post-hoc urgency.Source: §5.1 · pp. 7–8

5.2 Run-to-run stability

  • On ten runs over the same 50 scenarios, ALLMTS has κref = 0.955 versus 0.541 for Raw LLM.47 of 50 scenarios are correct on every ALLMTS run, versus 28 for Raw LLM; none is wrong on all ten ALLMTS runs, versus 12 for Raw LLM.Source: Table 3 · p. 8, §5.2 · p. 8

5.1 ALLMTS versus baselines

  • 0.90 ad-hoc disposition accuracy for the guideline-context ablation versus 0.95 for ALLMTS; structured traversal adds 0.02 to 0.05 in like-for-like cells.Holistic guideline reading loses the first-positive ordering; ad-hoc urgency-group over-triage is 0.09 for the ablation versus 0.02 for ALLMTS projected to that view.Source: §5.1 · p. 8

5.3 Comparison with prior methods

  • Both prior methods lean toward over-triage and the middle of the scale, while ALLMTS exhibits neither bias.The comparison reproduces the prior methods with gpt-5.2 and scores them in their native label spaces.Source: §5.3 · pp. 8–9, Table 4 · p. 9

5.4 Evaluation on related datasets

  • On external vignette subsets, ALLMTS reaches 0.41 accuracy for Levine et al. and 0.58 for Semigran et al.; experts judged the vignettes insufficiently detailed for guideline dispositions.The evaluated subsets included 43 of 48 Levine vignettes and 37 of 45 Semigran vignettes in scope.Source: §5.4 · p. 9

6 Limitations

ALLMTS is evaluated within the proprietary STCC guideline framework, with dataset and input assumptions that constrain the scope of its results. The paper also limits what can be reproduced and which component-level ablations are meaningful.

6 Limitations

  • The system and evaluation use STCC guidelines, creating acknowledged circularity and limiting reproducibility because the content is proprietary.The authors chose a dedicated, high-quality dataset despite the circularity.Source: §6 · p. 9
  • The evaluation excludes datasets with bedside vitals unavailable by phone, pediatric cases, and some vignettes experts judged too thin to assess without clarification.The guidelines cover adults only; external vignette sets are supplementary rather than the primary benchmark.Source: §6 · p. 9
  • The closed-world assumption requires sufficiently detailed intake notes because omitted findings are not distinguished from explicit negatives.An interactive version with an unknown state and clarification or nurse deferral is in development, but has not been evaluated.Source: §6 · p. 9
  • Per-stage ablations are omitted because removing a stage leaves the nurse-like traversal undefined, so it would evaluate a different system.The guideline-context ablation is reported; backbone, reasoning effort, and retriever configuration are the variations that preserve traversal.Source: §6 · p. 9

7 Conclusions

ALLMTS has an LLM execute an existing clinical standard, with dispositions determined by the first positive guideline question. It achieves higher accuracy than the raw-LLM baseline on the expert-validated scenarios, while the paper notes lower performance on external vignette sets.

7 Conclusions

  • 0.95 disposition accuracy for ALLMTS versus 0.47 for the raw-LLM baseline on 566 expert-validated scenarios; on the 50-scenario subset, accuracy is 0.98 versus the nurse panel’s 0.944 correct disposition rate.The scenarios span 14 disposition labels.Source: §7 · p. 10
  • ALLMTS makes the LLM execute an existing clinical standard, with each disposition determined by the first positive triage assessment question.The recommendation is traceable to a guideline question and answer.Source: §7 · p. 10
  • The selected guideline accounts for most of the observed improvement; explicit TAQ evaluation and SMAG re-routing together add gain on guideline-aligned scenarios.A holistic guideline-context ablation reaches 0.90 against ALLMTS at 0.95.Source: §7 · p. 10
  • ALLMTS performs lower on the two external vignette sets, indicating a need for further evaluation beyond guideline-aligned scenarios.Source: §7 · p. 10

A Label-view mapping

The paper defines independent label views for grouped evaluation and describes the scenario input, structured output, retrieval setup, and mappings used to compare against reproduced methods.

A Label-view mapping

  • Each fine-grained disposition maps independently to acuity, urgency, and urgency-group labels; “No positive TAQ found” remains unmapped.The table covers 14 dispositions in the full dataset and 10 in the subset.Source: Table 5 · p. 13

B Input format

  • Scenarios use nursing-intake headings, but the pipeline does not parse them and permits sections to be omitted or reordered; Figure 2 shows an example note.The example’s reference disposition is Go to ED Now.Source: B Input format · pp. 13–14, Figure 2 · p. 14

C Output schema

  • The structured prediction includes the disposition and settling TAQ, selected guideline, evaluated TAQ checklist, visit reasons, and retrieval and SMAG metadata.The output also includes an optional guideline rationale.Source: C Output schema · p. 14

D Retriever configuration

  • Experiments use title-based guideline embeddings; the scenario query is augmented with the primary visit reason, and the top 50 cosine-ranked guidelines go to selection.Similarity scores and retriever settings are stored for auditing.Source: D Retriever configuration · p. 14

E Output scale mappings

  • For the head-to-head comparison, the 10 dispositions in the 50-scenario set are projected through acuity and urgency into Levine’s four-level taxonomy and Gaber’s ESI.Levine is scored via urgency; Gaber via acuity, where ESI N aligns with acuity rating N.Source: E Output scale mappings · p. 15, Table 7 · p. 15

F Prior methods and external vignette sets

The section reports label distributions for reproduced methods and ALLMTS, external-vignette evaluation, and the shape of the full dataset’s labels and scenario lengths.

F Prior methods and external vignette sets

G Label distribution

  • The 566-scenario reference-label distribution is head-heavy, with a long thin tail in the fine-grained disposition space that coarsening collapses into populated bands.Source: Figure 4 · p. 16
  • Figure 5 presents scenario lengths in words with the median marked; the evaluation scenarios have a median length of 81 words.Source: Figure 5 · p. 16

H Run-to-run stability

The stability grids compare run-to-run predictions across methods. ALLMTS is correct on every run for most scenarios, while the Web Search baseline is the least consistent.

H Run-to-run stability

  • The grids mark missed reference acuity bands in red, group columns by reference acuity from most urgent first, and order scenarios by index.Titles report mean accuracy ± standard deviation, modal prediction consistency, and flip rate.Source: H Run-to-run stability · p. 17
  • 47 of 50 scenarios match the reference acuity band on all ten ALLMTS runs, and no scenario is wrong on every run.Figure 6 presents ALLMTS’s run-to-run stability grid.Source: Figure 6 · p. 17
  • 43 of 50 scenarios are correct on every guideline-context ablation run, while one is wrong on all ten; its reported consistency is 0.970 versus 0.974 for ALLMTS.The ablation is described as nearly as consistent as executing the guideline, but less accurate.Source: Figure 7 · p. 17
  • 0.26 is the Web Search flip rate, with predictions changing across runs for 13 of 50 scenarios.Web Search is the least consistent of the four methods.Source: Figure 9 · p. 18

I Confusion matrices

The confusion matrices show predictions concentrated near the diagonal for guideline-grounded methods, while both baselines spread toward more urgent predictions. The figures use different views for methods and evaluation sets.

I Confusion matrices

  • ALLMTS and the guideline-context ablation are shown on the disposition space, whereas both baselines are shown on the acuity view.Figures 10 and 11 use each method’s strongest view.Source: I Confusion matrices · p. 18
  • On the 50-scenario set, guideline-grounded methods cluster near the diagonal, while both baselines spread toward more urgent predictions, indicating over-triage tendency.Figure 10 shows the confusion matrices for this set.Source: Figure 10 · p. 18
  • On the full 566-scenario dataset, ALLMTS’s predictions stay on the diagonal across 14 dispositions, while both baselines again spread toward more urgent predictions.The trailing “No positive TAQ found” class collects cases with no positive TAQ.Source: Figure 11 · p. 19

Related papers

Loading 2610.32469v1…