Source-linked AI summary
DIASENTINEL: An Auditable Multi-Agent System for Guideline-Grounded Diabetes Risk Screening
Yung Wei Shueh, Zhi-Jie Chen, Chia-Hsuan Hsu, Hsin-Ling Hsu, Donghua Zhang, Chenwei Wu, Jun-En Ding, Tongze Zhang, Shihao Yang, Pengfei Hu, Fang-Ming Hung, Feng Liu
TL;DR
LLM-based clinical decision support for T2DM screening and reporting remains vulnerable to calibration, unsupported-content, and citation problems. DIASENTINEL addresses these issues with an on-premise multi-agent pipeline combining calibrated risk prediction, ADA-guideline retrieval, deterministic checks, and LLM entailment verification. The system demonstrates auditable screening and reporting, while its verification evaluation remains limited to synthetic manually constructed cases.
Problem
LLM-based clinical applications risk poorly calibrated probabilities, unsupported content, and citation errors, limiting reliable guideline-grounded decision support.
Method
DIASENTINEL combines calibrated Qwen2.5-14B risk prediction, ADA retrieval with RRF, deterministic clinical processing, and hybrid verification in an on-premise multi-agent system.
Results
72% sensitivity and 62% specificity were obtained at the high-risk threshold, while tier-level predicted and observed risks closely matched across high, medium, and low groups.
Takeaways & Limitations
DIASENTINEL demonstrates a practical framework for auditable, guideline-grounded, privacy-preserving LLM-assisted T2DM screening and clinical reporting.
Takeaways & Limitations
Verification was evaluated only on synthetic, manually constructed cases and therefore does not establish real-world clinical reliability.
Abstract
from arXiv · showhide
Large language models (LLMs) offer promising clinical decision support but remain vulnerable to hallucinated facts, unsupported recommendations, and citation errors. We present DIASENTINEL, a fully on-premise multi-agent system for one-year type 2 diabetes mellitus (T2DM) risk screening and guideline-grounded report generation from electronic health records (EHRs). The system integrates calibrated risk prediction, deterministic clinical signal extraction, Reciprocal Rank Fusion over American Diabetes Association (ADA) guidelines, and a hybrid verification layer combining rule-based checks with LLM entailment. The demonstration provides a real-time batch-screening dashboard and an interactive patient report interface with cited recommendations, verification results, and raw EHR comparison. DIASENTINEL demonstrates a practical framework for reliable, auditable, and privacy-preserving LLM-based clinical decision support.
1 Introduction
DIASENTINEL addresses clinical risks of LLM-based diabetes decision support by decomposing one-year T2DM screening and reporting into auditable, evidence-grounded subtasks. It combines calibrated risk prediction, ADA-guideline retrieval, and hybrid verification in a deployable, on-premise system.
- Motivation: T2DM has a modifiable high-risk period, creating a need for near-term risk indicators alongside routine clinical documentation.Relevant documentation includes HbA1c, fasting plasma glucose, BMI, blood pressure, and lipid profiles.
- Motivation: LLM-based clinical applications face risks from poorly calibrated probabilities, unsupported content, and citation errors.
- System contribution: DIASENTINEL decomposes the diagnostic workflow into auditable, evidence-grounded subtasks for deployable clinical decision support.The modular design is intended to improve transparency, traceability, and component-level retrieval and reasoning.
- System contribution: Its contributions include a calibrated Qwen2.5-14B risk predictor, an ADA retrieval pipeline using dense retrieval, reranking, and RRF, and hybrid verification.
- System contribution: DIASENTINEL supports batch screening of one-year new-onset T2DM risk and on-demand individualized reports while operating fully on premises.LLMs are used for risk estimation, report generation, and entailment verification; other components use deterministic or retrieval-only methods.
2 DIASENTINEL System Architecture
DIASENTINEL is an on-premise LangGraph pipeline that separates population screening from patient-specific reporting. It uses LLMs selectively while keeping extraction, numerical comparison, EHR retrieval, and evidence retrieval deterministic or retrieval-only, with auditable verification before display.
- Operating modes: The architecture provides batch risk screening and detailed report generation as two distinct operating modes.Batch screening assigns calibrated probabilities and risk tiers; report generation combines longitudinal EHR retrieval, signal extraction, guideline retrieval, synthesis, and verification.
- Component design: LLMs are restricted to prediction, report synthesis, and semantic verification, while EHR retrieval, factual extraction, numerical comparison, and evidence retrieval remain deterministic or retrieval-only.
- Clinical tracking: The system retrieves longitudinal SOAP records, laboratory results, and nursing assessments within a one-year window, then ranks structured clinical signals by priority.Signals include the variable, observed value, severity, and rationale; trend tracking compares earliest and most recent values over 365 days.
- Guideline grounding: The Evidence Agent grounds report generation in ADA Standards of Care in Diabetes through dense retrieval, cross-encoder reranking, and Reciprocal Rank Fusion.Retrieved recommendations retain source, section, and page metadata, while unsupported statements are rendered as generic, non-cited guidance.
- Verification: Four deterministic checks and an LLM entailment check compare reports with risk scores, EHR values, trends, and guideline citations before clinician presentation.Results are recorded as pass, flag, or skipped in an append-only JSONL audit log, and the verification agent annotates without modifying reports.
3 Demonstration
The demonstration presents a real-time batch-screening dashboard and per-patient report interface for reviewing risk, data completeness, extracted signals, guideline recommendations, verification findings, and raw EHR records. Human clinicians retain final authority through explicit review actions and disclaimers.
- Interfaces: The demonstration includes a daily batch-screening dashboard and a per-patient detailed report page for metabolic-care clinicians.
- Batch screening: Each patient receives a data-completeness score from 0 to 4 based on HbA1c, fasting glucose, BMI, and blood pressure availability.The interface labels completeness as high for 3–4 measurements and medium for 2 measurements; the supplied passage truncates the remaining label.
- Batch screening: The dashboard reports screening-run status, reuses prior results when no new laboratory data are detected, and excludes patients with confirmed diabetes.
- Patient reports: The detailed page combines a five-part report, verification checks, and raw EHR tables containing encounters, laboratory results, and nursing assessments.Raw EHR records are fetched directly with SQL queries and do not pass through an LLM.
- Patient reports: Reports present clinically prioritized signals, longitudinal changes, and ADA-grounded recommendations annotated with source and page metadata.Risk probabilities cached during batch screening are reused, and downstream agents rerun only when necessary for responsive on-demand generation.
- Human oversight: Verification findings appear in a collapsible panel, while physicians retain final diagnostic authority through ACCEPT and OVERRIDE actions.The panel expands automatically when issues are flagged and displays statuses with one-line rationales.
- Human oversight: An injected example changes a stable fasting-blood-glucose trend to rising, illustrating a tampered report case for verification testing.
4 System Evaluation
DIASENTINEL is evaluated across risk prediction, guideline retrieval, and verification reliability, with interfaces exposing calibrated screening, evidence-grounded reports, and audit checks. Results show calibrated risk stratification, strong retrieval performance, and perfect detection of the small synthetic verification set, while real-world verification reliability remains unestablished.
- Evaluation Metrics: Risk evaluation reports discrimination, calibration, and threshold metrics, emphasizing AUPRC for the approximately 6% low-prevalence outcome.Thresholds use Youden’s J for high-risk screening and a minimum-sensitivity constraint for medium-risk screening.
- Risk Function: 72% sensitivity and 62% specificity result from the high-risk threshold, while the medium-risk threshold is selected to achieve at least 90% sensitivity.The sensitivity-oriented design accepts additional false positives to prioritize identifying potential high-risk cases.
- Risk Function: Per-tier calibration closely matches observed one-year incidence: High 0.115 versus 0.117, Medium 0.048 versus 0.052, and Low 0.024 versus 0.022.The high-risk tier concentrates roughly twice the overall prevalence of approximately 0.060.
- Risk Function: 0.737 AUROC is achieved by the proposed model on the strict test set, comparable to XGBoost at 0.731 and above Logistic Regression at 0.697.XGBoost has higher AUPRC, 0.211 versus 0.146, while all three models have similar Brier scores after identical Platt calibration.
- Guideline Retrieval Performance: RRF outperforms dense retrieval on Recall@5 and MRR and achieves a chapter-level hit rate of 0.939 on a 50-question ADA-guideline evaluation set.The evaluation contains five query types and reports Cohen’s κ = 0.76 for annotation agreement.
- Verification Agent Evaluation: All 12 injected verification errors are detected and none of 12 clean controls is falsely flagged, yielding 100% sensitivity and 100% specificity.The entailment check achieves 80% sensitivity and 100% specificity, with identical verdicts across three temperature = 0 runs; the evaluation uses synthetic manually constructed cases.
5 Conclusion
DIASENTINEL is an auditable, on-premise LLM system for one-year T2DM risk screening and guideline-grounded clinical reporting. It combines calibrated prediction, evidence-grounded retrieval, and hybrid verification in dashboard and interactive report interfaces.
- DIASENTINEL supports one-year T2DM risk screening and guideline-grounded clinical reporting in an auditable, on-premise system.
- The system integrates calibrated risk prediction, evidence-grounded retrieval, and hybrid verification to improve transparency and reliability while keeping patient data within the hospital environment.
- A real-time screening dashboard and interactive patient reports demonstrate how these components operate together.
A.1 Model Development and Calibration
The Risk Function is trained and calibrated on de-identified real-world EHR data from one hospital, with dataset construction and threshold optimization supporting risk stratification. Calibration aligns predicted probabilities with observed disease prevalence for downstream use.
- Model Development and Calibration: AUROC 0.737 (95% CI [0.694, 0.773]) and Brier score 0.054 were attained under a strict validation/test split.
- Model Development and Calibration: Raw mean predicted probability was about 0.213 versus approximately 0.060 observed one-year T2DM incidence before Platt scaling.
- Model Development and Calibration: Platt scaling aligned the predicted-probability distribution more closely with observed prevalence and supported clinical interpretation of high-, medium-, and low-risk tiers.
- Model Development and Calibration: The dataset construction pipeline and train/test split were used to train and evaluate the Risk Function.
A.2 Model Evaluation Results
Evaluation covers clinical interpretation of the prediction model and synthetic-case verification performance. The supplied passages identify the held-out evaluation setting and the scope of verification assessment.
- Model Evaluation Results: N = 4,982 patients comprised the full held-out set evaluated at p = 0.5.
- Model Evaluation Results: Verification evaluation controlled for clean and error-injected synthetic cases to assess detection of unsupported claims and designed errors.
- Model Evaluation Results: Guideline-entailment performance was evaluated on synthetic claim–evidence pairs.
A.3 Key finding
Using a clinical-guideline corpus, the cross-encoder reranker alone degraded ranking quality because it favored longer passages over shorter answer-bearing recommendations or table-like chunks. RRF outperformed the dense-retrieval baseline, while the two missed questions were not solvable by retrieval itself.
- Key finding: MRR was 0.625 for the cross-encoder reranker alone, 0.034 below the baseline.
- Key finding: The reranker favored longer narrative passages, creating verbosity bias when answers appeared in shorter recommendation sentences or table-like chunks.
- Key finding: Only 2 questions were missed, and neither was solvable by the retrieval algorithm itself.
B Ethics and Limitations
DIASENTINEL is intended to assist rather than replace clinicians, keeping final judgment with trained professionals. Its current evaluation does not yet establish demographic subgroup calibration or robustness across clinical departments.
- Decision Support, Not Replacement: DIASENTINEL assists rather than replaces clinical decision-making, and final clinical judgment and treatment decisions remain with trained medical professionals.Reports carry a “for clinical reference only” disclaimer, while the verification layer flags and records potential issues without blocking output.
- Decision Support, Not Replacement: Physicians can accept or override each verification result, while the system avoids directly providing drug prescriptions or specific dosage recommendations.This human-in-the-loop design keeps decision authority with human professionals.
- Fairness and Subgroup Robustness: Demographic subgroup calibration and error-rate consistency across sexes remain unevaluated, and robustness across clinical departments has not yet been assessed.The data provide only sex as a demographic feature and do not include age or ethnicity.
Funding
The work was supported by the Far Eastern Memorial Hospital Innovation Research Project.
- The study was supported by the Far Eastern Memorial Hospital Innovation Research Project, Grant No. FEMH-2023-007.