Source-linked AI summary
Automatic Conversion of NICE Guidelines to an Executable Computational Model Using Large Language Models
Ashvin Gupta, Denys Prociuk, Alessandra Russo, Brendan C. Delaney
TL;DR
Clinical guidelines remain difficult to use computationally because they are written in natural language and existing formalization approaches require substantial manual engineering. The paper introduces a stepwise LLM-based pipeline that generates inspectable executable models and patient-specific recommendations, achieving an F1 score of 82.5% on pancreatic cancer vignettes. Its results support the feasibility of scalable automated guideline formalization, while guideline diversity and incomplete real-world data remain deployment boundaries.
Problem
NICE guidelines encode clinical reasoning in natural language, while existing computable representations require extensive manual knowledge engineering and lack widely adopted automatic conversion methods.
Method
A stepwise LLM pipeline converts NICE guideline text into a patient-agnostic ASP model, then translates patient records into facts for executable, traceable recommendations.
Results
82.5% F1 was achieved when the executable pancreatic cancer model generated recommendations for patient vignettes.
Takeaways & Limitations
LLMs can generate interpretable executable models that preserve guideline structure and support transparent inspection, modification, and patient-specific recommendations.
Takeaways & Limitations
The system is not ready for direct clinical integration because guideline diversity, data-to-predicate alignment, and incomplete real-world EHR data require further work.
Abstract
from arXiv · showhide
Introduction: NICE guidelines provide evidence-based recommendations for clinical care but remain largely in unstructured natural language. Existing approaches to converting them into computable representations often focus on individual diseases, require substantial manual encoding, and do not scale. Large language models (LLMs) may enable much of this translation to be automated. Methods: We present an end-to-end approach that converts textual clinical guidelines into executable models capable of generating explainable, patient-specific recommendations. A stepwise LLM-based transformation with in-context examples produces human-inspectable intermediate artifacts. We apply the approach to NICE pancreatic and lung cancer guidelines, use expert review to assess rule alignment, and evaluate the executable pancreatic cancer model on 20 patient vignettes. Results: Expert review showed strong alignment between the source guidelines and generated executable models. Most discrepancies were partial omissions rather than incorrect logic, while hallucinated or fundamentally incorrect rules were rare. On the patient vignettes, the executable model achieved an F1 score of 82.5%. Conclusion: LLMs can transform natural-language NICE guidelines into interpretable, executable models that preserve guideline structure, support transparent inspection and modification, and generate patient-specific recommendations. These findings demonstrate the feasibility of scalable automated generation of computable clinical guidelines.
1 | Introduction
NICE guidelines encode clinical reasoning in natural-language conditional statements, limiting direct use in decision-support systems. Existing computable-formalism approaches require extensive manual engineering, while automatic conversion must preserve fidelity, traceability, and consistency despite complex clinical language.
- NICE guidelines express clinical reasoning mainly as natural-language “if–then” statements rather than machine-executable rules.This limits their direct use in clinical decision-support systems.
- Existing guideline formalisms, including PROforma, Arden Syntax, and GLIF, require extensive manual knowledge engineering.
- Computable guideline models should provide fidelity, traceability, and consistency across guidelines.An intermediate logical representation is described as critical for scalable formalization.
- Automatic conversion methods must address linguistic complexity, conditional dependencies, and domain-specific semantics in real clinical guidelines.
- The paper extends modular in-context translation to NICE diagnostic guidelines while evaluating logical fidelity and downstream recommendation performance.
2 | Methods
The approach decomposes guideline conversion into inspectable stages that produce an executable, patient-agnostic ASP model, then converts patient narratives into facts for patient-specific reasoning. Expert review and vignette evaluation assess translation fidelity and recommendation performance.
- D2K extracts constants, defines predicate schemas, and generates ASP rules encoding conditional logic from NICE guidelines.The staged prompts are designed to support task decomposition and reliability.
- The resulting ASP model is patient-agnostic and encodes NICE guideline logic for later patient-specific inference.
- 2.6 | Knowledge to Performance: K2P translates an unstructured patient vignette into model-compatible facts, combines them with the executable model, and uses Clingo to compute traceable recommendations.
- D2K applies in-context examples from pancreatic cancer guidelines to pancreatic and lung cancer guidelines, including zero-shot and direct-generation ablations.
- Expert reviewers classify translations as absent, correct, incorrect, incomplete but clinically effective, or hallucinated, with free-text comments for additional issues.
- Clinical performance is evaluated by comparing recommendations from 20 pancreatic cancer vignettes with clinician-validated ground-truth interpretations.
3 | Results
The executable models performed less well on lung cancer guidelines, whose rule structures were more diverse and heterogeneous. D2K nevertheless produced fewer untranslated cases, making more guideline recommendations available for inspection and correction.
- Overall performance was lower for lung cancer guidelines, reflecting greater diversity and heterogeneity in their rule structures.
- The in-context baseline achieved slightly higher overall accuracy than D2K for lung cancer guidelines, without a statistically significant difference.
- D2K produced a notably lower proportion of untranslated lung cancer cases than the in-context baseline.The authors describe broader formalization as desirable because partially incorrect rules are easier to inspect and correct than entirely missing ones.
4 | Discussion
D2K produced executable guideline models with strong structural and downstream performance, but omissions, untranslated content, limited vignette coverage, and incomplete data alignment constrain clinical deployment.
- Weaknesses: Most errors were omissions within otherwise plausible rules, including underspecified predicates, omitted qualifiers, and limited higher-arity predicate use.The approach also retained semantically dense phrases as long-form constants rather than decomposing them compositionally.
- Weaknesses: Hallucinations were rare, with one pancreatic cancer case replacing generic “CT” with “pancreatic protocol CT,” likely through local overgeneralization.The stepwise pipeline may constrain unsupported generation by reusing previously generated constants and predicates.
- Weaknesses: Some recommendations were untranslated because general commentary and non-conditional statements do not map readily to patient-level ASP rules.The authors state that heterogeneous guideline structures and multiple valid encodings make human oversight and iterative refinement necessary.
- Implications: D2K’s modular, inspectable design supports stage-specific customization, reuse of patient-agnostic outputs, and maintainability without retraining.However, patient vignettes activated only approximately 64% of the full guideline rule set, and rare presentations and edge cases remained untested.
- Strengths: All generated rules were syntactically valid, logically coherent, and stable across the pancreatic and lung cancer guideline sets.With suitable in-context examples, the approach also captured numerical thresholds, choice constructs, and familial relationships.
- Implications: The system is not yet ready for direct clinical integration because clinician validation, robust data-to-predicate alignment, and accommodation of diverse guideline structures remain necessary.ASP’s closed-world assumption can treat missing EHR observations as false, so deployment requires distinguishing absent, unknown, and explicitly negated conditions.
Installation and Setup
The pipeline can be installed by cloning its repository and running it with Docker or a Python virtual environment, with Clingo and API configuration required.
- The pipeline supports Docker-based execution through the included Dockerfile and docker-compose.yml.
- Alternatively, users can create a virtual environment and install dependencies from requirements.txt.
- Clingo must be installed separately, and users must configure an API key in a .env file before running the pipeline.
Input Data Structure
Input files are organized by source guidelines, prompt templates, and human-authored ground truths within the repository’s input structure.
- The input_guidelines directory contains the lung and pancreatic cancer guideline text files used as natural-language source material.
- The D2K directory contains five prompt templates for constant extraction, predicate identification, rule generation, in-context prompting, and zero-shot prompting.
- The ground_truths directory contains human-authored ASP programs.
Configuration and Execution
Pipeline execution is controlled through YAML configuration files that specify the model, pipeline version, input paths, and supported execution modes.
- Separate YAML files configure lung and pancreatic cancer pipeline runs.
- The lung cancer configuration supports D2K-only execution, while the pancreatic configuration supports D2K-only, K2P-only, and D2K+K2P modes.
- Each configuration specifies the LLM model, pipeline version, and input paths.
- Reviewers execute the pipeline by passing the relevant configuration file to main.py.
Output Organization and Expected Results
Pipeline outputs and evaluation artifacts are centralized under the output directory, with separate materials for human rule review, patient-level accuracy, solver results, and explanations.
- Output Organization: Pipeline outputs are centralized in src/output_files/CLAUDE/, including evaluation materials in its reviews subdirectory.
- Expected Results: Human D2K review records independent annotations, categorical scores, and comments for generated rules.
- Expected Results: Pancreatic cancer K2P review stores per-patient prediction accuracy metrics in a CSV file.
- Output Organization: Pancreatic K2P outputs include Clingo-ready ASP rules, patient facts, solver results, and natural-language explanations.
Expected Output Formats and Content
The appendix documents structured outputs for guideline translation, hypothesis testing, patient-vignette evaluation, and pancreatic-cancer constant and predicate extraction.
- Expected Output Formats and Content: Generated outputs include typed predicates, valid ASP rules, fired-rule tracking, patient-organized atoms, solver results, and explanations of clinical rule firings.These artifacts support inspection of both model construction and patient-specific reasoning.
- Hypothesis Test: Welch’s t-test compared In-Context performance of 58.2 ± 6.8% with D2K performance of 54.8 ± 3.4% using three observations per group.The test accounted for unequal variances and the small sample size.
- Hypothesis Test: p ≈ 0.248, so the study failed to reject the null hypothesis at α = 0.05.The reported 3.4% higher In-Context mean was not statistically significant given the variance and sample size.
- Patient vignettes: The evaluation used pancreatic-cancer patient vignettes containing demographics, symptoms, imaging, pathology, genetic markers, and treatment decisions expressed in unstructured language.Table A1 reports predicted and ground-truth rules expected to fire for 20 vignettes.
- Constant Extraction for Pancreatic Cancer: Pancreatic-cancer constants were grouped into disease, symptoms, imaging, procedures, pancreatic findings, genetic disease, management, treatments, anatomy, patient information, and general findings.Examples include obstructive jaundice, CT, EUS, pancreatic abnormalities, BRCA1, surveillance, surgery, and diagnostic uncertainty.
- Constant Extraction for Pancreatic Cancer: The constant inventory includes anatomical structures, patient-risk details, diagnostic uncertainty, and pancreatic-management concepts such as surveillance, suspected-cancer referral, and urgent investigation.The appendix also lists treatments and additional prompt definitions for procedures, genetic disease, and anatomical structures.
- Predicate Extraction for Pancreatic Cancer: Predicates represent guideline relationships and actions: offer denotes recommended actions, have patient features, and findings diagnostic results or follow-up conditions.The listed predicate signatures include offer, consider, refer, have, findings, suspected, perform, drain, ask, and assess.
- Constant Prompt: The extraction prompts instruct the model to identify distinct medical concepts, classify them into supplied categories, preserve minimum exact wording, and output only categorized constants.Prompt categories cover disease, symptoms, imaging, procedures, findings, genetics, management, treatments, anatomy, patient information, and general findings.