PaperlayerView exact revision on arXiv

Source-linked AI summary

KREL: Automatic Medical Coding via Knowledge-Guided Reasoning over Clinical Evidence with LLMs

Xubin Chen, Yipeng Zhou, Wen Sun, Chengkai Huang, Xiaoming Fu, Quan Z. Sheng

arXiv:2608.20887v1cs.CLcs.AI

TL;DR

AMC must assign standardized ICD codes despite long clinical notes, a vast label space, and complex coding rules that existing models do not fully capture. KREL combines LLM-based clinical reasoning with structured ICD knowledge through retrieval and verification, and benchmark experiments report consistent improvements over strong baselines.

  • Problem

    AMC is challenging because clinical notes are long, the ICD taxonomy exceeds 70K labels, and coding guidelines are not inherently encoded in LLMs.

  • Method

    KREL extracts code-relevant queries, retrieves candidate ICD codes and guideline information with a knowledge graph, and uses LLM verification for final code assignment.

  • Results

    KREL consistently improves over competitive PLM-based and state-of-the-art LLM-based baselines across benchmark datasets, including full-label-space gains on MDACE and MIMIC-IV-Subset.

  • Takeaways & Limitations

    Integrating structured ICD knowledge with LLM reasoning supports scalable coding over the full ICD-10 label space and improves adherence to coding rules.

  • Takeaways & Limitations

    High candidate recall remains challenging for rare or highly specific codes, and the framework introduces LLM-related cost, latency, and reproducibility considerations.

Abstract

from arXiv · show

Automatic Medical Coding (AMC), which assigns standardized International Classification of Diseases (ICD) codes to clinical notes, is essential for medical reimbursement, quality reporting, and clinical research. Existing pre-trained language model (PLM)-based methods typically formulate AMC as an extreme multi-label classification problem over a predefined code set, while recent large language model (LLM)-based approaches instead frame it as generation or multi-step reasoning. However, key challenges remain, including the extreme length of clinical notes that hinders effective interpretation, the vast ICD label space, and complex coding rules that are not explicitly captured by LLMs. In this work, we propose Knowledge-Guided Reasoning over Clinical Evidence with LLMs (KREL), a framework that leverages LLMs for clinical text understanding and reasoning while integrating external ICD coding guidelines as structured knowledge. This design enables tight coupling between domain knowledge and LLM reasoning, reducing hallucinations and improving compliance with coding standards. Experiments on benchmark datasets show that KREL consistently outperforms strong PLM-based and state-of-the-art LLM-based baselines.

1 Introduction

Automatic medical coding assigns ICD diagnosis codes to clinical notes, but current approaches face large label spaces, long narratives, and coding rules that are difficult to capture. KREL addresses these challenges by combining LLM reasoning with external ICD knowledge and reports improvements across benchmark datasets.

  • Manual medical coding is essential but time-consuming and error-prone because it requires domain knowledge and complex rules.
  • Existing AMC methods use predefined-label classification, while the full ICD taxonomy contains over 70K codes.Typical predefined label spaces contain 50–100 labels, substantially fewer than the full taxonomy.
  • LLM-based coding struggles with concept-to-code mapping, rare codes, and coding constraints not inherently encoded in language models.
  • KREL uses query extraction, guideline-driven candidate retrieval, and code verification to connect LLM reasoning with structured ICD knowledge.The framework retrieves candidate codes, definitions, and coding rules before final LLM-based verification.
  • KREL scales to the full ICD coding space and is reported to improve AMC accuracy over competitive PLM-based and state-of-the-art LLM-based baselines.Experiments cover MDACE, ACI-BENCH, and MIMIC-IV.

2 Related Work

AMC research has progressed from manually maintained rule-based systems to neural classification, LLM generation, workflow systems, and retrieval-based approaches. KREL builds on these developments by using an ICD knowledge graph for hierarchy-aware retrieval and rule-aware verification.

  • Early rule-based and knowledge-driven AMC pipelines offered explicit control but required substantial maintenance across institutions and documentation styles.
  • Neural AMC systems shifted the task toward extreme multi-label classification as large EHR datasets became available.
  • Direct LLM prompting remains unreliable under large ICD label spaces because generated codes may be invalid, unsupported, or coding-inaccurate.
  • Standard RAG is less suited to AMC because ICD-10-CM is a hierarchical label space governed by coding guidelines rather than an unstructured document collection.
  • KREL uses an ICD knowledge graph to guide hierarchy-aware candidate search and convert coding-rule relations into verification context.

3 KREL Framework

KREL reformulates automatic medical coding as hierarchy-aware retrieval followed by rule-aware verification, using evidence-grounded queries and an ICD knowledge graph. Its staged design retrieves, reranks, and verifies candidate codes against clinical evidence and coding constraints.

  • Framework overview: KREL maps clinical notes to evidence-grounded queries, retrieves ICD candidates, and verifies them against the note, evidence spans, and coding rules.The query extractor identifies coding-relevant descriptions and rewrites them into concise queries paired with supporting spans.
  • ICD knowledge graph: The ICD knowledge graph encodes code descriptions, hierarchy, pairwise coding rules, and multi-code combination relations for downstream retrieval and verification.Hierarchy and code text support candidate search and reranking, while rule relations become verification hints and combination checks rather than ordinary candidate expansions.
  • Candidate recall: Hierarchy-Aware Beam Search follows ICD hierarchy paths to retrieve candidate leaf codes using embedding relevance and bounded search budgets.The method starts from high-scoring nodes, explores selected children, preserves a beam across depths, and forms the final candidate set from top-scoring leaves.
  • Candidate recall: HBS can outperform greedy retrieval in recall because it postpones pruning until leaf nodes, reducing premature removal of correct codes.Greedy methods prune parent codes earlier, whereas HBS retains alternative hierarchy paths longer.
  • Reranking and verification: A cross-encoder reranks retrieved candidates using query–code relevance derived from ICD descriptions, after which a note-level verification budget limits subsequent verification cost.The descriptions include symptoms, concepts, and inclusion terms, and the reranked lists are truncated under a shared budget.

4 Experimental Setup

The experiments evaluate KREL across three clinical datasets, two ICD label-space settings, and PLM- and LLM-based baselines. The setup emphasizes comparison under both benchmark inventories and the complete ICD-10-CM diagnosis space.

  • Datasets: KREL is evaluated on MDACE, ACI-Bench, and MIMIC-IV, covering annotated coding data, evidence-linked notes, and de-identified hospital records.ACI-Bench additionally supports evidence-based analysis through sentence-level links between text spans and ICD codes.
  • Label-space settings: The benchmark-label-space setting constrains predictions to each benchmark’s predefined inventory, whereas the full-label-space setting uses the complete ICD-10-CM space exceeding 70K labels.MDACE and ACI-Bench use benchmark inventories; fully labelled MDACE and a sampled MIMIC-IV subset support full-space evaluation.
  • Baselines: The comparison includes PLM-based extreme multi-label classifiers and LLM-based generation or multi-step reasoning systems.PLM baselines predict over predefined labels, while LLM baselines include direct prompting, chain-of-thought, and workflow-based methods.
  • Implementation: The implementation uses GPT-4o for query extraction, candidate selection, and verification, with Qwen3 embedding and reranking models supporting candidate selection.The ICD knowledge graph is built from the ICD-10-CM Tabular List and Official Coding Guidelines.
  • Metrics and comparison: Evaluation primarily reports micro-averaged precision, recall, and F1-score after converting outputs to encounter-level ICD predictions for fair comparison.Table 1 covers both benchmark-label-space and full-label-space settings.

5 Results and Analysis

KREL achieves strong coding performance across benchmark- and full-label-space settings, while ablations show that evidence-grounded querying, verification, and hierarchy-aware retrieval each support its effectiveness.

  • Main Results: KREL achieves the best F1 on MDACE and ACI-BENCH in the benchmark-label-space setting, reaching 0.49 and 0.70, respectively.The MDACE improvement is mainly recall-driven, whereas ACI-BENCH gains occur in both precision and recall.
  • Main Results: F1 improves from 0.32 to 0.51 on MDACE and from 0.33 to 0.39 on MIMIC-IV-Subset under the full-label-space setting.On MDACE, recall increases from 0.26 to 0.53; gains are consistent across datasets.
  • Ablation Study: Replacing the query extractor with medSpaCy NER reduces recall from 0.53 to 0.19 and F1 from 0.51 to 0.24.The replacement uses extracted medical terms directly as retrieval queries without evidence-grounded reformulation.
  • Ablation Study: Removing the verifier lowers precision from 0.49 to 0.10 while increasing recall to 0.62, whereas removing the reranker reduces F1 to 0.46.Removing coding rules slightly decreases F1 to 0.49, indicating contributions from all framework components.
  • Retrieval Strategy Analysis: HBS achieves 0.70 retrieval recall versus 0.67 for flat dense retrieval and 0.46 for greedy hierarchical search with a comparable candidate budget.HBS combines the ICD hierarchy with branch diversity, while greedy search can discard relevant subtrees prematurely.
  • Candidate Set Size Evolution: The pipeline reduces candidates from 470 to 49 after reranking and to 10 after verification, while precision rises from 0.01 to 0.48.Recall changes from 0.76 at retrieval to 0.62 after reranking and 0.53 after verification; F1 reaches 0.51.

6 Conclusion

KREL integrates structured ICD coding guidelines with LLM reasoning for automated medical coding. Experiments show consistent improvements over strong PLM-based and state-of-the-art LLM-based baselines.

  • KREL couples external ICD knowledge with LLM reasoning to interpret complex narratives, handle the large label space, and improve adherence to coding rules.The framework is designed to reduce hallucinations and enhance coding reliability.
  • KREL consistently outperforms strong PLM-based and state-of-the-art LLM-based baselines across benchmark datasets.

Limitations

KREL remains constrained by candidate recall and by practical limitations associated with LLM dependence and dataset coverage.

  • Missed candidates generally cannot be recovered later, making high candidate recall difficult under a fixed verifier budget, especially for rare or highly specific codes.
  • LLM-based query extraction and verification introduce cost, latency, and reproducibility considerations for deployment.Clinical deployment may require secure inference environments, locally hosted models, or institution-approved services.
  • Evaluation on public EHR datasets does not capture the full diversity of institutional documentation styles or coding workflows.Evidence annotations are unavailable for all datasets, limiting systematic evaluation of grounding quality.

Ethical Considerations

The study uses public or credentialed-access clinical datasets and conducts local computation alongside API-based LLM inference. Clinical use requires human validation and secure, institution-approved deployment.

  • MDACE and ACI-Bench are public datasets, while MIMIC-IV is accessed through PhysioNet credentialing within its authorized data-use scope.
  • KREL is not intended to replace professional coders or clinicians; outputs should undergo human validation before clinical, administrative, or billing use.Real-world deployment also requires secure infrastructure, approved models or services, and safeguards for protected health information.
  • Local retrieval, candidate generation, reranking, postprocessing, and metric computation ran on an Ubuntu server with one RTX 5090 GPU, while LLM stages used provider APIs.

B.1 Dataset and Evaluation Scope

The evaluation uses benchmark datasets and both restricted and full ICD-10-CM label spaces, with micro-averaged precision, recall, and F1 as primary metrics. The study also constructs an ICD knowledge graph from code metadata, hierarchy information, and coding-rule annotations.

  • Evaluation scope: The benchmark-label-space setting restricts predictions to each benchmark’s predefined inventory, whereas the full-label-space setting uses over 70K ICD-10-CM labels.The full-label-space evaluation uses fully labelled MDACE and a sampled MIMIC-IV subset.
  • Graph construction: The ICD knowledge graph combines code metadata, hierarchy information, coding-rule annotations, and multi-code combination relations.Code nodes store descriptions and related attributes, while hierarchy and rule structures represent coding dependencies.
  • Metrics: Methods are evaluated primarily with micro-averaged precision, recall, and F1 across notes.F1 summarizes the trade-off between accurate code assignment and recovering relevant diagnoses.

C.2 Error Analysis

KREL’s full-label-space error analysis shows broader code recovery with more predictions, while verifier and retrieval components help control unsupported or incomplete coding. Chapter-level gains extend across several clinically diverse ICD groups.

  • Error patterns: KREL predicts 9.67 codes per note, reducing false negatives to 255 versus 398 for prompting and 416 for the LLM-agent baseline.Its larger output also produces more false positives, reflecting a precision–recall trade-off.
  • Error patterns: KREL reaches 0.615 combination-code recall, compared with 0.077 for both baselines.The analysis links this gap to handling codes requiring information from multiple conditions or query blocks.
  • Error patterns: KREL’s sibling-code false-positive rate is 0.227 versus 0.219 for the LLM-agent baseline despite its larger predicted code set.The additional predictions therefore do not mainly reflect uncontrolled sibling-code expansion.
  • Chapter-level results: KREL improves recall and F1 across all five reported ICD chapters, with especially large gains in endocrine, genitourinary, nervous-system, musculoskeletal, and infectious diseases.It reaches 0.76 recall and 0.78 F1 in endocrine and metabolic diseases, and 0.73 recall and 0.76 F1 in genitourinary diseases.

D Evidence Alignment Analysis

The evidence-alignment analysis evaluates verifier evidence only for true-positive predictions. Returned evidence covers every evaluated prediction and usually contains or semantically matches the human-annotated support.

  • Evidence coverage: Among 286 true-positive code pairs, the verifier provides at least one evidence quote for every prediction, achieving 100% evidence coverage.Evidence quality is evaluated only for true-positive code predictions because false positives lack corresponding gold evidence annotations.
  • Evidence alignment: At least one returned quote contains the human-annotated clinical mention in 90.6% of true-positive cases.This measures mention-anchor coverage rather than merely the presence of an evidence quote.
  • Evidence alignment: The mean semantic cosine similarity between verifier evidence and gold evidence is 0.746.Similarity is computed from sentence-transformer embeddings using the maximum predicted–gold evidence-pair cosine for each true-positive code.
  • Verification design: The verifier prompt requires selected codes to come from the global candidate set and include one or two short verbatim note quotes.Decisions remain based on the clinical note, while evidence blocks serve as localization hints.
Loading 2608.20887v1…