PaperlayerView exact revision on arXiv

Source-linked AI summary

KREL: Automatic Medical Coding via Knowledge-Guided Reasoning over Clinical Evidence with LLMs

Xubin Chen, Yipeng Zhou, Wen Sun, Chengkai Huang, Xiaoming Fu, Quan Z. Sheng

arXiv:2608.20887v1cs.CLcs.AI

TL;DR

LLMs can mis-map clinical concepts, miss rare ICD codes, and lack encoded coding rules for automatic medical coding. KREL integrates structured ICD knowledge into evidence-grounded retrieval and verification, achieving the best full-label-space F1 on both evaluated datasets.

  • Problem

    For AMC, LLMs may mis-map clinical concepts, miss rare codes in the 70K-plus ICD taxonomy, and lack the encoded rules needed to satisfy coding constraints.

    Source: §1 · p. 2

  • Method

    KREL grounds LLM-based AMC in structured ICD knowledge, extracting evidence-based queries, retrieving and refining candidates with ICD hierarchy and rules, then verifying them against clinical notes.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    KREL achieved the best F1 across both full-label-space datasets, improving MDACE from the best baseline's 0.32 to 0.51 and MIMIC-IV-Subset from 0.33 to 0.39.

    Source: §5.1 · p. 7

  • Takeaways & Limitations

    KREL's explicit integration of structured ICD guidelines is presented as improving adherence to coding rules not inherently captured by LLMs.

    Source: §6 · p. 8

    Candidate recall remains challenging under a fixed verifier budget in the full ICD-10-CM label space, particularly for rare or highly specific codes.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Automatic Medical Coding (AMC), which assigns standardized International Classification of Diseases (ICD) codes to clinical notes, is essential for medical reimbursement, quality reporting, and clinical research. Existing pre-trained language model (PLM)-based methods typically formulate AMC as an extreme multi-label classification problem over a predefined code set, while recent large language model (LLM)-based approaches instead frame it as generation or multi-step reasoning. However, key challenges remain, including the extreme length of clinical notes that hinders effective interpretation, the vast ICD label space, and complex coding rules that are not explicitly captured by LLMs. In this work, we propose Knowledge-Guided Reasoning over Clinical Evidence with LLMs (KREL), a framework that leverages LLMs for clinical text understanding and reasoning while integrating external ICD coding guidelines as structured knowledge. This design enables tight coupling between domain knowledge and LLM reasoning, reducing hallucinations and improving compliance with coding standards. Experiments on benchmark datasets show that KREL consistently outperforms strong PLM-based and state-of-the-art LLM-based baselines.

1 Introduction

KREL addresses AMC’s scaling and coding-rule challenges by combining LLM reasoning with structured ICD knowledge, using retrieval, candidate refinement, and verification rather than direct code generation.

1 Introduction

  • Medical coding is essential but manual assignment is time-consuming and error-prone because it requires specialized knowledge and adherence to complex ICD guidelines.Human annotators typically follow World Health Organization ICD coding guidelines.Source: §1 · p. 1
  • PLMs map notes to predefined 50–100-code sets (Liu et al., 2022; Edin et al., 2023), while LLM approaches use generation or reasoning (Motzfeldt et al., 2025; Zheng et al., 2025; Yuan et al., 2025).Fixed label sets limit scalability; LLMs can mis-map concepts, miss rare codes in the >70K-label taxonomy, and violate coding rules.Source: §1 · pp. 1–2
  • KREL extracts structured queries from clinical notes, retrieves candidate codes, definitions, and rules from the full ICD taxonomy, then verifies candidates against clinical evidence.Its guideline-driven knowledge-graph retrieval-augmented generation module supports scalability to the full ICD-10 label space.Source: §1 · p. 2
  • Experiments on MDACE, ACI-BENCH, and MIMIC-IV show consistent improvements over competitive PLM-based and state-of-the-art LLM-based baselines, with interpretability also reported.The paper presents KREL as a retrieval, candidate-refinement, and verification process grounded in structured ICD knowledge and coding guidelines.Source: §1 · p. 2

2 Related Work

Prior AMC methods progress from classification to LLM workflows, but scaling across the ICD-10-CM label space and integrating retrieval, evidence grounding, and guideline-constrained verification remain challenges. KREL addresses these needs with an ICD knowledge graph for hierarchy-aware retrieval and rule-aware verification.

2.1 Automatic Medical Coding

  • AMC research shifted from maintenance-intensive rule-based pipelines to extreme multi-label classification, with neural encoders and later hierarchy, lexical-resource, and domain-adapted PLM methods improving evidence modeling and code distinctions.Classification methods remain difficult to scale to the complete ICD-10-CM taxonomy (Edin et al., 2023).Source: §2.1 · p. 2
  • Direct LLM prompting can generate invalid, unsupported, or coding-inaccurate codes, while prior workflows leave full-space retrieval, evidence grounding, and guideline-constrained verification uncombined.Code Like Humans and MedDCR use multi-step workflows with ICD resources or coding procedures to improve reliability (Motzfeldt et al., 2025; Zheng et al., 2025; Yuan et al., 2025).Source: §2.1 · p. 2

2.2 Knowledge-guided AMC

  • Standard RAG is less suited to AMC because ICD-10-CM is a hierarchical label space governed by coding guidelines, so candidate construction requires semantic matching and structured coding dependencies.The cited guidance includes CMS and NCHS, 2022; NCHS, 2022.Source: §2.2 · p. 3
  • KREL uses an ICD knowledge graph to guide hierarchy-aware candidate retrieval and convert coding-rule relations into verification context for evidence-grounded code selection.Knowledge graphs have supported concept-level tasks, but existing KG methods generally do not perform full-label-space ICD assignment.Source: §2.2 · p. 3

3 KREL Framework

KREL addresses the interdependent, guideline-constrained nature of ICD coding and the mismatch between free-text clinical notes and standardized concepts by reframing coding as hierarchy-aware retrieval followed by rule-aware verification.

3.1 Problem Formulation

  • AMC maps clinical notes to a code set from more than 70K ICD-10-CM labels, with interdependent codes and free-text expressions that may not align with standardized concepts.The paper focuses on ICD-10-CM; adapting KREL to other versions requires rebuilding the corresponding code hierarchy and guideline resources.Source: §3.1 · p. 3

3.2 Framework Overview

  • KREL extracts evidence-grounded queries, retrieves and reranks ICD candidates using hierarchy knowledge, builds rule checks, then verifies candidates against the note and evidence to produce codes.The framework overview describes three components: Query Extractor, Candidate Selector, and Code Verifier.Source: §3.2 · p. 3, Figure 2 · p. 4

3.3 Query Extractor

  • The Query Extractor sectionizes the note and uses an LLM to rewrite coding-relevant diagnoses, conditions, symptoms, and clinical status as queries paired with supporting evidence spans.Formally, it produces query-evidence pairs Q(x) = {(qi, Ei)}n.Source: §3.3 · p. 3

3.4.1 ICD Knowledge Graph Construction

  • The ICD knowledge graph combines code descriptions and inclusion terms, the ICD hierarchy, pairwise guideline relations, and multi-code combination-code relations.Hierarchy supports recall, code text supports semantic scoring and reranking, and rule relations become verification hints and combination checks rather than expanding ordinary candidate lists.Source: §3.4.1 · pp. 3–4, §3.4 · p. 3

3.4.2 Hierarchy-aware Candidate Recall

  • Hierarchy-Aware Beam Search retrieves query-specific codes by descending the ICD hierarchy, combining parent-path scores with local relevance, and postponing pruning until leaf codes.The paper states that HBS can outperform greedy-based retrieval in recall because greedy methods may prematurely prune correct parent codes.Source: §3.4.2 · pp. 4–5

3.4.3 Reranking

  • A cross-encoder LLM reranks recalled codes using their descriptions and query-code relevance, then a note-level budget retains top-ranked candidates across query lists for verification.Unused per-query budget slots are reassigned to the highest-ranked remaining candidates across queries.Source: §3.4.3 · p. 5

3.4.4 Augmenting Candidate Codes with Rules

  • KREL augments candidates with prioritized codeFirst and useAdditionalCode relations, falls back to codeAlso when needed, and supplies relation hints to guide final assignment.It also checks whether required codes for a combination code are present across the note’s candidate lists and adds the corresponding combination rule for verification.Source: §3.4.4 · p. 5

3.5 Evidence-grounded Code Verification

  • The verifier evaluates candidates against the full note, localized evidence, code descriptions, and rule dependencies, and retains codes rated SUPPORTED or POSSIBLE.It separately checks applicable combination rules against their required component codes and the clinical note.Source: §3.5 · p. 6

4 Experimental Setup

The experimental setup evaluates KREL across three clinical datasets and two ICD label-space settings, against PLM- and LLM-based baselines. It specifies the framework’s LLM and coding-knowledge components and uses micro-averaged precision, recall, and F1 for comparison.

4.1 Datasets

  • Experiments use MDACE, ACI-Bench, and MIMIC-IV, with benchmark inventories or the full ICD-10-CM space of over 70K labels.MDACE is manually annotated, ACI-Bench links note spans to codes, and MIMIC-IV contains de-identified hospital records. The full-label setting covers fully labelled MDACE and a sampled MIMIC-IV subset; MDACE and ACI-Bench use official splits and inventories.Source: §4.1 · p. 6

4.2 Baselines

4.3 Implementation Details

  • The implementation uses GPT-4o for KREL’s query extractor, candidate selector, and verifier, with ICD-10-CM coding resources and Qwen3 embedding and reranking models.The knowledge graph uses the ICD-10-CM Tabular List and 2022 Official Coding Guidelines (CMS and NCHS, 2022). HBS parameters are Kc=200, B=200, M=8, D=6, and Kf=30; at most KV=50 deduplicated codes per note proceed to final verification.Source: §4.3 · p. 6

4.4 Evaluation Metrics

  • Methods are evaluated with precision, recall, and F1-score, primarily micro-averaged, after outputs are converted to encounter-level ICD predictions for fair comparison.This aligns KREL’s retrieval-and-verification outputs with the encounter-level predictions used in prior work.Source: §4.4 · pp. 6–7

5 Results and Analysis

KREL achieves the best reported F1 in both benchmark-label-space datasets and improves over the best baselines in full-label-space tests. Ablations and pipeline analyses show how query construction, verification, reranking, coding rules, and hierarchical retrieval contribute to performance.

5.1 Main Results

  • In benchmark-label-space tests, KREL achieves best F1 of 0.49 on MDACE and 0.70 on ACI-BENCH; the MDACE gain is modest and mainly recall-driven.On ACI-BENCH, KREL gains in both precision and recall over PLM, prompting, and workflow baselines.Source: §5.1 · p. 7
  • In full-label-space tests, KREL raises MDACE F1 from 0.32 to 0.51 and recall from 0.26 to 0.53, and MIMIC-IV-Subset F1 from 0.33 to 0.39.The gains are smaller than in the benchmark setting but consistent across datasets.Source: §5.1 · p. 7

5.2 Ablation Study

  • Replacing the LLM query extractor with medSpaCy NER lowers recall from 0.53 to 0.19 and F1 from 0.51 to 0.24, underscoring the importance of evidence-grounded query construction.The replacement uses extracted medical terms directly as retrieval queries without evidence-grounded query reformulation.Source: §5.2 · p. 7, Table 2 · p. 7
  • Removing the verifier drops precision from 0.49 to 0.10 despite recall rising to 0.62; removing reranking lowers F1 to 0.46, and removing coding rules slightly decreases F1 to 0.49.The verifier filters unsupported retrieved candidates.Source: Table 2 · p. 7, §5.2 · p. 8

5.3 Retrieval Strategy Analysis

  • HBS achieves retrieval recall of 0.70 versus flat dense retrieval’s 0.67 at comparable candidate budget, while greedy hierarchical search reaches 0.46.HBS preserves branch diversity while controlling the search space; greedy search can prematurely discard relevant ICD subtrees.Source: §5.3 · p. 8, Figure 3 · p. 8

5.4 Candidate Set Size Evolution

  • Candidate retrieval prioritizes coverage, while reranking and verification progressively shrink the set and improve final code-assignment reliability.On MDACE, retrieval averages 470 codes per note at recall 0.76 and precision 0.01; reranking reduces this to 49 codes with precision 0.11, F1 0.19, and recall 0.62; verification outputs 10 codes with precision 0.48, F1 0.51, and recall 0.53.Source: §5.4 · p. 8, Figure 4 · p. 8

6 Conclusion

KREL integrates structured ICD coding guidelines into LLM reasoning for automated medical coding. The paper reports that this design improves handling of clinical narratives and coding rules, and performs consistently better than strong PLM- and LLM-based baselines on benchmark datasets.

6 Conclusion

  • KREL consistently outperforms strong PLM-based and state-of-the-art LLM-based baselines on benchmark datasets.The experiments are presented as evidence of its effectiveness and robustness for automated medical coding.Source: §6 · p. 8
  • Unlike PLM classification and LLM generation or multi-step reasoning approaches, KREL integrates structured ICD coding guidelines into LLM reasoning.The framework tightly couples external medical knowledge with LLM reasoning.Source: §6 · p. 8
  • This design supports interpretation of long, complex clinical narratives, handling of the large ICD label space, and adherence to coding rules not inherently captured by LLMs.The paper says the knowledge-guided design reduces hallucinations and enhances coding reliability.Source: §6 · p. 8

Limitations

KREL’s limitations concern candidate-code recall, the cost and reproducibility of LLM dependence, and evaluation that does not capture the full diversity of clinical data and coding workflows.

Candidate recall

  • KREL cannot recover missed retrieved candidates except through explicitly triggered combination-code checks, making high recall difficult for rare or highly specific codes.This remains challenging under a fixed verifier budget across the full ICD-10-CM label space.Source: Limitations · p. 9

Reliance on LLMs

Data Scarcity

  • Offline evaluation on public EHR benchmarks does not capture the full diversity of institutional documentation styles or coding workflows.Evidence annotations are unavailable for all datasets; future work should evaluate more diverse clinical datasets and incorporate professional medical-coder feedback.Source: Limitations · p. 9

Ethical Considerations … B Implementation Details

KREL uses public or credentialed clinical datasets and is intended to support, not replace, professional coding review. Experiments used a GPU-equipped Linux server, with retrieval and reranking local and LLM stages run through provider APIs.

Ethical Considerations

  • MIMIC-IV was used under PhysioNet credentialing and authorised data-use scope, while MDACE and ACI-Bench are public benchmarks.Source: Ethical Considerations · p. 9
  • KREL should support coding review, not replace medical coders or clinicians, and its outputs require human validation before use.Source: Ethical Considerations · p. 9
  • Clinical deployment would require secure inference, institution-approved models or services, and safeguards against unauthorised exposure of protected health information.Source: Ethical Considerations · p. 9

A Experimental Environment

B.1 Dataset and Evaluation Scope … C.1 Detailed Evaluation Metric

The experiments evaluate coding across benchmark-specific and full ICD-10-CM label spaces, while the framework represents ICD metadata and coding rules in a graph and reports micro-averaged precision, recall, and F1.

B.1 Dataset and Evaluation Scope

  • MDACE and ACI-Bench are public benchmarks derived from MIMIC clinical records; MIMIC-IV access required PhysioNet authorization, and experiments stayed within its permitted data-use scope.The benchmarks include additional task-specific annotation for medical coding evaluation.Source: §B.1 · p. 11
  • The full-label-space evaluation covers 72,750 terminal ICD-10-CM diagnosis codes, while benchmark-label-space evaluation uses each benchmark’s predefined inventory; MDACE includes ICD-10-CM and ICD-10-PCS labels.The full-label-space setting uses MDACE and MIMIC-IV and tests selection from the complete diagnosis space rather than benchmark-specific candidates.Source: §B.1 · p. 11

B.2 Graph Construction

  • The ICD knowledge graph includes code nodes with descriptions, inclusion terms, and version information, plus child-to-parent hierarchy edges and directed pairwise coding-rule edges.It is constructed from ICD-10-CM metadata, hierarchy information, and coding-rule annotations.Source: §B.2 · p. 11
  • For rules requiring multiple codes, the graph stores a combination relation linking required ICD codes to a target combination code through a rule node.The relation specifies required-code and target-code sets.Source: §B.2 · p. 12

C.1 Detailed Evaluation Metric

  • Evaluation reports micro-averaged precision, recall, and F1 over all notes; precision reflects correct predicted codes, while recall reflects recovered gold codes.Missing a reportable code or assigning an unsupported code can reduce coding quality.Source: §C.1 · p. 12
  • F1 is the harmonic mean of precision and recall, summarizing the trade-off between accurate code assignment and coverage of relevant diagnoses.Clinical notes may contain multiple reportable conditions.Source: §C.1 · p. 12

C.2 Error Analysis

KREL’s error analysis shows fewer missed codes and stronger combination-code recall, alongside recall and F1 gains across five ICD chapters.

C.2 Error Analysis

  • KREL reaches 0.615 combination-code recall versus 0.077 for each baseline, while its sibling-code FP rate is 0.227 versus 0.219 for LLM-agent.The sibling-code FP rate counts false-positive codes that are siblings of a gold code; the near-baseline rate suggests KREL’s added predictions are not mainly uncontrolled sibling-code expansion.Source: §C.2 · p. 12
  • KREL predicts 9.67 codes per note versus 5.98 for LLM prompting and 5.11 for LLM-agent, with 255 false negatives versus 398 and 416, respectively.This larger output also brings more false positives; the LLM-agent has fewer false positives but misses more gold codes, partly reflecting its more conservative output size.Source: §C.2 · p. 12, Table 4 · p. 13
  • Across all five reported ICD chapters, KREL improves recall and F1; examples include endocrine/metabolic at 0.76 recall and 0.78 F1, and genitourinary at 0.73 and 0.76.The largest gains also occur in nervous system, musculoskeletal, and infectious disease chapters, where baselines remain especially lower for nervous system and musculoskeletal codes.Source: §C.2 · p. 12, Figure 5 · p. 13

D Evidence Alignment Analysis · Generative AI Usage · E Prompt Template

The supplementary sections quantify verifier evidence alignment, disclose limited ChatGPT use, and document prompts for query extraction, embedding, reranking, and verification.

D Evidence Alignment Analysis

  • Evidence coverage tests for any quote, mention-anchor coverage for the annotated clinical mention, and semantic cosine compares verifier quotes with gold evidence sentences.For each true-positive prediction, cosine similarity uses sentence-transformers/all-MiniLM-L6-v2 and the maximum similarity across predicted–gold evidence pairs.Source: D Evidence Alignment Analysis · p. 15
  • 100% evidence coverage, 90.6% mention-anchor coverage, and 0.746 mean semantic cosine were reported across 286 true-positive code pairs on MDACE.Evidence quality was evaluated only for true-positive predictions; 90.6% of cases had a returned quote containing the annotated clinical mention.Source: Table 5 · p. 15, D Evidence Alignment Analysis · p. 15

Generative AI Usage

  • ChatGPT was used for grammar checking, language polishing, and limited debugging, but not to generate research ideas, results, or references; authors reviewed and verified suggestions.Source: Generative AI Usage · p. 15

E Prompt Template

  • Figure 6 provides a prompt for extracting and formatting queries from clinical note samples.Source: Figure 6 · p. 16
  • Figure 7 provides instructions for Qwen3-Embedding-8B, while Figure 8 provides instructions for Qwen3-Reranker-8B.Source: Figure 7 · p. 16, Figure 8 · p. 17
  • Figure 9 provides a prompt for rule-aware and evidence-grounded verification samples.Source: Figure 9 · p. 17

Related papers

Loading 2608.20887v1…