PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding the Limits of Agentic ICD Coding

Chong Yock Eng, Yushi Cao, Yiming Chen, Kezhi Mao, Hongchao Jiang

arXiv:2609.13806v1cs.CLcs.AIcs.LG

TL;DR

Standard ICD-10-CM benchmarks can obscure performance on rare codes and complex coding scenarios. The paper evaluates neural, workflow, and agentic systems on rarity-stratified and multi-step MIMIC-IV cases, finding distinct failure modes and partial recovery with structured tools and sequencing guidance.

  • Problem

    Aggregate benchmarks can obscure rare-code performance and provide limited evidence about failure modes in automated ICD coding, which must follow sequencing rules and chapter-specific conventions.

    Source: §1 · p. 1

  • Method

    The paper evaluates neural and agentic systems on rarity-stratified MIMIC-IV cases and tests a controlled agentic framework with structured tools for official coding references.

    Source: §1 · p. 1

  • Results

    Neural classifiers degrade on rare codes, workflows score near zero on injury and external-cause codes, and a tool-augmented agentic framework partially recovers performance on that subset.

    Source: §6 · p. 5

  • Takeaways & Limitations

    The multi-step failure is partially addressable by combining tool-layer discovery grounding with prompt-level sequencing guidance.

    Source: §4 · p. 4, §6 · p. 5

    The evaluation uses MIMIC-IV discharge summaries from one institution, whose code distributions may differ from those at other hospitals.

    Source: §7 · p. 5

Abstract

from arXiv · show

ICD-10-CM codes are alphanumeric codes used in the US to classify diagnoses and injuries for medical billing and epidemiological reporting. Standard ICD-10-CM benchmarks report aggregate metrics that obscure performance on complex coding scenarios. We evaluate neural, workflow, and agentic systems on a rarity-stratified set of MIMIC-IV discharge summaries and identify two orthogonal failure modes. Neural classifiers exhibit a 0.43 micro-F1 gap between rare and common codes. Workflow systems handle rare codes well but score near zero on injury and external cause codes that require multi-step guideline following. A tool-augmented agentic configuration with structured access to official ICD-10-CM reference materials recovers up to 0.34 micro-F1 on this subset. No single system dominates across all conditions.

1 Introduction

Automated ICD coding must handle a large, long-tailed label space and official coding rules, while current analyses leave important failure modes underexamined. This paper evaluates neural and agentic systems across rarity and multi-step complexity, including a controlled agentic framework with structured access to official references.

1 Introduction

  • Standard benchmarks can mask neural classifiers’ degradation on rare codes by oversampling frequent labels in the long-tailed ICD-10-CM space.Accurate coding also requires assigning codes from a vocabulary of over 70,000 labels while following sequencing rules and chapter-specific conventions.Source: §1 · p. 1
  • Fixed agentic workflows struggle with multi-step guideline-following because the needed lookup and consultation order depends on intermediate findings.The relevant steps include index lookup, tabular verification, and guideline consultation.Source: §1 · p. 1
  • The paper evaluates rarity and multi-step coding performance and introduces a controlled multi-turn framework with structured tools for official ICD-10-CM references.Its evaluations identify multi-step guideline-following failure as orthogonal to rarity bias and analyze systems’ complementary strengths and remaining gaps.Source: §1 · p. 1

2 Evaluation Setup

The evaluation defines coding as predicting diagnosis-code sets from discharge summaries and measures performance across rarity and multi-step cases. It compares a neural classifier, a fixed workflow, and a free-form agentic configuration with structured reference tools.

2.1 Task Formulation

  • The task predicts diagnosis-code sets from discharge summaries, where correct assignment requires index lookup, tabular verification, chapter conventions, and care-relevance filtering.Per-bin micro-F1 is computed independently by rarity bin, with true positives and false negatives assigned by ground-truth frequency and false positives by predicted-code frequency.Source: §2.1 · p. 2

2.2 Dataset and Tasks

  • The evaluation stratifies 150 MIMIC-IV notes into 50 each for rare, uncommon, and common codes, and separately tests 153 notes with Chapter 19/20 codes.The multi-step subset has 480 instances; LLM pre-training frequency is unmeasured and may differ from MIMIC-IV training frequency.Source: §2.2 · p. 2

2.3 Evaluated ICD Systems

  • The systems are PLM-ICD, CodeSeeker’s fixed Analyze-Locate-Assign-Verify workflow, and a free-form agent with structured tools for searching, verification, guideline retrieval, and grounded submission.The agent’s tools prevent submission of codes not previously discovered, while companion-code sequencing requirements are provided in the system prompt.Source: §2.3 · pp. 2–3

3 Results

Performance differs across both code rarity and task complexity: the neural classifier leads on common codes, while multi-step results expose weaknesses in the fixed workflow and limits to navigation alone.

3.1 All Codes Performance

  • PLM-ICD scores 0.227 on rare codes versus 0.656 on common codes; CodeSeeker with Claude Opus 4.6 achieves the highest rare-code score at 0.560, while PLM-ICD leads on common codes.The findings are consistent with training-distribution bias in PLM-ICD.Source: §3.1 · p. 3

3.2 Multi-Step Codes Performance

  • CodeSeeker with Claude Opus 4.6 scores 0.010/0.027/0.013 across multi-step rarity bins; the agentic framework has the highest rare and uncommon point estimates, but PLM-ICD outperforms both.The agentic framework’s rare and uncommon point estimates are 0.276 and 0.363, respectively, with wide confidence intervals; the comparison indicates that index traversal without sequential enforcement is insufficient.Source: §3.2 · p. 3

4 Discussion

The analysis locates multi-step coding errors mainly at discovery, while discovery order distinguishes correct from incorrect codes. PLM-ICD’s effects vary by model and code rarity.

4 Discussion

  • p < 10−13 across models, with Cliff’s δ ∈[0.20, 0.30]: correct codes are discovered earlier than incorrect codes after normalizing turn by session length.The authors suggest discovery order is a reliable signal for post-hoc precision improvement.Source: §4 · p. 3, Figure 2 · p. 4
  • Models verify 63K–178K candidate codes per run but submit less than 4%, and only 36–56% of submitted codes are correct; selection is the bottleneck.More tool calls do not translate into higher precision.Source: §4 · p. 3, Table 2 · p. 4
  • 66–86% of missed multi-step codes were never returned by search or browse, especially in Chapter 20; only 0–3% were wrong-7th-character errors.The decomposition traces misses through discovery, verification, submission, and specificity by chapter; 74–95% of false positives were verified and submitted.Source: Table 3 · p. 4, §4 · p. 4
  • The multi-step gain over CodeSeeker is attributed to External Cause Index access and prompt-level guidance on companion-code sequencing, both targeting discovery.Source: §4 · p. 4
  • PLM-ICD hurts Claude Opus 4.6 on rare codes but helps GPT-5.2; on multi-step codes it hurts both models on rare and uncommon codes and gives no gain on common codes.Source: Figure 3 · p. 4, §4 · p. 4

5 Ablation Studies

The ablations test the roles of prompt-level sequencing, external tools, and discovery grounding. Tool access improves performance, while the effect of sequencing varies by model and code frequency.

Prompt-level Sequencing Instructions

  • Removing the external-cause sequencing instruction leaves Opus 4.6 nearly unchanged, but GPT-5.2 common-code F1 drops from .322 to .091.Other GPT-5.2 rarity bins show the same pattern as Opus 4.6, with rare and uncommon results remaining similar.Source: §5 · p. 4, Table 4 · p. 5
  • The sequencing ablation removes only the mandatory external-cause instruction while keeping the tool pipeline, discovery gating, budgets, and other settings fixed.The Opus 4.6 F1 differences are at most .03 across rarity bins, with overlapping confidence intervals.Source: §5 · p. 4

External Tool Access

  • Without tools, rare-code F1 roughly halves across models, while Opus 4.6 common multi-step F1 falls from .280 to .050.Recall on common multi-step codes falls to .021–.028 without tools across all three models.Source: §5 · p. 5, Table 5 · p. 5

Discovery Grounding

  • Discovery gating was not ablated because removing it would re-admit hallucinated codes rather than isolate multi-step navigation.False-positive analysis finds 74–95% of false positives were verified and then submitted, consistent with a hallucination-filter role.Source: §5 · p. 5

6 Conclusion

The evaluation identifies rarity bias in neural classifiers and multi-step guideline-following failures in workflow systems. Tool-augmented agents partially recover performance on multi-step codes, but no system dominates every condition.

6 Conclusion

  • Neural classifiers degrade on rare codes, while workflow systems score near zero on injury and external-cause codes despite strong rare-code performance.The tool-augmented agentic framework combines structured guideline access with explicit prompt-level sequencing and partially recovers performance on this subset.Source: §6 · p. 5

7 Limitations

The evaluation is limited by its single-institution MIMIC-IV data, narrow multi-step diagnostic scope, and unisolated components of the agentic framework.

7 Limitations

  • The evaluation uses MIMIC-IV discharge summaries from one institution, whose code distributions may differ from those at other hospitals.MIMIC-IV labels can include coded conditions absent from notes, adding uncertainty to absolute F1; fixed model versions may also differ from newer releases.Source: §7 · p. 5
  • The multi-step diagnostic evaluation covers only ICD-10-CM Chapters 19 and 20, so it does not establish whether the patterns hold in other categories.The subset contains 153 notes and 480 code instances; the workflow-versus-agentic gap is large relative to this sample, while adjacent agentic systems are not statistically distinguishable.Source: §7 · p. 5
  • The study does not isolate discovery grounding or individual tools, and its tool ablation removes the entire environment.Fine-grained attribution, including the contribution of tools such as the External Cause Index, remains for future work.Source: §7 · p. 5

8 Ethics Statement

The section states an ethical boundary for use, situates the work among neural and LLM-based coding systems, and reports agent-loop settings and its tool environment.

8 Ethics Statement

  • Model outputs are not intended for direct clinical use, and code assignments require human expert review before deployment.The clinical notes come from MIMIC-IV under its established data use agreement; no new patient data was collected.Source: §8 · p. 6

A Related Work

  • Prior ICD-coding work includes neural extreme multi-label classifiers and LLM systems, including CLH's fixed Analyze-Locate-Assign-Verify workflow.CLH is characterized as a workflow because its stage order is predetermined, constraining re-consultation of guidelines or the Alphabetical Index for interdependent codes.Source: A Related Work · p. 7

B.1 Hyperparameters

  • Agentic configurations share limits of 200 turns, 200 tool calls, and 600 seconds per sample, with context compression triggered at 80% utilization.Compression preserves the system prompt, first user message, and five latest tool results; older results become one-line summaries.Source: Table 6 · p. 7, §B.1 · p. 7

B.2 Treatment of Failed Sessions

  • On timeout, model error, or stuck-loop detection, the harness submits accumulated codes and scores them under the same protocol..Source: §B.2 · p. 7

B.3 Tool Environment

  • The agent has nine tools across four categories, backed by CMS 2020 ICD-10-CM materials including the Tabular List, indexes, tables, and guidelines.Navigation tools populate the discovery set; verification and submission are gated to codes in that set.Source: §B.3 · p. 7, Table 7 · p. 7

C Full Tool-Access Ablation

The ablation reports precision, recall, and micro-F1 by rarity band, alongside a separate summary of resource usage. The supplied passages describe the metrics but do not give the ablation or resource-usage values.

C Full Tool-Access Ablation

  • The full tool-access ablation reports precision, recall, and micro-F1, with ground-truth code-instance counts per band and 95% bootstrap confidence intervals.The supplied passage does not include the metric values.Source: C Full Tool-Access Ablation · p. 8

D Resource Usage

  • Resource usage is summarized by wall-clock latency, estimated tokens per session, tool calls, voluntary submission rate, and failure rate.Failures combine timeouts, API errors, and stuck detection; no session reached the 200-turn or 200-tool-call limit.Source: D Resource Usage · p. 8, Table 10 · p. 8

E Discovery-Position Statistical Test

The statistical test compares where true-positive and false-positive codes are discovered across models. It reports sample counts, position differences, effect sizes, and one-sided test results.

E Discovery-Position Statistical Test

F Case Study: End-to-End Agentic Coding

The case study traces an agent’s coding decisions from index lookup through guideline consultation and exclusions to submission. The system prompt specifies tool-based verification and coding rules, including mandatory sequencing for external-cause cases.

F Case Study: End-to-End Agentic Coding

  • For the appendicitis-versus-colitis diagnosis, the agent used the index and tabular list, then applied the inpatient uncertain-diagnosis guideline to code both conditions as confirmed.It verified K35.80 rather than K35.2 or K35.3 because peritonitis was not documented.Source: F Case Study: End-to-End Agentic Coding · p. 8
  • The case trajectory excluded reactive lymph nodes and elevated white blood cells as findings integral to inflammation, and excluded undocumented nicotine history.OSA was included as a chronic past-medical-history condition.Source: F Case Study: End-to-End Agentic Coding · p. 8
  • The agent submitted {K52.9, K35.80, G47.33}, achieving precision 0.667, recall 1.000, and F1 0.800; both ground-truth codes were found in 14 tool calls across 10 turns.The false positive was OSA from past medical history; the authors describe the coding as defensible under inpatient conventions, while annotators excluded PMH-only conditions not actively managed.Source: F Case Study: End-to-End Agentic Coding · p. 8

G System Prompt

  • The prompt directs the agent to use reference tools rather than memorized codes, verify every candidate in the index and tabular list, and reread the note for missed conditions.The PLM-ICD priming step can be omitted in no-PLM configurations while coding rules remain identical.Source: G System Prompt · pp. 9–10
  • For injury or T-codes, the prompt requires cause/event, place-of-occurrence, and matching seventh-character sequencing, with Y92.9 used when the place is unstated.It also requires checking companion-code instructions such as “Use additional code,” “Code first,” and “Code also.”Source: G System Prompt · p. 9
  • Submission filters restrict coding to conditions that affected care during the encounter, excluding integral symptoms and incidental imaging findings not acted upon.The prompt also specifies rules for specificity, combination codes, and presumed relationships under “with” or “and” conventions.Source: G System Prompt · pp. 9–10
Loading 2609.13806v1…