Source-linked AI summary
Configurable Clinical Information Extraction with Agentic RAG: What Works, What Breaks, and Why
Osman Alperen Çinar-Koraş, Marie Bauer, Sameh Khattab, Merlin Engelke, Moon Kim, Stephan Settelmeier, Shigeyasu Sugawara, Fabian Freisleben, Felix Nensa, Jens Kleesiek
TL;DR
Clinical information extraction must handle incomplete metadata, interdependent records, and temporal conflicts across large patient contexts. This paper deploys an on-premise agentic RAG pipeline that grounds extracted values for clinician verification, achieving 96.5% clinician acceptance across 7,326 judgments.
Problem
Clinical information extraction lacks reliable document-level metadata for retrieving and temporally ordering information across heterogeneous, interdependent patient records.
Method
ACIE uses clinician-configured schemas and an on-premise agentic RAG pipeline that reasons over complete patient contexts and grounds every extracted value in source passages.
Results
96.5% of 7,326 clinician judgments accepted the extracted values, with 96.4% precision when the system produced a value and 96.8% correct abstention when it returned none.
Takeaways & Limitations
Grounded extraction shifts clinicians from compiling information to verifying it, while enabling batch processing outside working hours and roughly three-times-faster patient completion.
Takeaways & Limitations
Evaluation covered one retrospective study in one disease area, hospital, language, and 99 patients, so generalization to other settings remains untested.
Abstract
from arXiv · showhide
Patient contexts span hundreds of heterogeneous documents and thousands of structured data points, yet the document-level metadata that AI systems need for retrieval and triage is absent or incomplete. Standard retrieval-augmented generation fails on this data, mishandling temporal reasoning, cross-document dependencies, and missing metadata. We deploy ACIE (Agentic Clinical Information Extraction) at University Medicine Essen: an on-premise agentic RAG pipeline that reasons over complete patient contexts and grounds every answer in source passages for clinician verification. We quantify the metadata gap, trace the architectural decisions it shaped, and evaluate extraction alongside an independent retrospective lymphoma registry study, in which nuclear-medicine physicians verify every extracted value against its cited sources. Across 7,326 judgments, clinicians accepted 96.5\% of extractions, with per-type acceptance ranging from 80\% to 99\%.
1 Introduction
Clinical workflows require laborious reconstruction of structured patient data across thousands of heterogeneous records, while clinical IE remains difficult to deploy because adapting systems requires developer effort and external data transmission raises privacy and regulatory concerns. ACIE addresses these challenges with clinician-configured, on-premise agentic extraction that reasons over complete patient contexts and grounds every value in source passages for verification.
- Clinical information burden: Clinical workflows compile data across thousands of documents and tens of thousands of structured data points spanning multiple hospital systems.Lymphoma study enrollment may require reconstructing treatment histories and locating diagnostic markers across years of duplicated, misdated, or unrelated records.
- Deployment barriers: Recent clinical information-extraction systems still require developer effort for new workflows, while LLM-based clinical IE remains largely confined to research evaluations.The introduction identifies external transmission of patient data as an additional privacy and regulatory barrier.
- ACIE system: Nearly 2 billion FHIR resources support ACIE’s on-premise deployment at University Medicine Essen, where clinicians define typed extraction targets without developer involvement.Its agentic RAG pipeline reasons over complete patient contexts and grounds every extracted value in source passages for clinician verification.
- Evaluation: 7,326 judgments evaluate 74 clinician-configured fields across 99 patients in a clinician-verified agentic extraction and retrospective lymphoma registry study.Nuclear-medicine physicians accept or reject every extracted value and classify rejections using structured error and editorial categories.
- Contributions: ACIE’s study quantifies the metadata gap between AI requirements and clinical exports and examines architectural decisions shaped by real clinical data.These analyses are presented as contributions focused on the data reality and design trade-offs of building clinical systems.
2 Related Work
Clinical information extraction has progressed from developer-authored rules to domain-specific pretraining and few-shot LLMs, but deployment remains limited and targets are often static. Agentic RAG introduces iterative reasoning for complex clinical information needs, while recent clinical systems still leave structured extraction challenges open.
- From rules to domain-specific pretraining: Early systems shifted from developer-defined extraction targets toward expert-configured schemas, but Knowledge Author represented only 76% of concepts fully, with recall as low as 46%.cTAKES used engineered NLP pipelines, whereas Knowledge Author allowed domain experts to define schemas through a web interface; its rule-based backend limited expressiveness.
- LLMs shift what is possible but remain largely undeployed: LLMs enabled few-shot clinical extraction without task-specific training, yet clinical IE has rarely progressed beyond research evaluations.Local Llama 2 demonstrated on-premise feasibility for five fixed features from MIMIC-IV histories, but the targets were static and researcher-defined.
- LLMs shift what is possible but remain largely undeployed: On-premise LLM deployments have demonstrated clinical assistance, but the reported Epic and German university-hospital systems do not perform structured extraction.The Epic deployment involved 1,028 users; the German university-hospital system used an on-premise LLM.
- Agentic RAG as the emerging frontier: Agentic RAG combines evidence-grounded generation with iterative reasoning, and i-MedRAG outperformed single-pass RAG for medical question answering using a fixed iteration schedule on curated knowledge bases.ReAct exemplifies iterative reasoning over complex information needs, while agentic clinical information extraction has recently emerged.
3 System Overview
ACIE organizes complete patient contexts from heterogeneous clinical documents and structured FHIR data, then uses configurable schemas and an agentic search process to produce source-grounded extractions. Its retrieval, citation, and deployment design addresses unreliable metadata while keeping clinician verification central.
- Patient context: A patient context combines years of heterogeneous clinical documents with structured FHIR data points accumulated during care.Documents include discharge letters, radiology reports, laboratory findings, referral letters, and operative notes; structured data include laboratory results, medications, conditions, and observations.
- Data processing: ACIE ingests hospital data through FHIR, OCRs non-machine-readable documents, excludes low-quality scans, and creates coarse retrieval chunks plus fine-grained citation passages.The two granularities preserve surrounding context for retrieval while supporting atomic source citation across narrative, tabular, and form-like layouts.
- Retrieval: The length-penalized retrieval score dampens short-fragment bias, with τ = 40 and blend weights fixed on a development subset across tasks.Dense retrievers systematically favor short, low-information fragments produced by serializing heterogeneous clinical documents.
- Configuration: Clinicians configure typed extraction schemas, allowing the same system to support multiple use cases without code changes.The listed use cases are pre-procedure protocols, retrospective study data collection, and clinical documentation.
- Agentic extraction: For each target, a tool-calling agent searches the full patient context, iteratively gathers evidence, and returns every value attributed to specific source passages for clinician acceptance or rejection.Metadata-based filters proved insufficient because required metadata was unreliable or absent; clinicians review cited passages before values enter clinical documentation.
- Deployment: ACIE runs on-premise as a Kubernetes web application, keeping patient data within the hospital network while serving extraction on 4×H100 GPUs and OCR on one H100 GPU.The extraction model is Qwen 3.6 35B-A3B and scanned documents use PaddleOCR-VL 1.5.
4 Evaluation
ACIE was evaluated on the fragmented, poorly annotated, and temporally unreliable clinical data of 10,000 patients, alongside an independent lymphoma registry study. Clinicians accepted 96.5% of 7,326 extracted values, with most errors concentrated in date and tabular fields.
- Clinical data challenges: 10,000 patients contributed approximately 1.2M deduplicated documents for characterizing the challenges of automated clinical extraction.Patient contexts ranged from 1 to over 2,500 documents and from 0 to over 119,000 structured data points.
- Clinical data challenges: No document-level summaries existed for retrieval, document relationships appeared on only 0.52% of documents, and authorship metadata appeared for 1.9%.Subtype metadata appeared for 41.87%, while OCR rejection reached 52.0% for the worst patient and 10.3% at the median.
- Clinical data challenges: 56.5% of documents carried timestamps outside their linked encounter period, while same-day agreement with OCR- and LLM-extracted dates was only 58.8%.The encounter timeframe was absent from the export, requiring a priority cascade over available timestamps; 36.5% diverged by more than one day.
- Extraction evaluation: 84.3% of date fields and 79.8% of tabular fields were accepted, compared with 96.0–98.6% for categorical, numerical, Boolean, and free-text fields.Among non-empty tables, acceptance was 71.2%; tabular errors involved assembling multi-row timelines, while date errors often selected the wrong clinical event.
- Extraction evaluation: 241 of 253 rejections were extraction errors, while 3 were editorial adjustments and 9 were form configuration issues.Excluding configuration issues, extraction-attributable acceptance was 96.7%; 161 rejected values corrected produced values and 92 supplied values left empty.
5 Lessons from Deployment
Deployment revealed that clinical data quality falls short of AI requirements, especially because document-level metadata is sparse or absent. These gaps directly shaped ACIE’s architecture, including agentic content-based retrieval, query-relevant document summaries, and Markdown-based serialization.
- L1: Clinical data quality falls far short of what AI systems require.: Clinical data quality falls far short of what AI systems require, with document-level metadata sparse or absent even in a large-scale clinical FHIR repository.Missing or sparsely populated fields include encounter periods, document relationships, and authorship.
- L2: Architectural decisions shaped by data.: Data quality gaps shaped three design decisions in ACIE’s architecture.The decisions addressed retrieval, document triage, and input serialization.
- L2: Architectural decisions shaped by data.: Unreliable or absent metadata led ACIE to replace static filtering with an agent that reasons about relevant documents from content.Initial filters included encounter scope, date range, and category, but exhausting static combinations did not yield reliable retrieval.
- L2: Architectural decisions shaped by data.: Query-relevant summaries let the agent triage hundreds of documents without reading every document in full.Fine-grained chunking also creates fragments with orders-of-magnitude length variation that dense retrievers systematically favor when short.
- L2: Architectural decisions shaped by data.: Patient-specific malformed tool calls occurred deterministically when heterogeneous clinical metadata was serialized as JSON, motivating Markdown over JSON serialization.The paper relates this failure to input-side serialization effects on extraction and reasoning.
6 Conclusion
Deploying ACIE showed that incomplete and unreliable document-level metadata dictates the architecture, requiring reasoning within the retrieval loop. Source-grounded extraction shifts clinicians from compiling information to verifying it and enabled roughly three-times-faster patient completion.
- Architectural implications: Incomplete, unreliable, or unpropagated document-level metadata prevented structural retrieval filtering and forced reasoning into the retrieval loop.This remained true even at a site with large-scale FHIR integration.
- Clinical workflow: Grounding every extracted value in source passages shifts clinicians’ role from compiling information to verifying it.The cited passages support clinician verification of each extracted value.
- Clinical workflow: Roughly three times faster completion per patient was reported by study physicians when extraction ran in batches outside working hours.The workflow combines batch processing with source-grounded verification.
Limitations
The evaluation’s generalizability remains untested because it covers one retrospective study, disease area, hospital, language, and 99 patients, with single-reviewer grading and a more permissive setting than point-of-care use.
- Generalizability: 99 patients were studied at one hospital, in one disease area and language, leaving generalization to other settings untested.The evaluation was a single retrospective study.
- Evaluation design: Each field was graded by a single expert reviewer, so inter-rater reliability was not measured.
- Clinical deployment: The retrospective setting is more permissive than point-of-care use, where extracted values directly inform interventions.
Ethical Considerations · A Clinical Study Extraction Schema · B Rejection Categories
The paper frames ACIE as an on-premise, clinician-verified assistive system, defines a lymphoma registry extraction schema spanning the disease trajectory, and classifies rejected values by error, editorial, or configuration problems.
- Ethical Considerations: ACIE keeps patient data within the hospital network and requires clinician verification of every value against cited passages before documentation.Mandatory source review and the evaluated rejection workflow are safeguards against, but do not eliminate, automation bias.
- Ethical Considerations: The retrospective registry study followed applicable institutional and regulatory requirements at University Medicine Essen.
- Ethical Considerations: AI assistance was limited to language editing and LaTeX formatting, while the authors supplied and assumed responsibility for all research content and claims.
- A Clinical Study Extraction Schema: The lymphoma registry schema contains 74 AI-extracted fields, while four demographic fields read directly from FHIR are excluded from evaluation.Two nuclear-medicine physicians and a lymphoma hematologist designed the eCRF to cover the full disease trajectory.
- B Rejection Categories: Reviewers assign each rejected extraction a problem category organized into extraction errors, editorial adjustments, or form-configuration issues.
- B Rejection Categories: Extraction errors include incorrect, fully incorrect, outdated, missed, extraneous, hallucinated, and missing-reference values.These categories distinguish unsupported or absent information from values that are wrong despite attribution to a cited passage.
- B Rejection Categories: Editorial adjustments cover omitted, excessive, or incorrectly formatted information, whereas configuration errors reflect field options or definitions that do not match clinical reality.
C Error Analysis
The error analysis decomposes 253 rejections by field type, return-versus-abstention behavior, and rejection category. Errors are concentrated in a small subset of fields, while tabular fields and dates fail in opposite modes.
- Error analysis: The appendix organizes the 253 rejections across field-type acceptance, rejection-category placement, and error concentration.Tables 6, 7, and 8 provide these three breakdowns.
- Return-versus-abstention behavior: 71.2% of tabular-field failures occurred when the system produced a value, while abstentions were reliable.Table 6 separates acceptance by returned-value precision and abstention reliability.
- Return-versus-abstention behavior: 69.8% of date-field failures involved unreliable abstention, the reverse pattern from tabular fields.All other field types performed strongly in both return and abstention modes.
- Error concentration: 182 of the 253 rejections came from ten of the 74 fields, showing strong concentration of errors.The appendix uses Table 8 to quantify this concentration.
D Per-Patient Context Distributions … G Metadata and Timestamp Population
Patient contexts are heavily right-skewed, with extreme document and resource loads, dispersed encounter coverage, and corrupt timestamp maxima complicating agentic processing. Metadata and timestamp population rates quantify the sparsity underlying the retrieval challenge.
- D Per-Patient Context Distributions: Per-patient context statistics are consistently right-skewed, with means exceeding medians across all dimensions.The P1/P99 interval covers the central 98% of patients, while the P99-to-Max gap identifies the hardest cases.
- D Per-Patient Context Distributions: Date and table fields dominate the 253 rejections, while configuration errors occur only in categorical and Boolean fields.The ten most-rejected fields account for 182 rejections; dates are almost entirely incorrect-value errors.
- D Per-Patient Context Distributions: At least 937 documents and 37,074 structured resources occur in the top 1% of patients, reaching maxima of 2,542 and 119,191.History length reaches 739,726 days because of corrupt timestamps, whereas P99 is 9,775 days, approximately 26.8 years.
- E Encounter Coverage by Patient Complexity: For patients with 20+ case-level encounters, median top-encounter document coverage falls to 14.7%, while P99 coverage remains 53.5%.Patients with at most five encounters typically have all documents in one encounter, indicating increasing dispersion with complexity.
- F Patient History Length: Document counts and total FHIR resources by patient history length both show heavy right skew, with P99-to-Max gaps highlighting extreme cases.Tables 12 and 13 use this gap to characterize the cases the system must handle.
- G Metadata and Timestamp Population: Document-level metadata and timestamp population rates are reported for the two FHIR resource types carrying clinical documents.These rates quantify the metadata sparsity discussed in the clinical information-retrieval setting.
H Timestamp Cross-Validation
Across 15,142 documents, FHIR metadata timestamps aligned with OCR- and LLM-extracted clinical dates on only 58.8% of documents. Alignment stayed roughly 59% across resolved-date fields, indicating that no single metadata timestamp reliably captured clinical activity timing.
- Timestamp alignment: Roughly 59% alignment persisted regardless of which FHIR field supplied the resolved date.The comparison used the FHIR metadata timestamp resolved for each document against the clinical date extracted from its content.
- Timestamp alignment: No single metadata timestamp reliably represented when clinical activity occurred.This conclusion follows from the consistent alignment level across the alternative FHIR timestamp fields.
- Timestamp alignment: 58.8% overall same-day agreement was observed between FHIR metadata timestamps and clinical dates extracted from document content across 15,142 documents.Clinical dates were extracted using OCR and an LLM.