PaperlayerView exact revision on arXiv

Source-linked AI summary

Autonomous Structuring of Radiology Reports Across Modalities at Archive Scale Using an Open-Weight Large Language Model

Friedrich Puttkammer, Fabian Drexel, Marlene Fritzsche, Era Stambollxhiu, Miriam Kumpf, Lena Schmitzer, Lea Schumann, Lina Xu, Johannes Moll, Jannik Lübberstedt, Zeineb Ben Chaaben, Anirudh Narayanan, Hartmut Häntze, Renato Cuocolo, Antonios Billis, Alexander Löser, Jawed Nawabi, Marcus R. Makowski, Cosmin I. Bercea, Shahrooz Faghihroohi, Lisa C. Adams, Keno K. Bressem

arXiv:2610.04541v1cs.AIcs.CL

TL;DR

Prior LLM studies generally assumed a known template, although archives contain many examination types and metadata may not identify the needed template. This study applied a local open-weight LLM pipeline to a complete report archive; radiologists flagged unsupported content in 1.0% and 1.5% of corrected reports.

  • Problem

    Most prior LLM studies assumed a known template, while archives span hundreds of examination types and metadata may not identify the needed template.

    Source: Introduction · p. 2

  • Method

    A local open-weight LLM pipeline autonomously selects among 150 templates, rejects reports that fit none, and fills the selected templates.

    Source: Introduction · p. 2, Discussion · p. 9

  • Results

    2.1 million free-text reports across five modalities and sixteen years were structured; radiologists flagged unsupported content in 1.0% and 1.5% of corrected reports.

    Source: Discussion · p. 9

  • Takeaways & Limitations

    The structured archive makes sixteen years of imaging findings available for cohort identification, quality control, and training imaging models.

    Source: Discussion · p. 10

    Multi-region reports, 16% of the selection test set, accounted for 44% of selection errors and remain the main open template-selection problem.

    Source: Discussion · p. 9

Abstract

from arXiv · show

Purpose: To develop and evaluate an open-weight large language model (LLM) pipeline that converts an entire archive of free-text radiology reports into structured reports without human oversight. Materials and Methods: In this retrospective study, a pipeline with 150 hierarchically organized templates was developed at one center and tested at a second center on reports from 2010 to 2025. The open-weight model gpt-oss-120B selects the template in three constrained-decoding steps and fills it on one local graphics processing unit. Template selection was scored against expert labels on 914 randomly sampled reports of five modalities, structuring quality on 920 radiography and CT reports corrected field by field by five residents. The pipeline then processed the complete archive of the second center. Proportions are reported with Wilson 95% confidence intervals (CIs). Results: An optimal template set was selected for 74.4% of reports (680 of 914; 95% CI: 71.5%, 77.1%) and an appropriate set for 82.3% (752 of 914; 95% CI: 79.7%, 84.6%), 87.7% for single-region and 54.1% for multi-region reports. Macro semantic textual similarity between output and corrected reference was 0.95 for radiography and 0.97 for CT, residents left 88.7% of 24,638 fields unchanged, and unsupported content was flagged in 1.0% and 1.5% of reports. Of 2,186,982 archive reports, 96.5% received structured output, 2,401,544 structured reports, at 1,258 reports per hour on one graphics processing unit. Conclusion: An open-weight LLM pipeline structured a complete multimodality report archive without human oversight with high content fidelity. Multi-region reports remained the main source of template errors.

Introduction

Free-text radiology archives are difficult to evaluate automatically, and post-dictation structuring could make them usable without changing dictation workflows. Because archive-scale processing requires autonomous template selection, this study develops and tests a pipeline designed to address that need.

Introduction

  • Post-dictation structuring leaves the reporting workflow untouched and makes archived reports usable for automated evaluation.Radiology societies have recommended structured reporting, but longer reporting times, template disagreements, and disrupted dictation workflows have slowed adoption.Source: Introduction · p. 2
  • Most prior LLM studies assumed a known template, while archives span hundreds of examination types and metadata may not identify the needed template.Scaling to a whole archive therefore requires a fully autonomous structuring pipeline.Source: Introduction · p. 2
  • A whole-archive system must cover examinations consistently, handle large or unusual template choices and multi-examination reports, adapt across institutions, and protect sensitive data from costly commercial APIs.These four challenges motivate a locally run pipeline with adaptable templates and no human oversight.Source: Introduction · p. 2

Materials and Methods

The study developed templates at Charité and evaluated autonomous selection and structuring on TUM reports, then processed TUM’s complete archive. Evaluation used expert labels and resident-corrected reports, with field-level metrics and archive throughput assessed separately.

Study Design

  • Reports from Charité informed template development, while TUM reports from 2010–2025 supplied the test sets and complete-archive processing.The retrospective study used German reports; patient demographics were unavailable at report level.Source: Study Design · p. 2

Template Database

  • Radiologists defined 150 templates organized as base and derived templates, covering five modalities and 37 examination types.Derived templates add or replace base-template sections; 31 CT combination types specify examination types that may occur together, with one template assigned per examination type.Source: Template Database · p. 2

Pipeline

  • The pipeline selects modality, examination type, and template in successive constrained steps, then fills each selected template in a separate call.It switches to multi-label selection for reports describing several examinations and rejects stubs or examinations without a template; fields without source information receive “not mentioned”.Source: Pipeline · p. 3, Figure 2 · p. 4

Model

  • The same 4-bit quantized gpt-oss-120B model, with reasoning effort set to its lowest level, ran on one H100 GPU during development and one H200 GPU for testing and archive processing.The model had 117 billion total and 5.1 billion active parameters.Source: Model · p. 3

Test Datasets

  • The selection test set contained 914 reports, while the structuring reference comprised 920 appropriate radiography and CT reports corrected field by field by residents.Selection labels were reviewed by a second radiologist; unclear template ratings were adjudicated by two radiologists. The corrected reports covered 70 templates, sampled at 9–25 reports per template.Source: Test Datasets · pp. 3–4, Figure 1 · p. 3

Evaluation Metrics and Statistical Analysis

  • Template selection was scored per report as optimal when the selected set matched the expert label and appropriate when every selected template shared its base template with the labeled one.Structuring quality was scored field by field against the corrected reference; unchanged fields counted as correct, and semantic textual similarity and BLEU were macro-averaged over templates.Source: Evaluation Metrics and Statistical… · p. 5

Archive Processing

  • Archive processing counted a report as structured when it returned at least one template with valid JSON; throughput was measured on one H200 GPU.Rows without valid identifiers and imported-examination header rows were excluded before processing; throughput was also measured in a serial benchmark of 100 reports run twice.Source: Archive Processing · p. 5

Results

Across the external tests, template selection was less successful for multi-region reports, while structured output closely matched resident-corrected references. The pipeline produced structured output for most eligible reports in the complete archive.

Template Selection

Structuring Quality

  • Selected templates were rated appropriate for 87.5% of radiography reports and 93.1% of CT reports.The ratings include 41 of 44 initially unclear ratings adjudicated in consensus.Source: Structuring Quality · p. 5
  • Macro STS was 0.95 for radiography and 0.97 for CT, and residents left 88.7% of 24,638 scored fields unchanged.Table 3 and Figure 4 report agreement against the resident-corrected reference.Source: Structuring Quality · p. 6, Table 3 · p. 6, Figure 4 · p. 7

Hallucinations and Error Types

  • Unsupported content was flagged in 1.0% of radiography and 1.5% of CT reports; incomplete transcription dominated the lowest-STS cases.Among the 50 lowest-STS reports per modality, incomplete transcription was the primary error in 31 radiography and 32 CT reports.Source: Hallucinations and Error Types · p. 7, Figure 5 · p. 7

Archive-Scale Processing

Discussion

The open-weight pipeline structured a complete 2.1-million-report archive locally and without human oversight, with high content fidelity. Multi-region reports remained the main template-selection problem, while language and modality coverage limit the evaluation.

Discussion

  • An open-weight LLM converted a university hospital’s complete archive—2.1 million free-text reports across five modalities and sixteen years—into structured reports on local GPU infrastructure without a human in the loop.It chose among 150 templates, rejected reports that fit none, and filled chosen templates with content that radiology at the second center left largely unchanged.Source: Discussion · p. 9
  • At 1,258 reports per hour, local processing was estimated at about 1,680 GPU hours; equivalent API use was estimated at $48,000 mid-tier or $120,000 flagship.The authors conclude that local deployment favors both cost and privacy at archive scale.Source: Discussion · p. 9
  • Multi-region reports were 16% of the selection set but accounted for 44% of selection errors, making template selection for them the main open design problem.The templates center on one primary disease, while reports with several diseases may require a set of templates.Source: Discussion · p. 9
  • Base sections shared across diagnosis-specific templates give archive queries consistent field names, supporting comparability across reports.The paper links this design to common data elements, which define each element once for reuse across templates.Source: Discussion · p. 9
  • Unsupported content was flagged in 1.0% of radiography and 1.5% of CT corrected reports; all flagged reports were below their modality’s 30th STS percentile.The authors suggest embedding-based similarity could direct human review toward reports needing it; incomplete transcription was the primary error in about two thirds of the lowest-scoring reports.Source: Discussion · p. 9
  • Evaluation used German reports, assessed field-level structuring only for radiography and CT, and tested template selection on 914 reports spread across 150 templates.The authors note that English archives may have better results, while MRI, ultrasound, and interventional radiology remain untested at field level; accuracy is imprecise for templates with few test reports.Source: Discussion · p. 9
  • The structured archive makes sixteen years of imaging findings available for cohort identification, quality control, and imaging-model training.The authors state that the template hierarchy lets other institutions adapt the templates rather than the method.Source: Discussion · p. 10

Declarations

The declarations report the study’s funding, conflicts, data and code availability, AI use, and abbreviations.

Funding

  • The study received European Union funding through COMFORT and TWIN-X; the funder had no role in study design, data collection, analysis, or writing.The authors state that the views expressed do not necessarily reflect those of the European Union or HADEA.Source: Funding · p. 10

Conflicts of interest

  • Several authors report research or commercial relationships and European Union or BIH grant funding.Reported relationships include research collaboration agreements, speaker fees, and consultant fees.Source: Conflicts of interest · p. 10

Data and code availability

  • Report texts cannot be shared because they contain protected health information; the pipeline, German-language template definitions, and template index are available online.The repository is identified as RadFields.Source: Data and code availability · p. 10

Use of artificial intelligence

  • The authors used Claude for drafting, editing, analysis and figure code, and supplement assembly, and Codex to prepare the LaTeX preprint.The authors reviewed and verified all content and take full responsibility for it.Source: Use of artificial intelligence · p. 10

Abbreviations

  • The paper defines abbreviations including LLM, STS, BLEU, CI, IQR, GPU, and API.STS means semantic textual similarity; GPU means graphics processing unit.Source: Abbreviations · p. 10

Supplementary material

The supplement details template formats and hierarchy, constrained selection and structuring pathways, and supporting tables and figures.

Supplemental Methods S1. Template development and format

  • Each YAML template maps report sections to field types, generates a Pydantic response model for constrained decoding, and uses “not mentioned” for absent source information.Strings describe expected content, mappings define subsections, and lists support multiple fields or repeatable sections.Source: Supplemental Methods S1. Template… · p. 12

Supplemental Methods S2. Template index, inheritance, and combination types

  • The template index records each template’s name, description, file path, and hierarchy position; base and derived templates share sections, while CT combination types select templates for constituent examinations.At runtime, derived templates update base templates; the index and files can be edited without changing pipeline code.Source: Supplemental Methods S2. Template… · p. 12

Supplemental Methods S3. Template selection pathways and constrained decoding

  • Classification uses constrained decoding over allowed options, with reserved responses for no findings, no matching template, and multiple examinations.Multi-exam reports proceed through multi-label examination and template selection; combination-type CT reports select a template for each constituent examination type.Source: Supplemental Methods S3. Template… · p. 12, Figure S2 · p. 21

Supplemental tables

Supplemental figures

  • Supplementary tables also cover selection ablations and templates used in the archive; figures depict the CT template index, hierarchical selection, and runtime template composition.The composition figure shows derived-template values replacing base values and derived-only sections being added.Source: Supplemental tables · pp. 16, 19, Figure S1 · p. 20, Figure S2 · p. 21

Supplement 1. CLAIM 2024 checklist

The CLAIM 2024 checklist covers 44 items; 35 are reported, one is unreported because demographic characteristics were unavailable at report level, and eight are not applicable.

Supplement 1. CLAIM 2024 checklist

  • 35 of 44 CLAIM items are reported, one is unreported because report-level demographics were unavailable, and eight are not applicable.The checklist identifies item 32 as the unreported item.Source: Supplement 1. CLAIM 2024 checklist · p. 23
  • The checklist reports software and libraries, including Python, LangChain 1.2.18, vLLM, Ollama, Llama-Embed-Nemotron-8B, and SciPy.Source: Supplement 1. CLAIM 2024 checklist · p. 24
  • Statistical analysis used Wilson 95% confidence intervals, percentile bootstrap with 10,000 resamples, exact McNemar tests, and P < .05.Source: Supplement 1. CLAIM 2024 checklist · p. 24
  • The checklist points to Figure 1 for test-set flow and Figure 6A for archive flow; performance is reported in Tables 2–4 and Tables 2–3 and Figures 3–5, respectively.Source: Supplement 1. CLAIM 2024 checklist · p. 24
  • The checklist marks study limitations and safety and bias considerations as reported, including hallucination and omission rates, review flags for low-similarity outputs, and language dependence.Demographic subgroups were not analyzed.Source: Supplement 1. CLAIM 2024 checklist · p. 24
  • Conflicts of interest and data sharing are reported; pipeline materials and evaluation scripts are available on GitHub, but report texts are not shareable.Source: Supplement 1. CLAIM 2024 checklist · p. 24
Loading 2610.04541v1…