Source-linked AI summary
Multi-Agent AI System for Radiology Report Structuring and Quality Assurance with Independent Radiologist Evaluation
Iryna Hartsock, Cesar Lam, Christopher Otteni, Aliya Qayyum, Robert Gatenby, Cyrillo Araujo, Ghulam Rasool
TL;DR
Radiology reports vary in organization and can contain clinically relevant inconsistencies, motivating automated structuring and quality assurance. This study developed a locally deployed multi-agent system that preserved report text while performing both tasks, with favorable radiologist evaluations and QA rated excellent or good in 84% of reviewed reports.
Problem
Variable report organization and reporting discrepancies can complicate clinical communication and limit consistent quality monitoring and secondary use.
Method
A locally deployed multi-agent pipeline assigned report sentences to standardized anatomical sections while performing automated quality assurance for clinically relevant inconsistencies.
Results
Radiologist review rated overall QA performance as excellent or good in 38 of 45 reports (84%), with favorable performance across structuring and QA tasks.
Takeaways & Limitations
Integrating report structuring and quality assurance in one workflow may support standardized reporting while preserving original content and accommodating varied writing styles.
Takeaways & Limitations
Evaluation was limited to 45 reports, an intentionally QA-flag-enriched subset that was not designed to provide unbiased estimates of overall QA performance.
Abstract
from arXiv · showhide
Purpose: To develop and evaluate a locally deployed multi-agent AI system for radiology report structuring and quality assurance. Materials and Methods: This retrospective study included 638 radiology reports from CT examinations of the chest, abdomen, and pelvis dictated by 15 board-certified radiologists in 2023 and 2024. A multi-agent AI pipeline was developed to perform report structuring and quality assurance (QA). The system structured the report into standardized anatomical sections at the sentence level using regex rules and local large language models. It also detected mismatches between the Findings and Impression sections, or within sections; gender-anatomy conflicts; and undocumented communication of critical findings. Two board-certified radiologists independently evaluated a 45-report subset. Results: The multi-agent system structured the Findings sections of all reports (22,270 sentences) into a predefined anatomical format while retaining the original report content. The system flagged 90 (14.1%) reports, most commonly for section mismatches (80 reports, 12.5%). In the radiologist evaluation, both reviewers agreed that 31 (69%) were correctly restructured, 2 reports (4%) were incorrectly restructured, and disagreed on the remaining 12 reports (27%). Both reviewers agreed that no clinically important information was omitted and no fabricated content was introduced. Overall QA performance was rated as "excellent" or "good" in 84% of the evaluated reports, with the remaining reports rated as "fair". Conclusion: A locally deployed multi-agent AI system combined radiology report structuring and quality assurance within a single workflow. The system demonstrated favorable performance in radiologist evaluation. Such systems may support standardization of reporting and quality assurance in radiology practice.
1 INTRODUCTION
Radiology reports may lack consistent organization and contain discrepancies or communication errors that hinder clinical interpretation. This work proposes a multi-agent AI system that standardizes report organization and performs automated quality assurance while preserving original text.
- Reporting challenges: Inconsistent or absent report structure can cause referring physicians to overlook critical information or spend time searching for it.Finding placement and organization may vary across radiologists, complicating communication with referring clinicians.
- Prior approaches: Structured templates, NLP, and LLM approaches have been proposed to improve consistency or extract structured information from narrative reports.Template adoption has varied because of concerns about workflow burden and reporting flexibility.
- Reporting challenges: Reporting discrepancies and communication errors can cause misinterpretation, delayed treatment, or inappropriate clinical management.Voice-recognition transcription may introduce errors or unintended wording changes, increasing the need for quality control.
- Proposed system: The proposed multi-agent AI system assigns report sentences to predefined anatomical sections without rewriting the original content and performs automated quality assurance.QA targets mismatches between Findings and Impression, discrepancies within sections, gender–anatomy conflicts, and critical findings without documented communication.
2 MATERIALS AND METHODS
This retrospective study used 638 CT radiology reports to evaluate a locally deployed four-agent pipeline that structured Findings sections and performed report-level quality assurance. Structuring combined rule-based sentence classification with local language models, followed by independent review of 45 reports by two radiologists.
- Study population: 638 complete CT radiology reports from chest, abdomen, and pelvis examinations were collected from 15 board-certified radiologists’ dictations between 2023 and 2024.Reports ranged from 146 to 1,594 words, averaging 438 words.
- AI pipeline: A four-agent pipeline organized Findings sentences into predefined anatomical sections, while a fourth agent performed report-level quality assurance.The first three agents handled structuring; the fourth identified clinically meaningful inconsistencies.
- AI pipeline: Structuring used approximately 600 domain-specific regex patterns, LLaMA-3–8B for unresolved cases, and DeepSeek-R1-Distill-Llama-70B for selected context-dependent sentences.The regex rules were developed with input from a board-certified radiologist, and the language models were quantized for local inference.
- Quality assurance: The QA agent assessed laterality, negation, polarity, organ or segment, size or unit, gender–anatomy, and critical-finding communication inconsistencies.It operated at the whole-report level and suppressed clinically acceptable summarization differences.
- Clinical evaluation: Two independent board-certified radiologists evaluated a fixed sample of 45 reports, including three reports per radiologist-writer with author identities redacted.The evaluating radiologists were distinct from those who authored the original reports.
3 RESULTS
The multi-agent system processed 638 reports and 22,270 Findings sentences, primarily using regex-based classification while flagging potential QA inconsistencies. Independent radiologist evaluation showed favorable restructuring and QA performance, with some reviewer disagreement.
- Sentence classification: 18,493 sentences (83.0%) were classified by Agent 1 (regex), while Agent 2 (LLaMA 3) classified 2,729 sentences (12.3%).Agent 2 primarily handled cases where simple pattern matching was insufficient.
- Sentence classification: 93.3%, 90.0%, and 90.3% were the highest Agent 1 proportions for reports by radiologists 3, 6, and 13, respectively.Radiologists 2 and 4 had the greatest combined use of Agents 2 and 3, at 26.7% and 27.2%, respectively.
- Quality assurance: 90 reports (14.1%) were flagged for potential inconsistencies or errors, including 80 reports (12.5%) with section mismatches.Flagged findings were categorized as major severity in 40 reports (6.3%), minor in 44 (6.9%), and critical in 6 (0.9%).
- Processing time: 55.6 seconds per report was the overall average processing time, with Agent 3 averaging 28.62 seconds and Agent 4 averaging 25.13 seconds.Agent 1 was fastest at 0.54 seconds per report, followed by Agent 2 at 1.31 seconds.
- Radiologist evaluation: 31 reports (69%) were judged correctly restructured, 2 reports (4%) incorrectly restructured, and reviewers disagreed on 12 reports (27%).The evaluation covered 45 reports and disagreements primarily involved borderline section assignments or organizational preferences.
- Radiologist evaluation: 37 reports (82%) had correctly assessed mismatches between or within Findings and Impression sections.The QA evaluation also recorded 5 false-positive mismatch flags and 1 incorrect gender–anatomy discrepancy flag.
4 DISCUSSION
The multi-agent system integrated report structuring and automated quality assurance by combining deterministic rules with locally deployed LLMs. Radiologist review showed favorable performance, although evaluation was limited and enriched for QA-flagged reports.
- System contribution: The pipeline standardized heterogeneous radiology reports while preserving original text and simultaneously performing automated QA analysis.It combined rule-based methods with locally deployed large language models.
- System contribution: Deterministic classification with selective LLM reasoning enabled consistent organization while limiting reliance on computationally intensive models.The system used LLM-based reasoning for more ambiguous findings rather than generating or rewriting structured report content.
- Quality assurance: QA predominantly identified mismatches between Findings and Impression sections, while also assessing within-section inconsistencies, gender–anatomy conflicts, and undocumented critical-finding communication.The latter error categories extended the scope of prior radiology QA approaches described in the discussion.
- Quality assurance: 7 of 45 evaluated reports contained false-positive QA flags, while one reviewer identified an additional subtle discrepancy missed by both the QA agent and the second reviewer.All but one identified mismatch were detected by the QA agent.
- Limitations: The clinical evaluation included only 45 reports reviewed by two radiologists, with 41 flagged and 4 unflagged reports, so it was not designed to estimate unbiased overall QA performance.A larger evaluation cohort would strengthen the findings, but comprehensive review would require substantial time and effort.
- Conclusion: Further validation is needed before agent-based AI approaches can be considered for improving reporting consistency.The discussion concludes that automated structuring and QA can be integrated within a single workflow across a heterogeneous dataset.
Radiologist Reports, n Sentences, n
Across 638 radiology reports, the multi-agent pipeline performed report-level quality assurance and report structuring, with results summarized by radiologist, error category, processing time, and independent evaluation. A 45-report subset was independently assessed by two radiologists.
- Quality assurance: QA error categories included section mismatches, gender–anatomy discrepancies, and missing communication of critical findings.Error categories were not mutually exclusive, so one report could contain multiple error types.
- Processing time: Processing times were summarized across 15 radiologists for each structuring and quality-assurance agent, including overall average time per report.The table reports agent-level averages and an overall average total processing time for each radiologist.