PaperlayerView exact revision on arXiv

Source-linked AI summary

Extraction of clinical findings from mammography and breast ultrasound reports: a comparison between specialists and Artificial Intelligence

Lorenzo Farias, Hanna Reckziegel, Daniela Duarte da Silva Bagatini, Daniel Schulz, Gabriela de Andrade Monteiro, Letícia Zanatta, Ana Laura Brill Thum, Priscila Schmidt Lora, Débora Oliveira da Silva, Ana Paula Wernz da Cunha Müller, Cristiane Drebes Pedron

arXiv:2609.31974v1cs.CLcs.AI

TL;DR

Delayed mammography reports are relevant to screening adherence, and this study compares LLM extraction with manual review of mammography and breast ultrasound findings. Using a few-shot prompt and Gemini 2.5 Flash, the model achieved higher overall F1 scores than manual extraction in the reported comparison.

  • Problem

    Delayed report release is relevant to screening adherence, motivating comparison of automated and manual extraction of findings from Brazilian Portuguese breast-imaging reports.

    Source: §1 · p. 2

  • Method

    The study compared manual extraction with an LLM using an iteratively developed prompt and a four-example few-shot strategy.

    Source: §1 · p. 2, §3 · p. 3, §3.3.2 · p. 5

  • Results

    Macro F1 was 0.91 versus 0.72 and Micro F1 was 0.98 versus 0.95 for AI versus human extraction; AI detected the test set’s only microcalcification.

    Source: §6 · p. 13

  • Takeaways & Limitations

    Prompt-engineered LLMs can assist with structuring clinical findings from non-standardized Brazilian Portuguese reports at low computational cost.

    Source: §1 · p. 2, §6 · p. 13

    Evaluation covered 33 examinations from one partner clinic, and only two invited professionals responded to the validation form.

    Source: §5.2 · pp. 12–13

Abstract

from arXiv · show

Breast cancer is the leading cause of cancer-related death among women in Brazil, and the time between the request and the release of mammography reports directly influences adherence to screening, making the agility in processing these reports a critical factor for early diagnosis. In this context, this study compares the performance of a Large Language Model (LLM) with manual extraction performed by a team of health researchers in identifying clinical findings from mammography and breast ultrasound reports written in Brazilian Portuguese. Named Entity Recognition (NER) was applied through Prompt Engineering using a few-shot strategy, employing the Gemini 2.5 Flash model, selected from preliminary exploratory tests with four candidate models. The Gemini 2.5 Flash model demonstrated the best performance, achieving a Macro F1 of 0.91 and a Micro F1 of 0.98. The subjective validation, in which 29 exams of different formats were evaluated by health researchers using a Likert scale, yielded an agreement index of 93.1%. The model outperformed human extraction in overall Macro F1 (0.91 vs. 0.72), as in four reports the model correctly identified information that had been omitted or incorrectly recorded during manual extraction, demonstrating its potential as a complementary verification tool alongside specialists. The results confirm the hypothesis that LLMs, when instructed through Prompt Engineering, can achieve performance comparable to or superior to manual extraction by health professionals.

1 Introduction

Delayed breast-imaging reports are relevant to screening adherence, motivating automated extraction of clinical findings from mammography and breast ultrasound reports. The study compares an LLM with manual extraction and frames prompt-engineered LLMs as potential support for structuring non-standardized Brazilian Portuguese reports.

1 Introduction

  • Breast cancer accounted for 16.5% of oncological deaths in Brazil, and the 2023–2025 estimate was 73,610 new cases annually.The estimated adjusted incidence rate was 41.89 cases per 100,000 women.Source: §1 · p. 1
  • Only 48.8% of screening mammography reports were released within 30 days in 2023, while about 36% took more than 60 days.The study links report-release time to screening adherence.Source: §1 · p. 2
  • The study compares an LLM with health researchers for identifying pre-selected findings in mammography and breast ultrasound reports.Its objectives also include mapping findings, refining a structured prompt, quantitative evaluation, and Likert-scale validation.Source: §1 · p. 2
  • The article presents empirical evidence that adequately instructed LLMs can support structuring data from non-standardized Brazilian Portuguese reports.It situates this contribution in digital health.Source: §1 · p. 2

2 Related Work

Related work contrasts training-intensive clinical NLP approaches with adaptable LLM prompting, including studies of report structuring and extraction in several languages. The cited studies report varied strategies and performance outcomes for mammography and ultrasound reports.

2 Related Work

  • Earlier clinical-data extraction approaches such as BioBERT require extensive training data containing local clinical terminology, a challenge across linguistic and cultural contexts.Source: §2 · p. 2
  • For Spanish mammography laterality, GPT-4o with few-shot prompting achieved an F1 of 0.77 and outperformed GPT-3.5-turbo.The analysis categorized errors as positioning, format, and severe non-correctable errors.Source: §2 · p. 2
  • For Chinese breast-ultrasound report structuring, Baichuan2-7b achieved 97.63% precision and 96.22% attribute-value matching after LoRA fine-tuning.Dividing reports into logical sections improved performance, and 400 training samples was the reported optimal volume.Source: §2 · p. 2
  • For Turkish mammography extraction, many-shot prompting with Gemini 1.5 Pro achieved macro F1 of 0.99 under relaxed matching and 0.84 under exact matching [13].The approach used 165 annotated examples in a 26,000-token prompt across five clinical entities.Source: §2 · p. 3

3 Methods and Procedures

The study uses an iterative Design Science Research approach to develop and evaluate an LLM-based extractor against manually adjudicated findings. It maps clinical variables, tests models and prompt configurations, and combines quantitative metrics with professional validation.

3 Methods and Procedures

  • The study combines iterative experimentation, quantitative evaluation, and expert validation within a five-stage Design Science Research approach.The final system is evaluated against a manually adjudicated reference using classification metrics and Likert-scale validation.Source: §3 · p. 3

3.1 Procedures

  • The procedure moves from identifying the Brazilian Portuguese extraction gap through variable mapping and exploratory development to quantitative and qualitative evaluation.Exploratory tests compared file formats, prompting strategies, and four LLMs; Prompt Engineering was selected over RAG for sufficient efficacy with lower complexity, and a Python application produced structured JSON.Source: §3.1 · p. 3

3.2 Study Sample

  • The sample comprised 33 examinations: 29 for prompt testing and comparison, plus four representative examples used in the final few-shot prompt.Reports came from one partner clinic and included different laboratories and radiology services, with varied terminology and writing styles.Source: §3.2 · p. 3

3.3 Data Mapping

  • Researchers extracted report findings into a spreadsheet, then evaluated agreement with a specialist to establish consistency in the structured reference data.Figure 1 depicts the mapping process, including FHIR standardization and final validation.Source: §3.3 · p. 4, Figure 1 · p. 4

3.3.1 AI Model Selection

  • Gemini 2.5 Flash was selected after evaluating four candidates on qualitative performance, API cost and availability, latency, and LMArena score.It had the second-highest benchmark score at 1,394 points, input cost of $0.15/M tokens, and average latency of 8.15 s; its qualitative exploratory performance was best.Source: §3.3.1 · pp. 4–5

3.3.2 Parameter Definition and Prompt Testing

  • Few-shot prompting with four examples was selected, while TXT input was used to preserve essential report text and mitigate PDF-related hallucinations.Zero-shot produced structural inconsistency in JSON, whereas multi-shot could dilute attention and degrade extraction quality.Source: §3.3.2 · p. 5

3.3.3 Comparative Analysis and Statistical Evaluation

  • AI extractions were cross-validated by a specialist and researcher, and F1 was the central quantitative metric alongside accuracy, precision, and recall.Disagreements between AI and manual extraction were jointly reviewed against the original report to establish the reference ground truth.Source: §3.3.3 · p. 7

4 Results

On the adjudicated test set, Gemini 2.5 Flash achieved high extraction performance and outperformed manual extraction on Macro F1. Professional ratings were strongly positive overall, while Exam 4 exposed a case where professionals had greater interpretive flexibility.

4.1 Quantitative Performance and Adjudication

  • Macro F1 was 0.91 and Micro F1 was 0.98 for Gemini 2.5 Flash after adjudication.Across 116 binary instances, the model recorded 39 TP, 75 TN, 1 FP, and 1 FN.Source: §4.1 · pp. 7–8, Table 2 · p. 8
  • Accuracy was 1.00 for BI-RADS categories 1 to 6, with all 29 examinations correctly classified and no confusion between classes.BI-RADS 2 was most frequent (n = 15).Source: §4.1 · p. 8, Figure 4 · p. 9

4.2 Comparison Between AI and Human

  • AI Macro F1 was 0.91 versus 0.72 for human extraction; Micro F1 was 0.98 versus 0.95.The model had one false negative compared with three for human extraction; the comparison used an adjudicated gold standard.Source: §4.2 · p. 9, Table 3 · p. 9
  • The AI corrected manual false positives and omissions, including calcifications and nodules missed in Exams 10, 16, and 70.In Exam 4, it classified a calcification as a microcalcification using a size criterion, unlike the human morphological criterion.Source: §4.2 · p. 10

4.3 Subjective Validation (Agreement)

  • The Global Acceptance Index was 93.1% across 58 professional assessments, while Exam 4 showed interpretive divergence over a punctiform calcification.Professionals showed greater flexibility in atypical cases such as Exam 4.Source: Figure 5 · p. 10, §4.3 · p. 10

5 Discussion

The error analysis reports full agreement in representative cases and a format-sensitive result for Exam 15, alongside a residual calcification-classification disagreement. The study is bounded by a small single-clinic dataset, prompt specificity, and limited researcher validation.

5.1 Error Analysis

  • In Exam 4, AI classified a punctiform calcification as a microcalcification using a < 0.5 cm size criterion, while professionals used morphological and clinical criteria; prompt adjustment could correct the disagreement.The figure compares the AI and human extraction for this case.Source: §5.1 · p. 11, Figure 7 · p. 11
  • In Exam 15, extraction achieved full agreement across clinical variables with .txt input, although the report was selected for its tendency to induce hallucinations in PDF format.The input format is the reported condition for the full-agreement result.Source: §5.1 · p. 11, Figure 8 · p. 12
  • Exams 6 and 14 matched the gold standard across entity presence or absence and BI-RADS; similar mammography tasks reported Macro F1 between 0.84 and 0.99.The figures show the representative comparisons for Exams 6 and 14.Source: §5.1 · p. 12, Figure 9 · p. 12, Figure 10 · p. 12

5.2 Limitations

  • The dataset was limited to 33 examinations from one partner mastology clinic.The passage attributes this constraint to ethical restrictions under Brazil’s LGPD.Source: §5.2 · p. 12
  • The prompt was calibrated only to mammography and breast-ultrasound report patterns, and only two of the three researchers and the partner mastologist responded to validation.The validation form had low adherence among the invited researchers and mastologist.Source: §5.2 · p. 13

6 Conclusion

The study concludes that prompt-instructed LLMs can assist with extracting and summarizing clinical findings from mammography and breast-ultrasound reports. It reports higher overall F1 scores than human extraction, notes potential for complementary checking, and identifies future validation and adaptation work.

6 Conclusion

  • The authors conclude that prompt-instructed LLMs can effectively assist in extracting and summarizing clinical findings from mammography and breast-ultrasound reports at low computational cost.They describe the reports as semi-structured clinical data.Source: §6 · p. 13
  • 0.91 vs. 0.72 Macro F1 and 0.98 vs. 0.95 Micro F1: model scores exceeded human extraction, while subjective validation yielded 93.1% acceptance.The authors also report that AI identified the test set’s only microcalcification case, which human extraction missed.Source: §6 · p. 13
  • Reports from varied Rio Grande do Sul laboratories and radiology services supported the authors’ claim of robustness to formatting, terminology, and writing-style variation.The authors suggest this indicates adaptability across Brazilian-Portuguese mammography and breast-ultrasound reports.Source: §6 · p. 13
  • The authors plan to enlarge the data corpus, adapt prompts to other examination types and report standards, and include more health professionals in validation.Source: §6 · p. 13

7 Ethics Statement

The study reports that examination acquisition complied with Brazil’s data-protection law and received research ethics approval. Reports were consented, anonymized, and screened against inclusion and exclusion criteria.

7 Ethics Statement

  • The examinations were obtained through a mastology clinic under Research Ethics Committee approval via Plataforma Brasil, CAAE No. 88036825.9.0000.5343.Source: §7 · p. 13
  • Included reports required patient authorization by signed informed consent and compliance with data-protection regulations.Source: §7 · p. 13
  • Personal identifiers were removed or anonymized before AI analysis; incomplete, unsuitable-format, or non-consented reports were excluded.Source: §7 · p. 13

Data Availability Statement

The dataset comprises mammography and breast-ultrasound reports from a partner mastology clinic and is not publicly available under privacy and data-protection constraints.

Data Availability Statement

Declaration of Competing Interests

The reports were provided by Clínica de Mastologia Dra. Ana Paula Muller Ltda., whose external evaluator also contributed to qualitative validation and variable-search definition. The authors report no competing interests and limited, supervised AI assistance in manuscript preparation.

Declaration of Competing Interests

  • The clinic provided the study reports, and Dr. Ana Paula Wernz da Cunha Müller evaluated qualitative validation and helped define the variable search process.Source: Declaration of Competing Interests · p. 14
  • External collaborators had no influence on study design, data collection, analysis, interpretation, writing, or the decision to submit; remaining authors declared no competing interests.Source: Declaration of Competing Interests · p. 14

Declaration of Generative AI and AI-Assisted Technologies in the Manuscript Preparation Process

Related papers

Loading 2609.31974v1…