PaperlayerView exact revision on arXiv

Source-linked AI summary

Error Detection for PET/CT Radiology Reports: Domain-Specific vs Large Language Models

Hermione Warr, Harry Anthony, Lilli J Freischem, Yasin Ibrahim, Daniel R McGowan, Konstantinos Kamnitsas

arXiv:2608.30021v1cs.LGcs.AI

TL;DR

Radiology report errors can affect clinical management, while PET/CT error detection remains challenging and under-evaluated beyond chest X-rays. This study compares compact domain-specific models with large open-source LLMs using a synthetic PET/CT error benchmark, finding that compact models achieve stronger performance with far fewer parameters.

  • Problem

    Automated PET/CT report quality assurance is needed because subtle reporting errors can alter clinical management, yet LLM-based error detection has been evaluated mainly on chest X-ray datasets.

  • Method

    The study builds a 30,633-report FDG PET/CT dataset and an 11,500-report held-out benchmark with clinically motivated synthetic inconsistencies, then compares domain-specific BERT models with open-source LLMs.

  • Results

    A 15M-parameter BERT achieved 94.4% balanced accuracy and a 5.8% false-positive rate, outperforming the evaluated larger generic LLM baselines.

  • Takeaways & Limitations

    Domain-specific training can outperform much larger generic models for PET/CT report error checking, supporting compact models for this task.

Abstract

from arXiv · show

Errors in radiology reports can adversely affect patient treatment, yet automated report quality assurance remains challenging because errors are often subtle and require domain expertise to detect. Although large language models (LLMs) have recently been proposed for radiology report verification, their ability to detect clinically meaningful errors beyond chest X-ray datasets remains under-explored. To this end, we present the first systematic evaluation of language models for PET/CT report error detection, comparing compact domain-specific models with SOTA open-weight LLMs. We collected 30,633 oncology FDG PET/CT reports from 23 radiologists over 10 years. We trained domain-specific BERT models to detect clinically motivated synthetic reporting errors and evaluated alongside zero-/few-shot Qwen3-32B, Gemma-3-27B and Llama-3.3-70B on a held-out benchmark of 11,500 reports. A 15M-parameter model achieved 94.4% balanced accuracy with a 5.8% false-positive rate, compared with 84.0% for the strongest prompted LLM. Task-specific adaptation of Llama-3.3-70B closed this performance gap (94.4%) but retained substantially greater computational requirements. Our results suggest that domain-specific training matters more than model scale for PET/CT report error detection, supporting compact models as an accurate and computationally efficient approach to automated radiology report quality assurance.

1 Introduction

The study addresses the under-explored problem of detecting clinically meaningful errors in lengthy PET/CT reports, where subtle semantic distinctions and governance constraints challenge generic LLM approaches. It compares compact domain-specific models with large open-source LLMs for automated report quality control.

  • Radiology-report errors occur in approximately 3–5% of interpretations and can alter clinical management.PET/CT errors may involve subtle distinctions such as physiological uptake versus malignancy or changes in disease burden.
  • PET/CT report verification remains under-explored because prior LLM evaluations largely focused on chest X-ray datasets, while PET/CT requires integrated metabolic and anatomical reasoning.PET/CT reports are typically more than twice as long as chest X-ray reports.
  • Hospitals’ privacy and governance requirements motivate locally deployable domain-specific models rather than dependence on cloud-hosted LLMs.The study therefore asks whether billion-parameter LLMs are necessary for PET/CT quality control.
  • The authors collected 30,633 oncology FDG PET/CT reports and created an 11,500-report held-out benchmark with clinically relevant synthetic inconsistencies.Domain-specific BERT models were compared with state-of-the-art open-source LLMs.
  • A 15M-parameter domain-specific BERT achieved 94.4% balanced accuracy and a 5.8% false-positive rate, outperforming the evaluated generic LLM baselines.The study reports that compact models contained around 1000× fewer parameters than the compared LLMs.

2 Methods and Materials

The study formulates PET/CT report checking as findings–conclusion consistency classification and builds a controlled benchmark by perturbing conclusions from real reports. It evaluates five clinically motivated error types across patient-level data splits and binary or multi-class prediction settings.

  • Methods and Materials: Each report is represented as findings F and conclusion C, and the model predicts whether C is supported by F.The label space contains consistent versus inconsistent classes in binary classification and one consistent plus five error classes in multi-class classification.
  • Synthetic Error Generation Framework: Synthetic inconsistencies are generated from real reports by modifying only the conclusion while leaving the findings unchanged.Original findings–conclusion pairs serve as consistent controls, and perturbations create the inconsistent dataset.
  • Synthetic Error Generation Framework: Five perturbation categories cover laterality, anatomical site, disease severity, disease status, and whole-conclusion mismatch.Keyword perturbations use predefined antonym vocabularies, while whole-conclusion perturbations replace conclusions from another report or disease-status class.
  • Synthetic Error Generation Framework: Disease-status labels classify conclusions as No Findings or Disease using cosine similarity to 22 canonical no-findings phrases.Conclusions with similarity ≥0.8 and no more than 15 words were labelled No Findings; manually verified labels were used for evaluation.
  • PET/CT radiology report dataset: The retrospective dataset contains whole-body FDG PET/CT reports collected over 10 years and split approximately 80:10:10 at the patient level to prevent leakage.The held-out benchmark used 3,000 test reports and yielded 11,500 evaluation reports across consistent examples and applicable perturbations.
  • Error detection models: All models were evaluated on binary error detection and multi-class error classification, with generic LLMs additionally assessed for prompt sensitivity and confidence.Experiments used one NVIDIA RTX PRO 6000 GPU with 96 GB memory, and sequence lengths were set to fit 99.9% of reports or prompts.

A. Prompt engineering tree

The prompt engineering study organizes successive prompt additions and evaluates them alongside locally deployable generic LLMs and domain-specific BERT models. It also describes model architectures, training, evaluation metrics, calibration measures, and severity-term generalisation.

  • A. Prompt engineering tree: P0–P9 are successive prompt versions formed by systematically adding and reordering instructions, while few-shot examples are added at the start of the user prompt.
  • A. Prompt engineering tree: Generic LLMs were evaluated locally with deterministic decoding, receiving findings and conclusion sections while prompt variants altered label ordering, expert roles, error descriptions, task framing, and related instructions.
  • A. Prompt engineering tree: Domain-specific models use BERT-style encoders with a linear classification head applied to concatenated findings and conclusion sections.
  • A. Prompt engineering tree: The binary classifier applies a sigmoid to the [CLS] representation, whereas six-class classification uses softmax for consistency plus five error categories.
  • A. Prompt engineering tree: The four BERT configurations jointly scale encoder layers, attention heads, and hidden dimension, ranging from BERT4 to BERT12 with a 600-token maximum sequence length.
  • A. Prompt engineering tree: Training proceeds through masked language modelling, a consistency-aware mismatched-pair objective, and supervised fine-tuning on synthetic error detection.
  • A. Prompt engineering tree: Robustness strategies balance classes and error categories, up-weight rarer laterality, site, and severity corruptions, and reduce lexical cues through tiered severity synonyms.
  • 2.4 Evaluation and robustness experiments: Balanced accuracy averages macro-averaged recall across five error categories with true-negative recall, while ECE summarizes weighted accuracy–confidence gaps across probability bins.

3 Results

Domain-specific models consistently outperformed generic LLMs on PET/CT report error detection, with performance driven largely by in-domain pre-training and remaining robust across error categories. Fine-tuning a large LLM recovered comparable accuracy but required substantially more computation, while confidence and lexical baselines exposed important robustness limitations.

  • Generic vs Domain-specific: 94.6% balanced accuracy was achieved by BERT12 versus 84.0% for the strongest generic LLM, while BERT6 reached 94.4% with a 5.8% false-positive rate.Performance largely saturated beyond BERT6, indicating limited benefit from additional model scale once sufficient capacity and in-domain pre-training were available.
  • Generic vs Domain-specific: Domain-specific models maintained higher detection rates and lower false-positive rates across synthetic error categories than generic LLMs, especially for semantically specific perturbations.Generic LLM performance declined on several site and severity errors, whereas BERT6–12 maintained high recall.
  • Robustness: 93.7% balanced accuracy followed BERT6 masked-language-model pre-training, increasing to 94.4% after consistency-aware pre-training from a 56.5% baseline.Masked-language-model pre-training remained crucial across model sizes, while the incremental benefit of the second objective diminished as models grew.
  • Generic vs Domain-specific: 94.4% balanced accuracy was reached by LoRA-fine-tuned Llama, matching domain-specific BERT performance but requiring over 200M trainable parameters plus a much larger backbone.Fine-tuning closed the prompting gap without recovering the computational efficiency of the 15M-parameter BERT.
  • Robustness: BERT6 and BERT12 retained substantial ability to detect severity errors involving unseen term pairs, although recall declines ranged from 1.5–24.5 and 8.8–25.9 percentage points, respectively.The largest reductions occurred when withholding frequently represented mild(ly) perturbations, while rarer faint(ly)/intense(ly) pairs caused less degradation.
  • Robustness: 95.9% and 97.3% lexical-null performance was accompanied by false-positive rates of 23.4% and 7.9%, indicating that model performance was not explained by simple lexical cues alone.Lexical baselines remained poor across individual perturbation types.

4 Conclusion

Compact domain-specific models outperformed zero-/few-shot generic LLMs on five synthetic PET/CT inconsistency types, while task-adapted Llama closed the gap at much greater computational cost. The results suggest domain-specific training matters more than model scale, but external validation on naturally occurring errors remains future work.

  • 94.4% balanced accuracy and a 5.8% false-positive rate were achieved by a 15M-parameter BERT, matching the task-adapted 70B LLM.The compact model matched the accuracy of fine-tuned Llama3.3-70B on the controlled benchmark.
  • Fine-tuning Llama3.3-70B closed the performance gap with compact domain-specific models but required substantially greater computational cost.
  • Domain-specific training appears more important than model scale for PET/CT report error detection and can substantially lower computational cost.
  • Future work will evaluate the findings on larger collections of naturally occurring errors and external datasets.

A.1 Synthetic Error Generation: Perturbation type breakdown

The benchmark introduces five clinically motivated perturbation types that alter findings–conclusion consistency while preserving report fluency. These include lexical swaps, disease-status changes, and whole-conclusion replacements, with automated disease-status labeling checked against human review.

  • Five perturbation operators introduce controlled inconsistencies spanning laterality, anatomical site, disease severity, disease status, and whole-conclusion mismatch.
  • Laterality perturbations swap left and right descriptors, while anatomical-site perturbations replace terms with antonyms from predefined location pairs.
  • Disease-severity perturbations replace terms across manually defined severity mappings and tiered, morphology-matched synonym groups.
  • Disease-status perturbations swap conclusions describing active disease with conclusions describing no evidence of active disease, or the reverse.
  • Disease-status labels were assigned using similarity to 22 canonical no-findings phrases and assessed against an independently human-reviewed subset.
  • Whole-conclusion mismatch replaces a report’s conclusion with another report’s conclusion, preserving authentic linguistic characteristics while creating report-level inconsistency.

A.2 Error generator coverage

Error-generator coverage varied across perturbation types because some lexical keywords were absent from reports, so training used oversampling and prompt variants to address imbalance and test sensitivity.

  • Laterality, site, and severity generators applied to 56%, 48%, and 52% of conclusions, whereas disease-status and cross-pair generators applied to all reports.
  • Applicable conclusions contained a median of one keyword, with left/right, upper/lower, and markedly/mildly among the most frequent terms.
  • Underrepresented keyword-swap errors and no-findings reports were oversampled to improve training balance.
  • The prompt study varied label ordering, task framing, error examples, role assignment, context, and few-shot demonstrations.
  • The prompts operationalized consistency as agreement between findings and conclusion and identified errors involving laterality, site, or disease severity.

A.4 Experimental details

The experiments used patient-level data splitting, domain-specific BERT training details, and held-out evaluations spanning binary detection, category breakdowns, severity generalization, and computational requirements.

  • 30,633 reports were retained after excluding reports without both findings and conclusions, with patient-level partitioning preventing cross-split leakage.
  • Domain-specific models used a roughly 10k-token vocabulary, 600-token context, and staged masked-language and consistency-head pretraining.
  • Report lengths were characterized using 95th, 99th, and 99.9th percentiles, and LLM inputs were slightly longer because of tokenization and prompts.
  • Binary results were also broken down by the five error types and the no-error class using balanced accuracy based on macro recall and true-negative rate.
  • QLoRA-adapted Llama3.3-70B matched compact models without surpassing them, while inference took over four hours versus 30–60 seconds for domain-specific models.
  • BERT12 errors commonly involved confusion between whole-conclusion swaps and no-findings-versus-disease swaps because the categories overlap.

B.5 Calibration Analysis

Calibration analysis found that most models made highly confident predictions, with scores concentrated near 0 and 1. BERT models with at least six layers were better calibrated than the evaluated LLMs, while BERT12 was the most confident.

  • 80–98% of predictions for most models fell in edge bins near 0 or 1, indicating highly concentrated confidence scores.BERT-4L was the exception, with 39% of predictions in the edge bins.
  • ECE was 0.027–0.033 for BERT models with ≥6 layers, compared with 0.085 for Qwen3-32B, 0.137 for Llama3.3-70B, and 0.185 for Gemma3-27B.The 4-layer BERT model had substantially worse calibration, with ECE 0.165.
  • LLM error probabilities were concentrated near the extremes, with margins close to one and little probability mass in the middle.The plots used confidence prompting to compare LLM confidence with self-reported confidence.
  • BERT12 produced the most confident predictions among the domain-specific models.

B.6 Analysis of BERT models decision influence

The analysis tested whether clinically meaningful anatomical-site and severity terms were associated with BERT-predicted inconsistency probabilities. Several site and severity terms showed differing associations, but these surface-form effects do not establish causal importance.

  • Keyword lifts compared mean predicted inconsistency probabilities between reports containing a term and reports lacking it.The analysis used terms occurring anywhere in the findings and conclusion sections.
  • Only keywords with at least 10 present-case reports were retained to reduce instability from rare terms.The analysis measures behavioural association between surface forms and model outputs rather than causal importance.
  • Proximal, lower, and thoracic terms were associated with higher predicted inconsistency scores, whereas distal, upper, and abdominal terms were associated with lower scores.Paired antonyms often showed opposite effects.
  • Mild, moderately, and marked severity terms increased predicted inconsistency, while faint, extremely, and moderate terms decreased it.Morphological variants of the same root could show opposite effects.

B.7 Error incidence plots

Error-incidence plots examined how detection rates and false-positive rates varied with the number of potentially confusing keywords. Increasing keyword presence caused a small BERT degradation but a larger LLM degradation, particularly in detection rate.

  • Increasing keyword presence caused a small degradation in BERT performance but a larger degradation for LLMs, particularly in detection rate.The analysis covered laterality, anatomical-site, and severity keyword swaps.
  • Each erroneous report contained one deliberately inserted error, although keyword swaps could provide multiple potentially confusing keyword locations.
  • The plots report BERT true-positive and false-positive rates by incidence for laterality, disease-severity, and anatomical-site errors.Results are separated by 6-, 8-, and 12-layer BERT models, with marker size indicating the number of erroneous conclusions.
  • The LLM plots report true-positive and false-positive rates by incidence for the three keyword-error types across Gemma, Qwen, and Llama.The displayed LLM results use prompt P7.
Loading 2608.30021v1…