PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Site Real-World Performance of Commercial AI for Pulmonary and Incidental Pulmonary Embolism Detection

Aawez Mansuri, Mohammadreza Chavoshi, Theodorus Dapamede, Wasif Bala, Beatrice Brown-Mulry, Rohan Isaac, Bardia Khosravi, Hanzhou Li, Frank Li, John T. Moon, Chad Robichaux, Dan I. G. Cohen-Addad, Ninad V. Salastekar, Janice Newsome, Judy W. Gichoya, Hari Trivedi

arXiv:2609.37750v1cs.CVcs.AI

TL;DR

The real-world performance of FDA-cleared PE AI remains incompletely characterized. This study evaluates two models from one commercial platform on dedicated CTPA and routine CT, reporting task-specific sensitivity and specificity.

  • Problem

    Real-world performance of FDA-cleared PE models remains incompletely characterized, including across embolus characteristics and patient settings.

    Source: Introduction · p. 3

  • Method

    The study retrospectively evaluates one commercial platform’s PE triage model on dedicated CTPA and its iPE model on routine non-angiographic CT.

    Source: Introduction · p. 4

  • Results

    86.8% sensitivity and 99.1% specificity for dedicated CTPA PE detection; routine-CT iPE detection reached 73.5% sensitivity and 99.8% specificity.

    Source: AI Model Performance · p. 6

  • Takeaways & Limitations

    The findings underscore the need for standardized frameworks for continuous AI performance monitoring across diverse patient populations.

    Source: Clinical Impact Analysis · p. 11

    Follow-up concordance reflects a clinically selected subgroup, not a validated estimate of standalone effects on missed diagnoses or patient outcomes.

    Source: Statistical Analysis · p. 15

Abstract

from arXiv · show

Pulmonary embolism (PE) is a leading cause of cardiovascular mortality, yet the real-world performance of FDA-cleared AI detection models remains incompletely characterized. We retrospectively evaluated two FDA-cleared AI algorithms from a single commercial platform (Aidoc Medical BriefCase), one for PE triage on dedicated CT pulmonary angiography (CTPA; n = 30,678) and one for incidental PE (iPE) detection on routine contrast-enhanced CTs (n = 37,191), across a 17-facility academic health system. Reference-standard labels were extracted from radiology reports using a validated LLM pipeline (97% accuracy, kappa = 0.94). The PE model achieved 86.8% sensitivity and 99.1% specificity, with sensitivity declining from 99.3% for saddle emboli to 72.9% for subsegmental PE, and from 89.7% for acute to 65.3% for non-acute PE. The iPE model achieved 73.5% sensitivity and 99.8% specificity. Both models demonstrated lower sensitivity than FDA-clearance benchmarks while exceeding cleared specificity, with diminishing performance for peripheral and non-acute emboli mirroring known human reader limitations and underscoring the need for standardized post-market surveillance of AI-enabled medical devices.

Introduction

This study evaluates two Aidoc models for pulmonary embolism detection across dedicated CTPA and routine CT imaging, addressing the limited characterization of their real-world performance. It describes a two-stage assessment of LLM-derived labels and AI diagnostic performance, including clinically relevant subgroups and a follow-up-imaging analysis in a clinically selected subgroup.

Introduction

  • PE is a major cause of cardiovascular death, and incidental PE on contrast-enhanced CT can be overlooked when radiologists focus on the primary finding [1, 5–8].The passage estimates 60,000 to 100,000 PE deaths annually in the United States.Source: Introduction · p. 3
  • Nearly 60% of patients experience diagnostic delays, while rising CTPA volumes strain radiology capacity and motivate AI-assisted triage [9–15].Prior evidence cited in the passage found AI assistance increased radiologist sensitivity for iPE from 80.0% to 96.2% and reduced report turnaround times [7].Source: Introduction · p. 3
  • Real-world evaluations report PE-model sensitivities ranging from 63.0% to 96.8% and specificities from 95.0% to 99.9%, with limited subgroup coverage [15, 20].The passage identifies patient setting, thrombus location and subtype, and demographic groups as important evaluation gaps.Source: Introduction · p. 3
  • The study evaluates a cleared Aidoc CTPA model and a separate cleared iPE model on routine CTs, addressing gaps in the clearance summary across clinical, demographic, and anatomical subgroups [21, 22].The CTPA model's FDA validation set of 499 cases reported 94.86% sensitivity and 94.04% specificity [21]. The two models belong to one platform and are not competing products.Source: Introduction · pp. 3–4

Methods

  • The same frozen, zero-shot GPT-4o prompt pipeline extracted hierarchical PE labels from CTPA reports and binary PE-presence labels from routine CT reports.The binary prompt was developed on the primary CTPA cohort and reused verbatim for iPE extraction; all 250 independently annotated iPE reports were reserved for validation.Source: Ground Truth Determination via LLM… · pp. 13–14
  • LLM label extraction was evaluated against a manually annotated 200-report validation set using classification metrics and agreement analyses.The study reports label-specific performance in Table S4, an overall confusion matrix in Figure S1, and validation-set label prevalence in Table S13.Source: Statistical Analysis · p. 14
  • AI diagnostic performance was assessed against LLM-derived labels using standard classification metrics, subgroup analyses, imaging-feature regression, and examination-level bootstrap confidence intervals.Indeterminate or suboptimal exams were excluded; regression examined correct versus incorrect predictions among PE-positive primary-cohort exams, and repeated CTPAs were treated as independent diagnostic events.Source: Statistical Analysis · pp. 14–15
  • Follow-up concordance for AI-flagged iPE was assessed among secondary-cohort patients receiving dedicated PE imaging within 30 days, but this selected subgroup cannot validate standalone missed-diagnosis or outcome effects.Selection for follow-up imaging depended on clinical suspicion, symptoms, and report findings.Source: Statistical Analysis · p. 15

5. Management of the Incidental Pulmonary Embolism. doi:10.2214/AJR.16.17201

Reference 6 is a meta-analysis on the prevalence and clinical history of incidental, asymptomatic pulmonary embolism.

  • Reference 6 is a meta-analysis on the prevalence and clinical history of incidental, asymptomatic pulmonary embolism.Source: §5 · p. 17

Sensitivity

The supplied passage reports a sensitivity value for Hispanic patients, while the other passages identify subgroup-performance table scopes.

Sensitivity

  • 83.56% sensitivity is reported for Hispanic patients, with the interval beginning at 76.31%.Source: Sensitivity · p. 26
  • Table 4 covers model performance across different PE subgroups.Source: Table 4 · p. 27
  • Table 5 covers performance across intersections of age and patient-class subgroups.Source: Table 5 · p. 28
Loading 2609.37750v1…