Source-linked AI summary
STEP: A Modular Silent Trial Engine for Operational Evaluation of Digital Pathology AI in Routine Workflow
Gabriele Campanella, Matthew Croken, Olga Lukatskaya, Jane Houldsworth, Ricky Kwan, Peter Schüffler, Chad Vanderbilt
TL;DR
Prospective silent trials need reliable infrastructure to evaluate pathology AI on live clinical data without influencing care, but existing evaluations and implementations are often retrospective or application-specific. The paper presents STEP, a modular platform that separates common trial orchestration from institution-specific data and compute integrations. Deployed across heterogeneous environments, STEP supports durable, auditable trial execution while remaining limited in clinical-effectiveness evidence and broader generalizability.
Problem
Pathology AI lacks sufficient prospective evidence of reliable real-world operation, while silent-trial workflows require complex coordination across data, imaging, compute, and inference systems.
Method
STEP uses modular adapters to separate reusable silent-trial orchestration from institution-specific data access, compute infrastructure, and result production.
Results
STEP provides scheduled case discovery, per-slide inference submission, deterministic idempotency, failure recovery, result and ancillary-data ingestion, persistent state, and audit logging across heterogeneous pathology environments.
Takeaways & Limitations
STEP provides a practical reusable foundation for auditable real-world evaluation of computational pathology AI before interventional clinical deployment.
Takeaways & Limitations
STEP does not establish clinical effectiveness, model safety, or generalizability, and broader evaluation across institutions, models, and use cases remains necessary.
Abstract
from arXiv · showhide
Prospective silent trials provide an important bridge between retrospective validation of artificial intelligence (AI) models and their use in clinical care by evaluating model performance and operational reliability on live clinical data without influencing patient management. In computational pathology, conducting silent trials requires integration across laboratory information systems, digital pathology infrastructure, computational resources, and model inference pipelines, and these workflows are often implemented using application-specific software. We developed the Silent Trial Engine for Pathology (STEP), a reusable software platform for orchestrating prospective silent trials of computational pathology AI models across heterogeneous clinical and computational environments. STEP separates common trial orchestration from institution-specific data access and compute infrastructure through modular adapter interfaces. The platform supports scheduled case discovery, per-slide inference submission, deterministic idempotency, failure recovery, result and ancillary-data ingestion, persistent trial and run state, and audit logging, with compute adapters supporting local execution and high-performance computing environments using LSF and Slurm. STEP was deployed at three institutions to support prospective silent evaluation of EAGLE, an AI model for predicting EGFR mutation status from hematoxylin and eosin-stained whole-slide images. By separating trial-level workflow logic from site-specific integrations, STEP enables a common execution framework to operate across heterogeneous pathology environments while maintaining durable and auditable trial state. This approach may reduce duplicated engineering effort and facilitate systematic real-world evaluation of computational pathology AI before interventional clinical deployment.
1. Introduction
Computational pathology AI has advanced rapidly, but clinical adoption remains limited by insufficient prospective evidence and complex real-world deployment requirements. STEP addresses this gap with reusable orchestration for silent trials across heterogeneous pathology environments.
- Motivation: Clinical adoption of pathology AI remains limited, with pathology representing less than 0.5% of FDA-approved software-as-a-medical-device products.Existing cleared applications address relatively narrow diagnostic use cases, while implementation remains constrained by incomplete real-world validation and evolving regulation.
- Motivation: Most computational pathology evaluations are retrospective, limiting evidence about reliable end-to-end operation on live clinical data.Prospective deployment must address population shifts, production-pipeline discrepancies, data availability, case identification, inference execution, and clinically actionable turnaround times.
- Silent-trial paradigm: Silent trials execute frozen AI systems prospectively on live clinical data while keeping predictions invisible to clinicians and separate from patient care.This enables evaluation of performance, operational reliability, data availability, and temporal actionability under authentic conditions.
- Silent-trial paradigm: Prospective silent evaluation remains exceptionally limited in computational pathology despite growing adoption across clinical AI.A scoping review identified 75 silent evaluations across 16 countries, while pathology deployments require coordination across LIS, digital pathology, storage, compute, and clinical workflows.
- Engineering gap: Application-specific scripts for case discovery, data retrieval, inference, failure handling, and result aggregation limit maintainability, reproducibility, and auditability.Although infrastructure differs between institutions, silent trials share a common progression from case identification and data acquisition through inference and completion.
- Proposed solution: STEP provides a reusable orchestration layer that separates common trial execution from institution-specific data access and model implementation.The platform supports scheduling, compute portability, failure recovery, and auditable state transitions for prospective pathology AI evaluation.
2. STEP System Design
STEP combines reusable silent-trial control flow with adapter interfaces for local data, compute, and ancillary information sources. It persists trial state and supports scheduled discovery, per-slide execution, result ingestion, retries, and auditability across deployment environments.
- System design: STEP supports scheduled and on-demand case discovery, slide-level inference, delayed ancillary-data ingestion, durable state, audit logging, and concurrent independent trials.Common control flow is separated from local data access, compute infrastructure, and result production.
- Architecture: The layered architecture combines an API control plane, runtime schedulers, an orchestration service, adapters, and a MongoDB repository.The repository stores trial configuration, cases, jobs, results, and audit events, while the API exposes operational views and controls.
- Execution lifecycle: The orchestration loop scans configured data sources, upserts cases, checks prior submissions, delegates execution, ingests completed artifacts, updates statuses, and records audit events.The same scan-to-persistence sequence repeats according to each trial’s schedule.
- Adapters: Adapters isolate institution-specific interfaces for pathology data, compute backends, and ancillary feeds while leaving orchestration logic unchanged.Supported compute modes include local execution and HPC schedulers, and new adapters can be added through the corresponding interfaces and factory builders.
- Trial management: Trial registration persists configuration and creates scheduled jobs and artifact directories, allowing active trials to be listed, stopped, or restored.The runtime can restore active trials after process restart because registered configurations are stored in MongoDB.
- Execution lifecycle: For each eligible slide, STEP derives an input hash and idempotency key, skips matching existing jobs, and otherwise creates a run, writes its payload, and submits inference.This deterministic process prevents duplicate submissions during repeated scans.
- Execution lifecycle: A standardized worker contract requires a payload path and run-specific result JSON, while ingestion persists results, timing, errors, scores, metadata, logs, and ancillary records.The API also supports manual cycles, scheduler inspection, failed-run retries, run-history queries, and result views.
3. Discussion
STEP provides reusable infrastructure for prospective, non-interventional evaluation by separating portable trial orchestration from site-specific integrations and preserving operational provenance. The platform supports recovery and operational measurement, but broader validation and additional integrations remain necessary.
- Contribution: STEP addresses the need for reusable infrastructure to evaluate pathology AI prospectively before models influence patient care.Its common orchestration layer combines case discovery, data access, inference execution, failure recovery, and provenance across models and institutions.
- Portability: Separating trial orchestration from institution-specific infrastructure allows common control flow, persistence, and idempotency mechanisms to be retained across sites.Site-specific integrations are implemented at the adapter boundary despite heterogeneous institutional architectures.
- Operational evaluation: Persistent run state and audit history support recovery from failures and measurement of case capture, inference success, failure rates, and turnaround time.STEP’s scope is trial-level orchestration and provenance, complementing rather than replacing model-serving or general-purpose workflow platforms.
- Limitations: STEP does not establish clinical effectiveness, model safety, or generalizability, and deployments still require institution-specific integration development and validation.Current compute support covers local execution, LSF, and Slurm; other schedulers, cloud environments, and container platforms require additional adapters.
- Future directions: Future work includes additional adapters, standards-based clinical data exchange, and richer monitoring and reporting of operational endpoints.The authors frame STEP as a foundation for evaluating computational pathology models under real-world conditions before interventional deployment.
Declarations
The authors report publicly available source code and documentation, no competing interests, specified contributions across the team, and institutional computational and translational-science support.
- Data and materials: Source code and documentation are available on GitHub.
- Competing interests: The authors declare that they have no competing interests.
- Author contributions: The contributions covered platform design, manuscript drafting, implementation at participating institutions, implementation support, and manuscript approval.
- Acknowledgements: The work received support from Minerva computational and data resources, Scientific Computing and Data at Mount Sinai, and CTSA grant UL1TR004419.