Source-linked AI summary
Surgical Data Science -- from Concepts toward Clinical Translation
Lena Maier-Hein, Matthias Eisenmann, Duygu Sarikaya, Keno März, Toby Collins, Anand Malpani, Johannes Fallert, Hubertus Feussner, Stamatia Giannarou, Pietro Mascagni, Hirenkumar Nakawala, Adrian Park, Carla Pugh, Danail Stoyanov, Swaroop S. Vedula, Kevin Cleary, Gabor Fichtinger, Germain Forestier, Bernard Gibaud, Teodor Grantcharov, Makoto Hashizume, Doreen Heckmann-Nötzel, Hannes G. Kenngott, Ron Kikinis, Lars Mündermann, Nassir Navab, Sinan Onogur, Raphael Sznitman, Russell H. Taylor, Minu D. Tizabi, Martin Wagner, Gregory D. Hager, Thomas Neumuth, Nicolas Padoy, Justin Collins, Ines Gockel, Jan Goedeke, Daniel A. Hashimoto, Luc Joyeux, Kyle Lam, Daniel R. Leff, Amin Madani, Hani J. Marcus, Ozanan Meireles, Alexander Seitel, Dogu Teber, Frank Ückert, Beat P. Müller-Stich, Pierre Jannin, Stefanie Speidel
TL;DR
Surgical Data Science addresses persistent integration, interoperability, and data-access challenges in surgery, where translational success stories remain limited. The paper reviews current practice and initiatives, then outlines goals and a roadmap for advancing clinical translation; experts now view AI as a key enabling technique for the future operating room.
Problem
Persistent needs for technology integration, common standards, improved operating-room information access, and equipment interoperability continue to constrain data-driven surgery.
Method
An international workshop review and four-round Delphi process with clinical and technical stakeholders from more than 50 institutions examine SDS infrastructure, annotation, sharing, analytics, products, and translation goals.
Results
Experts reported no disruptive changes in the operating room since 2004, while AI is now viewed as a key enabling technique for the future operating room.
Takeaways & Limitations
SDS development requires stronger interdisciplinary coordination, clearer communication with computer vision and machine-learning communities, and new transdisciplinary career paths.
Takeaways & Limitations
Perioperative data remain distributed across specialized information systems, making effective linking and exchange more complicated.
Abstract
from arXiv · showhide
Recent developments in data science in general and machine learning in particular have transformed the way experts envision the future of surgery. Surgical Data Science (SDS) is a new research field that aims to improve the quality of interventional healthcare through the capture, organization, analysis and modeling of data. While an increasing number of data-driven approaches and clinical applications have been studied in the fields of radiological and clinical data science, translational success stories are still lacking in surgery. In this publication, we shed light on the underlying reasons and provide a roadmap for future advances in the field. Based on an international workshop involving leading researchers in the field of SDS, we review current practice, key achievements and initiatives as well as available standards and tools for a number of topics relevant to the field, namely (1) infrastructure for data acquisition, storage and access in the presence of regulatory constraints, (2) data annotation and sharing and (3) data analytics. We further complement this technical perspective with (4) a review of currently available SDS products and the translational progress from academia and (5) a roadmap for faster clinical translation and exploitation of the full potential of SDS, based on an international multi-round Delphi process.
1. Introduction
Surgical Data Science (SDS) emerged to apply data capture, organization, analysis, and modeling across the interventional care process, including procedural data. This paper reviews persistent challenges and proposes concrete measures for clinical translation.
- The field addresses a persistent need for integrated technologies, shared standards, accessible operating-room information, and interoperable equipment.
- SDS aims to improve interventional healthcare through the capture, organization, analysis, and modeling of data.
- Unlike general biomedical data science, SDS includes procedural data alongside patient, caregiver, and technology-related information.
- In 2019, no commonly recognized surgical data science success stories existed, motivating a review of the field and measures for clinical success.
- The paper examines infrastructure, data annotation and sharing, analytics, clinical translation, and a Delphi-based roadmap for future goals.
2. Lack of success stories in surgical data science
Surgical Data Science has advanced conceptually and technically, but surgery still lacks widely recognized success stories. The authors use workshop discussions and a multi-institution Delphi process to identify obstacles and goals for moving the field forward.
- Success stories in machine learning healthcare applications are widespread outside surgery, while striking examples remain lacking in surgery.
- The 2019 workshop expanded the initiative’s scope from defining SDS and its challenges to reviewing research initiatives, industrial perspectives, and early success stories.
- Participants identified representative annotated data as the field’s main obstacle and the main reason for previous SDS project failures.
- EndoVis, Cholec80, and JIGSAWS were the most useful publicly available datasets, but limited size and representativeness remained core issues.
- The paper organizes the field around infrastructure, annotation and sharing, analytics, and clinical translation, then formulates goals through a Delphi process involving experts from more than 50 institutions.
3. Technical infrastructure for data acquisition, storage and access
Surgical data infrastructure remains limited by incomplete capture, storage, documentation, exchange, and regulatory governance. The paper identifies technical and organizational priorities for acquiring, linking, and using perioperative data in clinical settings.
- Current practice: OR teams use sensory and interpersonal information that current infrastructure often fails to acquire, including vision, touch, hearing, and communication.The authors note that infrastructure addressing these data sources is not yet widely available.
- Current practice: High-resolution surgical video can exceed 50 GB per video when stored uncompressed, while healthcare information technology is not designed for such prospective storage.Modern stereoscopic endoscopes can generate two Full HD streams at 60 Hz, with larger files possible for 4K and additional sensors.
- Current practice: Clinical documentation often satisfies legal requirements but remains largely inaccessible to computation, while surgical decisions and procedure details may be undocumented or handwritten.Hospital-specific recording parameters also create missing values when datasets are merged.
- Current practice: Perioperative data are distributed across specialized systems, and strict semantic annotation is needed for retrievability and interoperability; consequently, exchange between systems is rare.The paper describes PACS, RIS, and LIS as examples of systems organized around different data types and workflows.
- Governance: Regulatory and ethical constraints complicate SDS data use through variable security requirements, immature surgical data governance, and data-access arrangements that can create power imbalances.The paper highlights GDPR-related obligations and concerns about democratizing access to surgical data.
- Technical constraints: An uncompressed Full HD video stream at 60 fps and 24-bit color requires 2.98 Gbps, exceeding typical 1-Gbps Ethernet capacity.More modern 10-Gbps installations exist but remain expensive and are typically reserved for data-center networks; wireless networks are slower.
- Data storage and distribution: Cloud storage may reduce local storage and maintenance needs, but introduces privacy, access-speed, and large-dataset download-cost concerns.The paper emphasizes that institutions must align provider privacy options with strict legal and institutional requirements.
- Current challenges and next steps: The proposed infrastructure priorities include prospective capture of perioperative data and long-term outcomes, interoperable exchange, standardized storage and annotation, and improved tissue imaging.The paper warns that data lakes without metadata management can become data swamps and calls for standards developed with clinical and industrial stakeholders.
4. Data annotation and sharing
Surgical data annotation and sharing depend on representative, interoperable datasets, consistent standards, and efficient access to expert knowledge. Current resources and initiatives provide foundations, but datasets remain small and heterogeneous, while widely accepted annotation standards are still lacking.
- Data requirements: Annotated datasets should span multiple centers, use defined acquisition and annotation protocols, link to patient outcomes, and support validation and replication.They also need to be representative of the task they address.
- Data requirements: Existing curated SDS datasets are useful starting points but remain relatively small, often single-institution, and highly diverse in structure, nomenclature, and target procedure.
- Annotation: Surgical annotations vary by spatial, temporal, and spatio-temporal granularity, including classification, segmentation, and regression tasks.
- Annotation: Expert knowledge is the main annotation bottleneck, motivating crowdsourcing, active learning, staged expert review, and educational integration of annotation.Non-experts can contribute when tasks are designed for meaningful annotation and experts review their output.
- Standards and sharing: OntoSPM and related consensus efforts aim to establish shared vocabularies and standards for surgical actions, instruments, actors, temporal models, anatomy, and software structures.These initiatives support interoperable annotation and data sharing across surgical applications.
- Challenges: No widely accepted annotation standards yet exist, and complex temporal boundaries, varied techniques, expert disagreement, and dataset versioning remain practical challenges.Exceptions include established protocols for skill assessment and Critical View of Safety documentation.
- Bias: Representative surgical datasets are difficult to construct because patient, procedure, device, protocol, and surgeon variables can produce selection and confounding bias.The paper states that fully representative datasets are only possible in a multi-center setting.
- Data sharing: Anonymization enables data sharing but removing metadata may be insufficient when individuals can be recognized directly from image data, such as facial features in head MRI.Pseudonymization is weaker because re-identification remains possible through separately held data.
5. Data analytics
Surgical data analytics covers descriptive, diagnostic, predictive, and prescriptive uses across heterogeneous perioperative data. Translation remains constrained by data sparsity, real-time requirements, validation gaps, explainability, regulatory demands, and the lack of scaled surgical success stories.
- Scope: Data analytics processes perioperative data to address clinical needs spanning prevention, training, diagnosis, treatment assistance, and follow-up.
- Analytics types: SDS analytics are commonly classified as descriptive, diagnostic, predictive, or prescriptive according to whether they summarize, explain, forecast, or guide decisions.
- Data complexity: Surgical analytics must accommodate heterogeneous 2D/3D/4D imaging, video, time series, laboratory results, patient history, and genomic information.The surgical process also varies substantially compared with the more regular diagnostic data-acquisition flow.
- Translation: Commercial healthcare analytics initiatives have largely focused outside surgery, and industrial success stories in surgery at scale remain lacking.The paper notes limitations of deployed systems, including Watson’s reported poor performance in one breast-cancer setting.
- Current challenges: Data sparsity is addressed through crowdsourcing, synthetic data, self-supervised learning, and semisupervised learning using unlabeled data or pseudo-annotations.
- Current challenges: Out-of-distribution inputs create epistemic uncertainty because the algorithm may lack a meaningful basis for inference and face a high probability of error.
- Current challenges: Real-time assistance requires both suitable hardware and communication infrastructure and algorithms optimized for application-specific latency constraints.Moving high-resolution video between devices or displays can introduce delays.
- Current challenges: Validation and evaluation must establish both whether a system performs as designed and whether it provides short-, mid-, and long-term added value.Evaluation on data from the same distribution as training data is identified as a problem.
6. Clinical translation
Clinical translation of Surgical Data Science remains limited by weak evidence, inconsistent surgical practice, and regulatory and governance challenges. The paper identifies practical opportunities, infrastructure and collaboration needs, and priorities for advancing SDS into clinical use.
- Current evidence: Retrospective studies with small, single-center, poorly representative datasets and methodological shortcomings limit the quality of clinical evidence for SDS.Reported shortcomings include inadequate comparator variability analysis, missing quantitative error analysis, and failure to separate training and test data.
- Current products: Modest clinical translation has been achieved, predominantly through endoscopic decision support, alongside OR safety algorithms and computer-vision-based data extraction.Examples include systems for detecting cancerous lesions, although one reported endoscopic system struggled with low positive predictive value.
- Low-hanging fruit: Descriptive SDS applications may offer lower-risk translation routes than prescriptive decision systems by informing surgeons without assigning surgical responsibility to AI.Suggested applications include reporting procedural steps, recognizing anatomy and instruments, and providing moment-to-moment risk stratification.
- Low-hanging fruit: Predictive tools for OR logistics and object-removal warnings could provide operational or safety value while requiring relatively little annotation and posing low patient risk.Procedure-time prediction can support OR volume and cost optimization, while video-based recognition could warn about objects introduced but not removed.
- Low-hanging fruit: Next-generation surgical robotics may facilitate systematic data capture and guidance, with automated camera control presented as a comparatively low-risk opportunity.Camera repositioning is disruptive for surgeons, whereas correcting it has lower patient risk than automating invasive tasks such as suturing.
- Challenges and next steps: Clinical translation requires a cultural shift toward governed data work, multi-institutional collaboration, interdisciplinary training, and stronger external validation.The paper calls for hospital resources, representative data for development and validation, stakeholder networks, SDS career paths, and more external validation studies.
7. Discussion
The 2019 discussion revisits persistent operating-room challenges and finds that AI has become the key enabling technology, while disruptive clinical change remains limited. It identifies interoperability, data integration, translation, and interdisciplinary collaboration as central priorities for SDS.
- Operating-room progress: The 2019 workshop found no disruptive change in the operating room compared with 2004; improvements were largely incremental and lacked relevant AI or ML components.Examples included better visualization, safer tissue dissection, and more sophisticated staplers, while anastomotic leakage remained relevant.
- Operational efficiency and workflow: Persistent workflow problems include fragmented information systems, difficult research-data access, and insufficiently centralized infrastructure for acquisition, annotation, and processing.Standards alone are insufficient without hospital requirements for data accessibility and standardized regulatory workflows.
- Systems integration and technical standards: Interoperability remains incomplete: operating-room systems still cannot reliably exchange information across machines and imaging modalities.Standards such as SDC can enable data exchange, but platforms supporting dynamic reactions and complex interactions still require development.
- Telecollaboration: Telecollaboration has advanced only slightly, constrained by technical limitations, coordination problems, and user knowledge gaps.A genuine breakthrough in telecollaboration had not yet been achieved.
- Changing vision: AI is now viewed as a key enabling technique for the future operating room, replacing the earlier emphasis on devices.The paper therefore centers on technical challenges in applying AI and ML to surgery.
- Clinical translation and interdisciplinarity: Clinical translation is hindered by low surgeon engagement and limited prospective evidence, making interdisciplinary collaboration and clinician-led application development essential.The proposed direction includes shared task forces and stronger participation from surgery, informatics, computer vision, and machine learning.
Conflicts of interest
The authors disclose employment, advisory, consulting, research-support, and founder relationships with companies relevant to surgical technology and data science.
- Conflicts of interest: Several authors report industry relationships, including employment, advisory or consulting roles, research support, and company founding activities.The disclosed organizations include Mimic Technologies, KARL STORZ, CMR Surgical, Caresyntax, Johnson & Johnson, Verily Life Sciences, Activ Surgical, Olympus, the Intuitive Foundation, 10 Newtons, Digital Surgery, Odin Vision, and Surgical Safety Technologies.
A. Publicly accessible and annotated surgical data repositories
Table A.1 catalogs publicly accessible, annotated surgical data repositories across six application categories, spanning robotic, laparoscopic, endoscopic, microscopic, sensor-enhanced, and other surgical settings.
- Repository categories: The repositories are assigned to six categories: robotic minimally-invasive surgery, laparoscopic surgery, endoscopy, microscopic surgery, sensor-enhanced operating-room surgery, and other.Each repository appears only once, although categories can overlap.
- Data modalities: Modalities include images, video, RGB-D, MRI, intraoperative ultrasound, fluoroscopy, device signals, and kinematic data.The table therefore represents multimodal surgical data rather than video alone.
- Data settings: Repositories include in-vivo human data as well as ex-vivo, phantom, synthetic, porcine, and goat data.This range covers both clinical recordings and controlled experimental settings.
- Annotation types: Datasets provide annotations for instruments, anatomy, workflow phases, actions, skills, pathologies, landmarks, and disease regions.Annotation formats include segmentation, bounding boxes, polygons, center points, classification labels, and phase or action labels.
B. Surgical Data Science standards & tools
This section catalogs selected standards and tools supporting SDS data acquisition, access, storage, communication, and intersecting technical disciplines.
- Table B.1 lists selected standards relevant to SDS data acquisition, access, storage, and communication.
- The standards span serialization formats, patient-centric health-data architectures, resource and relationship data models, and web-service principles.
- Table B.3 identifies representative software tools used across disciplines intersecting with SDS.
C. Surgical Data Science annotation tools & services
This section presents selected tools for annotating spatial, spatio-temporal, and temporal data, alongside companies offering managed human annotation workforces.
- Table C.1 selects annotation tools for spatial, spatio-temporal, and temporal annotations.
- Together, the tables cover both annotation software and externally provided human annotation services.
- Table C.2 lists leading companies providing dataset annotations through managed human workforces.
D. Published SDS clinical studies - perioperative
This section selects perioperative SDS clinical studies identified through database searches and manual evaluation for studies analyzing systems with machine-learning components.
- Table D.1 presents a selection of perioperative SDS clinical studies.
- The searches combined surgery with machine learning, deep learning, artificial intelligence, decision support, or surgical data science.
- Studies were included when they analyzed a perioperative SDS system with a machine-learning component.
E. Registered SDS clinical studies
This section catalogs registered SDS clinical studies at ClinicalTrials.gov, covering studies that test SDS systems or components and studies collecting data to develop and test them.
- Table E.1 reports registered SDS clinical studies at ClinicalTrials.gov as of October 2020.
- The registry search combined surgery with machine learning, deep learning, artificial intelligence, decision support, data science, or surgical data science.
- Included studies either tested an SDS system or component or collected data to create and test one.
F. Stakeholder importance
The Delphi process assessed the importance of stakeholders relevant to Surgical Data Science using a five-level importance scale. Healthcare information technology and data-generating units in healthcare institutions are among the identified stakeholder groups, alongside clinical, research, regulatory, industry, patient, and public-health stakeholders.
- Stakeholder importance was determined through a Delphi process using ratings from “Very important” to “Unimportant.”
- Healthcare information technology is identified as a stakeholder group in the assessment.
- Data-generating units in healthcare institutions include imaging departments, laboratories, and centers of clinical studies.
- The stakeholder landscape includes surgical teams, medical professional bodies, researchers, research institutions, and scientific societies.Examples include surgeons, nurses, anesthesiologists, SAGES, EAES, clinician scientists, university hospitals, and MICCAI.
- Other identified stakeholders include funding agencies, medtech companies of different sizes, regulatory agencies, institutional review boards, public and private stakeholders, patients or their representatives, and public health organizations.Examples include the ERC, FDA, and WHO.