Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmark

1. Introduction

• Surgical workflow and skill analysis are framed as prerequisites for cognitive assistance systems that could support safer, more efficient operations, skill development, and semi-autonomous assistance.Such systems would need to understand workflow and learn from skilled surgeons.Source: §1 · p. 5
• Existing clinical datasets are limited in multi-center diversity and annotation variety, while JIGSAWS lacks real patient data, constraining their transferability to clinical settings.The introduction also notes that surgical video annotation is laborious and that few surgical video datasets are openly available.Source: §1 · pp. 6–7
• The study presents an open benchmark for surgical workflow and skill analysis, using a dataset of 33 annotated laparoscopic cholecystectomies from three different surgical centers in the sub-challenge.Source: §1 · p. 7
• The study asks whether independent researchers can reproduce previously reported surgical phase and instrument recognition performance on the new dataset.Source: §1 · p. 7

• Surgical actions are often brief and subtle, while skill assessment is holistic and considers the whole video.Source: §1 · p. 7

2.1. Data collection

• The dataset contains 33 laparoscopic cholecystectomy videos from three German centers, totaling 22 hours, split by stratified random assignment into 24 training and 9 test videos.The training set equally represents Heidelberg and Salem, while the test set includes three videos from each of the three centers.Source: §2.1 · p. 8

2.2. Data annotation

• Medical experts annotated anonymized videos framewise using explicit rules, with phase labels independently assigned by three students and skill ratings by two students, resolving disagreements by consensus.Scenes outside the abdominal cavity were censored, and videos were anonymized.Source: §2.1 · p. 8, §2.2 · p. 9

2.2.1. Phase

• Each frame received one of seven phase labels: preparation, calot triangle dissection, clipping and cutting, gallbladder dissection, packaging, cleaning and coagulation, or retraction.The phases did not necessarily occur in a fixed order.Source: §2.2.1 · p. 10

2.2.2. Action

• Four actions—grasp, hold, cut, and clip—were annotated alongside whether the performer was the surgeon’s left or right hand or the assistant’s hand.Action labels indicate whether each action is being performed in a frame.Source: §2.2 · p. 9, §2.2.2 · p. 10

2.2.3. Instrument

• The dataset annotates 21 instruments grouped into seven categories, with instruments from different vendors across centers; staplers are absent from the test set.An instrument was marked visible when its characteristic tip appeared, with shaft-only visibility generally continued after tip disappearance.Source: §2.2.3 · p. 10

2.2.4. Skill

• Modified GOALS informed skill scoring for depth perception, bimanual dexterity, efficiency, and tissue handling; autonomy was omitted, and ratings from 1 to 5 covered the full operation, P1, and P3.The dataset describes skill annotation across five dimensions, while the skill subsection specifies that autonomy was omitted.Source: §2.2 · p. 10, §2.2.4 · p. 11
• Skill assessment from intraabdominal video alone was considered insufficient without information about what was spoken during the operation or senior-surgeon assistance.Source: §2.2.4 · p. 11
• Operation difficulty (S5) was additionally annotated using Chang’s adaptation of GOALS, with inflammatory signs, adhesions, and individual anatomical conditions used to objectify skill assessment.Source: §2.2.4 · pp. 11–12
• Skill was annotated in five ranking components for the complete operation and additionally for Calot triangle dissection (P1) and gallbladder.Source: §2.2.4 · p. 12

3.1 EndoVis challenge 2019

• Using the dataset, 12 international teams trained and evaluated their algorithms; participants could submit results for a subset of categories rather than every category.Participants could augment the provided training data with public third-party data.Source: §3.1 · pp. 12–13

3.2.1. Team CAMI-SIAT (Phase)

• CAMI-SIAT used a Pseudo-3D ResNet to recognize phases, combining frame-sequence information with phase-occurrence probabilities estimated from training data.The team did not use additional data for pre-training.Source: §3.2.1 · p. 13

3.2.2. Team CAMMA (Phase & Action)

• CAMMA combined I3D and Inception-ResNet features with LSTM predictions and majority voting for phase recognition, and used a ResNet with a convolutional LSTM for action recognition.Binary instrument labels provided an auxiliary training task for workflow recognition; ImageNet and Kinetics were used for pre-training.Source: §3.2.2 · pp. 13–14

3.2.3. Team CareSyntax (Instrument & Skill)

• CareSyntax used localization maps and spatial pooling for instrument detection, and a segmented-video 3D-ResNet with rounded outputs for skill assessment.ImageNet and Cholec80 were used for instrument-model pre-training, while Kinetics was used for the skill model.Source: §3.2.3 · p. 14

3.2.4. Team CUHK (Phase & Instrument)

• CUHK used a multitask ResNet for phase and instrument recognition, with elapsed-time input and phase-order post-processing for phase predictions.The post-processing used a median filter and the PKI strategy to detect and correct impossible phase changes.Source: §3.2.4 · pp. 14–15

3.2.5. Team HIKVision (Phase & Instrument)

• HIKVision first trained a ResNet for joint phase and instrument recognition, then replaced its task-specific fully connected layers with an LSTM and two output layers.PKI was used to post-process predictions with phase order and consistency in mind.Source: §3.2.5 · p. 15

3.2.6. Team IGITech (Phase & Instrument)

• IGITech used ResNet visual features, a fully connected layer for instrument detection, and a support vector machine for phase recognition.The paper states that it is unclear whether IGITech used pre-training data.Source: §3.2.6 · p. 15

3.2.7. Team Konica Minolta (Action & Instrument)

• Konica Minolta used a 101-layer ResNeXt with Squeeze-and-Excitation blocks and separate layers for instrument presence, action, and instrument super-class detection.An instrument super-class generally represented instruments that regularly occurred together.Source: §3.2.7 · p. 15
• Konica Minolta’s action-and-instrument approach used a ResNeXt pretrained on ImageNet; the supplied passage also mentions grasper and scissors.Source: §3.2.7 · p. 15

3.2.8. Team MEVIS (Phase)

• MEVIS combined ResNet visual features with an LSTM and task-specific fully connected outputs, using zero-weight clip-start frames for warm-up and adding clips from 16 Cholec80 videos to training.Source: §3.2.8 · p. 15

3.2.9. Team NCT (Phase, Action & Instrument)

• NCT used a multi-task ResNet–LSTM model to predict phase, action, and instrument presence from video, with separate fully connected task outputs and pretraining on Cholec80.Source: §3.2.9 · p. 16

3.2.10. Team VIE-PKU (Phase, Action & Instrument)

• VIE-PKU ran 2D ResNet and 3D architectures in parallel, branched features for action and instrument predictions, combined them to predict phase, and post-processed phase results with PKI and learned probabilities.Source: §3.2.10 · p. 16

3.2.11. Team Wintegral (Phase, Action & Instrument)

• Wintegral trained single-task, multi-task, and one-vs-all models on a ResNet basis, then aggregated their outputs into one predictor per task.Source: §3.2.11 · p. 16

3.3. Assessment Method

• Challenge scores used F1 for phase, action, and instrument tasks, averaging across classes and videos for equal weighting, while skill assessment used mean absolute error.Source: §3.3 · p. 20
• Ranking stability was assessed by recalculating ranks on 1,000 bootstrap test sets and comparing them with Kendall’s tau; pairwise Wilcoxon signed-rank tests with Holm adjustment tested performance differences.Source: §3.3 · p. 20

4.1. Data set for Surgical Workflow and Skill Analysis

• The dataset comprises 33 videos spanning 22 hours across three centers, with annotations for seven phases, four actions, 21 instruments in seven categories, and 495 skill classifications in five dimensions.The annotations include 250 phase transitions, 5514 action occurrences, and 6980 instrument occurrences.Source: §4.1 · pp. 20–21

4.1.1. Phase

• In 28 videos (84.8%), all seven phases occurred; the mean number of phase transitions was 7.6 ± 1.8 per video, and mean phase lengths ranged from 1.4 min ± 1.2 min to 17.2 min ± 9.6 min.Operations ended with gallbladder retraction (P6) in 84.8% of videos or cleaning and coagulation (P5) in 15.2%; P5 was omitted in five videos (15.1%).Source: §4.1.1 · p. 21

4.1.2. Action

• Grasp (A0) was the most frequent action (n=2830); hold (A1) occurred less often (n=2124) but had a mean total action length per operation of 974.2 s ± 879.0 s.Source: §4.1.2 · p. 22

4.1.3. Instrument

• Grasper presence averaged 25.1 min ± 12.3 min per operation, the highest instrument-category presence, followed by coagulation instruments at 18.1 min ± 8.4 min.Grasper and coagulation instruments were also the most present categories in the test data; specimen bags were prevalent in the last third of operations, corresponding to gallbladder packaging (P4).Source: §4.1.3 · p. 22

4.1.4. Skill

• In the skill assessment based on ranking components, grades ranged from medium grade 3 to best grade 5; grades 1 and 2 were never assigned, and the dataset contained surgeries of each difficulty level.Source: §4.1.4 · p. 23

4.1.5. Differences between centers

• All phases and actions occurred in every center, but instrument presence varied by center and Heidelberg cases were rated more difficult than cases at the smaller hospitals.The argon beamer was used exclusively at Heidelberg, while crocodile forceps were used only at Salem and Sinsheim; no Salem or Sinsheim cases received difficulty grade 5.Source: §4.1.5 · pp. 23–24

4.2. EndoVis challenge 2019

• The algorithms’ performance was compared across individual recognition tasks, and an algorithm ranking with ranking uncertainty was presented; results included CAMMA 1 and 2, which submitted after the challenge deadline.They were not considered for the challenge awards.Source: §4.2 · p. 24

4.2.1. Phase

• HIKVision and CUHK achieved the top phase-recognition F1-scores, up to 65.4% and 65.0%, respectively; because their difference was small and CUHK excelled in most videos, both were declared winners.Highest rates were 79.4% for preparation (P0) and 82.8% for calot triangle dissection (P1); the lowest was 50.2% for gallbladder packaging (P4).Source: §4.2.1 · pp. 25–26
• 0.93 was the mean Kendall’s tau across bootstrap samples for phase rankings, which were relatively stable although the second- and third-ranked algorithms were close.The median tau was 0.91 and the interquartile range was (0.91,0.96).Source: §4.2.1 · p. 26

4.2.2. Action

• The best team, Wintegral, reached an F1 score of 23.3%, while all challenge algorithms almost exclusively recognized the hold action (A1).Grasp (A0) occurred during 1.2% of an operation on average and participants recognized it during 1.1%; hold (A1) occurred during 73.3% of an operation.Source: §4.2.2 · p. 27

4.2.3. Instrument

• Konica Minolta was the best team in the instrument-presence detection task, and it and runner-up CUHK outperformed all other algorithms; rankings were very stable across bootstrap samples.Ranks 6 and 7 were often interchanged, and the first two algorithms were close together.Source: §4.2.3 · p. 28

4.2.4. Skill

• 0.78 was the average absolute error for skill assessment by the only participating team, CareSyntax.Across videos, depth perception (S1), bimanual dexterity (S2), and efficiency received scores of 4; tissue handling (S4) received 3, and difficulty (S5) received 2.Source: §4.2.4 · p. 29

4.3. HeiChole benchmark

• The multi-task HeiChole dataset and challenge results establish a reference benchmark against which future algorithms can be compared.The training set will be published, while the test set is withheld to avoid fraud; future submissions can be evaluated on it and reported on an online leaderboard upon request.Source: §4.3 · p. 29

5.1. HeiChole Benchmark

• HeiChole provides an open dataset and comparative validation of algorithms for automatic phase, instrument, action, and skill recognition in laparoscopic cholecystectomy.The discussion introduces the benchmark alongside results from the 2019 EndoVis challenge sub-challenge for surgical workflow and skill analysis.Source: §5.1 · p. 30

5.2. Data set quality

• The dataset offers reference annotations reflecting variation across hospitals, including differences in video resolution, frame rate, and instruments.It includes the first frame-by-frame clinical surgical-action annotations, skill assessment based on the GOALS score, and open publication for reuse.Source: §5.2 · p. 31
• The dataset’s phase and skill annotations use established references and multiple raters who followed precise rules and resolved disagreements by consensus.Phase annotation was based on Cholec80, and the skill assessment used the established GOALS score.Source: §5.2 · p. 31
• HeiChole’s annotations do not capture the full diversity of vascular anatomy, and action or instrument labels can be difficult when tools leave view or smoke obscures them.The authors suggest that medical-device sensor data could improve annotation reliability; they also caution that resolution and frame-rate differences may affect algorithm performance and confound comparisons across datasets.Source: §5.2 · p. 31

5.3. Surgical phase recognition

• 65% F1-score was the best phase-recognition result in the challenge, compared with 91% average precision previously reported on Cholec80; HeiChole lacked a rigid phase order.Clipping and cutting (P2) and gallbladder packaging (P4) were particularly challenging because their characteristic instruments also appeared in neighboring phases.Source: §5.3 · p. 32
• Phase-recognition results were best on Salem, worst on UKHD, and intermediate on Sinsheim, which was excluded from training; UKHD cases tended to be more complex.The discussion reports that the top three teams combined CNNs with LSTMs and that recurrent approaches generally outperformed methods based on 3D convolutions.Source: §5.3 · p. 33

5.4. Surgical action recognition

• The leading action-recognition algorithm detected almost exclusively the common action hold (A1), and participating methods had similar performance despite using varied approaches.A1 was also assigned to frames labeled as other actions or no action; the study reports that cholecystectomy actions were annotated and recognized for the first time in the literature.Source: §5.4 · p. 33
• Action recognition was complicated by class imbalance, brief and subtle simultaneous actions, and the lack of existing reference results or training datasets from other groups.The authors note that larger training datasets and further research may improve results.Source: §5.4 · p. 33
• F1 scores for “Hold” ranged from 0.84 to 0.91, while average F1 scores for every other action were well below 0.03.Because “Hold” was the most common action, the authors conclude the training data did not adequately represent variation in other actions.Source: §5.4 · p. 33

5.5. Instrument presence detection

• Instrument presence detection can be as challenging as phase recognition, with multiple instrument categories per frame and moving-camera blur impeding detection of small tips.Unlike phase recognition, which can use the whole surgical scene and patient anatomy, instrument detection focuses on small, moving targets.Source: §5.5 · p. 34
• Instrument categories showed phase-linked class imbalance, while submitted results suggested temporal information mattered less than for phase recognition.The highest-performing Konica Minolta method used no temporal information and learned instrument co-occurrences; all five top methods used data augmentation, unlike the bottom three.Source: §5.5 · p. 34

5.6. Skill assessment

• The skill-assessment results did not sufficiently reflect the reference, but were not representative because only one team participated.Raters did not use the full score range, potentially reflecting either the surgeons’ best-effort selection or an error of central tendency.Source: §5.6 · p. 35

5.7. Reducing bias in the challenge

• The HeiChole benchmark was validated through the EndoVis challenge, enabling comparison of international teams’ algorithms for phase, instrument, action, and skill recognition.Challenge performance should not be directly compared across datasets with different resolutions or frame rates without considering their effects; the paper’s plots are approximate representations.Source: §5.7 · pp. 35–36

5.9. Future Research Directions

• Future work should test these tasks on more complex procedures and use larger datasets, ideally supplemented by medical-device sensor data and time- and cost-effective annotation tools.The authors also call for algorithms to incorporate surgical knowledge, including correlations between instruments and phases.Source: §5.9 · pp. 36–37

6. Conclusion

• The authors conclude that surgical workflow and skill analysis are promising technologies to support the surgical team, but remain unsolved.This conclusion is based on their comparison of machine learning algorithms on the HeiChole benchmark.Source: §6 · p. 37
• Continued creation of open, high-quality datasets is described as important for developing accurate and robust machine learning algorithms.Source: §6 · p. 37
• Machine learning algorithms are presented as a foundation for artificial intelligence and cognitive robots in surgery.Source: §6 · p. 37

User Notes

• The dataset is available under a Creative Commons Attribution-NonCommercial-ShareAlike license.Source: User Notes · p. 37
• The dataset is publicly available for non-commercial use.Source: User Notes · p. 37
• Users must cite the paper, and new creations must use the same terms as the current dataset version.Source: User Notes · p. 37

• The disclosures also report work with KARL STORZ in InnOPlan and Surgomics projects.Source: Conflicts of interest · p. 37
• Several authors disclosed employment ties to Caresyntax, KARL STORZ, Konica Minolta, and Wintegral, a Richard Wolf subsidiary.Source: Conflicts of interest · p. 37
• Felix Nickel received conference travel support and course equipment from KARL STORZ, Johnson & Johnson, Intuitive Surgical, Cambridge Medical Robotics, and Medtronic.Source: Conflicts of interest · pp. 37–38

• Conceptualization, methodology, and validation are credited to M. Wagner, B. Müller, and colleagues.Source: CRediT authorship contribution statement · p. 38
• Other listed roles include formal analysis, investigation, visualization, software, project administration, and writing.Source: CRediT authorship contribution statement · p. 38
• Data curation, review and editing, and data analysis are among the listed author contributions.Source: CRediT authorship contribution statement · p. 38
• The statement also credits the senior authors with challenge organization, supervision, resources, and funding acquisition.Source: CRediT authorship contribution statement · p. 38

• Support included the National Center for Tumor Diseases Heidelberg, Helmholtz Incubator, ANR, BPI CONDOR, and Richard und Annemarie Wolf-Stiftung.Source: Funding · p. 39
• Other listed support included German federal ministries, DFG's CeTI cluster, a DAAD scholarship, and the Helmholtz Imaging Platform.Source: Funding · p. 39
• Additional support came from Guangdong and Shenzhen research programs and the Hong Kong RGC TRS project.Source: Funding · p. 39
• The authors state that funding sources had no influence on study design, data handling, reporting, or the decision to submit.Source: Funding · p. 39

Appendix A - Annotation rules

• Annotations span at least two frames because Anvil does not allow independent single-frame annotation.Source: Appendix A · p. 45

Phase annotation rules

• The seven phases run from preparation through retraction, and their sequence is not required to be fixed.Source: Phase annotation rules · p. 45
• For clipping and cutting, one main cystic artery and duct are assumed, excluding additional vessels from that phase annotation.Source: Phase annotation rules · p. 46

Action annotation rules

• Grasp, hold, cut, and clip have distinct boundaries; grasp begins with grasper closure, while cutting requires tissue between the closing blades.Source: Action annotation rules · p. 47
• Hold persists through obscured or off-screen handling, with tissue tension and slipping used to judge release.Source: Action annotation rules · p. 47

Instrument annotation rules

• The protocol labels 21 instruments across seven categories and uses an undefined-shaft label when the instrument cannot be identified.Source: Instrument annotation rules · p. 48
• Instrument presence starts when the characteristic tip appears and continues while its associated shaft remains visible, including brief smoke concealment.Source: Instrument annotation rules · p. 48

• Skill assessment used a modified GOALS score with five ranking components.Source: Skill annotation rules · p. 48
• The four retained GOALS domains were depth perception, bimanual dexterity, efficiency, and tissue handling.Source: Skill annotation rules · p. 48
• Autonomy was omitted because valid assessment requires information beyond intraabdominal video, including speech or senior-surgeon assistance.Source: Skill annotation rules · p. 48
• Skill was annotated for the full operation and phases P1 and P3; difficulty (S5) used Chang’s adaptation based on inflammation, adhesions, and anatomy.Source: Skill annotation rules · p. 48

• The phase-recognition results cover preparation, calot triangle dissection, clipping and cutting, gallbladder dissection, gallbladder packaging, and cleaning and coagulation.Source: Appendix B · p. 49
• The action-recognition results display grasp, hold, cut, and clip for all participants.Source: Appendix B · p. 50
• Instrument-presence results display six categories for all participants; the stapler category is omitted because it was absent from the test dataset.Source: Appendix B · pp. 50–51