Source-linked AI summary
Looking Beyond the Scale: Do Surgical Skill Models Learn Transferable Representations Across Assessment Rubrics?
Hanna Hoffmann, Felix von Bechtolsheim, Stefanie Speidel, Rebecca Hisey
TL;DR
Whether surgical skill models learn transferable representations across assessment rubrics remains unclear. This paper evaluates bidirectional transfer between JIGSAWS and LASANA using supervised, ASAM, and augmentation-based pretraining methods, finding effective transfer to LASANA but consistent failure on JIGSAWS.
Problem
Whether vision-based skill models learn transferable representations across GOALS and OSATS, rather than dataset-specific visual patterns, has not been systematically studied.
Method
The study evaluates bidirectional transfer between JIGSAWS and LASANA using supervised, ASAM, self-supervised, and contrastive training with target-specific head adaptation.
Results
CCC 0.77 to 0.80 on LASANA closely matched the 0.81 end-to-end baseline, whereas transfer to JIGSAWS failed consistently across methods.
Takeaways & Limitations
The findings support generalizable spatiotemporal skill representations across visual domains and assessment scales when the target dataset is sufficiently large and well-annotated.
Takeaways & Limitations
Replication on additional well-annotated datasets is needed to establish these conclusions more firmly.
Abstract
from arXiv · showhide
Vision-based surgical skill assessment has shown strong in-domain results, yet a fundamental question remains unasked: do these models learn transferable representations of surgical proficiency, or do they merely encode dataset-specific visual patterns? This paper systematically analyzes what limits cross-domain skill transfer between the GOALS and OSATS assessment scales using the LASANA and JIGSAWS datasets. Each evaluated method serves a targeted diagnostic purpose: end-to-end training to test whether supervised skill learning transfers directly, Adaptive Sharpness-Aware Minimization (ASAM) to probe whether flatter loss landscapes improve generalization, and augmentation-based self-supervised and contrastive learning to assess whether domain-invariant pretraining decouples skill from visual context. Transfer is evaluated in both directions using a disjoint-participant held-out test set for JIGSAWS. Results reveal an asymmetry: backbones pretrained on JIGSAWS achieve CCC values of 0.77 to 0.80 on LASANA, closely matching the end-to-end baseline, showing cross-rubric transfer is feasible when the target domain provides consistent supervision. Transfer to JIGSAWS fails across all methods, likely due to annotation inconsistencies. Control experiments with a Kinetics-pretrained backbone suggest task-specific heads carry the majority of the skill prediction burden, while the backbone need only provide adequate spatiotemporal features. These findings offer a new perspective on vision-based skill assessment: the central question of whether skill representations transfer across scoring systems has not been previously investigated. Results indicate the visual component is dominant but not solely responsible for skill prediction; further work is needed to conclusively disentangle transferable skill features from those bound to a specific visual domain.
1 Introduction
The introduction motivates standardized, simulation-based surgical education while noting persistent limitations in manual assessment. It then frames the study around bidirectional transfer across visually and contextually distinct datasets under stricter participant-disjoint evaluation.
- Motivation: Surgical education has shifted from apprenticeship-based observation and supervised operating-room repetition toward simulation-based curricula for deliberate practice.Simulation offers repeatable, low-risk environments for training.
- Motivation: OSATS and GOALS were introduced to reduce evaluator bias and provide criterion-referenced feedback, although manual assessment remains limited.The passage links these instruments to standardized assessment while indicating that shortcomings persist.
- Study scope: LASANA and JIGSAWS differ substantially in visual appearance, surgical context, and annotation scheme, enabling bidirectional transfer analysis under domain shift.The analysis also probes visual versus more abstract features in skill prediction.
- Evaluation design: A disjoint-participant held-out test set for JIGSAWS provides a stricter generalization setting than commonly used cross-validation protocols.This design is intended to evaluate generalization without participant overlap.
2 Related Work
Prior surgical skill assessment progressed from kinematic and sensor-based methods to video-based models, but domain shift remains a major challenge. Existing adaptation work shows limited generalization, while augmentation-based self-supervised transfer across differing visual domains and scoring rubrics remains systematically unstudied.
- Vision-Based Skill Assessment: Early skill assessment relied on robotic kinematics and external sensors, whereas newer methods extract skill-relevant features directly from surgical video.JIGSAWS was the predominant benchmark for kinematic-based approaches.
- Domain Shift: Domain shift in surgical computer vision reflects differences in anatomy, instrumentation, camera viewpoints, and institutional practices, but skill-assessment adaptation remains limited.Domain adaptation has been applied more extensively to surgical phase recognition and instrument segmentation.
- Domain Adaptation: 0.46 and 0.41 Spearman Correlation Coefficients were achieved by CoRe-DA on target datasets without labeled target data.CoRe-DA uses contrastive regression, relative score supervision, target-domain self-training, and background mixing to learn domain-invariant representations.
- Domain Adaptation: Existing state-of-the-art skill-assessment models generalize poorly under domain shift.This limitation is reported in the context of cross-domain technical skill assessment and CoRe-DA’s results.
- Self-Supervised Learning: Augmentation-based self-supervised learning may disentangle skill-relevant temporal patterns from domain-specific visual characteristics.Such methods encourage invariance to visual perturbations and can use abundant unlabeled surgical video.
- Open Problem: The utility of augmentation-based self-supervised pretraining for transfer across domains differing in visual appearance and scoring rubric has not been systematically studied.The passage identifies this as an open gap in prior work.
3 Methodology
The methodology evaluates surgical skill assessment across the JIGSAWS and LASANA datasets using a multitask extension of an X3D-based spatiotemporal model. JIGSAWS provides OSATS-based ratings, while LASANA supplies a large-scale laparoscopic video benchmark.
- Datasets: The study uses two surgical skill datasets: JIGSAWS and LASANA.
- Datasets: JIGSAWS contains synchronized stereo video and robot kinematic data from eight participants performing suturing, needle passing, and knot tying on the da Vinci Surgical System.
- Datasets: JIGSAWS skill is annotated with six OSATS categories scored from 1 to 5, plus a modified GRS summing those categories over a range of 6 to 30.
- Datasets: LASANA comprises 1270 stereo recordings of four laparoscopic training tasks performed by 70 participants, including medical students and clinicians.
- Model architecture: The proposed model extends Funke et al.’s single-task approach into multitask learning with an X3D spatiotemporal 3D CNN backbone pretrained on Kinetics-400 and multiple prediction heads.
Workflow
The workflow uses a shared model architecture across supervised and unsupervised approaches, with training differences defining the learning procedure. After training, prediction heads are adapted and fine-tuned on the target dataset when applicable.
- The same model architecture is used for supervised and unsupervised learning, while the training procedure is the primary difference.
- Supervised: Supervised models train end-to-end with a Huber loss, summing the separately calculated losses from each prediction head.
- Unsupervised: Unsupervised training first pretrains the backbone with self-supervised or contrastive learning, then freezes it while fine-tuning model heads on the target dataset.
4 Experiments and Results
Experiments test whether surgical skill models transfer between GOALS and OSATS using supervised, ASAM, and representation-pretraining strategies under consistent evaluation protocols. Results indicate that ASAM does not improve transfer, while cross-domain performance depends strongly on the direction of transfer and supervision available in the target domain.
- Experimental Protocol: All experiments compare methods against a multitask baseline using consistent data splits and evaluate performance with CCC and MAE.CCC is the primary metric because it captures both correlation and agreement.
- Baselines: The in-domain end-to-end model provides the upper-bound reference, while the cross-domain baseline reuses the trained model with prediction heads replaced for the target domain.These baselines establish whether transfer can occur without changing the learned backbone.
- Supervised Training Study: ASAM produces slightly lower performance than supervised training for LASANA in-domain evaluation and transfer in either direction.The experiment tests whether flat minima yield more generalizable visual representations across heterogeneous domains and assessment scales.
- Self-Supervised and Contrastive Pretraining: Self-supervised and contrastive pretraining aim to learn appearance-invariant representations that separate domain-specific visual characteristics from skill-relevant features.The hypothesis is that features not tied to single-domain task labels will transfer more effectively across visual contexts.
- Results Tables: Tables 1 and 2 report test-set results for models pretrained on LASANA and JIGSAWS, respectively, with finetuned-head results shown by target dataset.Values are reported as means over three trials with standard deviations in parentheses.
5 Discussion
Cross-dataset transfer is strongly asymmetric: JIGSAWS-pretrained backbones reach LASANA CCC values of 0.77–0.80, near the 0.81 end-to-end baseline, whereas transfer to JIGSAWS fails amid annotation inconsistencies. A Kinetics control further indicates that task-specific heads can carry much of skill prediction, though additional well-annotated datasets are needed to confirm these conclusions.
- Cross-dataset transfer: CCC 0.77–0.80 on LASANA closely matches the 0.81 LASANA end-to-end Supervised baseline across all four transfer methods.Standard deviations range from 0.01 to 0.05.
- Cross-dataset transfer: Transfer to JIGSAWS is asymmetric, with annotation discrepancies likely disrupting reproduction of score rank structure despite similar absolute score ranges.JIGSAWS MAE values are 0.16 to 0.19 for transfer and 0.18 to 0.23 for baselines, while CCC fluctuates.
- Representation analysis: CCC 0.78 on LASANA and 0.17 on JIGSAWS from a Kinetics-pretrained backbone show that heads can learn skill predictions without surgical or skill-related pretraining.The control uses a backbone pretrained on a large-scale action-recognition dataset and trains heads directly on each target dataset.
- Limitations: Further replication on additional well-annotated datasets is necessary to establish the conclusions more firmly.The limitation specifically concerns confirming the interpretation of transferable features and dataset effects.
6 Conclusion
The study investigated bidirectional skill transfer between JIGSAWS and LASANA, which differ in visual domain, assessment framework, and scale. Transfer was feasible and effective when the target dataset was sufficiently large and well annotated.
- Conclusion: The study evaluated bidirectional skill transfer between JIGSAWS and LASANA.The datasets differ in visual domain, assessment framework, and scale.
- Conclusion: 0.77 to 0.80 CCC: JIGSAWS-pretrained backbones achieved these values on LASANA.This result supports effective cross-dataset transfer when the target dataset is sufficiently large and well annotated.
- Conclusion: 0.81 baseline: LASANA performance closely matched the end-to-end baseline.The passage reports the end-to-end baseline as 0.81.
Financial Disclosure
The authors acknowledge support from German research programs and the German Research Foundation, with TUD Dresden University of Technology identified separately.
- Financial Disclosure: The work acknowledges support from BMFTR’s Communication Systems program and DFG funding under Germany’s Excellence Strategy.The cited projects include 6G-life (16KIS2413K) and the CeTI Cluster of Excellence (EXC 2050/2; Project ID 390696704).
- Financial Disclosure: TUD Dresden University of Technology is listed in the disclosure.
Conflicts of Interest
The authors declare no conflicts of interest.
- No conflicts to declare.