Looking Beyond the Scale: Do Surgical Skill Models Learn Transferable Representations Across Assessment Rubrics?

1 Introduction

• OSATS and GOALS provide structured assessment, but manual evaluation requires expert time, limiting scalable feedback despite the link between technical proficiency and patient outcomes.The introduction situates these rubrics amid a shift from traditional apprenticeship-based surgical education.Source: §1 · pp. 1–2
• Vision-based models may learn domain-specific cues, such as robotic instruments or task geometry, rather than generalizable proficiency features, limiting their relevance across clinical settings.Understanding this distinction is presented as important for assessment tools that generalize across procedures, environments, and evaluation frameworks.Source: §1 · p. 2
• The study addresses an unexplored question: whether vision-based skill representations transfer across GOALS and OSATS or mainly encode rubric- and domain-specific visual patterns.Prior work largely trained and evaluated models within a single assessment framework; cross-rubric transfer is proposed as a diagnostic of what models learn about skill.Source: §1 · p. 2
• Using JIGSAWS and LASANA, the analysis tests transfer in both directions across datasets differing in visual appearance, surgical context, and annotation scheme.JIGSAWS evaluation uses a disjoint-participant held-out test set, a stricter generalization setting than commonly used cross-validation and, to the authors’ knowledge, the first such multitask evaluation on JIGSAWS [20].Source: §1 · p. 2
• The authors explore ASAM during pretraining and augmentation-based self-supervised and contrastive learning to encourage more transferable surgical-skill representations.The analysis also probes the relative contribution of visual versus more abstract features under domain shift.Source: §1 · p. 2

2.1 Vision-Based Surgical Skill Assessment

• Video-based models assess surgical skill from video alone, and temporal modeling consistently outperforms frame-level approaches [23] [24].AIxSuture extended video-only prediction to open surgery beyond the robotic domain.Source: §2.1 · p. 2

2.2 Domain Adaptation and Transfer Learning in Surgical AI

• Prior transfer work spans kinematic SSL [25], ophthalmic video, few-shot pretraining [27], and CoRe-DA [28], but source-only models generalize poorly under domain shift.CoRe-DA reports target-domain Spearman Correlation Coefficients of 0.46 and 0.41 without labeled target data [28]; this paper instead studies transfer across GOALS and OSATS and what vision models learn.Source: §2.2 · pp. 2–3

2.3 Self-Supervised Learning for Surgical Video

• Augmentation-based contrastive SSL encourages invariance to visual perturbations, but its utility for transfer across different assessment rubrics has not been systematically studied.Prior work used contrastive principles and background mixing to partially align low-level visual statistics [28]; this paper explores SSL across GOALS and OSATS.Source: §2.3 · p. 3

3.1 Datasets

• JIGSAWS provides 103 videos across three tasks, with OSATS category ratings and a participant-disjoint 70/10/20 train/validation/test split.The modified GRS sums six categories scored from 1 to 5; the split stratifies participants by quantile-binned mean GRS and includes all tasks.Source: §3.1 · pp. 3–4
• LASANA contributes 314 suture-and-knot videos rated on modified GOALS, selected to reduce visual-domain differences from JIGSAWS.Four assessors’ maximum-normalized ratings are averaged; predefined splits contain 232 training, 32 validation, and 50 test recordings.Source: §3.1 · p. 4

3.2 Model

• The model uses an X3D backbone [30], an LSTM, average clip pooling, and rubric-specific prediction heads to predict category scores and their summed GRS.X3D is pretrained on Kinetics-400; clips contain 16 frames sampled every fifth frame, center-cropped to 224×224 and ImageNet-normalized.Source: §3.2 · p. 4

• Both approaches use the same architecture, with target-dataset prediction-head adaptation and finetuning after training when applicable.Source: Workflow · p. 4

Supervised

• In the supervised setting, end-to-end training uses a Huber loss calculated separately for each head and summed across heads.Source: Workflow · p. 4

Unsupervised

• In the unsupervised setting, the backbone is pretrained with VICReg-based SSL or triplet-margin CL, then frozen while heads are supervised-finetuned on the target dataset.SSL uses augmented views of the same sample; CL uses an augmented anchor as the positive and an unaugmented sample from another video as the negative.Source: Workflow · p. 4

Experiments and Results

• Experiments compare approaches for transfer between GOALS and OSATS using consistent data splits and the multitask baseline; CCC is primary, with MAE reported alongside it.CCC captures correlation and agreement between predicted and true skill levels, while MAE conveys absolute prediction error magnitude.Source: §4 · p. 5

Baseline

• The in-domain multitask baseline achieves CCC 0.81 and MAE 0.07 on LASANA, versus CCC -0.03 and MAE 0.23 on JIGSAWS.Extensive trials varying hyperparameters, seeds, splits, and model components produced similar JIGSAWS results, leading the authors to suspect a dataset-level issue.Source: §4 · p. 5

Supervised Training Study

• ASAM does not meaningfully improve transfer in either direction, and its performance is slightly below supervised training for in-domain LASANA evaluation.The authors suggest flat-minima optimization alone is insufficient to address the domain gap; conclusions involving JIGSAWS are limited by annotation inconsistencies.Source: §4 · p. 5

Self-Supervised and Contrastive Pretraining

• For LASANA-to-JIGSAWS transfer, both SSL and contrastive learning outperform fully supervised methods, while both underperform the supervised cross-domain model when transferring to LASANA.The SSL model pretrained on JIGSAWS slightly outperforms the in-domain LASANA SSL model; interpretations involving JIGSAWS require caution.Source: §4 · p. 5

Discussion

• CCC values 0.77 to 0.80 for JIGSAWS-to-LASANA transfer closely match LASANA’s end-to-end Supervised baseline of 0.81, while reverse transfer yields only 0.06 to 0.20.The JIGSAWS end-to-end Supervised model achieves CCC -0.03 (± 0.08), and the best JIGSAWS baselines, ASAM and SSL, reach only 0.22; thus poor JIGSAWS results are not unique to transfer.Source: §5 · p. 6
• MAE values remain 0.16 to 0.19 for JIGSAWS transfer and 0.18 to 0.23 for baselines while CCC fluctuates, consistent with score-range prediction but poor rank-order prediction.Hyperparameter searches, architectural changes, and training adjustments did not meaningfully improve results; annotation review found experts with unexpectedly low scores and novices with scores comparable to or above more experienced operators, as reflected in GRS distributions in Figure 2.Source: §5 · pp. 6–7
• Hendricks et al. found substantial overlap and inconsistent ordering of JIGSAWS GRS distributions across skill classes, while robot-side kinematic indicators, particularly acceleration, correlated with GRS.The participant-disjoint held-out protocol may expose label limitations that LOSO protocols masked through participant-specific patterns; the resulting absolute scores are not directly comparable to published JIGSAWS benchmarks.Source: §5 · p. 7
• 103 versus 314 videos and 8 versus 70 participants, together with JIGSAWS target training of about 70 videos from 5 participants, complicate transfer comparisons.LASANA’s larger, well-structured target data supports stable GOALS head training, whereas small sample size and questionable labels limit JIGSAWS target learning; visual domains and OSATS/GOALS skill constructs also differ, but their effects cannot be disentangled.Source: §5 · p. 7
• CL reaches the highest LASANA-to-JIGSAWS transfer CCC at 0.20 (± 0.06), versus Supervised at 0.08 and ASAM at 0.06, but remains far below useful prediction.In the reverse direction, all methods converge to CCC 0.77 to 0.80, suggesting pretraining strategy matters less when the target dataset has sufficient quality labels for head training.Source: §5 · p. 7
• CCC 0.78 on LASANA and 0.17 on JIGSAWS from a Kinetics-pretrained backbone indicate that general video features can support skill prediction when target labels are adequate.The Kinetics result is close to JIGSAWS-pretrained transfer on LASANA (CCC 0.77 to 0.80) and the Supervised baseline (0.81), suggesting task-specific heads carry most predictive capacity; JIGSAWS cannot validate this inference, and replication on other well-annotated datasets is needed.Source: §5 · p. 8

6 Conclusion

• CCC values of 0.77 to 0.80 on LASANA from JIGSAWS-pretrained backbones closely matched the end-to-end baseline of 0.81, while transfer to JIGSAWS failed across all methods.The authors relate the JIGSAWS failure to Hendricks et al., who found that its GRS scores lack a consistent relationship with operator experience.Source: §6 · p. 8
• A Kinetics-pretrained backbone achieved a CCC of 0.78 on LASANA, nearly matching surgical-domain transfer methods, as task-specific heads carried most of the skill-learning burden.The authors suggest that lightweight head training on well-annotated target data could reduce the need for costly domain-specific pretraining and large surgical training corpora.Source: §6 · p. 8
• Future work should validate transfer and head-driven learning on more well-annotated datasets and establish larger benchmarks for rigorous held-out-test evaluation.The authors also call for revisiting established baseline assumptions so evaluation reflects predictive capability rather than dataset-structure artifacts.Source: §6 · p. 8

• BMFTR support was provided through the research program “Souverän. Digital. Vernetzt.” and joint project 6G-life (16KIS2413K).Source: Financial Disclosure · p. 9
• DFG funded the Centre for Tactile Internet with Human-in-the-Loop (CeTI) at TUD Dresden University of Technology under Excellence Strategy project EXC 2050/2 (390696704).Source: Financial Disclosure · p. 9

Conflicts of Interest

• The authors declare no conflicts to declare.Source: Conflicts of Interest · p. 9