Source-linked AI summary
SurgSkill-Bench: A Benchmark for Multimodal Surgical Skill Assessment
Chaohui Dang, Zheheng Jiang, James Glasbey, David Luke, Theodoros Arvanitis, Le Zhang
TL;DR
Objective surgical skill assessment remains labor-intensive and is poorly supported by approaches that jointly model video, structured scores, and evaluator feedback. SurgSkill-Bench introduces a video-score-text benchmark with video-only and post hoc comment-assisted prediction settings, finding improved video-only performance with content-adaptive sampling and additional score-related signal from comments, with best mean AUROC of 0.88 under dataset-specific median dichotomization.
Problem
Surgical skill assessment depends on labor-intensive expert review, while existing automated approaches provide limited standardized support for jointly modeling operative video, OSATS scores, and evaluator comments.
Method
The paper introduces SurgSkill-Bench and controlled frozen-backbone baselines for video-only and post hoc expert-comment-assisted six-dimensional OSATS prediction.
Results
Under internal video-level validation, CA-Frame was associated with improved video-only performance, evaluator comments added score-related signal in assisted prediction, and best mean AUROC reached 0.88 under dataset-specific median dichotomization.
Takeaways & Limitations
The benchmark establishes a baseline for studying multimodal surgical skill assessment while separating autonomous video-only evaluation from post hoc comment-assisted prediction.
Takeaways & Limitations
The findings remain constrained by dataset scale, incomplete participant metadata, dataset-specific AUROC thresholds, and the assisted nature of comment-based prediction.
Abstract
from arXiv · showhide
Objective assessment of surgical technical skill is important for surgical training and structured feedback, but current workflows remain dependent on labor-intensive expert review. Existing automated approaches primarily focus on visual inputs and provide limited support for jointly studying operative performance, structured skill scores, and evaluator feedback. We introduce SurgSkill-Bench, an initial video-score-text benchmark-style dataset containing 214 surgical training simulation videos, six-dimensional OSATS scores, and expert free-text comments. We define two evaluation settings: video-only OSATS prediction for automated assessment and post hoc expert-comment-assisted prediction, where evaluator comments are available as auxiliary information. We provide controlled baseline experiments using representative frozen visual backbones, content-adaptive key-frame sampling, and a simple video-text co-attention fusion module. Under internal video-level validation, content-adaptive sampling improves video-only performance in this dataset, while evaluator comments provide additional score-related signal in the assisted setting. The best mean AUROC reaches 0.88 under dataset-specific median dichotomization. We further discuss evaluation constraints related to dataset scale, metadata completeness, and the interpretation of comment-assisted prediction. Code will be released publicly at a later date.
1 Introduction
SurgSkill-Bench addresses the need for scalable, standardized surgical skill assessment that combines operative video, structured OSATS scores, and evaluator feedback. It establishes a pilot benchmark with controlled evaluation settings and baselines.
- OSATS assessment is clinically relevant but requires labor-intensive expert review and remains vulnerable to inter-rater variability and subjective bias.
- Existing surgical video approaches face difficulty capturing subtle skill cues because training videos contain long low-information intervals and repeated movements.
- SurgSkill-Bench contains 214 surgical training simulation clips with six-dimensional OSATS ratings and evaluator free-text comments.
- The benchmark defines video-only OSATS prediction and post hoc expert-comment-assisted OSATS prediction settings.
- The work provides controlled baselines using frozen visual backbones, shared regression heads, and content-adaptive sampling rather than claiming a new state-of-the-art architecture.
2 Dataset & Benchmark
The benchmark pairs 214 surgical simulation clips with consensus OSATS annotations and evaluator comments, supporting video-only and post hoc comment-assisted prediction. Its evaluation uses continuous regression targets alongside dataset-specific median-based discrimination analysis.
- SurgSkill-Bench includes 214 curated surgical training simulation clips, each linked to six-dimensional OSATS rating events and available evaluator comments.The clips are approximately one minute long and are intended for postoperative skill assessment rather than intraoperative decision support.
- Two or more expert raters score six OSATS dimensions on a 1–5 ordinal scale, with mean scores forming continuous video-level consensus targets.
- Evaluator comments describe technical performance, but comment-assisted prediction is interpreted as post hoc because the same evaluators supplied comments and scores.
- Models regress continuous consensus scores using MAE and MSE, while supplementary AUROC uses median-based binarization for each dimension.
- The assisted framework combines frozen video and text encoders through bidirectional co-attention before six OSATS regression heads, whereas video-only removes the text branch and fusion module.
3 Methodology
The methodology uses modular frozen-encoder baselines for video-only and comment-assisted OSATS prediction, with CA-Frame selecting visually distinct frames before temporal regression. Comment features are fused through controlled bidirectional co-attention.
- Expert-comment-assisted fusion: The benchmark framework extracts video and evaluator-comment representations, fuses them, and predicts six OSATS scores; video-only omits text and fusion.
- Content-Adaptive Key-Frame Extraction: CA-Frame selects candidate key-frames using InceptionV3 features and cosine distance, then falls back to uniform sampling when too few key-frames are available.The method uses threshold τ = 0.05 and preserves chronological order when selected frames suffice.
- Content-Adaptive Key-Frame Extraction: CA-Frame may miss subtle skill cues or fine-grained motion continuity because it relies on ImageNet-pretrained InceptionV3 features.
- Expert-comment-assisted fusion: Evaluator comments are encoded with frozen GPT-2, pooled into embeddings, and aggregated with multi-head attention before fusion and regression.Missing comments use a neutral placeholder; projection, aggregation, fusion, and regression layers remain trainable.
- Expert-comment-assisted fusion: The co-attention block is a controlled fusion baseline that operates on projected video and comment feature sequences.The fused features undergo residual connections, layer normalization, temporal pooling, and feed-forward processing.
- Multi-Task Regression and Optimization: Frozen visual and textual encoders leave projection layers, fusion modules, and six regression heads trainable for continuous OSATS prediction.Regression is used because averaged rater scores produce continuous consensus labels.
4 Experiments
The experiments compare frozen visual backbones, frame-sampling strategies, and expert-comment-assisted prediction for six-dimensional OSATS assessment. Content-adaptive sampling improves video-only performance in internal validation, while comments add score-related signal in the assisted setting.
- Baselines: Experiments evaluate frozen pretrained visual backbones with comparable projection layers and six OSATS regression heads.The evaluated backbones include ViViT, VideoMAE, DINOv3, V-JEPA 2, X-CLIP, and a surgical SSL baseline.
- Experimental Settings: Table 1 compares video-only models under standard and key-frame extraction, while Table 2 evaluates the corresponding post hoc comment-assisted setting.The tables report mean ± standard deviation for the two extraction conditions.
- Evaluation Metrics: MAE and MSE are primary metrics for ordinal 1–5 OSATS regression, with quadratic weighted Cohen’s kappa as a secondary agreement statistic.Metrics are macro-averaged across the six OSATS dimensions; kappa uses rounded and clipped predictions and labels.
- Frame Sampling: CA-Frame is associated with improved video-only performance for most backbones under the internal video-level protocol.CA-Frame selects candidate key-frames using feature-space change and preserves chronological order, falling back to uniform sampling when necessary.
- Quantitative Results: 0.88 mean AUROC is reached by ViViT and V-JEPA 2 with key-frame inputs under dataset-specific median dichotomization.VideoMAE improves from 0.57 to 0.86 AUROC and ViViT from 0.55 to 0.85 AUROC under the reported video-only comparison.
- Comment Assistance: Evaluator comments improve several models in the post hoc assisted setting, but the benefit is not uniform across backbones and metrics.The assisted setting should not be conflated with autonomous video-only assessment.
- Qualitative Visualization: Selected attention maps visually appear more concentrated around instruments, hands, and tissue interactions for the CA-Frame-guided surgical SSL encoder than for a generic baseline.These are qualitative examples rather than quantitative evidence of interpretability or clinically meaningful reasoning.
5 Conclusion
The paper introduces SurgSkill-Bench as a benchmark combining surgical videos, six-dimensional OSATS scores, and evaluator comments, with separate video-only and assisted settings. Internal validation associates CA-Frame with improved video-only performance and comments with additional score-related signal, while several dataset and evaluation constraints remain.
- Conclusion: SurgSkill-Bench combines surgical training videos, six-dimensional OSATS scores, and evaluator free-text comments in an initial benchmark.The benchmark is designed to establish a reproducible setting and baseline protocol rather than claim a new state-of-the-art architecture.
- Conclusion: The benchmark defines separate video-only and post hoc expert-comment-assisted prediction settings with controlled baselines.The baselines use frozen visual backbones, content-adaptive sampling, and video-text fusion.
- Conclusion: Internal video-level validation associates CA-Frame with improved video-only performance, while evaluator comments provide additional score-related signal in the assisted setting.These findings establish a benchmark baseline within the reported evaluation protocol.
- Limitations: The findings remain constrained by dataset scale, incomplete participant metadata, dataset-specific AUROC thresholds, and the assisted nature of comment-based prediction.Planned future work includes dataset expansion, participant-level validation, text-leakage controls, and human-agreement analysis.