SurgSkill-Bench: A Benchmark for Multimodal Surgical Skill Assessment

• OSATS assessment is clinically relevant but requires labor-intensive expert review and remains vulnerable to inter-rater variability and subjective bias.Source: §1 · p. 2
• Existing surgical video approaches face difficulty capturing subtle skill cues because training videos contain long low-information intervals and repeated movements.Source: §1 · p. 2
• SurgSkill-Bench contains 214 surgical training simulation clips with six-dimensional OSATS ratings and evaluator free-text comments.Source: §1 · p. 2
• The benchmark defines video-only OSATS prediction and post hoc expert-comment-assisted OSATS prediction settings.Source: §1 · p. 2
• The work provides controlled baselines using frozen visual backbones, shared regression heads, and content-adaptive sampling rather than claiming a new state-of-the-art architecture.Source: §1 · p. 2

• SurgSkill-Bench includes 214 curated surgical training simulation clips, each linked to six-dimensional OSATS rating events and available evaluator comments.The clips are approximately one minute long and are intended for postoperative skill assessment rather than intraoperative decision support.Source: §2.1 · pp. 2–3
• Two or more expert raters score six OSATS dimensions on a 1–5 ordinal scale, with mean scores forming continuous video-level consensus targets.Source: §2.2 · p. 3
• Evaluator comments describe technical performance, but comment-assisted prediction is interpreted as post hoc because the same evaluators supplied comments and scores.Source: §2.2 · p. 3
• Models regress continuous consensus scores using MAE and MSE, while supplementary AUROC uses median-based binarization for each dimension.Source: §2.3 · p. 3
• The assisted framework combines frozen video and text encoders through bidirectional co-attention before six OSATS regression heads, whereas video-only removes the text branch and fusion module.Source: Figure 1 · p. 4

Expert-comment-assisted fusion

• The benchmark framework extracts video and evaluator-comment representations, fuses them, and predicts six OSATS scores; video-only omits text and fusion.Source: §3 · p. 4

Content-Adaptive Key-Frame Extraction

• CA-Frame selects candidate key-frames using InceptionV3 features and cosine distance, then falls back to uniform sampling when too few key-frames are available.The method uses threshold τ = 0.05 and preserves chronological order when selected frames suffice.Source: §3.1 · p. 4, Figure 2 · p. 5
• CA-Frame may miss subtle skill cues or fine-grained motion continuity because it relies on ImageNet-pretrained InceptionV3 features.Source: §3.1 · p. 5

Expert-comment-assisted fusion

• Evaluator comments are encoded with frozen GPT-2, pooled into embeddings, and aggregated with multi-head attention before fusion and regression.Missing comments use a neutral placeholder; projection, aggregation, fusion, and regression layers remain trainable.Source: §3.2 · p. 5
• The co-attention block is a controlled fusion baseline that operates on projected video and comment feature sequences.The fused features undergo residual connections, layer normalization, temporal pooling, and feed-forward processing.Source: §3.2 · p. 5

Multi-Task Regression and Optimization

• Frozen visual and textual encoders leave projection layers, fusion modules, and six regression heads trainable for continuous OSATS prediction.Regression is used because averaged rater scores produce continuous consensus labels.Source: §3.3 · p. 6

Baselines

• Experiments evaluate frozen pretrained visual backbones with comparable projection layers and six OSATS regression heads.The evaluated backbones include ViViT, VideoMAE, DINOv3, V-JEPA 2, X-CLIP, and a surgical SSL baseline.Source: §4 · pp. 6–7

Experimental Settings

• Table 1 compares video-only models under standard and key-frame extraction, while Table 2 evaluates the corresponding post hoc comment-assisted setting.The tables report mean ± standard deviation for the two extraction conditions.Source: Table 1 · p. 7, Table 2 · p. 8

Evaluation Metrics

• MAE and MSE are primary metrics for ordinal 1–5 OSATS regression, with quadratic weighted Cohen’s kappa as a secondary agreement statistic.Metrics are macro-averaged across the six OSATS dimensions; kappa uses rounded and clipped predictions and labels.Source: §4 · p. 7

Frame Sampling

• CA-Frame is associated with improved video-only performance for most backbones under the internal video-level protocol.CA-Frame selects candidate key-frames using feature-space change and preserves chronological order, falling back to uniform sampling when necessary.Source: §4 · p. 7

Quantitative Results

• 0.88 mean AUROC is reached by ViViT and V-JEPA 2 with key-frame inputs under dataset-specific median dichotomization.VideoMAE improves from 0.57 to 0.86 AUROC and ViViT from 0.55 to 0.85 AUROC under the reported video-only comparison.Source: §4 · p. 7

Comment Assistance

• Evaluator comments improve several models in the post hoc assisted setting, but the benefit is not uniform across backbones and metrics.The assisted setting should not be conflated with autonomous video-only assessment.Source: §4 · p. 7

Qualitative Visualization

• Selected attention maps visually appear more concentrated around instruments, hands, and tissue interactions for the CA-Frame-guided surgical SSL encoder than for a generic baseline.These are qualitative examples rather than quantitative evidence of interpretability or clinically meaningful reasoning.Source: §4 · p. 8, Figure 3 · p. 8

Conclusion

• SurgSkill-Bench combines surgical training videos, six-dimensional OSATS scores, and evaluator free-text comments in an initial benchmark.The benchmark is designed to establish a reproducible setting and baseline protocol rather than claim a new state-of-the-art architecture.Source: §5 · p. 8
• The benchmark defines separate video-only and post hoc expert-comment-assisted prediction settings with controlled baselines.The baselines use frozen visual backbones, content-adaptive sampling, and video-text fusion.Source: §5 · p. 8
• Internal video-level validation associates CA-Frame with improved video-only performance, while evaluator comments provide additional score-related signal in the assisted setting.These findings establish a benchmark baseline within the reported evaluation protocol.Source: §5 · pp. 8–9

Limitations

• The findings remain constrained by dataset scale, incomplete participant metadata, dataset-specific AUROC thresholds, and the assisted nature of comment-based prediction.Planned future work includes dataset expansion, participant-level validation, text-leakage controls, and human-agreement analysis.Source: §5 · p. 9