Semantic RGB--Depth Based Surgical Skill Assessment in Microscopic Stereo Videos

• Microsurgery demands precise bimanual control in confined anatomy, while expert observation is subjective, time-consuming, and susceptible to inter-rater variability.Source: §1 · p. 1
• Video-based assessment operates on recordings without dedicated tracking hardware, supporting retrospective analysis across clinical settings.Source: §1 · p. 2
• RGB-only video methods largely omit 3D instrument positioning and anatomy relationships, while high-magnification stereo matching can yield sparse or noisy depth.Source: §1 · p. 2
• The framework separates instruments and anatomy into semantic streams, then uses hierarchical attention to model instrument use and instrument–anatomy interactions.Each stream has a dedicated temporal convolutional network, allowing grasper, scissors, and anatomy branches to learn distinct temporal dynamics before global fusion.Source: §1 · p. 2
• The semantic RGB–Depth approach achieved the strongest overall performance, with depth providing complementary information to RGB appearance.Source: §1 · p. 2
• Regression-based fusion combines sparse stereo depth with dense monocular estimates to recover metrically consistent depth maps from microscopic stereo video.Source: §1 · p. 2

2.1 Dense Metric Foreground Reconstruction

• Prompt-based first-frame masks identify the grasper, scissors, and foreground anatomy, then propagate through the sequence to isolate each semantic stream.The propagated masks support separate RGB and depth feature extraction for each scene component.Source: §2.1 · p. 3
• Sparse stereo measurements are aligned with dense monocular relative depth within the foreground to recover dense metric depth while retaining stereo scale.Stereo measurements serve as metric references where available; monocular estimates supply dense relative structure.Source: §2.1 · pp. 3–4

2.2 Semantic-Stream Skill Representation

• Each instrument and anatomy stream combines projected RGB and depth features and receives its own dilated TCN, allowing distinct temporal dynamics before global fusion.The independent temporal encoders model the grasper, scissors, and anatomy branches separately.Source: §2.2 · p. 4
• Stream attention weights each semantic representation within a frame, global temporal attention weights fused frames, and weighted local skill scores form the trial prediction.The global temporal attention is learned from the stream-fused representation, independently of local stream-specific temporal attention.Source: §2.2 · p. 5
• Spatial maps visualize attention within the scissors and grasper masks around stream-specific peak moments, sampled over seven-frame windows.The maps are upsampled to the original image resolution; warmer colors indicate greater attention.Source: Figure 3 · p. 6

3 EXPERIMENTS AND RESULTS

• The dataset comprised 33 laryngeal microsurgery videos from six surgeons, evaluated with leave-one-surgeon-out cross-validation.The surgeons comprised three attending surgeons and three residents; participant-level training status defined the binary classification labels.Source: §3 · pp. 6–7
• Mean Dice was 0.9582 at initialization and 0.9731 across CUTIE-propagated sequences in one representative trial per participant.The evaluation covered six videos: three expert and three resident cases.Source: Table 2 · p. 7, §3 · p. 7
• The semantic RGB + SPP-depth model achieved the lowest MAE and highest F1 among the evaluated representations.SPP-based depth alone also performed strongly, while ASPP depth was less effective.Source: Table 3 · p. 8, §3 · p. 8
• Removing semantic decomposition reduced F1 from 0.938 to 0.343, while removing temporal attention and auxiliary supervision reduced it to 0.652 and 0.533, respectively.The ablations indicate that all three components contribute to performance under participant-disjoint evaluation.Source: Table 3 · p. 8, §3 · p. 8
• The adapted USSA baseline predicted held-out expert trials predominantly as residents and had zero expert-class precision, recall, and F1.The participant-wise results show that this baseline predicted all trials as resident.Source: §3 · p. 8, Table L · p. 9
• The strongest model weighted anatomy at 0.47 ± 0.38, scissors at 0.28 ± 0.36, and grasper at 0.24 ± 0.37.Semantic RGB emphasized the grasper most, while depth representations distributed attention more broadly across instruments and anatomy.Source: §3 · p. 8, Table 5 · p. 9
• With SPP depth alone, resident temporal-attention peaks were near the sequence end at 0.99 ± 0.01, but shifted to 0.59 ± 0.36 with RGB fusion.Expert peaks were 0.49 ± 0.26 for SPP depth and 0.57 ± 0.44 for SPP depth combined with semantic RGB.Source: §3 · p. 8, Table L · p. 9
• Around peak-attention frames, experts showed a pronounced anatomy contribution, whereas residents had lower, more evenly distributed stream contributions.An independent attending reviewer identified instrument tips and pivot regions as prominent and clinically plausible spatial-attention areas.Source: §3 · pp. 8–9, Figure 5 · p. 9

4 CONCLUSIONS

• The framework combines sparse stereo and dense monocular depth, then jointly models scene geometry, instrument appearance, motion, and instrument–tissue interactions with hierarchical attention.Source: §4 · p. 9
• Evaluation used leave-one-surgeon-out validation on ex vivo transoral microlaryngeal procedures.Source: §4 · p. 9
• Depth-derived geometric information provided complementary information for skill assessment beyond RGB appearance alone.Source: §4 · p. 10
• Attention patterns offered qualitative views of spatial, temporal, and semantic weighting, but their causal influence on predicted skill levels was not established.Source: §4 · p. 10
• The study covered one ex vivo dataset and did not evaluate reconstructed depth against dense per-pixel metric ground truth.The authors identify larger, multiinstitutional and in vivo datasets, plus more comprehensive geometric ground truth, as future validation directions.Source: §4 · p. 10