Video and Accelerometer-Based Motion Analysis for Automated Surgical Skills Assessment

1 Introduction

• Manual assessment remains subjective and resource-limited, with only a few expert surgeons available to score trainees and provide feedback.OSATS is widely adopted and includes criteria such as Respect for Tissue, Time and Motion, and Overall Performance.Source: §1 · p. 2
• The proposed entropy-based features quantify predictability and regularity in surgical motion to assess skills.The paper presents this approach as a response to the time-consuming and subjective nature of manual assessment.Source: §1 · p. 2
• The study compares video and acceleration features for basic surgical skills assessment, including suturing and knot tying, in what the authors call the largest study of its kind.The authors compare existing techniques on both modalities and tasks.Source: §1 · p. 2

2 Background

• Earlier RMIS skill-assessment work used Hidden Markov Models, linear dynamical systems, and bag-of-words models.These approaches focused on robotic minimally invasive surgery.Source: §2 · p. 2
• Earlier video methods modeled motion as event sequences, frame-kernel textures, or sequential motion textures; later work also used DFT and DCT features.The cited video literature included an exhaustive OSATS analysis, but its reported results used video data only.Source: §2 · pp. 2–3
• Prior studies used few participants, focused on motion granularity or repetitiveness, and did not address disorder or wearable motion sensing such as accelerometers.The authors note that small samples limit the ability to capture wide variation in surgical skills.Source: §2 · p. 3
• Entropy features with pose had been used to predict Olympic diving scores, inspiring the paper’s use of entropy to encode predictability in surgical motion.Source: §2 · p. 3
• The authors propose entropy features to encode orderliness in motion and report that they outperform other feature types for video and acceleration data.Their comparative analysis uses the largest participant group published thus far, to the authors’ knowledge.Source: §2 · p. 3

3 Methodology

• Video motion is encoded as a K-by-N motion-class time series, while two accelerometers’ x, y, and z signals are concatenated into a 6-by-Q time series.Both modalities’ time series are used for feature extraction and skill prediction.Source: §3 · p. 3
• Sequential forward feature selection reduces feature dimensionality, and a Nearest-Neighbor classifier performs classification.The processing flow is shown in Fig. 1.Source: §3 · p. 4, Figure 1 · p. 4

3.1 Entropy Features for Skill Assessment

• ApEn measures time-series regularity, while XApEn measures asynchrony between two time series to represent motion predictability and coordination.ApEn can capture repetitive patterns; XApEn is proposed to capture how synchronized a surgeon’s hands and tools are.Source: §3.1 · pp. 4, 6
• ApEn decreases as signal-to-noise ratio increases, consistent with less noisy sinusoids being more predictable.The demonstration varies SNR from 1 to 50 and examines different radius values.Source: §3.1 · p. 5, Figure 2 · p. 5
• XApEn peaks at about 0.5π phase difference and returns to zero at π, where the signals are out of phase but synchronized.The figure compares sinusoids with the same SNR and phase differences from 0 to π.Source: §3.1 · p. 5, Figure 2 · p. 5

4.1 Data Set

• The dataset grew from 18 to 41 participants and included two instances each of suturing and knot tying, with 74 videos per task after exclusions.Video was recorded at 30 frames per second; the dataset added 23 participants to the prior 18-participant dataset.Source: §4.1 · p. 6
• Accelerometer recordings used two sensors per task, with 54 knot-tying and 62 suturing recordings retained after noisy or detached-sensor cases were removed.Sensors were placed on both wrists for knot tying, and on the dominant wrist and needle-holder for suturing.Source: §4.1 · p. 7

4.2 Parameter Selection

• Video motion classes were tested at K values from 2 to 20, while accelerometer features used a fixed six-dimensional time series.The tested K values were 2–10 and 12, 14, 16, 18, and 20.Source: §4.2 · p. 7
• ApEn was calculated across six radius values from 0.1 to 0.25 times the time-series standard deviation, producing a 6K-dimensional feature vector.K is six for acceleration data and variable for video data.Source: §4.2 · p. 7
• XApEn used the same radius values for accelerometer data but only r = 0.2 for video, where values varied little and computation time rose with dimensionality.The study also evaluated concatenating ApEn and XApEn before classification and used LOOCV with a nearest-neighbor classifier after SFFS feature selection.Source: §4.2 · p. 8

5 Results and Discussion

• Combined ApEn+XApEn achieved the highest average video accuracy for both suturing and knot tying across applicable OSATS criteria.Figure 4 compares average accuracy across K values; the results are averaged across OSATS criteria.Source: §5 · p. 8, Figure 4 · p. 8
• Combining both accelerometers performed better than either individual sensor for both tasks and all evaluated feature types.The sensors were placed at the wrist and needle-holder for suturing, and at both wrists for knot tying.Source: §5 · p. 8, Figure 5 · p. 9
• Sensor-position analysis may help identify task-relevant motions and provide surgeons with more targeted performance-improvement feedback.The authors present this as a potential use of the analysis.Source: §5 · p. 9
• Entropy features performed better for video and suturing accelerometer data, but not for knot-tying accelerometer data, where the authors cite limited time-series dimensionality.The authors attribute the accelerometer limitation to its three-axis data, which may capture insufficient information for entropy features, especially XApEn.Source: §5 · p. 9, Table 2 · p. 9
• Video outperformed accelerometer data for all feature types, consistent with video capturing more motion than the hands and needle-holder’s 3-D acceleration.Figure 6 reports individual OSATS-criterion results using each feature’s optimal K.Source: §5 · p. 9, Figure 6 · p. 10
• ApEn+XApEn achieved the highest performance for each task; fusion improved knot-tying performance but slightly reduced suturing performance versus video alone.DCT and DFT performance deteriorated with fusion; the comparisons use the available paired data: 62 suturing and 54 knot-tying videos.Source: §5 · p. 10, Table 3 · p. 11, Table 4 · p. 11
• ApEn+XApEn outperformed DCT and DFT under 2-, 5-, and 10-fold cross-validation schemes.For these comparisons, the best-performing modality was used for each feature: video for DCT and DFT, and video for suturing or video plus accelerometer for knot tying with ApEn+XApEn.Source: §5 · p. 10, Figure 7 · p. 11
• The authors suggest entropy features may generalize beyond suturing and knot tying because they capture motion irregularity rather than only repetitiveness.This is presented as an expectation, not a tested result for other procedures.Source: §5 · p. 10

• Video features provide more skill-relevant information than accelerometer features, while fusion improves performance and combined ApEn/XApEn outperforms state-of-the-art features.Source: §6 · p. 11
• Videos are better than accelerometer data for extracting skill-relevant information.Source: §6 · p. 11
• Automated assessment could improve training quality, enable more practice with valuable feedback, and save expert surgeons time spent assessing trainees.Source: §6 · p. 11