Recognizing Surgical Activities with Recurrent Neural Networks

• Automated activity recognition can support objective skill assessment and targeted trainee feedback; maneuver-level feedback aligns with faculty surgeons' subjective feedback.Source: §1 · p. 1
• Prior work focused on short gestures,,,, whereas maneuvers comprise gesture sequences such as knot tying.Source: §1 · p. 1
• Earlier HMM/CRF systems use local-neighborhood unary terms; LSTM learns memory operations for sequence-wide unaries, and independent-label predictions remain temporally smooth without post-processing.Source: §1 · pp. 1–2
• One RNN model and hyperparameter set matches state-of-the-art gesture recognition and improves maneuver recognition in both accuracy and edit distance.Source: §1 · p. 2

2 Methods

• The task is supervised mapping from kinematic sequences to activity labels, with the goal of generalizing to users absent from training.The models predict from past and present inputs online, or from the full sequence offline.Source: §2 · p. 2

2.1 Recurrent Neural Networks

• A generic RNN uses its previous hidden state and current input to produce a new hidden state and output; a specified nonlinear block gives a vanilla RNN.The vanilla RNN's parameters are shared over time.Source: §2.1 · pp. 2–3, Figure 2 · p. 3
• For multiclass classification, a linear layer and softmax produce class probabilities, while bidirectional RNNs can use future inputs to improve offline performance.The bidirectional model runs forward and backward RNNs, concatenates their hidden states, and computes outputs jointly.Source: §2.1 · p. 3

2.2 Long Short-Term Memory

• LSTM was designed to address the vanilla RNN vanishing-gradient training problem, and the LSTM parameters are shared across time.The paper gives the recursive equations for its LSTM block.Source: §2.2 · pp. 3–4
• Assuming each label is independent of other labels given the kinematics makes maximizing log likelihood equivalent to minimizing sequence-level cross-entropy.The paper trains by stochastic gradient descent and obtains gradients with backpropagation.Source: Loss · p. 4, Training · p. 4
• The parameter update takes a step opposite the loss gradient, with the learning rate setting the step size.The update is applied after gradients are obtained for a free parameter.Source: Training · p. 4

3.1 Datasets

• JIGSAWS provides 39 suturing trials with ten gesture labels, while the MISTIC-SL experiments use 39 trials labeled with four maneuvers.Both datasets use da Vinci kinematics; MISTIC-SL trials include suture throws and surgeon’s knots.Source: §3.1 · pp. 4–5

3.2 Experimental Setup

• Evaluation averages leave-one-user-out runs: eight for JIGSAWS and eleven for MISTIC-SL; performance is measured by frame accuracy and normalized segment-level edit distance.Accuracy does not account for temporal consistency, while edit distance measures operations needed to transform predicted segments into ground-truth segments.Source: §3.2 · p. 5

3.3 Hyperparameter Selection and Training

• A batch size of five sequences was used because larger batches lacked stochasticity and converged to bad local optima.Training ran for approximately 80 epochs, with the learning rate halved every five epochs after the first 40.Source: §3.3 · p. 5
• Hyperparameters were selected on four MISTIC-SL validation users and shared across datasets; one 1024-unit hidden layer with dropout gave the lowest edit distance and high accuracy.JIGSAWS’s fixed test setup precluded using it for hyperparameter selection without training on test users.Source: §3.3 · pp. 5–6

3.4 Results

• Table 1 compares forward and bidirectional LSTMs with four prior methods for gesture recognition and maneuver recognition.The comparisons include MsM-CRF, SDSDL, SC-CRF, and LC-SC-CRF.Source: §3.4 · p. 6, Table 1 · p. 6
• Bidirectional LSTM differences from LC-SC-CRF in accuracy and edit distance were significant on MISTIC-SL but not JIGSAWS.The paired-sample permutation test used a p-value of 0.05.Source: §3.4 · p. 6
• Across both datasets, the figure shows highest-, median-, and lowest-accuracy trials, with ground truth above predictions.Predicted label sequences are smooth despite the assumption that labels are independent given the kinematics.Source: Figure 3 · p. 7, §3.4 · p. 7

4 Summary

• MISTIC-SL accuracy rose from 81.7% to 89.5%, while normalized edit distance fell from 29.7% to 19.5%.The reported improvement is for maneuver recognition.Source: §4 · p. 7
• A single model and hyperparameter set matched state-of-the-art JIGSAWS gesture recognition and advanced state-of-the-art MISTIC-SL maneuver recognition.The comparison covers both accuracy and edit distance.Source: §4 · p. 7
• The approach jointly segments and classifies activities from robot kinematics, predicting both low-level gestures and higher-level maneuvers with recurrent neural networks.It replaces HMM- or CRF-based mapping from inputs to labels.Source: §4 · p. 7