Source-linked AI summary
RSDNet: Learning to Predict Remaining Surgery Duration from Laparoscopic Videos Without Manual Annotations
Andru Putra Twinanda, Gaurav Yengera, Didier Mutter, Jacques Marescaux, Nicolas Padoy
TL;DR
Preoperative surgery-duration prediction is difficult when patient, surgeon, and intraoperative factors vary, and estimating remaining time from video is also challenging because different-looking frames may correspond to the same remaining duration. RSDNet uses laparoscopic video and automatically generated labels to estimate remaining surgery duration without manual training annotations. It achieves performance comparable to a method requiring expert phase labels on cholecystectomy and is tested on bypass surgeries as well.
Problem
Preoperative surgery-duration prediction is difficult because patient conditions, surgeon skills, and intraoperative circumstances vary, while visual frames with different appearances can share the same remaining-duration label.
Source: §I · p. 1Method
RSDNet uses a CNN for visual features and an LSTM for temporal prediction, jointly estimating progress and remaining duration from laparoscopic video with automatically generated labels.
Source: §I · p. 2, A. Deep Architecture · pp. 2–3Results
On 120 cholecystectomy videos, RSDNet performs comparably to an approach requiring expert phase labels; it is also tested on 170 bypass videos.
Source: §VII · p. 10Takeaways & Limitations
The reported bypass results support RSDNet’s potential use for remaining-duration estimation in other laparoscopic surgery types.
Source: §VII · p. 10
Abstract
from arXiv · showhide
Accurate surgery duration estimation is necessary for optimal OR planning, which plays an important role in patient comfort and safety as well as resource optimization. It is, however, challenging to preoperatively predict surgery duration since it varies significantly depending on the patient condition, surgeon skills, and intraoperative situation. In this paper, we propose a deep learning pipeline, referred to as RSDNet, which automatically estimates the remaining surgery duration (RSD) intraoperatively by using only visual information from laparoscopic videos. Previous state-of-the-art approaches for RSD prediction are dependent on manual annotation, whose generation requires expensive expert knowledge and is time-consuming, especially considering the numerous types of surgeries performed in a hospital and the large number of laparoscopic videos available. A crucial feature of RSDNet is that it does not depend on any manual annotation during training, making it easily scalable to many kinds of surgeries. The generalizability of our approach is demonstrated by testing the pipeline on two large datasets containing different types of surgeries: 120 cholecystectomy and 170 gastric bypass videos. The experimental results also show that the proposed network significantly outperforms a traditional method of estimating RSD without utilizing manual annotation. Further, this work provides a deeper insight into the deep learning network through visualization and interpretation of the features that are automatically learned.
I. INTRODUCTION
Preoperative duration estimates are difficult amid variable patient, surgeon, and intraoperative factors, while staff communication about remaining time can interrupt surgical workflow. RSDNet estimates remaining surgery duration from laparoscopic video without manual training annotations.
I. INTRODUCTION
- Accurate duration estimates support anesthesia planning and OR management, while overestimates underuse resources and underestimates cause overtime and longer patient waits.The paper links overestimation to underutilization of expensive resources and underestimation to overtime and high patient waiting time.Source: §I · p. 1
- Preoperative prediction is difficult because patient conditions, surgeon skills, and intraoperative circumstances vary and are difficult to incorporate into prediction models.The cited examples report average underestimates of 31 minutes by general surgeons and 35 minutes by anesthesiologists.Source: §I · p. 1
- RSDNet estimates remaining duration intraoperatively from endoscope video, avoiding verbal staff interruptions that may disrupt workflow and compromise OR safety.The method focuses on laparoscopic procedures and visual information from the endoscope.Source: §I · p. 1
- RSDNet trains a CNN for progress estimation and an LSTM jointly for progress and remaining-duration regression, using automatically generated labels instead of manual annotations.Both progress and RSD labels can be generated from the video sequences, removing the need for manual annotation during training.Source: §I · p. 2
- The paper reports testing RSDNet on cholecystectomy and bypass, comparing it with other approaches, and interpreting learned features through LSTM-cell visualization.Its stated contributions include generalization to bypass and extensive comparisons with other approaches.Source: §I · p. 2
II. RELATED WORK
Prior intraoperative duration methods may require staff input or manually annotated signals. RSDNet instead uses laparoscopic video and a CNN–LSTM pipeline trained with automatically generated progress and duration labels.
II. RELATED WORK
- Prior intraoperative approaches use staff input or signals such as tool usage and low-level task representations, often relying on manual annotation; RSDNet instead predicts continuously from laparoscopic video without human intervention.One cited electrosurgical-device approach starts detection after 15 minutes and assumes patient preparation 25 minutes before surgery ends.Source: §II · p. 2
- RSDNet directly regresses remaining duration rather than deriving it from predicted progress, and the paper reports this direct-estimation approach performs better experimentally.The authors also distinguish their evaluation on long sequences with high duration variance, including cholecystectomy videos whose longest video is eight times the shortest.Source: §II · p. 2
III. METHODOLOGY
- RSDNet’s CNN extracts frame features, while its LSTM incorporates temporal information for prediction.The architecture is illustrated in Fig. 1.Source: §III · p. 2, Figure 1 · p. 3
A. Deep Architecture
- ResNet-152 is fine-tuned for progress estimation, a regression task estimating surgery progress as a percentage, by replacing its final layer with a single-node fully connected layer.The paper describes ResNet-152 as state-of-the-art in computer vision.Source: A. Deep Architecture · p. 2
- The LSTM jointly predicts progress and RSD; its output is concatenated with elapsed time before separate regression heads, and the two smooth L1 losses are summed equally.The paper states that elapsed time is essential because it relates to total duration and progress; both target labels are automatically generated, so manual annotation is unnecessary.Source: A. Deep Architecture · p. 3
B. Training Strategy
RSDNet uses a two-step training strategy to address memory constraints, then normalizes RSD targets to mitigate overfitting. Hyperparameter search determines the training setup, including dropout and dataset-specific normalization scales.
B. Training Strategy
- A 152-layer ImageNet-pretrained ResNet is finetuned with batch size 48, while CNN and LSTM training use stochastic gradient descent with momentum 0.9.CNN finetuning runs for 50k iterations; the LSTM runs for 30k iterations on Cholec120 and 50k on Bypass170.Source: B. Training Strategy · p. 3
- The CNN is finetuned first, then extracts visual features that are passed to an LSTM trained on complete sequences, avoiding end-to-end training under memory constraints.The pipeline is trained and tested at 1 fps.Source: B. Training Strategy · p. 3
- RSD targets are divided by a scalar during training and denormalized at testing to address overfitting associated with low regularization.The selected scales are snorm = 5 for Cholec120 and snorm = 10 for Bypass170, restricting targets to 0–20.Source: B. Training Strategy · p. 3
- Hyperparameter search on training and validation subsets selects dropout of 0.3 on both CNN input features and LSTM output features.The search covered normalization scale, regularization, weight decay, learning and dropout rates, and LSTM state count.Source: B. Training Strategy · p. 3
IV. EXPERIMENTAL SETUP
The Cholec120 evaluation uses 120 cholecystectomy recordings and a four-fold cross-validation setup. Videos are divided into training, validation, and evaluation subsets, with surgery-duration ranges marked by quartiles.
IV. EXPERIMENTAL SETUP
- Cholec120 contains 120 cholecystectomy recordings, evaluated with the same four-fold cross-validation setup used in prior work.The dataset combines Cholec80 with 40 additional videos.Source: A. Cholec120 Dataset · p. 3
- The short, medium, and long surgery boundaries are the first and third duration quartiles, Q1=27.4 mins and Q3=46.8 mins.The distribution is shown for Cholec120.Source: Figure 2 · p. 4
- Each split assigns T1 to CNN training, T1+T2 to LSTM training, V to validation, and E to evaluation.Four-fold evaluation uses all videos, with subsets selected to retain duration distributions similar to the full dataset.Source: A. Cholec120 Dataset · pp. 3–4
B. Bypass170 Dataset
The Bypass170 dataset tests RSD estimation on bypass procedures without manually labeling the dataset. The baselines range from historical-duration statistics to phase-based and learned approaches.
B. Bypass170 Dataset
- Bypass170 contains 170 bypass videos totaling over 327 hours, with an average duration of 115 (± 29) mins.The dataset was collected to test generalizability to another surgery type and was not manually labeled for RSD prediction.Source: B. Bypass170 Dataset · p. 4
- The Bypass170 evaluation fixes T1+T2, V, and E at 120, 10, and 40 videos, while varying CNN-training subset T1 from 40 to 80 videos.All T1+T2 videos train the LSTM.Source: B. Bypass170 Dataset · p. 4
C. Baselines
- The naïve baseline estimates remaining duration as max(0, referential duration minus elapsed time), using a dataset mean or median.It uses historical statistics without intraoperative information.Source: C. Baselines · p. 4
- The phase-inferred baseline estimates remaining duration from elapsed time in the current phase and reference durations for the remaining phases.Phase information may come from manual ground-truth labels or a CNN-LSTM phase recognizer; the latter still requires manual training annotations.Source: C. Baselines · p. 4
- TimeLSTM uses a ResNet trained for phase recognition before LSTM RSD prediction, so its CNN training requires manual labels.The single-task LSTM removes the progress-estimation task from RSDNet’s LSTM training.Source: C. Baselines · pp. 4–5
- The progress-derived RSD baseline computes remaining duration from progress estimates, enabling comparison with RSDNet’s direct RSD prediction.This approach has also been used to estimate remaining activity duration.Source: C. Baselines · p. 5
D. Evaluation Metrics
RSD estimation is evaluated with mean absolute error and comparisons across surgery-duration categories. The paper also discusses model behavior beyond quantitative performance.
D. Evaluation Metrics
- RSD performance is measured using mean absolute error in minutes, chosen for interpretability to clinicians and standard comparison.The authors describe MAE as a natural metric for this regression task.Source: D. Evaluation Metrics · p. 5
- Methods are also compared across short, medium, and long surgeries, comprising 25%, 50%, and 25% of the dataset, respectively.Short cases fall below Q1, long cases above Q3, and the rest are medium; this tests performance across workflow variation.Source: D. Evaluation Metrics · p. 5
- The paper’s discussion also examines CNN finetuning, approach reliability, and deeper analysis of the LSTM.These discussions extend beyond quantitative analysis.Source: D. Evaluation Metrics · p. 5
V. EXPERIMENTAL RESULTS
RSDNet performs similarly to phase-informed approaches and TimeLSTM while avoiding manual training annotations. It outperforms the naïve baseline, and its category-wise results remain comparable to the best method for each duration range.
V. EXPERIMENTAL RESULTS
- RSDNet significantly outperforms the naïve approach on the complete Cholec120 dataset (p < 0.005).The naïve approach does not use intraoperative information.Source: A. Cholec120 Dataset · p. 5
- Automatic phase recognition and manually annotated phase labels yield similar results, with no performance sacrifice from removing the expert observer.The phase-recognition pipeline achieves 87% accuracy.Source: A. Cholec120 Dataset · p. 5
- The direct RSDNet estimate significantly outperforms deriving RSD from progress estimates (p < 0.05), and progress training slightly improves over the single-task network.The progress-derived approach follows Eq. 1.Source: A. Cholec120 Dataset · p. 6
- RSDNet yields performance similar to TimeLSTM and phase-inferred approaches, while its progress labels are automatically generated and reduce annotation effort.The authors note that this makes the architecture easier to adapt to other surgery types.Source: A. Cholec120 Dataset · p. 6
- Across short, medium, and long Cholec120 surgeries, RSDNet’s results are comparable to the best-performing method in each category.Which method performs best varies by category.Source: A. Cholec120 Dataset · p. 6
B. Bypass170 Dataset
On Bypass170, RSDNet generalizes beyond cholecystectomy and significantly outperforms the annotation-free Naïve approach overall, especially for short and long surgeries. Training the CNN with 40 videos produced the best complete-dataset result.
B. Bypass170 Dataset
- RSDNet significantly outperforms the Naïve approach on Bypass170 overall, with MAE advantages greater than 5 minutes for short surgeries and 15 minutes for long surgeries.For medium surgeries, the Naïve approach is about 2 minutes better, consistent with its referential-duration design.Source: B. Bypass170 Dataset · p. 6
- The Bypass170 results show generalization to bypass surgery and potential for estimating RSD on other laparoscopic surgeries.The comparison is limited to the Naïve approach because other methods require phase labels unavailable in this dataset.Source: B. Bypass170 Dataset · p. 6
- Using 40 videos to train the CNN yielded the best complete-dataset result; using 80 degraded performance, while 40 and 60 showed no significant difference.Using half of the total training videos for CNN training also seemed effective on Cholec120.Source: B. Bypass170 Dataset · p. 6
A. RSD Prediction Analysis
RSDNet is compared with the Naïve approach using MAE across predicted and ground-truth remaining-duration values. It performs better on short and long surgeries, while medium-surgery comparisons and some late-stage bypass errors require qualification.
A. RSD Prediction Analysis
- Figure 4 evaluates reliability by plotting MAE against predicted RSD values from 5 to 30 minutes for both methods and datasets.For short Bypass170 surgeries, the Naïve approach makes no prediction at 15 minutes or lower; the corresponding MAE is undefined and omitted.Source: Figure 4 · p. 6, A. RSD Prediction Analysis · pp. 6–7
- RSDNet outperforms the Naïve approach on short and long surgeries but performs similarly on medium surgeries across the prediction analysis.The short- and long-surgery improvement is more pronounced on Bypass170, which has greater variation in surgery duration.Source: A. RSD Prediction Analysis · pp. 6–7, Figure 5 · p. 7
- Figure 5 evaluates accuracy by plotting MAE against ground-truth RSD values from 5 to 30 minutes; the 30-minute value is undefined for short Cholec120 surgeries.All short Cholec120 surgeries last less than 30 minutes.Source: Figure 5 · p. 7, A. RSD Prediction Analysis · p. 7
- For long Bypass170 surgeries at 25 minutes remaining, the Naïve approach predicts completion, making its low late-surgery MAE values uninformative.At a 25-minute ground-truth RSD, its MAE is 25 (±0) minutes.Source: A. RSD Prediction Analysis · p. 7
- For medium Bypass170 surgeries, RSDNet has greater MAE standard deviations; the Naïve approach’s slight advantage comes with considerably poorer short- and long-surgery performance.The Naïve method is effectively handcrafted for medium-duration surgeries, while RSDNet models progression across durations.Source: A. RSD Prediction Analysis · p. 7
B. Performance Analysis
The Bypass170 analysis describes how RSDNet’s over- and under-estimation varies across surgery duration and time within surgery. The CNN task-formulation experiments identify progress regression as the most effective formulation among those tested.
B. Performance Analysis
- Short surgeries are usually over-estimated, long surgeries under-estimated, and RSDNet tends to over-estimate RSD in the final quarter.The authors conclude that small RSD predictions indicate the surgery is likely to end within the predicted time.Source: B. Performance Analysis · p. 7, Table III · p. 8
C. CNN Task Formulation
- RSD classification achieved around 15% validation accuracy, with bin imbalance proposed as a possible explanation.The formulation uses three-minute bins and caps the number of bins at 20.Source: C. CNN Task Formulation · p. 8, Figure 6 · p. 8
- Progress classification reached 30% validation accuracy despite balanced classes, with equal penalties for near and distant bin errors identified as a likely issue.The authors note that predicting bin-10 instead of bin-1 should incur a greater penalty than predicting bin-2.Source: C. CNN Task Formulation · p. 8
- Direct RSD regression did not converge, which the authors attribute in part to the CNN ignoring temporal information and dissimilar visuals across frames with similar targets.Frames with 30 minutes remaining can show substantially different scenes across videos.Source: C. CNN Task Formulation · p. 8
- Progress regression converged with approximately 15% validation MAE, but indirectly deriving RSD from progress yielded approximately 11 minutes MAE.The authors believe progress regression works better because it avoids classification’s improper penalties and encourages similar-target frames to resemble one another.Source: C. CNN Task Formulation · pp. 8–9
D. Importance of CNN Finetuning
Removing CNN finetuning substantially worsens performance on Cholec120. LSTM cell activations exhibit patterns associated with progress, halfway points, and specimen-bag appearances.
D. Importance of CNN Finetuning
- The non-finetuned ImageNet-pretrained ResNet pipeline yielded 12.3±6.4 minutes MAE on Cholec120, significantly worse than RSDNet.It also performed generally worse than the Naïve approach; the authors suggest sub-optimal visual features may make LSTM training difficult to converge.Source: D. Importance of CNN Finetuning · p. 9
E. LSTM Cell Visualization
- LSTM cell-410 activation increased and cell-427 activation decreased over surgery, patterns that the authors say mimic progress and remaining duration.They consider these cells intuitively important for accurate RSD prediction.Source: E. LSTM Cell Visualization · p. 9
- Cell-450 activation splits videos around the predicted 50% progress point, although the four videos reach that point in different phases and with different tools.The authors report no specific visual cue explaining the predicted halfway point.Source: E. LSTM Cell Visualization · p. 9, Figure 8 · p. 9
- Cell-376 responds strongly to specimen-bag presence, which the authors say signals that surgery is nearing its end and lowers the RSD estimate.The pattern is shown for two cholecystectomy videos.Source: E. LSTM Cell Visualization · p. 9
VII. CONCLUSIONS
RSDNet estimates remaining surgery duration from laparoscopic video without manual training labels, and the authors report results on cholecystectomy and bypass datasets. They identify two-step training and reliance on visual information alone as limitations.
VII. CONCLUSIONS
- RSDNet uses automatically generated labels rather than manual annotations and performs comparably to a phase-label-dependent approach on 120 cholecystectomy videos.The authors also report generalization to bypass surgery on a dataset of 170 videos.Source: §VII · pp. 9–10
- The authors propose that RSDNet could support intraoperative management of remaining surgery durations, with potential benefits for OR throughput, patient comfort, and safety.They connect those patient benefits to reduced waiting time and duration of anesthesia.Source: §VII · p. 10
- Two-step training prevents LSTM loss from backpropagating to the CNN, leaving its visual features without temporal context; the pipeline also uses visual information alone.The authors suggest incorporating preoperative patient-condition and surgeon-skill information as additional features.Source: §VII · p. 10