PaperlayerView exact revision on arXiv

Source-linked AI summary

EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos

Andru P. Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel de Mathelin, Nicolas Padoy

arXiv:1602.03012v2cs.CV

TL;DR

Surgical phase recognition needs useful visual features without relying on handcrafted representations or manually collected tool signals. The paper introduces EndoNet, a CNN that learns visual features from cholecystectomy videos while jointly detecting tool presence and surgical phase. Extensive experiments report state-of-the-art performance for both tasks and better visual features than single-task architectures and ground-truth binary tool signals for phase recognition.

  • Problem

    Existing phase-recognition methods often use handcrafted visual features or tool signals obtained through manual annotation or additional equipment.

  • Method

    EndoNet is a CNN that learns visual features from cholecystectomy videos and jointly performs tool presence detection and phase recognition.

  • Results

    EndoNet yields state-of-the-art results for phase recognition and tool presence detection, with phase-recognition features outperforming prior features, single-task architectures, and ground-truth binary tool signals.

  • Takeaways & Limitations

    The results suggest that laparoscopic images contain recognition-relevant characteristics beyond simple tool presence information and that EndoNet retrieves them.

  • Takeaways & Limitations

    Cholecystectomy videos came from one hospital, limiting institutional variability and potentially increasing over-fitting and reducing generalizability.

Abstract

from arXiv · show

Surgical workflow recognition has numerous potential medical applications, such as the automatic indexing of surgical video databases and the optimization of real-time operating room scheduling, among others. As a result, phase recognition has been studied in the context of several kinds of surgeries, such as cataract, neurological, and laparoscopic surgeries. In the literature, two types of features are typically used to perform this task: visual features and tool usage signals. However, the visual features used are mostly handcrafted. Furthermore, the tool usage signals are usually collected via a manual annotation process or by using additional equipment. In this paper, we propose a novel method for phase recognition that uses a convolutional neural network (CNN) to automatically learn features from cholecystectomy videos and that relies uniquely on visual information. In previous studies, it has been shown that the tool signals can provide valuable information in performing the phase recognition task. Thus, we present a novel CNN architecture, called EndoNet, that is designed to carry out the phase recognition and tool presence detection tasks in a multi-task manner. To the best of our knowledge, this is the first work proposing to use a CNN for multiple recognition tasks on laparoscopic videos. Extensive experimental comparisons to other methods show that EndoNet yields state-of-the-art results for both tasks.

I. INTRODUCTION

Surgical workflow recognition can support operating-room and video-database applications, but existing approaches rely on handcrafted visual features or tool information that is difficult to obtain automatically. EndoNet addresses these limitations by learning visual features with a CNN while jointly performing phase recognition and tool presence detection.

  • Motivation: Workflow recognition could support context-aware monitoring, operating-room scheduling, automated video indexing, and alerts for probable upcoming complications.These applications depend on segmenting surgical workflows and analyzing video content.
  • Limitations of existing features: Tool usage signals and surgical triplets are often manually annotated, making them virtually impossible to obtain at test time.Reliable automatic acquisition of this information remains an open question.
  • Limitations of existing features: Handcrafted visual features can lose significant image characteristics because they are empirically designed to capture selected information.Examples include pixel values, intensity gradients, spatio-temporal features, and color, texture, and shape combinations.
  • Proposed approach: EndoNet learns inherent visual features from cholecystectomy videos with a CNN and jointly performs phase recognition and tool presence detection.The approach relies on visual information that is typically readily available in the operating room.
  • Evaluation: 80 cholecystectomy videos were used for validation, with additional experiments on seven EndoVis videos to assess feature generalizability.The authors also report that multi-task training produces a better network than single-task training.
  • Evaluation: The paper reports state-of-the-art results for phase recognition and tool presence detection using solely visual features.It also presents CNN-based visual feature extraction and multi-task recognition for laparoscopic videos as contributions.

II. RELATED WORK

Prior work on surgical workflow recognition uses tool signals, surgical triplets, and visual features, while tool presence detection often targets related tasks or requires additional equipment. Existing visual-feature approaches include handcrafted methods and constrained automatic feature learning, motivating a visual approach tailored to laparoscopic procedures.

  • Tool presence detection: Automatic tool-presence literature remains limited, with prior approaches often addressing tool detection, pose estimation, or tracking instead.Many methods are tested only on short sequences, unlike the complete procedures considered here.
  • Tool presence detection: RFID-tagged tools can support detection and tracking but are complex to integrate into the operating room, encouraging investigation of video-based cues.A cited laparoscopic-image method automatically recognizes tool types but consists of many steps.
  • Feature types: Surgical workflow studies have explored tool usage signals, surgical action triplets, and visual features across cataract, neurological, and laparoscopic surgeries.The paper focuses its literature discussion on methods using visual features because its method relies solely on them.
  • Phase recognition: An HMM-based phase-recognition method combined tool signals with two visual cues but required tool signals that are not always immediately available in the operating room.The visual cues indicated camera location and whether clips were visible.
  • Phase recognition: A cataract-surgery framework achieved 91% accuracy with combined visual information and HMM, but cataract procedures differ substantially from cholecystectomy surgeries.Cholecystectomy videos are generally longer and present additional visual challenges.
  • Phase recognition: Genetic-programming feature learning for cholecystectomy was limited by predefined operators and sometimes failed to outperform handcrafted features.The cited approach therefore did not consistently improve recognition over handcrafted alternatives.

C. Convolutional Neural Networks

The proposed pipeline fine-tunes EndoNet and then uses its outputs for tool presence detection and phase recognition. EndoNet extends AlexNet with task-specific branches and is designed to learn more discriminative phase features through multi-task training and automatically generated tool signals.

  • Pipeline: The complete pipeline first fine-tunes EndoNet, then uses the trained network for tool presence detection and phase recognition.For phase recognition, extracted visual features are passed to an SVM and Hierarchical HMM for final phase estimation.
  • Design assumptions: EndoNet assumes that multi-task fine-tuning can learn more discriminative phase-recognition features than optimizing for phase recognition alone.The architecture is explicitly designed to perform phase recognition and tool presence detection together.
  • Design assumptions: EndoNet also assumes that automatically generated tool signals can improve phase recognition features.This assumption builds on prior use of tool signals for phase recognition.
  • EndoNet architecture: EndoNet extends AlexNet with a tool-presence branch, a concatenated feature layer, and a phase-classification branch.The tool branch has seven outputs, and the phase branch has seven outputs corresponding to dataset-defined tools and phases.

B. Fine-Tuning

EndoNet is fine-tuned with separate losses for tool presence detection and phase recognition, then combines these objectives into a weighted total loss. The architecture uses task-specific outputs for seven tools and seven phases.

  • Task losses: Tool presence detection is formulated as seven binary classification tasks trained with cross-entropy loss.Each task corresponds to one tool, and the network output represents that tool’s presence confidence.
  • Task losses: Phase recognition is formulated as a multi-class classification task using the softmax multinomial logistic loss.The phase output represents confidence over seven phases.
  • Joint optimization: The final objective sums the tool and phase losses as L = a·LT + b·LP.The weighting coefficients were set to a = b = 1 because preliminary experiments found no improvement when varying them.
  • Joint optimization: Setting either loss weight to zero is equivalent to optimizing a CNN for only phase recognition or only tool presence detection.This provides the single-task counterparts used to contrast with the multi-task design.

C. SVM and Hierarchical HMM

The phase-recognition pipeline converts EndoNet features into phase confidence values and applies a two-level HHMM to enforce surgical-workflow temporal structure. The datasets differ in phase definitions and annotation scope, constraining cross-dataset interpretation.

  • SVM and HHMM: EndoNet’s fc8 output is used as an image feature to compute seven-dimensional phase confidence values with a one-vs-all multiclass SVM.The confidence vector is obtained without temporal information.
  • SVM and HHMM: The HHMM models inter-phase dependencies at the top level and intra-phase dependencies at the bottom level, using SVM confidence values as observations.Offline recognition uses the Viterbi algorithm to select the most likely state path.
  • SVM and HHMM: EndoNet’s fc phase output can directly provide confidence values, but the SVM step is retained for fair comparison with other feature representations.Preliminary experiments found only a slight performance difference between fc phase and SVM-derived confidence values before and after HHMM processing.
  • Datasets: Cholec80 contains 80 cholecystectomy videos labeled with phase and tool-presence annotations, while its evaluation subset is separate from the fine-tuning subset.The dataset is split into two equal 40-video subsets; the fine-tuning subset contains approximately 86K annotated images.
  • Datasets: EndoVis contains seven cholecystectomy videos with phase definitions that differ from Cholec80, including a preparation phase that incorporates calot triangle dissection.Some EndoVis phase labels may require external video information because the relevant phase is not always visible laparoscopically.

B. Fine-Tuning, SVM and HHMM Parameters

EndoNet is fine-tuned from an ImageNet-pretrained AlexNet, then evaluated with linear SVM and HHMM components configured for the surgical phases and feature types. Training uses differentiated learning rates and staged decay.

  • Fine-Tuning: EndoNet is trained by fine-tuning AlexNet, which was pretrained on ImageNet.
  • Fine-Tuning: The network is fine-tuned for 50K iterations with batches of 50 images, using learning rates of 10^-3 for existing layers and 10^-2 for randomly initialized task layers.All learning rates decrease by a factor of 10 every 20K iterations.
  • Computational Cost: Training takes roughly 11 hours per network, feature extraction takes approximately 0.2 second per image, and HHMM training takes approximately 15 seconds.SVM training ranges from 0.1 to 90 seconds depending on feature size.
  • SVM: Feature-based phase recognition uses a one-vs-all linear SVM, while handcrafted features additionally pass through a histogram intersection kernel.Preliminary nonlinear-kernel experiments did not improve results for the other features.
  • HHMM: The HHMM uses seven top-level states, data-driven bottom-level states, and Gaussian mixtures to model SVM outputs.Five Gaussians are used per feature, except for the binary tool signal, which uses one Gaussian; covariances are diagonal.

C. Baselines

The experiments compare EndoNet against object-detection and single-task neural baselines, and compare phase-recognition inputs spanning manual tool signals, handcrafted features, pretrained features, and task-specific features.

  • Tool Presence Detection: Tool presence detection compares EndoNet’s fc tool output with Deformable Part Models and a single-task ToolNet.ToolNet isolates the tool-presence task to test the effect of multi-task fine-tuning.
  • Phase Recognition: Phase recognition uses four-fold cross-validation on Cholec80 and full cross-validation on EndoVis, averaging results over five experimental runs.Multiple runs address variability from randomly initialized components in the recognition pipeline.
  • Visual and Tool Baselines: The phase-recognition baselines include manually annotated binary tool information and handcrafted SIFT, HOG, RGB, and HSV histogram features.The handcrafted features are also evaluated after CCA-based processing.
  • Neural Baselines: Additional baselines use ImageNet AlexNet fc7 features and features from single-task PhaseNet.These represent pretrained and phase-specific fine-tuned visual features, respectively.
  • Proposed Features: EndoNet features use fc8 outputs, with EndoNet-GTbin additionally concatenating EndoNet fc7 features with ground-truth binary tool information on Cholec80.This comparison tests whether automatically extracted tool information is sufficient for phase recognition.

D. Evaluation

Tool presence detection is evaluated with average precision, while phase recognition uses precision, recall, and accuracy. The study also examines phase-recognition performance before and after HHMM processing and in practical applications.

  • Metrics: Tool presence detection is measured with average precision, defined as the area under the precision-recall curve.
  • Metrics: Phase recognition is evaluated using precision, recall, and accuracy, with precision and recall averaged across phases.Accuracy is the percentage of correct detections over the complete surgery.
  • Baselines: Single-task CNN architectures provide comparison networks for tool presence detection and phase recognition, and they are trained using transfer learning.
  • Evaluation Design: The evaluation computes phase-recognition metrics both before and after applying the HHMM to measure the contribution of temporal modeling.

V. RESULTS

EndoNet performs strongly on tool presence detection and phase recognition using visual features, with multi-task learning improving phase-recognition features. Performance varies with phase-transition difficulty and the amount of fine-tuning data.

  • Tool Presence Detection: EndoNet gives the best tool presence detection results and achieves 81% mean AP across seven tools on 40 Cholec80 evaluation surgeries.Binary tool annotations suffice for training, without explicit segmentation or region-of-interest selection.
  • Phase Recognition: Multi-task EndoNet features further improve phase recognition beyond AlexNet, handcrafted features, binary tool annotations, and single-task PhaseNet features.EndoNet-GTbin improves accuracy by only approximately 0.1% over EndoNet, indicating that EndoNet’s visual features are sufficient without ground-truth tool information.
  • Phase Recognition: HHMM consistently improves phase-recognition results across all feature types, while offline recognition outperforms online recognition because it can access the complete video.The feature-performance trend remains consistent between offline and online modes.
  • Phase Recognition: Offline top-5 phase-recognition results exceed 98% accuracy, whereas top-5 online results remain above 92%.Offline bottom-5 errors are associated with jumps between P5 and P6, while online estimates show more frequent jumps because future data are unavailable.
  • Phase Recognition: Recognition performance decreases for P5 and P6, likely because their transitions are non-sequential and not always clearly defined.
  • Fine-Tuning Subset Size: Performance increases proportionally as the fine-tuning subset grows from 10 to 40 videos.

B. EndoVis Dataset

On EndoVis, EndoNet improves phase recognition despite being fine-tuned on Cholec80, supporting generalization across datasets. Its tool-confidence outputs also contribute discriminative information even when the tool sets differ.

  • EndoVis Results: EndoNet features significantly improve phase recognition on EndoVis, although gains are smaller than on Cholec80 because fine-tuning used Cholec80 videos.
  • EndoVis Results: EndoNet features produce the best phase-recognition results across all reported EndoVis cases, indicating generalizability to another dataset.
  • Feature Transfer: EndoVis uses tools different from Cholec80, so EndoNet’s seven tool-confidence outputs act as additional scalar features rather than direct tool matches.These values help construct more discriminative features.
  • Practical Applications: EndoNet is presented as applicable to practical computer-assisted-intervention tasks involving surgical video indexing and tool monitoring.

A. Automatic Surgical Video Database Indexing

EndoNet supports automatic surgical video indexing through accurate offline phase-boundary estimation. It also detects bipolar and clipper appearance blocks with useful temporal precision, though performance differs between tools.

  • Automatic Video Indexing: 89% of phase boundaries are detected within 30 seconds, while only 6% have errors exceeding 2 minutes.The authors regard the errors as acceptable because annotated boundaries are stricter than practical visual boundaries.
  • Automatic Video Indexing: EndoNet’s phase-boundary estimates require relatively few corrections, making surgical video indexing faster and easier.
  • Bipolar and Clipper Detection: Tool-block evaluation merges same-tool ground-truth blocks separated by gaps shorter than 15 seconds and counts a block as detected if any frame is recognized.Detection thresholds prioritize high precision while retaining sensitivity to tool blocks.
  • Bipolar and Clipper Detection: Over 90% of bipolar tool blocks are detected within 5 seconds, with a 3.8% false positive rate.The distinctive blue shaft is cited as supporting bipolar detection.
  • Bipolar and Clipper Detection: For clipper detection, 80% of blocks are detected within 5 seconds and 97% within 30 seconds.The clipper’s higher false-positive rate may relate to its limited annotations and brief appearance in surgeries.

VII. DISCUSSION AND CONCLUSIONS

EndoNet learns visual features directly from laparoscopic images while jointly performing tool presence detection and phase recognition, outperforming prior feature approaches. The study also identifies dataset and temporal-modeling limitations that constrain generalizability and phase estimation.

  • EndoNet jointly performs tool presence detection and phase recognition using visual features learned directly from raw laparoscopic images.The architecture is designed as a multi-task CNN rather than a single-task feature extractor.
  • EndoNet visual features outperform previously used features and ground-truth binary tool signals for phase recognition.The comparison suggests that laparoscopic images contain useful information beyond simple tool presence.
  • 80 videos from 13 surgeons formed the large cholecystectomy dataset used to train and evaluate EndoNet.The dataset was presented as a large-scale resource for these recognition tasks and included substantial variability in phase durations.
  • Cholec80 videos came from one hospital, so limited institutional variability may cause overfitting and reduce network generalizability.The authors recommend including videos from additional medical institutions to increase dataset variability.
  • EndoNet was demonstrated in two video-database-management applications, while live-surgery assistance and other laparoscopic tasks remain future directions.Suggested applications include completion-time estimation, surgical-video classification, and anatomy recognition.
  • The phase-recognition pipeline still relies on a separately trained HHMM to enforce temporal constraints.Consequently, EndoNet features lack temporal information and are not optimized jointly for the complete phase-recognition task.
Loading 1602.03012v2…