Source-linked AI summary
Multi-Task Recurrent Convolutional Network with Correlation Loss for Surgical Video Analysis
Yueming Jin, Huaxia Li, Qi Dou, Hao Chen, Jing Qin, Chi-Wing Fu, Pheng-Ann Heng
TL;DR
Tool presence detection and surgical phase recognition are correlated but challenging tasks that prior methods often handled independently. MTRCNet-CL jointly models them with shared early features, an LSTM phase branch, and correlation-loss regularization. On Cholec80, it consistently outperformed state-of-the-art methods on both tasks.
Problem
Prior methods commonly treated tool presence detection and phase recognition independently despite their clinically grounded correlation, leaving their relationship insufficiently exploited.
Method
MTRCNet-CL uses shared early convolutional features, task-specific higher layers, LSTM-based phase modeling, and correlation loss to align the two task predictions.
Results
MTRCNet-CL consistently outperformed state-of-the-art methods on tool presence detection and phase recognition in extensive Cholec80 experiments.
Takeaways & Limitations
The framework captures task relatedness through both shared representations and prediction-level constraints for surgical video analysis.
Takeaways & Limitations
Multi-task learning requires simultaneous annotations for both tasks on the same surgical-video dataset, restricting development to some extent.
Abstract
from arXiv · showhide
Surgical tool presence detection and surgical phase recognition are two fundamental yet challenging tasks in surgical video analysis and also very essential components in various applications in modern operating rooms. While these two analysis tasks are highly correlated in clinical practice as the surgical process is well-defined, most previous methods tackled them separately, without making full use of their relatedness. In this paper, we present a novel method by developing a multi-task recurrent convolutional network with correlation loss (MTRCNet-CL) to exploit their relatedness to simultaneously boost the performance of both tasks. Specifically, our proposed MTRCNet-CL model has an end-to-end architecture with two branches, which share earlier feature encoders to extract general visual features while holding respective higher layers targeting for specific tasks. Given that temporal information is crucial for phase recognition, long-short term memory (LSTM) is explored to model the sequential dependencies in the phase recognition branch. More importantly, a novel and effective correlation loss is designed to model the relatedness between tool presence and phase identification of each video frame, by minimizing the divergence of predictions from the two branches. Mutually leveraging both low-level feature sharing and high-level prediction correlating, our MTRCNet-CL method can encourage the interactions between the two tasks to a large extent, and hence can bring about benefits to each other. Extensive experiments on a large surgical video dataset (Cholec80) demonstrate outstanding performance of our proposed method, consistently exceeding the state-of-the-art methods by a large margin (e.g., 89.1% v.s. 81.0% for the mAP in tool presence detection and 87.4% v.s. 84.5% for F1 score in phase recognition). The code can be found on our project website.
1. Introduction
Surgical video analysis supports intra- and post-operative context-aware applications, but tool and phase recognition remain difficult and are often treated independently despite their correlation. MTRCNet-CL jointly models both tasks through shared features, temporal modeling, and correlation-based regularization, achieving strong results on Cholec80.
- Motivation: Tool and phase recognition can support real-time warnings, clinical decision-making, operating-room resource management, and surgery completion-time estimation.These applications span intra-operative assistance and post-operative workflow support.
- Challenges: Surgical videos challenge recognition systems through tool variety, occlusion, blur, camera artifacts, and small differences between phases.These conditions also create substantial variation within individual phases.
- Research Gap: Existing methods often address tool and phase recognition separately, leaving their intrinsic association and close yet subtle correlations insufficiently captured.The paper identifies this as an open problem for surgical video analysis.
- Task Relatedness: Surgical phases and tool usage are highly correlated because specified operations use corresponding instrument sets.For example, hooks are associated with dissection, while clippers and scissors are used during clipping and cutting.
- Proposed Method: MTRCNet-CL jointly tackles tool presence detection and phase recognition with an end-to-end multi-task architecture.The framework shares early convolutional layers while retaining task-specific higher layers and uses LSTM for phase sequential dependencies.
- Results: 89.1% v.s. 81.0% mAP for tool presence detection and 87.4% v.s. 84.5% F1 score for phase recognition demonstrate consistent gains over existing state-of-the-art approaches.The method was extensively validated on the Cholec80 surgical video dataset.
2. Related Work
Prior surgical video methods commonly separated tool presence detection and phase recognition, using handcrafted features, statistical models, or deep networks based on different input signals. Related multi-task approaches began sharing features or modeling label relationships, motivating more effective exploitation of complementary task information.
- Tool Presence Detection: Tool presence detection methods evolved from handcrafted shape, color, and texture features to CNN-based and multi-label approaches.Some later methods modeled tool co-occurrence or combined deep CNN architectures.
- Phase Recognition: Surgical phase recognition methods used manually annotated signals, sensor signals, or combinations of both.Examples include binary instrument usage, anatomical structure, surgical motion, robot kinematics, and video-based features.
- Deep Learning: Deep CNN and RNN methods extracted discriminative features from online sensor signals while reducing reliance on labor-intensive manual annotations.These approaches improved phase-recognition performance using learned representations and temporal modeling.
- Motivation for Joint Learning: Separate treatment of tool and phase tasks misses complementary information, although tool usage has been shown useful as an input for phase recognition.This motivates joint learning of the two tasks.
- Multi-Task Learning: Multi-task learning in natural and medical image analysis has simultaneously addressed related prediction problems across multiple streams or shared representations.Reported applications include detection, classification, localization, segmentation, and quantitative assessment.
- Surgical Multi-Task Models: EndoNet jointly performed tool detection and phase recognition using shared early layers, with a hierarchical HMM enforcing temporal constraints afterward.The paper positions its approach within this emerging line of surgical multi-task research.
- Relationship Modeling: Other work learned relationships through matrix spaces, label embeddings, auxiliary-task predictions, or additional regularization.These strategies provide precedents for modeling dependencies between disparate labels or tasks.
3. Methods
MTRCNet-CL jointly recognizes surgical tools and phases using shared visual features, task-specific branches, LSTM-based temporal modeling, and correlation-based prediction regularization.
- Network Architecture: MTRCNet-CL uses two task-specific branches that share early convolutional feature encoders while retaining higher layers for tool and phase recognition.The phase branch incorporates an LSTM to encode sequential dependencies, while the tool branch uses a fully connected CNN layer.
- Objective Functions: Tool presence detection is formulated as multi-label classification because multiple tool categories may appear in one frame.The tool branch uses multi-label logistic loss, with binary labels indicating whether each tool is present.
- Objective Functions: Phase recognition is trained as multi-class classification with softmax cross-entropy while the LSTM updates hidden states from current and previous frames.This recurrent module jointly learns video sequential dynamics and visual representations.
- Correlation Loss: The correlation loss minimizes bidirectional KL divergence between tool predictions and tool probabilities inferred from phase-branch features.This regularization encourages the phase branch to encode tool information and the tool branch to incorporate phase representations.
- Overall Loss and Training: The overall joint objective combines tool detection loss, phase recognition loss, correlation loss, and weight decay, balanced by hyperparameters λ1, λ2, and λ3.Shared low-level features and high-level correlation constraints are optimized end-to-end using stochastic gradient descent.
- Overall Loss and Training: The framework is trained with a three-step strategy that initializes the large shared backbone using an ImageNet-pretrained model.Branch-specific weights are randomly initialized before subsequent training steps.
4. Experiments
Experiments on Cholec80 evaluate MTRCNet-CL through ablations, component analyses, training-strategy comparisons, and comparisons with existing methods. Results show benefits from longer temporal inputs, multi-task learning, correlation loss, and the proposed training design for both recognition tasks.
- Dataset and metrics: Cholec80 contains 80 cholecystectomy videos with seven surgeon-annotated phases and seven tool categories annotated after 1 fps resampling.Videos were recorded at 25 fps with resolutions of 1920 × 1080 or 854 × 480.
- Dataset and metrics: Phase recognition uses phase-wise precision, recall, and video-level accuracy, while tool recognition uses mean average precision.Reported results include averages and standard deviations across test videos.
- Effectiveness of key components: 85.3% to 86.4%: SingleNet phase accuracy improves as input length increases from 4 to 10 seconds; MTRCNet accuracy rises from 85.9% to 87.3%.These results support learning longer temporal dependencies for phase recognition.
- Effectiveness of key components: MTRCNet improves phase metrics over SingleNet across input lengths, while shared spatial-temporal features also improve tool detection; correlation loss further raises both tasks, reaching 89.2% phase accuracy.The reported 10-second MTRCNet phase metrics are 85.0% precision, 85.1% recall, and 87.3% accuracy.
- Effectiveness of key components: MTRCNet-CL improves phase precision in most phases, especially P3, phase recall particularly in P1 and P5, and average precision across all seven tools.Figure 4 shows increasing diagonal recall probabilities from SingleNet to MTRCNet-CL.
- Detailed ablation study: Separate mapping-matrix pretraining improves phase accuracy and tool mAP over joint training from the beginning, while MTRCNet-CL marginally exceeds TS2.The results are attributed to better initialization and balancing the learning difficulties of network components.
- Comparison with state-of-the-art methods: 89.1% versus 81.0% mAP and 87.4% versus 84.5% phase F1: MTRCNet-CL exceeds state-of-the-art methods on tool presence detection and phase recognition.The method also achieves superior performance in most tool categories, with especially large improvement for T4 Scissors.
5. Discussion
MTRCNet-CL jointly optimizes tool presence and phase recognition through shared features, temporal modeling, and correlation-based supervision. The discussion emphasizes real-time and postoperative utility while noting annotation scarcity as a central constraint.
- Discussion: MTRCNet-CL jointly optimizes tool and phase recognition using shared early convolutional layers, task-specific higher layers, and an LSTM phase branch.The framework is trained end-to-end, introducing temporal information throughout training.
- Discussion: The learned mapping matrix and correlation loss reinforce interaction between the two tasks by providing additional supervision for prediction consistency.The authors report that this additional penalization improves correlation capture and task performance.
- Discussion: The tool branch omits LSTM because clinical annotations define tool presence from a single frame and require at least half of the tool tip to be visible.This task-specific design contrasts with temporal modeling in the phase branch.
- Discussion: Post-processing is not used for the reported MTRCNet-CL results, although an additional experiment examines how prior knowledge inference affects performance.The main results therefore come from the end-to-end network itself.
- Discussion: Limited surgical-video data and the need for simultaneous tool and phase annotations constrain multi-task learning development.The authors validate the method on Cholec80 and note potential extension toward semi-supervised learning with fewer annotations.
- Discussion: 0.3s per frame with one GPU supports potential real-time context-aware applications including warning generation, process monitoring, and staff scheduling.The paper also identifies postoperative uses such as report writing, video indexing, skill assessment, and review.
6. Conclusion
MTRCNet-CL is an end-to-end multi-task architecture that jointly detects surgical tools and recognizes phases using shared features, temporal modeling, and correlation loss. Experiments on a large surgical-video dataset validate its effectiveness and show performance exceeding state-of-the-art methods.
- Conclusion: MTRCNet-CL jointly detects surgical tools and recognizes phases with shared early features, task-specific higher layers, an LSTM phase branch, and correlation loss.The correlation loss uses a learned mapping matrix to enforce consistency between the two task predictions.
- Conclusion: The learned mapping matrix encourages interaction between the branches and helps the framework capture the close relatedness between tool presence and surgical phase.The architecture combines low-level feature sharing with high-level prediction correlation.
- Conclusion: MTRCNet-CL outperforms state-of-the-art methods on the Cholec80 surgical-video dataset.The conclusion reports extensive experimental validation of the method's effectiveness.