Source-linked AI summary
Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos
Chinedu Innocent Nwoye, Tong Yu, Cristians Gonzalez, Barbara Seeliger, Pietro Mascagni, Didier Mutter, Jacques Marescaux, Nicolas Padoy
TL;DR
Surgical action triplet recognition aims to identify instruments, verbs, targets, and their associations, but requires simultaneous multi-label recognition and instance association. The paper introduces RDV, combining CAGAM and MHMA, and contributes the CholecT50 dataset. RDV surpasses state-of-the-art methods on CholecT50, while target localization and multiple identical triplet instances remain limitations.
Problem
Action triplet recognition must simultaneously classify instruments, verbs, and targets and associate them correctly in surgical scenes.
Method
RDV combines instrument-conditioned CAGAM for verb and target detection with Transformer-inspired MHMA using self- and cross-attention for component association.
Results
RDV surpasses state-of-the-art methods for surgical action triplet recognition on CholecT50.
Takeaways & Limitations
CholecT50 and RDV provide a validated basis for fine-grained surgical triplet recognition and analysis of its clinical relevance.
Takeaways & Limitations
Target localization is not yet achieved, and the dataset cannot represent multiple instances of most identical triplets.
Abstract
from arXiv · showhide
Out of all existing frameworks for surgical workflow analysis in endoscopic videos, action triplet recognition stands out as the only one aiming to provide truly fine-grained and comprehensive information on surgical activities. This information, presented as <instrument, verb, target> combinations, is highly challenging to be accurately identified. Triplet components can be difficult to recognize individually; in this task, it requires not only performing recognition simultaneously for all three triplet components, but also correctly establishing the data association between them. To achieve this task, we introduce our new model, the Rendezvous (RDV), which recognizes triplets directly from surgical videos by leveraging attention at two different levels. We first introduce a new form of spatial attention to capture individual action triplet components in a scene; called Class Activation Guided Attention Mechanism (CAGAM). This technique focuses on the recognition of verbs and targets using activations resulting from instruments. To solve the association problem, our RDV model adds a new form of semantic attention inspired by Transformer networks; called Multi-Head of Mixed Attention (MHMA). This technique uses several cross and self attentions to effectively capture relationships between instruments, verbs, and targets. We also introduce CholecT50 - a dataset of 50 endoscopic videos in which every frame has been annotated with labels from 100 triplet classes. Our proposed RDV model significantly improves the triplet prediction mean AP by over 9% compared to the state-of-the-art methods on this dataset.
1. Introduction
Surgical action triplet recognition seeks fine-grained understanding of instruments, verbs, targets, and their relationships in laparoscopic scenes. This paper introduces CholecT50 and the attention-based RDV model to address component recognition and association.
- Motivation: Surgical workflow methods need fine-grained action descriptions because phase labels omit important instrument–verb–target activities.Examples include graspers holding landmarks, clippers applying clips, and scissors cutting the cystic duct.
- Challenge: Action triplet recognition simultaneously identifies instruments, verbs, targets, and the relationships connecting them.Triplets are instrument-centric: anatomy is a target only when acted upon, and verbs are defined by instrument actions.
- Dataset: CholecT50 contains 50 cholecystectomy videos annotated with 161K instances from 100 triplet classes.Examples include grasper–retract–gallbladder, clipper–clip–cystic-artery, and scissors–cut–cystic-duct.
- Method: RDV combines CAGAM for instrument-conditioned verb and target detection with MHMA for learning triplet-component interactions.MHMA mixes self-attention and cross-attention in a Transformer-inspired decoder.
- Evaluation: The paper validates Attention Tripnet and RDV through ablations and comparisons with baseline and state-of-the-art methods.Attention Tripnet is Tripnet upgraded with CAGAM alone.
2. Related Work
Prior surgical workflow research ranges from coarse phase recognition to fine-grained but incomplete action labels. This work positions triplet recognition and attention-based modeling as ways to represent and associate instruments, verbs, and targets more fully.
- Surgical Workflow Analysis: Earlier surgical workflow methods often describe scenes coarsely through surgical phases, while some fine-grained methods classify only verbs.Other datasets model verb–anatomy relations without identifying the instrument performing the action.
- Surgical Action Triplet Recognition: Surgical action triplets represent an action as the used instrument, its verb, and the anatomy acted upon.Tripnet was the first deep learning model designed to recognize these triplets directly from surgical videos.
- Related Interaction Models: Existing HOI research uses human–verb–object triplets and datasets with spatial annotations or binary presence labels.Surgical instruments must be categorized because they carry semantic information about surgical actions.
- Attention Mechanism: Many attention mechanisms focus networks on crucial context, but several HOI approaches rely on expensive spatial annotations.Weakly supervised saliency-based attention provides an alternative without additional supervision.
- Attention Mechanism: CAGAM improves instrument-conditioned verb and target detection by combining channel attention for verbs with position attention for targets.It redesigns saliency-guided attention around class activation maps.
- Attention Mechanism: RDV uses self- and cross-attention in multi-head layers to decode interactions between detected instruments and tissues.The design adapts Transformer-style attention to online surgical action triplet recognition.
3. CholecT50: Cholecystectomy Action Triplet Dataset
CholecT50 extends fine-grained surgical action-triplet research with 50 annotated cholecystectomy videos and 100 clinically selected triplet classes. Its labels encode instrument–verb–target interactions, supported by clinician-mediated class selection and dataset statistics.
- Dataset composition: CholecT50 contains 50 laparoscopic cholecystectomy videos, extending CholecT40 with 45 Cholec80 videos and 5 in-house videos.Two surgeons annotated action intervals with instrument, verb, and target labels.
- Class selection: Class construction combines semantic grouping, clinical relevance ratings, and occurrence counts to select clinically useful triplets.Clinical experts grouped semantically equivalent labels and ranked compositions using clinician ratings and frequency.
- Triplet labels: The dataset uses 100 triplet classes formed from 6 instruments, 10 verbs, and 15 target classes.Classes follow the ⟨instrument, verb, target⟩ format.
- Dataset statistics: Down-sampling to 1 fps yields 100.86K annotated frames and 161K triplet instances, with videos divided into training, validation, and test sets.The split preserves the same surgeon distribution across subsets.
4. Methodology
Rendezvous separates triplet-component detection from component association using two attention mechanisms. CAGAM conditions verb and target recognition on instrument cues, while MHMA models their semantic interactions.
- Task formulation: Action triplet recognition requires simultaneous multi-label detection of instruments, verbs, and targets plus association across multiple triplet instances.The task therefore combines component recognition with relationship modeling.
- Component detection: CAGAM uses tool type and location information to highlight discriminative features for verb and target detection, respectively.Attention Tripnet replaces Tripnet’s class activation guide with CAGAM.
- Interaction decoding: MHMA resolves triplet-component association using multiple heads of self- and cross-attention over instrument, verb, target, and triplet representations.Cross-attention operates on instrument, verb, and target heads, while self-attention operates on the triplet head.
- Rendezvous architecture: RDV combines a feature-extraction backbone, an encoder for component detection, a decoder for relationships, and a classifier.The encoder includes weakly supervised instrument localization, CAGAM, and a bottleneck collecting unfiltered low-level features.
- Component detection: CAGAM combines channel attention for verbs, which are mainly affected by instrument type, with position attention for targets, which depend on instrument position.The mechanism enhances verb and target features using instrument-derived affinity information.
5. Experiments
The experiments train and evaluate component and triplet predictions on standardized video splits using multi-task and association losses. Performance is assessed with component AP, interaction AP, and video-averaged mAP.
- Experimental setup: Videos are resized to 256 × 448, augmented with random scaling and brightness/contrast shifts, and split into 35 training, 5 validation, and 10 test videos.The models are evaluated using the specified dataset split.
- Training objectives: Weighted sigmoid cross-entropy losses train the instrument, verb, and target component tasks, with learnable uncertainty weighting balancing their contributions.Class-balancing weights are used for the component losses.
- Training objectives: The total objective adds a sigmoid cross-entropy association loss to the component losses and uses an 18-epoch warm-up for component learning.The regularization weight is λ = 1e−5.
- Evaluation metrics: Component average precision measures instrument, verb, and target detection, while triplet average precision measures instrument–verb, instrument–target, and complete triplet recognition.Component probabilities can be decomposed from triplet predictions by taking the maximum probability among triplets sharing a component class.
- Evaluation metrics: Mean AP is computed by averaging category-level AP scores across videos and serves as the main evaluation metric.The protocol computes video-specific AP before averaging across the dataset.
6. Results and Discussion
The experiments use ablation studies and comparisons with baseline and state-of-the-art methods to evaluate Attention Tripnet and RDV. The supplied passages identify the evaluation strategy but provide no numerical findings.
- Ablation studies: The study evaluates individual components of Attention Tripnet and RDV through ablation studies.These experiments examine the contribution of task-attention components.
- Comparative analysis: The study compares the proposed methods with baseline and state-of-the-art approaches to assess their performance.The supplied passage characterizes this as a comparative validation.
1 Dual = (channel + position) attentions
The study evaluates dual spatial attention and mixed-attention decoding for surgical triplet recognition, reporting improvements in component detection, association, and overall performance. The analyses also examine prediction quality, class distributions, significance, and qualitative localization.
- 1 Dual = (channel + position) attentions: Channel attention suits verb detection while position attention suits target detection; combining both yields the highest balanced improvement.+12.4% verbs and +12.0% targets are reported for the matched configuration.
- 1 Dual = (channel + position) attentions: The hybrid decoder combines self- and cross-attention, and RDV outperforms self-attention-only alternatives in learning triplet associations.The Multiple Self ablation uses self-attention across instrument, verb, target, and triplet contexts.
- 1 Dual = (channel + position) attentions: 9.9% mAPIVT improvement over SOTA is achieved by RDV, with association gains of +9.7% for instrument-verb and +10.5% for instrument-target.Attention Tripnet improves verb detection by 5.7%, target detection by 5.3%, and overall triplet recognition by 3.4%.
- 1 Dual = (channel + position) attentions: At top 20 predictions, the model reaches approximately 95% AP, while confidence decreases as more predictions are considered.The decrease suggests that many triplet classes are closely related.
- 1 Dual = (channel + position) attentions: RDV detects clinically relevant and rare instrument-action-target combinations, with every top-10 prediction recognized above 50% AP.The reported examples include clipper or scissors actions on the cystic duct or artery, which are relevant to safety monitoring.
- 1 Dual = (channel + position) attentions: Across all 100 classes, the proposed models maintain higher median, upper-quartile, and upper-whisker AP performance than baselines.The box plots show both central performance and the spread above the interquartile range.
7. Conclusion
The paper presents attention mechanisms for surgical action-triplet recognition and discusses their validation, applications, and remaining limitations. It identifies scalability, inference cost, target localization, and multi-instance triplet recognition as boundaries for future work.
- The proposed methods surpass the state of the art for surgical action-triplet recognition by combining instrument-guided attention with hybrid self- and cross-attention.CAGAM focuses on verbs and targets through instrument class activation maps, while the hybrid mechanism models component associations.
- Qualitative results suggest possible extensions to automated surgical report generation and spatial action segmentation.
- Qualitative results compare top-5 triplet predictions from the best baseline, SOTA, and proposed models.
- The CAGAM attention maps show how the module focuses on targets in CholecT50 test images.
- Scalability to unseen triplets and inference speed remain challenges, because deeper models improve performance but are computationally costly.The paper suggests zero-, one-, or few-shot learning for unseen triplets and lighter Rendezvous models to reduce costs.
- Target localization is not yet achieved with the same weakly supervised technique used for instruments, and binary presence annotations prevent counting multiple identical triplets.The latter limitation affects future triplet localization rather than recognition.
- The method and CholecT50 dataset are positioned as valuable for safety monitoring, skill evaluation, and objective reporting.Future work also considers temporal modeling because some verbs may be better recognized from tool-tissue interaction dynamics.
Appendix A. Additional Dataset Statistics
The appendix provides dataset statistics for instrument-verb and instrument-target co-occurrences in CholecT50.
- Dataset statistics summarize the distribution of instrument-verb co-occurrences.
- Dataset statistics summarize instrument-target co-occurrences and map target ids 0 through 14 to anatomical structures, fluid, specimen-bag, and null.
Appendix B. More Quantitative Results
The appendix reports per-class detection, association, and ablation results for instruments, verbs, targets, and attention-sequence modeling.
- The proposed models correctly detect all surgical-instrument categories at performance higher than 80.0%.
- The proposed models correctly predict dominant verbs including retract, dissect, coagulate, clip, and cut over 70.0% of the time.
- The target-detection breakdown maps target ids 1 through 14 to anatomical structures, fluid, specimen-bag, and null.
- The proposed models show stronger detection of common and clinically relevant instrument-verb combinations.
- The proposed models predict clipping and cutting of the cystic artery and cystic duct among their top detected instrument-target labels.
- The attention-sequence ablation evaluates class-wise mapping against the contemporally patch-based Vision Transformer sequence.
- The auxiliary-loss ablation indicates that jointly learning individual triplet components helps the network understand triplets.
- The reported metrics include APIV, APIT, and APIVT across model layer sizes.
Appendix C. More Qualitative Results
Appendix C adds further qualitative examples of top-5 triplet predictions from the best baseline, SOTA, and proposed models.
- Additional qualitative results compare top-5 triplet predictions across the best baseline, SOTA, and proposed models.
Appendix D. Supplementary Video
Supplementary material includes an online video demonstrating qualitative RDV results across triplet prediction, action-region localization, and attention maps.
- The supplementary video presents qualitative results for RDV triplet prediction.
- It also demonstrates action-region localization produced by the RDV model.
- Attention maps are included as an additional qualitative visualization.