PaperlayerView exact revision on arXiv

Source-linked AI summary

Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets

Chinedu Innocent Nwoye, Cristians Gonzalez, Tong Yu, Pietro Mascagni, Didier Mutter, Jacques Marescaux, Nicolas Padoy

arXiv:2007.05405v1eess.IVcs.CVcs.LG

TL;DR

Fine-grained recognition of instrument–tissue interactions as triplets is challenging and had not previously been addressed directly from video. The paper introduces CholecT40 and Tripnet, which combines instrument-guided verb and target recognition with learned triplet associations. Tripnet outperforms the baselines in instrument–tissue interaction recognition.

  • Problem

    Fine-grained instrument–verb–target interactions are challenging to recognize directly from video, a task not previously addressed by the cited work.

    Source: §1 · p. 2

  • Method

    Tripnet uses multitask recognition with instrument activation maps to guide verb and target prediction, plus a trainable 3D interaction space to learn triplet relationships.

    Source: §1 · p. 3, §3 · p. 4, Proposed Model · p. 6

  • Results

    15.6% minimum improvement over all baselines in instrument-tissue interaction recognition; the authors also report gains across all metrics.

    Source: Quantitative Results · p. 8, §5 · p. 9

  • Takeaways & Limitations

    Including target anatomy adds substantial semantics to recognized action and instrument information.

    Source: §1 · p. 2

Abstract

from arXiv · show

Recognition of surgical activity is an essential component to develop context-aware decision support for the operating room. In this work, we tackle the recognition of fine-grained activities, modeled as action triplets <instrument, verb, target> representing the tool activity. To this end, we introduce a new laparoscopic dataset, CholecT40, consisting of 40 videos from the public dataset Cholec80 in which all frames have been annotated using 128 triplet classes. Furthermore, we present an approach to recognize these triplets directly from the video data. It relies on a module called Class Activation Guide (CAG), which uses the instrument activation maps to guide the verb and target recognition. To model the recognition of multiple triplets in the same frame, we also propose a trainable 3D Interaction Space, which captures the associations between the triplet components. Finally, we demonstrate the significance of these contributions via several ablation studies and comparisons to baselines on CholecT40.

1 Introduction

Surgical workflow recognition supports context-aware decision tools, and this paper addresses the more fine-grained task of recognizing instrument–tissue interactions as action triplets. It introduces CholecT40 and methods for recognizing triplets directly from video while learning their component relationships.

1 Introduction

  • Triplets capture instrument, action, and target anatomy, extending beyond earlier workflow-recognition tasks; the target adds substantial semantics, while fine-grained recognition is challenging.The EndoVis 2019 action-recognition task covered only four verbs.Source: §1 · p. 2
  • CholecT40 contains 135K annotated action triplets from 40 Cholec80 cholecystectomy videos, spanning 128 classes across 6 instruments, 8 verbs, and 19 targets.Examples include grasper–retract–gallbladder and scissor–cut–cystic duct.Source: §1 · p. 2
  • The Class Activation Guide uses weak localization from instrument activation maps to guide verb and target recognition without spatial annotations.Triplets are instrument-centric: an action is performed only when an instrument is present.Source: §1 · pp. 2–3
  • A trainable 3D interaction space learns triplet-component relationships, addressing the greater data-association complexity of triplets compared with bipartite interactions.The paper frames triplet association as tripartite graph matching, an NP-hard optimization problem.Source: §1 · p. 3

2 Cholecystectomy Action Triplet Dataset

CholecT40 is a surgeon-annotated dataset of action triplets from 40 Cholec80 cholecystectomy videos. The paper describes its annotation process, class composition, and train/validation/test split.

2 Cholecystectomy Action Triplet Dataset

  • The dataset has 135K action-triplet instances across 128 classes, comprising 6 instruments, 8 verbs, and 19 target classes.Videos were downsampled to 1 fps, yielding 83.2K annotated frames.Source: §2 · p. 3
  • A surgeon labeled action start and end frames and their instrument, verb, and target; a second clinician mediated annotations and grouped classes.For idle tools, both verb and target are set to null.Source: §2 · p. 3
  • The random split assigns 25 videos to training, 5 to validation, and 10 to testing, with 82.4K, 15.9K, and 37.1K triplets, respectively.The corresponding frame counts are 50.6K, 10.2K, and 22.5K.Source: §2 · p. 3
  • Table 1 reports occurrence frequencies for instruments, verbs, and targets; target IDs 0–18 map to the named target classes.Source: Table 1 · p. 4

3 Methodology

Tripnet combines multitask learning with an instrument-guided CAG unit and a trainable 3D interaction space to recognize action triplets. Instrument activation maps guide verb and target recognition, while the interaction space models component associations.

Multitask Learning

  • A shared ResNet-18 backbone feeds separate instrument, verb, and target branches; the instrument branch learns class activation maps for weak localization.The instrument branch uses global max pooling, while the verb and target branches each use two convolutional layers and a fully connected layer.Source: Multitask Learning · p. 4

Class Activation Guide

  • The CAG unit concatenates instrument class activation maps with verb and target features, conditioning their search on instrument appearance.The instrument CAM is weakly supervised and serves as a cue for likely action regions.Source: Class Activation Guide · pp. 4–5

3D Interaction Space

  • A trainable 3D outer-product projection associates instrument, verb, and target logits in an m × n × p grid.Learnable vectors project the three components, and each grid axis represents one triplet component.Source: 3D Interaction Space · p. 5
  • Triplet points with probabilities above a threshold are treated as valid, while points absent from the training-set triplets are masked during training and testing.Source: 3D Interaction Space · p. 6

Proposed Model

  • Tripnet integrates the CAG unit and 3D interaction space into a multitask model for action-triplet recognition.The model is trained end-to-end, with a warm-up period before instrument cues guide the verb-target subnet.Source: §3 · p. 4, Figure 2 · p. 5, Proposed Model · p. 6

4 Experiments

Experiments compare Tripnet with baselines on instrument detection and triplet recognition, then assess its CAG and trainable 3D interaction space through ablations and qualitative results. Tripnet outperforms the baselines on instrument-tissue interaction recognition, while the ablations show gains from combining its two novel components.

Experimental Setup

Quantitative Results

  • Tripnet exceeded the MTL baseline by 15.1% mean AP in instrument detection, while MTL and Tripnet had competing performance.The naive model performed poorly because it learned triplet IDs without modeling their components.Source: Quantitative Results · p. 7, Table 2 · p. 7
  • Tripnet outperformed all baselines in instrument-tissue interaction recognition by a minimum of 15.6%, with APIVT increasing by 12.5% on average.The MTL baseline performed only slightly above the naive model on triplet metrics, despite its high instrument detection performance.Source: Quantitative Results · pp. 7–8, Table 3 · p. 7
  • Instrument-verb recognition was easier than instrument-target recognition, likely because verbs associate more directly with instruments and targets are more numerous and harder for non-experts to distinguish.Source: Quantitative Results · p. 8
  • The authors place the action-recognition results in the context of other challenging datasets, citing 21.2% HICO-DET state-of-the-art performance and a 23.3% F1 score from a four-verb surgical challenge.Source: Quantitative Results · p. 8

Ablation Studies

  • Combining CAG with the trained 3D interaction space gave the best results across all four metrics.CAG improved APIV and APIT by approximately 2.0% and 1.0%; a trainable 3D projection improved APIVT by 6.0% over an untrained space or FC layer.Source: Ablation Studies · p. 8, Table 4 · p. 8

Qualitative Results

  • Qualitative results show accurate localization of action regions; most incorrect predictions involve one incorrect triplet component, while instruments are usually predicted and localized correctly.Source: Qualitative results · pp. 8–9, Figure 3 · p. 9

5 Conclusion

The paper presents CholecT40 and a model for recognizing action triplets directly from surgical videos; experiments show the model outperforms baselines across all metrics.

5 Conclusion

  • The proposed model outperforms the baselines by a substantial margin across all metrics.Source: §5 · p. 9
  • The recognition model uses instrument class activation maps to learn verbs and targets, with a trainable 3D interaction space for triplet associations.Source: §5 · p. 9
  • CholecT40 contains 135K action triplets across 40 videos.Source: §5 · p. 9

Appendix I : Co-occurence Distribution of the Triplets

The appendix presents dataset statistics for instrument–verb and instrument–target co-occurrence frequencies.

Appendix I : Co-occurence Distribution of the Triplets

  • Table 5 presents dataset statistics for instrument–verb occurrence frequency.Source: Table 5 · p. 12
  • Table 6 presents dataset statistics for instrument–target occurrence frequency.Source: Table 6 · p. 12

Appendix II : Qualitative Results

The appendix provides qualitative results that include both successful and failed cases.

Appendix II : Qualitative Results

  • Figure 4 shows additional qualitative results, including success and failure cases.Source: Figure 4 · p. 13
Loading 2007.05405v1…