PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimize Surgical Triplet Recognition: A Knowledge-Driven Mixture-of-Experts Solution

Yiyi Zhang, Yuchen Yuan, Ying Zheng, Jialun Pei, Jinpeng Li, Zheng Li, Pheng-Ann Heng

arXiv:2608.22972v1cs.CV

TL;DR

Surgical triplet recognition is challenged by conflicting component objectives, severe category imbalance, and limited domain-knowledge guidance. MoeCo combines a knowledge-driven mixture of experts with component-tailored adaptation and coordinated gradient learning; it achieves state-of-the-art performance on CholecT45 and CholecT50, with MoeCo-B reaching 41.7% average APIV T.

  • Problem

    Surgical triplet recognition must jointly identify instruments, verbs, targets, and their combinations despite component-level conflicts, severe imbalance, and limited domain-knowledge guidance.

  • Method

    MoeCo integrates MLLM-guided knowledge through activated experts, specializes component features across spatial-temporal regimes, and rebalances positive-negative gradients for rare categories.

  • Results

    41.7% average APIV T is achieved by MoeCo-B, outperforming the second-best method by 2.6% on the reported comparison.

  • Takeaways & Limitations

    MoeCo provides a knowledge-enriched co-optimization framework for handling hierarchical conflicts in surgical triplet recognition.

  • Takeaways & Limitations

    Limited public annotated data for diverse surgical scenarios remains a bottleneck, motivating future contextual understanding and zero-shot application work.

Abstract

from arXiv · show

Surgical action triplet recognition constitutes a critical task in context-aware robot-assisted surgery, facilitating automatic surgical action perception by identifying instrument, verb, target, and their association. However, existing works struggle to analyze such complex surgical scenes due to three main issues: (1) component-level optimization conflicts caused by entangled feature spaces, (2) category-level optimization conflicts arising from severe data imbalance, and (3) lack of domain knowledge guidance that limits model interpretability and robustness. To address these challenges, we propose a Mixture-of-Experts-guided Co-Optimization (\textit{MoeCo}) framework powered by knowledge-driven learning. Within the co-optimization pipeline, to first mitigate component-level conflicts, we introduce a component-tailored adapter that disentangles task-specific features across spatial-temporal regimes, facilitating effective component specialization. Next, we develop a coordinated gradient learning strategy to handle category-level conflicts, which adaptively rebalances positive-negative gradients to enhance the perception of rare categories. Notably, inspired by surgical domain expertise, we introduce a knowledge-driven mixture-of-experts mechanism that dynamically integrates multimodal large language model-guided knowledge via activated experts, thereby enriching the co-optimization pipeline with more expressive and robust representations. Extensive experiments on the public CholecT45 and CholecT50 datasets confirm the effectiveness of the proposed co-optimization pipeline and the superiority of dynamic priors integration via the knowledge-driven mixture-of-experts mechanism.

I. INTRODUCTION

Surgical triplet recognition must jointly identify instruments, verbs, targets, and their combinations, but entangled features, severe category imbalance, and missing domain priors create hierarchical optimization conflicts. MoeCo addresses these conflicts through specialized feature adaptation, coordinated gradients, and knowledge-driven experts, achieving state-of-the-art performance on CholecT45 and CholecT50.

  • Motivation: Surgical triplet recognition identifies instrument, verb, target, and their interactions, with any component error producing an incorrect overall outcome.The task supports fine-grained understanding of physician-patient interactions in surgical procedures.
  • Challenges: Shared instrument-verb associations pull examples together while distinct targets and triplets require separating them, creating component-level optimization conflicts.The examples <grasper, retract, gallbladder> and <grasper, retract, gut> illustrate this opposing objective.
  • Challenges: Severe triplet-category imbalance lets head-class gradients dominate optimization, leaving rare categories poorly optimized.This imbalance is tied to the uneven frequency of triplet categories in surgical procedures.
  • Challenges: Surgical structural-functional priors constrain valid instrument-verb-target combinations and can narrow reasoning despite noisy visual cues.Instrument design and intended use provide semantic constraints on plausible associations.
  • Proposed framework: MoeCo uses a component-tailored adapter for spatial-temporal feature specialization and coordinated gradient learning to address component- and category-level conflicts.The adapter supports task-specific prompt tuning, while coordinated gradients balance positive-negative optimization.
  • Proposed framework: A knowledge-driven mixture-of-experts mechanism dynamically integrates MLLM-mined knowledge, enriching representations, and experiments on CholecT45 and CholecT50 report state-of-the-art performance.The mechanism uses activated experts for dynamic knowledge integration.

II. RELATED WORK

Prior surgical triplet-recognition methods use multitask learning, weak instrument cues, and imbalance remedies, but often overlook fine-grained structure and gradient-level conflicts. MoeCo combines knowledge-driven experts, component-tailored representations, and coordinated gradient learning to model these issues jointly.

  • Surgical Triplet Recognition: Surgical triplet recognition is a complex multitask problem involving instrument, verb, target, and combination recognition under a long-tailed distribution.The long-tailed distribution makes balanced recognition difficult across triplet categories.
  • Surgical Triplet Recognition: Existing methods use weak instrument localization or activation cues to improve verb and target recognition, but often overlook fine-grained structural patterns within instruments.Examples include class activation-guided attention and MedSAM-based pseudo-labels.
  • Surgical Triplet Recognition: Re-sampling, re-weighting, teacher guidance, and memory banks address imbalance, yet prior approaches mainly emphasize instance-level rather than gradient-level rebalancing.The CholecT45 frequency gap ranges from over 40,000 samples for the most frequent category to 8 for the least frequent.
  • MoeCo: MoeCo comprises knowledge-driven MoE, a component-tailored adapter, and coordinated gradient learning for domain knowledge, task-specific representations, and optimization-conflict mitigation.The knowledge-driven MoE captures instrument-anchored dependencies through activated experts.
  • Knowledge-Driven Mixture-of-Experts: The framework converts surgical domain knowledge into phrase-based textual embeddings and integrates relevant semantics through a knowledge-driven mixture-of-experts mechanism.The approach uses visual-language-model generalizability to distill textual priors into feature modeling.
  • Knowledge-Driven Mixture-of-Experts: Representative images and systematic MLLM prompting provide fine-grained expert knowledge for each surgical instrument class.At least three representative images are presented for each instrument class.

1) MLLM-guided Knowledge Generation:

The knowledge-generation process constructs instrument-centered semantic priors from multimodal large language model descriptions, organizes them into structural experts, and uses probabilistic gating to activate relevant knowledge during training.

  • Knowledge-base construction: Three images per instrument class are selected to cover the tip, wrist, and shaft from informative viewpoints, with four images used for grasper.The selection process seeks coverage while minimizing interference from occlusion, motion blur, smoke, or blood.
  • Knowledge-base construction: Human consensus pruning reduces the average number of candidate attributes per class from 11.33 to 6.67 before compiling the knowledge base.Researchers remove redundant, generic, or visually non-discriminative attributes.
  • Knowledge representation: The resulting knowledge sets are converted into text-encoder vectors using templates for attributes such as “forked” and “not present” instrument descriptions.Template formulations improve compatibility between decoupled knowledge and textual-visual representation alignment.
  • Knowledge-driven expert design: Three instrument experts are predefined from semantic priors describing tip, shaft, and wrist components.Each expert specializes in a distinct structural component of surgical instruments.
  • Dynamic knowledge activation: During training, probabilistic pooling evaluates image embeddings against expert-specific Gaussian models, and a gating network activates the top-k attributes from each expert.Gaussian components are established offline from attribute-associated image features and textual attribute representations.
  • Dynamic knowledge activation: The activated semantic knowledge dynamically adapts representations to structural attributes, while a consistently active shared expert captures common knowledge across contexts.The shared expert output is combined with specialized expert knowledge and concatenated with the original visual feature.

C. Component-Tailored Adapter

The component-tailored adapter specializes task representations by combining temporal- and spatial-wise prompt tuning, providing parameter-efficient adaptation without separate network architectures for each sub-task.

  • Motivation: Prompt learning provides task-specific feature specialization without adding parallel network branches for each task.This addresses instability associated with task-specific branches that use distinct classifiers but fail to model distinct task features.
  • Adapter design: CTA learns temporal- and spatial-wise task-specific features in a parameter-efficient manner without requiring multiple network architectures for each sub-task.The adapter is designed to mitigate component-level optimization conflicts through specialized representations.
  • Spatial-temporal prompting: Temporal prompting captures long-range context, while spatial prompting models information across video features for more comprehensive task-specific feature interaction.The design addresses the insufficiency of spatial-only prompting for surgical triplet identification and combines the resulting prompts through element-wise addition.

D. Coordinated Gradient Learning

Coordinated Gradient Learning (CGL) addresses long-tailed optimization conflicts by rebalancing positive and negative gradients, reducing head-tail discrepancies while preserving head-class learning.

  • Motivation: Severe long-tailed distributions cause head classes to overfit while tail categories remain underfitted, producing asynchronous optimization dynamics.Under BCE, tail classes are dominated by negative gradients.
  • CGL Design: CGL adjusts the positive-to-negative gradient ratio by suppressing selected tail-category negative losses and discarding selected head-category positive gradients.The modulation occurs with probability γ and targets the most severe head-tail imbalance.
  • Optimization Effects: Compared with BCE, CGL substantially reduces the discrepancy between tail- and head-class optimization behavior.Tail-class gradients become closer to head-class behavior, while positive head-class learning remains highly confident with low gradient magnitude.
  • CGL Design: CGL leaves medium-category losses unchanged while modulating gradients associated with head and tail categories.This focuses optimization changes where imbalance is most severe.
  • Overall Loss: The overall recognition loss combines CGL for triplet recognition with individual BCE losses for instrument, verb, and target recognition.This preserves the decomposed auxiliary-task formulation while applying coordinated gradients to the triplet task.

A. Datasets and Implementation Details

Experiments use the public CholecT45 dataset and its official five-fold cross-validation protocol for training, validation, testing, ablation, and sensitivity analyses.

  • Dataset: CholecT45 contains 45 laparoscopic cholecystectomy videos, 100.9K frames, and 161K triplet instance labels.Each frame is annotated with 100 binary action triplets spanning 6 instruments, 10 verbs, and 15 targets.
  • Protocol: The experiments follow the official 5-fold cross-validation split, using 31 videos for training, 5 for validation, and 9 for testing.The same experimental setup supports ablation studies and sensitivity analyses.

1) Introduction of Datasets:

The evaluation compares the proposed models with state-of-the-art methods on CholecT45 and CholecT50, reporting strong triplet and component-level performance.

  • CholecT45: On CholecT45, MoeCo-T achieves 40.5% APIV T, exceeding CurConMix-T by 2.8% and TERL-T by 4.8%.MoeCo-B reaches 41.7%, outperforming the second-best CurConMix-B by 2.6%.
  • CholecT45: MoeCo-Ens achieves 42.6%, outperforming SelfD by 4.1% despite SelfD using larger backbone networks in its ensemble.The ensemble averages the sigmoid probabilities of MoeCo-T and MoeCo-B.
  • Component Recognition: The method achieves 94.7% API, 72.5% APV, and 52.3% APT on individual component metrics.The reported results also indicate advantages in instrument-related association recognition.
  • CholecT50: On CholecT50, MoeCo-B and MoeCo-T achieve 40.5% and 39.5% APIV T, with MoeCo-B surpassing the second-best CoT by 2.3%.MoeCo-B also attains 95.1% API, 69.5% APV, and 49.7% APT.

2) Experimental Analysis on Cholect50 Dataset:

Analyses show improved rare-category performance, balanced head-tail convergence, and complementary contributions from KD-MoE, CTA, and CGL.

  • Per-Category Analysis: MoeCo-T achieves higher median performance than the baseline across all four task metrics and raises the distributions’ lower bounds.The lower-bound improvement indicates more consistent recognition of rare or difficult categories.
  • CGL Analysis: Under BCE, head classes learn faster than tail classes, whereas CGL makes tail-class convergence comparable to head-class convergence.BCE’s tail-class imbalance risks head-class overfitting and tail-class underfitting.
  • Ablation Study: The ablation score rises from 38.3% APIV T for the baseline to 40.3% with KD-MoE, then 42.3% after adding CTA and CGL.KD-MoE, CTA, and CGL contribute respectively to knowledge integration, task-specific features, and balanced gradient learning.

2) Ablation on the Semantic Content of KD-MoE:

KD-MoE benefits from semantically meaningful, visually activated priors rather than arbitrary feature augmentation, while CGL improves long-tailed recognition and CTA adds task-specific representations with modest overhead.

  • Semantic Content of KD-MoE: 40.3% APIV T: KD-MoE improves the vision-only baseline from 38.3%, while oracle activation reaches 40.9%.KD-MoE approaches the oracle without ground-truth attributes during inference, indicating that learned gating selects relevant structural priors from visual evidence.
  • Semantic Content of KD-MoE: Random features provide only limited improvement over the vision-only baseline, whereas random prior activation performs better but remains below full KD-MoE.The comparison indicates that MLLM-generated attribute prototypes contain useful domain knowledge and provide regularization beyond simply adding feature inputs.
  • Integration Layer: 42.3% APIV T: integrating KD-MoE at MSA layer l = 2 gives the best performance, and KD-MoE remains better than no integration across tested layers.The results support robustness across different feature latent spaces.
  • Coordinated Gradient Learning: 40.4% APIV T: CGL achieves the highest triplet recognition performance and the best result on five of six metrics among tested long-tailed losses.Compared with BCE, Focal Loss, CB Loss, and EQ Loss, CGL improves APIV T by 2.1%, 1.3%, 1.9%, and 1.4%, respectively.
  • Component-Tailored Adapter: CTA achieves higher APIV T than VPT and ST-Adapter with comparable parameter and computational scale.Its attention-based interactions learn task-specific features for instrument, verb, target, and triplet prediction from shared representations.
  • Computational Efficiency: KD-MoE increases FLOPs only slightly from 4.57G to 4.60G, while MLLM knowledge and Gaussian statistics are constructed offline.The MLLM is not used during training or inference.

E. Visualization

Visualizations show that KD-MoE activates semantic attributes when corresponding instrument structures are present and improves component recognition in challenging surgical scenes, though fine-grained verb confusion remains.

  • Attribute Activation: KD-MoE activates “forked,” “metallic,” and “flexible” attributes only when the corresponding tip, shaft, and wrist structures are visually present.In visually inconsistent cases, the attributes are suppressed, supporting visually responsive rather than random gating.
  • Complex Surgical Scenarios: In smoke- and motion-blurred scenarios, instruments provide the initial step for recognizing other sub-tasks and eliminating false positives.The visualization emphasizes instruments’ distinguishable visual features and their role in guiding component recognition.
  • Complex Surgical Scenarios: The proposed method demonstrates improvements over the baseline in recognizing each triplet component in the showcased complex cases.Fig. 10 compares baseline and MoeCo-T predictions against ground-truth annotations, with correct and incorrect predictions visually marked.
  • Failure Cases: Confusion between “dissect” and “retract” remains a failure case because their visual or semantic differences are minimal.This identifies a boundary for precise verb discrimination despite the qualitative gains.

F. Sensitivity Analysis

Sensitivity analysis examines CTA and CGL hyperparameters and shows that CGL remains effective across relative head/tail definitions, with performance varying only modestly across tested thresholds.

  • Hyperparameters Selection: CGL is intended to improve tail classes while preserving head-class performance.The sensitivity analysis frames this balance as a critical requirement for handling imbalanced surgical triplet categories.
  • Head/Tail Definition: The original CholecT45 absolute head/tail split corresponds to relative thresholds of > 8% and < 0.8%.The paper transitions from absolute counts to percentile-based thresholds to assess robustness across datasets of varying scales.
  • Sensitivity to Head/Tail Splits: 41.6%–42.3% APIV T: performance fluctuates only slightly across relative threshold settings, and all configurations outperform the 41.2% baseline without CGL.The best result uses the > 8%/< 0.8% criterion corresponding to the original experimental split.
  • Head/Tail Definition: For new datasets, the authors recommend relative class frequencies for defining head and tail classes, leaving medium-frequency categories unmodulated by CGL.The proposed interpretation treats very frequent categories as head classes and extremely infrequent categories as tail classes.
  • Conclusion: MoeCo addresses component- and category-level optimization conflicts with CTA and coordinated gradient learning, while integrating domain knowledge through mixture-of-experts.The conclusion presents these mechanisms as the framework’s combined co-optimization design.
Loading 2608.22972v1…