PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Memory Relation Network for Workflow Recognition from Surgical Video

Yueming Jin, Yonghao Long, Cheng Chen, Zixu Zhao, Qi Dou, Pheng-Ann Heng

arXiv:2103.16327v1cs.CVcs.AI

TL;DR

Surgical workflow recognition needs temporal context beyond the short fixed ranges used by prior joint models, while separately learned long-term cues disrupt joint spatio-temporal learning. TMRNet combines a long-range memory bank, multi-scale temporal variation layer, and non-local bank operator for end-to-end fusion, consistently outperforming state-of-the-art methods on M2CAI and Cholec80; one reported Cholec80 comparison is 78.9% versus 67.0% Jaccard.

  • Problem

    Surgical workflow recognition is difficult because prior methods either use short fixed-range temporal information or separately learn visual and long-term temporal cues.

  • Method

    TMRNet stores long-range features in a memory bank, enhances them with multi-scale temporal convolutions, and attentively relates them to present features through a non-local bank operator during end-to-end training.

  • Results

    78.9% versus 67.0% Jaccard on Cholec80, with TMRNet consistently outperforming state-of-the-art methods on both evaluated datasets.

  • Takeaways & Limitations

    Relating present features to long-range, multi-scale temporal supports improves surgical workflow recognition while retaining joint spatio-temporal learning.

  • Takeaways & Limitations

    For cholecystectomy, increasing input length does not continuously improve performance; around 30-second supportive features provide the best result for the studied minute- to hour-level surgeries.

Abstract

from arXiv · show

Automatic surgical workflow recognition is a key component for developing context-aware computer-assisted systems in the operating theatre. Previous works either jointly modeled the spatial features with short fixed-range temporal information, or separately learned visual and long temporal cues. In this paper, we propose a novel end-to-end temporal memory relation network (TMRNet) for relating long-range and multi-scale temporal patterns to augment the present features. We establish a long-range memory bank to serve as a memory cell storing the rich supportive information. Through our designed temporal variation layer, the supportive cues are further enhanced by multi-scale temporal-only convolutions. To effectively incorporate the two types of cues without disturbing the joint learning of spatio-temporal features, we introduce a non-local bank operator to attentively relate the past to the present. In this regard, our TMRNet enables the current feature to view the long-range temporal dependency, as well as tolerate complex temporal extents. We have extensively validated our approach on two benchmark surgical video datasets, M2CAI challenge dataset and Cholec80 dataset. Experimental results demonstrate the outstanding performance of our method, consistently exceeding the state-of-the-art methods by a large margin (e.g., 67.0% v.s. 78.9% Jaccard on Cholec80 dataset).

I. INTRODUCTION

Surgical workflow recognition supports context-aware operating-room systems but remains difficult because short temporal views miss long-range dependencies and phases contain actions with variable durations. TMRNet addresses these challenges with long-range memory, multi-scale temporal modeling, and attentive end-to-end fusion, outperforming prior methods on two benchmark datasets.

  • I. INTRODUCTION: Surgical workflow recognition enables operating-room systems to monitor and optimize procedures, provide context-aware support, warn of deviations, and support post-operative applications.The cited applications include team coordination, documentation, education, skill evaluation, and patient monitoring.
  • I. INTRODUCTION: Short-term video models collect at most 10 seconds because of computational limits, which is insufficient for recognizing phases that require long-range temporal context.The paper illustrates that a current frame can be misclassified from a short-term view but correctly recognized using long-range cues.
  • I. INTRODUCTION: Surgical phases comprise actions at different granularities whose temporal durations vary, including homogeneous repetitive motions and more compositional activities.This variability makes fixed temporal scopes inadequate for representing all phase components.
  • I. INTRODUCTION: TMRNet relates long-range and multi-scale temporal supports to the current frame through a memory bank, temporal variation layer, and non-local bank operator while preserving end-to-end training.The memory bank stores time-indexed past representations, the temporal layer applies multi-scale temporal convolutions, and the non-local operator attentively fuses supportive context.
  • I. INTRODUCTION: 78.9% v.s. 67.0% Jaccard on Cholec80 and 74.3% v.s. 68.5% on M2CAI, with TMRNet surpassing state-of-the-art approaches by large margins.The method was extensively validated on the M2CAI challenge dataset and Cholec80 dataset.

II. RELATED WORK

Related work on surgical workflow recognition includes phase, gesture, step, and activity recognition, using visual-temporal modeling and auxiliary information. Existing temporal approaches include recurrent and convolutional models, while multi-task and multi-modal methods exploit related signals such as tool presence.

  • II. RELATED WORK: Surgical workflow research spans phase recognition and finer-grained gesture, step, and activity recognition, with this paper focusing on surgical phase recognition from video.Video is emphasized as a routinely collectable data source that does not disrupt minimally invasive surgery.
  • II. RELATED WORK: Some temporal methods rely on pre-defined dependencies that are insufficient for subtle motions with strong nonlinear dynamics.This limitation concerns conventional temporal modeling rather than the auxiliary-information stream.
  • II. RELATED WORK: Deep CNN-RNN approaches jointly learn spatial and temporal representations, including SV-RCNet, which integrates a residual network with LSTM.Earlier methods also used handcrafted intensity, gradient, shape, color, and texture descriptors.
  • II. RELATED WORK: Auxiliary-information approaches use multi-task or multi-modal learning, such as jointly recognizing surgical phases and tool presence to complement network training.Prior systems combine tool-recognition branches or predictions with recurrent phase-recognition models.

III. METHODOLOGY

TMRNet augments a recurrent convolutional surgical-video model with a long-range memory bank and temporal variation processing. These components preserve present-frame features while representing distant context and variable-duration motion patterns.

  • III. METHODOLOGY: A long-range memory bank stores time-indexed information from the surgical procedure to provide distant temporal context for current-frame prediction.The design targets videos with complex, lengthy context where distant information can benefit recognition.
  • III. METHODOLOGY: The memory bank is constructed with SV-RCNet, combining a 50-layer ResNet and LSTM to encode each frame with preceding clip-level temporal cues.Each memory feature is a 512-dimensional spatio-temporal representation generated from the current frame and its preceding frames.
  • III. METHODOLOGY: Surgical phases contain fine-grained actions with highly variable durations, making fixed-size temporal kernels too rigid for complex workflow dynamics.The need for temporal variation is especially strong for long-range memory features, which contain more temporal possibilities than short-term inputs.
  • III. METHODOLOGY: The temporal variation layer applies temporal convolutions with kernel sizes 3, 5, and 7, max-pooling, and a short connection while preserving the temporal-channel count.It processes long-range features to capture distinctive multi-scale temporal patterns and retain original information.

C. TMRNet with Non-local Bank Operator

The non-local bank operator uses the current feature to attend to temporally enhanced memory features and add the selected support back to the current representation. This provides long-range contextual fusion within an end-to-end trainable recognition pipeline.

  • C. TMRNet with Non-local Bank Operator: The long-range clip L_t contains the preceding memory-bank features up to the current time, and the temporal variation layer outputs refined representations L̃_t with compositional motion information.In online mode, the clip is formed from the current time index and preceding L−1 frame features.
  • C. TMRNet with Non-local Bank Operator: The operator computes similarities between the current feature c_t and enhanced long-range features L̃_t, then uses soft attention to select supportive temporal cues.The similarity is formed in an embedding space using linear projections and matrix multiplication.
  • C. TMRNet with Non-local Bank Operator: The non-local operation transforms attended memory features through a learned embedding, applies normalization and Softmax attention, and resums them with c_t through a shortcut connection.Layer normalization and dropout refine the attended feature before it is combined with the current representation.
  • C. TMRNet with Non-local Bank Operator: The resulting representation r_t is concatenated with c_t into a 1024-dimensional vector and passed through fully connected layers for workflow prediction.The current feature therefore appears both in the shortcut-fused representation and in the final concatenation.

D. Objective Function and Training Details

TMRNet is trained with a softmax cross-entropy objective that jointly optimizes its backbone, temporal variation layer, and non-local bank operator. Because back-propagation through the full long-range memory bank is computationally infeasible, the bank is treated as fixed and features are obtained offline initially.

  • Objective Function: The softmax cross-entropy loss trains frame-level phase predictions using the local clip and corresponding long-range memory feature.The objective jointly optimizes the backbone, temporal variation layer, and non-local bank operator parameters.
  • Training Details: The full memory bank is not jointly back-propagated because its long-range computational cost is infeasible.The first training stage therefore treats the bank as fixed and computes its features offline; an alternating online update produced no obvious improvement.

2) Training Procedure:

The experiments evaluate online surgical workflow recognition on two public datasets using standard phase-level and video-wise metrics. The study includes backbone comparisons and follows the datasets’ established evaluation settings.

  • Training Procedure: Experiments evaluate workflow recognition on the M2CAI and Cholec80 public surgical-video benchmarks, with method comparison and ablation analysis.Two backbones, ResNet and ResNeSt, are implemented to assess the method’s general efficacy.
  • Datasets: The M2CAI benchmark contains 41 cholecystectomy videos segmented into 8 phases, whereas Cholec80 contains 80 videos annotated with 7 phases.Both datasets use videos recorded at 25 fps, with resolutions specified separately for Cholec80.
  • Evaluation Metrics: Precision, recall, and Jaccard measure phase-level performance, while accuracy measures the percentage of correctly classified frames video-wise.Phase-level metrics are averaged over phases, and all experiments operate online without access to future frames.

B. Comparison with State-of-the-arts

TMRNet outperforms prior surgical workflow recognition methods on both benchmark datasets, including approaches with longer predictions or additional tool annotations. Its gains are attributed to jointly learned spatio-temporal features and longer temporal support.

  • Comparison with State-of-the-arts: Accuracy rises from 85.2% to 86.3% on M2CAI and from 87.3% to 89.2% on Cholec80 over phase-label-based approaches.The method also achieves Jaccard gains exceeding 5% on both datasets.
  • Comparison with State-of-the-arts: TMRNet outperforms OHFM and SV-RCNet on both datasets, supporting joint spatio-temporal learning with longer temporal support.OHFM separates spatial and temporal encoding, while SV-RCNet is an end-to-end recurrent convolutional baseline.
  • Comparison with State-of-the-arts: On Cholec80, TMRNet improves precision from 86.9% to 89.7% and recall from 88.0% to 89.5% despite competing with tool-aware methods.The compared multi-task methods require extra tool-presence labels, whereas the cited comparison emphasizes TMRNet’s longer-range temporal cues.

C. Effectiveness of Key Components

Ablations show that long-range memory, multi-scale temporal kernels, and non-local fusion each improve surgical workflow recognition, with the complete TMRNet performing best and remaining stable across runs.

  • C. Effectiveness of Key Components: 3%-7% gains from adding long-range memory raise ResNet performance from 85.3% AC and 70.4% JA to 88.6% AC and 77.9% JA.The comparison covers four evaluation metrics, not only AC and JA.
  • C. Effectiveness of Key Components: 89.2% AC and 78.9% JA are achieved by full TMRNet with ResNet, while ResNeSt reaches 90.1% AC and 79.1% JA.Long-range cues contribute more than multi-scale patterns, and the non-local bank operator outperforms weighted averaging.
  • C. Effectiveness of Key Components: Five random-initialization runs differ from reported ResNet results by at most 0.2%, with standard deviations below 0.4.The repeated experiments indicate stable and reproducible performance.
  • C. Effectiveness of Key Components: p < 0.05 is observed for Baseline versus TMRNet− across all evaluation metrics, with most other ablation comparisons also significant.These tests support the contributions of long-range cues, multi-scale information, and non-local operation.
  • C. Effectiveness of Key Components: Color-coded ribbons show progressively smoother and more consistent phase predictions as long-term and multi-scale contexts are added.The improvement is especially visible for longer phases such as P2 and difficult noisy frames.

2) Qualitative comparison:

Qualitative analyses show that TMRNet improves phase-level recognition, reduces confusion, and identifies most phase transitions close to their ground-truth timing.

  • 2) Qualitative comparison:: Most phase transitions are identified with less than 10 seconds of deviation from ground truth, and transitions P1→P2, P2→P3, and P5→P6 differ by less than 5 seconds.The challenging worst-Jaccard example still improves notably for P1 and P2, although P2 remains difficult.
  • 2) Qualitative comparison:: All key components consistently increase prediction confidence toward correct labels for representative surgical actions.The comparison uses prediction probabilities under different ablation settings.
  • 2) Qualitative comparison:: TMRNet increases diagonal recall probabilities and reduces noisy misclassification patterns across phase-level confusion matrices.Errors involving P5 and P6 assigned to P4 and P7 assigned to P1 are especially alleviated.
  • 2) Qualitative comparison:: TMRNet dominates the competing settings in Jaccard across almost all seven phases, with consistent improvements in most phase-level precision and recall values.Precision gains are especially evident for P1 and P3, while recall increases notably for P2, P5, and P6.

D. Importance of Long-range Information

Longer memory-bank support generally improves recognition, but excessive temporal context can introduce irrelevant information and slightly reduce some metrics.

  • D. Importance of Long-range Information: AC rises from 85.3% to 88.6% as memory support increases from 0 to 40 seconds on Cholec80.The result demonstrates the value of integrating long-term temporal dependencies, while recurrent memory allows shorter supports to cover longer cues.
  • D. Importance of Long-range Information: RE decreases by about 0.2% and JA by 0.1% when support grows from 30 to 40 seconds.The authors attribute this possible degradation to irrelevant noise and incompatible appearances within phases, and select 30-second support.
  • D. Importance of Long-range Information: Supportive features with windows from 10 to 40 seconds can represent approximately 20 to 50 seconds of preceding information through the LSTM memory bank.These qualitative comparisons omit multi-scale kernels.

2) Different manners to increase temporal cues:

Memory-bank support is more effective than simply lengthening the LSTM input, while the preferred multi-scale integration fashion yields a small advantage and smaller fixed kernels can work well.

  • 2) Different manners to increase temporal cues:: Directly expanding pure SV-RCNet input clips to 20, 30, or 40 seconds fails to deliver large gains and degrades performance at 30 and 40 seconds.The authors relate this to difficulty preserving long-range cues in the LSTM hidden state and to harder LSTM training.
  • 2) Different manners to increase temporal cues:: I2 generally outperforms I1 by around 0.1% when incorporating multi-scale temporal information.The paper attributes I1's weaker performance to late fusion potentially disrupting learned attended structures.
  • 2) Different manners to increase temporal cues:: Fixed temporal kernels of size 3 or 5 can outperform multi-kernel settings, consistent with short-duration frequent actions in cholecystectomy.The comparison varies incorporation fashion, kernel type, and fusion style.

V. DISCUSSION

TMRNet addresses short-term or separately staged temporal modeling by relating long-range, multi-scale context to current surgical-video features. Its memory-based design improves recognition across backbones and supports real-time use, while performance does not improve indefinitely with longer temporal support.

  • Architecture independence: Consistent gains with two backbones on public datasets indicate that the scheme is architecture-independent and can integrate with existing 2D and 3D CNN video models.This extends the method beyond a single backbone or video-model family.
  • Temporal-range limitation: Increasing temporal support does not continuously improve Cholec80 performance because longer ranges can include irrelevant lens-cleaning noise and smoke or blood artifacts.Around 30 seconds can already cover complete fine-grained motions, whereas excessive context may degrade recognition.
  • Post-processing results: 92.1% PR, 91.1% RE, and 93.7% AC were achieved on Cholec80 with PKI post-processing, exceeding refined state-of-the-art results of 91.6%, 90.1%, and 93.3%.The reported TMRNet results before post-processing were produced by a single network; the higher values use prior knowledge inference.
  • Clinical usability: Offline precomputation and online memory updates support real-time recognition while avoiding repeated training computation.The memory bank is pre-calculated over complete procedures for training and updated incrementally during inference.
  • Architecture: TMRNet relates present features to long-range, multi-scale temporal context through a memory bank, temporal variation layer, and non-local bank operator.The design preserves joint spatio-temporal learning while incorporating supportive temporal features.
Loading 2103.16327v1…