PaperlayerView exact revision on arXiv

Source-linked AI summary

OphBiWSSD: Scaling Temporal Action Localization in Ophthalmic Surgeries with Bidirectional Weight-tied State Space Duality

Yang Liu, Qionghong Ma, Joongwon Chae, Lihui Luo, Yibing Shen, Yulin Zhuo, Yingting Zhu, Jiashu Chang, Xiaoyun Zhong, Dongmei Yu, Peter E. Lobie, Peiwu Qin, Chengming Yang

arXiv:2609.12409v1cs.CV

TL;DR

Ophthalmic temporal action localization must preserve fine action boundaries while modeling long-range dependencies that challenge attention-based architectures. OphBiWSSD uses bidirectional, weight-tied state-space duality to combine preceding and succeeding context with linear complexity. On OphNet, it reports state-of-the-art phase and operation localization results.

  • Problem

    Attention-based temporal localization faces quadratic computational demands on high-frame-rate ophthalmic videos, while causal state-space models cannot use succeeding context that may resolve action ambiguity.

    Source: §I · pp. 1–2

  • Method

    OphBiWSSD uses bidirectional selective scanning and weight-tied state-space duality to synthesize preceding and succeeding surgical context with linear complexity.

    Source: §I · p. 2

  • Results

    44.42% mAP on phases and 43.08% mAP on operations establish OphBiWSSD’s state-of-the-art OphNet performance and surpass established baselines.

    Source: §I · p. 2, §VI · p. 9

  • Takeaways & Limitations

    OphBiWSSD’s bidirectional scan captures non-causal temporal dependencies, while weight tying shares parameters across directions and maintains computational efficiency.

    Source: §VII · p. 10

    Wrong Label Err reduces mAPN by 6.3%, and miss rates exceed 23% for Extra Small instances with brief gestures.

    Source: §VI · p. 9

Abstract

from arXiv · show

High-frequency surgical maneuvers in ophthalmology necessitate high-fidelity temporal modeling, yet characterizing long-range procedural dependencies remains computationally prohibitive for attention-based architectures. Existing models often require aggressive temporal downsampling, which compromises the detection of fine-grained action boundaries and instrument-tissue interactions. To address these scalability constraints, we present OphBiWSSD, a framework that reformulates surgical temporal action localization leveraging Bidirectional State Space Duality. By employing a weight-tied selective scan mechanism that incorporates both preceding and succeeding surgical contexts, our approach facilitates the global synthesis of non-causal temporal cues with linear complexity. This streamlined architecture is well-suited to capture the bidirectional dependencies present in ophthalmic workflows, effectively bridging the gap between local boundary precision and long-range procedural context without incurring the quadratic memory overhead of traditional Transformers. Extensive experiments on the OphNet benchmark demonstrate that OphBiWSSD achieves state-of-the-art temporal localization performance, with mean Average Precisions of 44.42% on phases and 43.08% on operations, surpassing the baselines by 6.80% and 6.66%, respectively. Empirical validation indicates that our approach ensures precise temporal localization and offers a computationally viable pathway for deploying surgical intelligence systems in clinical environments. The code is publicly available at https://github.com/yo3nglau/OphBiWSSD.

I. INTRODUCTION

Ophthalmic temporal action localization must capture rapid gesture boundaries and long-range context without the quadratic cost of attention. OphBiWSSD uses bidirectional, weight-tied state-space modeling, while the introduction situates the task within broader temporal localization and surgical workflow analysis.

I. INTRODUCTION

  • High-frame-rate ophthalmic videos require fine boundary localization, but attention scales quadratically and aggressive temporal decimation can obscure dense boundaries.Causal state-space models use only preceding information, although later surgical events can help resolve action ambiguity.Source: §I · pp. 1–2
  • OphBiWSSD combines bidirectional selective scanning with weight-tied state-space duality to synthesize preceding and succeeding context at linear computational complexity.The weight-tied formulation is described as enforcing direction-invariant feature learning across scanning paths.Source: §I · pp. 1–2
  • 44.42% mAP on phases and 43.08% on operations are reported as state-of-the-art OphNet results, improving on previous attention-based architectures.The proposed framework is motivated by the need to model surgical context in both temporal directions with linear complexity.Source: §I · p. 2

A. Temporal Action Localization

  • Temporal action localization has shifted from proposal-and-refinement pipelines toward anchor-free, single-stage models, while attention-based encoders retain quadratic sequence complexity.ActionFormer uses a multi-scale Transformer encoder, and TriDet adds a Trident head for boundary confidence.Source: A. Temporal Action Localization · p. 2

B. Surgical Workflow Analysis

  • Ophthalmic workflow analysis addresses rapid maneuvers and subtle instrument-tissue cues; OphNet provides a hierarchical benchmark and baselines for this setting.The cited context links surgical video analysis to intraoperative safety and objective skill assessment.Source: B. Surgical Workflow Analysis · p. 2

C. State Space Models

State-space models offer a route to efficient long-sequence processing, and OphBiWSSD builds on selective scanning and state-space duality. The preliminaries describe how continuous and discrete state-space formulations support this approach.

C. State Space Models

  • State-space models, including Mamba and SSD, are presented as efficient sequence-modeling alternatives, with SSD linking structured state spaces to semiseparable matrices.The cited work describes this link as enabling faster training through optimized matrix multiplications.Source: C. State Space Models · p. 2

A. Preliminaries

  • The continuous state-space formulation maps input x(t) through latent state h(t) to output y(t), and Zero-Order Hold discretizes its parameters for digital sequence processing.A and B are the state-transition and input matrices, while C is the projection matrix.Source: A. Preliminaries · p. 2
  • Mamba’s Selective Scan Mechanism makes B, C, and ∆ functions of input x_k, enabling selective propagation or attenuation of information over time.Source: A. Preliminaries · p. 3
  • Structured SSD constrains A to a scalar-times-identity form, creating a recurrent-to-matrix-multiplication duality that supports efficient training and O(L) inference.Source: A. Preliminaries · p. 3

B. Overall Architecture

OphBiWSSD transforms ophthalmic video features into discrete action segments through a hierarchical bidirectional state-space architecture. Its processing combines forward and reversed sequences, fuses their outputs, and applies a temporal mask.

B. Overall Architecture

  • The architecture maps high-dimensional temporal features from ophthalmic video to action segments with temporal boundaries through a hierarchical BiWSSD transformation.The input sequence has length L and feature dimension D; Ω is a binary mask for variable-length sequences.Source: B. Overall Architecture · pp. 3–4
  • BiWSSD forms weight-tied projections from forward and reversed features, scans and splits the outputs, restores backward temporal order, then fuses and gates them.The fused features pass through a linear projection to produce the output.Source: B. Overall Architecture · p. 3
  • The overall transformation is organized as a multi-stage backbone that moves from local feature extraction to global procedural modeling and multi-scale temporal features.Source: B. Overall Architecture · p. 4

1) Local Feature Embedding:

The feature pipeline preserves local temporal detail before modeling global procedural context and constructing a multi-scale hierarchy. A joint objective trains action classification and temporal boundary regression.

1) Local Feature Embedding:

  • Masked convolutional embedding projects the input into an embedding space while preserving initial temporal resolution and encoding local instrument-tissue interactions.Source: 1) Local Feature Embedding · p. 4

2) Constant-Resolution Stem:

  • At native temporal scale, the constant-resolution stem uses MaskBiWSSD blocks to integrate forward and backward scans for non-causal modeling.The bidirectional context supports boundary localization where post-action context is important alongside preceding signals.Source: 2) Constant-Resolution Stem · p. 4

3) Multi-Scale Branch:

  • Strided downsampling creates a hierarchy of feature maps with progressively reduced temporal resolution to accommodate actions of varying durations.The resulting feature hierarchy is passed to a detection head for classification and boundary refinement.Source: 3) Multi-Scale Branch · p. 4
  • The training objective is a composite multi-task loss that jointly optimizes action classification and temporal boundary regression; Lcls and Lreg represent Focal Loss and IoU Loss, respectively.At feature level l and temporal position t, c^l_t > 0 identifies foreground samples, while c^l_t = 0 denotes background samples.Source: 3) Multi-Scale Branch · p. 4

C. Bidirectional Selective Scan Mechanism

The mechanism scans the sequence in both temporal directions to combine preceding and succeeding surgical cues with linear complexity. It fuses the resulting contexts into a joint representation.

C. Bidirectional Selective Scan Mechanism

  • Forward and backward latent states evolve from preceding and succeeding inputs, respectively, providing bilateral context without quadratic self-attention scaling.The states are mapped to the output space through discretized projection matrices.Source: C. Bidirectional Selective Scan… · p. 4
  • Concatenating original and reversed sequences along the batch dimension lets one hardware-aware kernel process both scan directions.The transformation uses temporal reversal and shared state-space parameters θ.Source: C. Bidirectional Selective Scan… · p. 4
  • The forward and backward outputs are fused by concatenation or addition, then linearly projected into the model space.Concatenation projects from a doubled feature dimension, while addition uses the original inner dimension.Source: C. Bidirectional Selective Scan… · p. 4

D. Weight-tied State Space Duality

Weight-tied State Space Duality shares projection logic across forward and backward scans to encourage direction-invariant features. The section also formulates OphNet localization as predicting action categories and temporal boundaries.

D. Weight-tied State Space Duality

  • Weight-tied State Space Duality uses a shared transformation for both scan directions, rather than independent directional operators.The shared parameterization is presented as a structural regularizer for direction-invariant feature learning.Source: D. Weight-tied State Space Duality · p. 4
  • Shared weights reduce the input-projection parameter footprint and are intended to make latent representations more robust to temporal variation.The resulting representations support boundary regression and action classification.Source: D. Weight-tied State Space Duality · p. 5

A. Dataset and Task Formulation

  • OphNet is a publicly available benchmark with a hierarchical taxonomy, framing localization as action-category prediction plus temporal-boundary identification.Its constrained microscopic field and frequent tool interactions make some gesture transitions ambiguous, motivating use of future cues.Source: A. Dataset and Task Formulation · p. 5

1) Temporal Phase Localization:

The evaluation covers coarse procedural phases and fine-grained instrument-tissue operations, using mAP across temporal-overlap thresholds. Implementation details specify the sequence, backbone, and localization-head settings.

1) Temporal Phase Localization:

  • 14,674 instances across 52 procedural milestones define phase localization, which targets longer-duration stages and global workflow consistency.Examples include Phacoemulsification and IOL Insertion.Source: 1) Temporal Phase Localization · p. 5

2) Temporal Operation Localization:

  • 17,508 instances across 107 instrument-tissue gestures define operation localization, emphasizing precise boundaries for subtle, high-frequency actions.Examples include suturing maneuvers and capsulorhexis segments.Source: 2) Temporal Operation Localization · p. 5

B. Evaluation Metrics

  • mAP averages category-level AP across predefined tIoU thresholds, with a true positive requiring a matching action category and overlap above threshold α.Reported thresholds typically range from 0.3 to 0.7.Source: B. Evaluation Metrics · p. 5

C. Implementation Details

  • Training uses sequences up to length 2304 and a hierarchical backbone; the Trident head performs multi-scale classification and boundary regression with 16-bin refinement.The stated DropPath rate is 0.3, and Soft-NMS uses a voting threshold of 0.7.Source: C. Implementation Details · p. 6

V. RESULTS

OphBiWSSD delivers strong phase and operation localization on OphNet, with further gains reported against Mamba-based and attention-based comparisons.

A. Temporal Phase Localization

  • 44.42% average mAP on OphNet phases; OphBiWSSD exceeds ActionFormer by 6.80% and TriDet by 5.20%.Its advantage also grows at higher tIoU thresholds, reaching 33.18% average mAP at α = 0.7 versus 26.04% for ActionFormer and 28.61% for TriDet.Source: A. Temporal Phase Localization · p. 6
  • OphBiWSSD outperforms the Mamba-based ActionMamba by 3.89%.With the CLTDR-GMG detection head and loss, OphBiWSSD† improves by 3.79%.Source: A. Temporal Phase Localization · p. 7

B. Temporal Operation Localization

  • 43.08% average mAP on OphNet operations, a 6.66% improvement over established benchmarks.The enhanced OphBiWSSD† variant reaches 44.41% average mAP and improves 4.01% over CLTDR-GMG at 40.40%.Source: B. Temporal Operation Localization · p. 7

C. Ablation Study

The ablations examine bidirectionality, weight tying, fusion, and SSD head count; they favor the proposed bidirectional SSD, weight tying, concatenation, and 32 heads.

1) Bidirectional Strategy:

  • Bidirectional SSD improves average mAP by 1.49% over the causal baseline.Random sequence flipping and paired forward/reverse blocks degrade performance; the text attributes this to noise or insufficient fine-scale fusion of bilateral cues.Source: 1) Bidirectional Strategy · p. 7

2) Weight-tied Strategy:

  • Weight-Tied SSD reduces parameters by approximately 34.8% and inference latency by 44.8% versus the fully untied baseline, while maintaining performance parity.Untied Proj+Tied SSD improves average mAP by 0.57% but requires 34.7% more parameters.Source: 2) Weight-tied Strategy · p. 7

3) Bidirectional Feature Fusion:

  • Concatenation improves average mAP by 0.95% over element-wise summation.The authors attribute the gain to better preservation of semantic signatures from the forward and backward scans.Source: 3) Bidirectional Feature Fusion · p. 7

4) Number of SSD Heads:

  • P = 32 yields 0.17% higher average mAP than P = 16 and 0.48% higher than P = 64.The authors identify 32 heads as the optimal set point; fewer may limit representational capacity, while more may over-parameterize and risk overfitting.Source: 4) Number of SSD Heads · pp. 7–8

5) Detection Head:

The section evaluates detection heads and computational efficiency, then examines qualitative localization and error patterns across instance characteristics.

5) Detection Head:

  • The CLTDR detection head improves average mAP by 2.50% over the baseline convolutional head, while the Trident head improves it by 2.10%.The authors associate these gains with task decoupling and multi-scale refinement.Source: 5) Detection Head · p. 8

6) Computational Analysis:

  • OphBiWSSD runs 2.3× faster than DyFADet and reduces multiply-accumulate operations by 41.7%.It also reduces latency by 41.7% and computational overhead by 9.4% versus CLTDR-GMG; relative to ActionMamba, average mAP improves by 3.89% for phases and 3.25% for operations.Source: 6) Computational Analysis · p. 8

D. Qualitative Visualization

  • Qualitative OphNet visualizations show category assignments and action onset and offset predictions aligned with ground truth, including in dense sequences.Colored segments denote action categories, while blank segments denote invalid regions or gaps.Source: D. Qualitative Visualization · p. 8

E. Error Analysis

  • The DETAD analysis uses normalized mAP to compare uneven subsets by coverage, duration, and per-video instance frequency.Its false-positive breakdown separates true positives and five error categories; the false-negative analysis groups misses by coverage, length, and frequency.Source: E. Error Analysis · p. 8, Figure 4 · p. 9, Figure 5 · p. 9

1) Sensitivity Profile:

  • OphBiWSSD localizes long-range actions effectively but performs less well on transient gestures with minimal duration.Average precision also falls for M and L instance-frequency categories, where similar gestures or intermediate-duration actions create temporal ambiguity.Source: 1) Sensitivity Profile · pp. 8–9

2) False Positive Profile:

OphBiWSSD achieves strong OphNet localization results, while its error profiles identify wrong labels and very brief gestures as remaining challenges. The discussion attributes gains to bidirectional context and weight-tying; the conclusion describes integration of preceding and succeeding context with direction-invariant feature learning.

2) False Positive Profile:

  • Wrong Label Err reduces performance by 6.3%, whereas Localization Err has minimal impact.The discussion identifies Wrong Label Err as the primary bottleneck, accounting for a 6.3% reduction in mAPN.Source: 2) False Positive Profile · p. 9, §VI · p. 9

3) False Negative Profile:

  • Extra Small instances have miss rates of 23.9% for Coverage and 25.4% for Length, versus 6.7% and 6.4% for Extra Large instances.The passage attributes the remaining difficulty for atomic instrument-tissue interactions to limited visual evidence in subsecond gestures.Source: 3) False Negative Profile · p. 9

VI. DISCUSSION

  • 44.42% phase mAP and 43.08% operation mAP on OphNet surpass established baselines, including CLTDR-GMG and ActionMamba.The bidirectional selective scan improves mAP by 1.49%, and weight-tying reduces parameters by 34.8% without compromising representational capacity.Source: §VI · p. 9
  • Wrong-label errors and Extra Small gesture misses remain limitations; the discussion suggests specialized classification decoders and locally perceptive modules as possible improvements.The reported Extra Small miss rates exceed 23%.Source: §VI · p. 9

VII. CONCLUSION

  • OphBiWSSD integrates preceding and succeeding contexts at each time step, while structural parameter sharing supports direction-invariant feature learning and computational efficiency.The conclusion states that the framework enables precise automated procedure recognition and objective skill assessment in clinical settings.Source: §VII · p. 10

Related papers

Loading 2609.12409v1…