Source-linked AI summary
Multimodal Shared Latent Representation of Narration, Microscope and iOCT Images for Phase Recognition in Vitreoretinal Surgery
Onur Izmitlioglu, Shervin Dehghani, Tarek Ghannoum, Benedikt Schworm, Nassir Navab
TL;DR
Vitreoretinal phase recognition lacks synchronized multimodal data combining microscope views, iOCT, and narration. The framework uses microscope images as a shared anchor across real and synthetic datasets, improving macro-phase recognition while exploring micro-phase estimation with limited real-data validation.
Problem
Vitreoretinal phase recognition is difficult because subtle tool-tissue interactions and scarce synchronized multimodal datasets limit context-aware surgical analysis.
Method
The framework aligns narration, microscope, and synthetic tool-aligned iOCT representations through microscope-anchored shared embeddings, then uses a dual-head MS-TCN++ for macro- and micro-phase prediction.
Results
Macro-phase recognition achieved a mean F1 of 53%, with mean precision of 50.3% and mean recall of 57.9%, outperforming the zero-shot baseline.
Takeaways & Limitations
Microscope images can bridge disjoint real and synthetic datasets, allowing structural priors from synthetic iOCT to inform real surgical video without synchronized tri-modal recordings.
Takeaways & Limitations
Micro-phase targets were validated quantitatively only on synthetic data, while real-video estimates were assessed qualitatively; the study also lacks alignment ablations.
Abstract
from arXiv · showhide
Surgical phase recognition is key to context-aware computer-assisted feedback in vitreoretinal procedures, yet the scarcity of synchronized multimodal intraoperative data, particularly microscope views and intraoperative OCT, limits approaches that aim to replicate the multimodal integration surgeons perform naturally. Surgical narration, by contrast, is abundantly available online and offers rich semantic supervision. Prior work has mainly explored pairwise contrastive learning (e.g., intraoperative OCT-microscope or microscope-narration), leaving the joint modeling of all three modalities largely unexplored. We introduce a framework that uses microscope views as a shared anchor to bridge surgical narrations and intraoperative OCT (iOCT) without requiring a fully synchronized tri-modal dataset, leveraging real microscope-narration videos and a synthetic dataset of synchronized microscope video and tool-aligned iOCT pairs. Contrastive alignment transfers structural priors from the synthetic domain to real videos lacking iOCT, and a dual-head MS-TCN++ integrates the resulting embeddings for joint macro- and micro-phase prediction. Evaluated on real vitreoretinal surgeries, our framework improves macro-phase recognition over a zero-shot baseline (mean F1 0.38 to 0.53) and provides an exploratory route to estimating fine-grained instrument-tissue measurements that are not directly observable in real microscope video alone; these micro-phase estimates are validated quantitatively on synthetic data and shown only qualitatively on real surgery. To our knowledge, this is the first work to unify microscope view, iOCT B-scans, and surgical narrations in a shared latent space for surgical phase recognition.
1 Introduction
The paper addresses scarce synchronized multimodal data for vitreoretinal phase recognition by using microscope views to connect narrated videos with synthetic tool-aligned iOCT data. The shared latent space supports macro-phase prediction and exploratory inference of fine-grained instrument-tissue measurements from microscope views.
- The framework uses three modality-specific encoders for narration, microscope views, and iOCT B-scans, followed by dual-head MS-TCN++ prediction of macro- and micro-phases.
- Surgical phase recognition is foundational for context-aware systems but remains difficult because vitreoretinal procedures contain subtle tool-tissue interactions and scarce annotated multimodal datasets.
- Joint datasets containing surgical narrations, microscope views, and iOCT B-scans are largely absent because synchronized acquisition across heterogeneous intraoperative systems is challenging.
- The framework uses microscope views as a common anchor to align separate microscope-narration and microscope-iOCT datasets in one latent space without fully annotated tri-modal data.
- Joint supervision allows a microscope view alone to support exploratory approximation of measurements such as tool-to-retina distance when iOCT is unavailable.
2 Related Works
Related work has established deep temporal, multimodal, and language-supervised surgical analysis, but vitreoretinal procedures remain comparatively underrepresented in narrated and context-rich datasets. The paper builds on vision-language pretraining and contrastive alignment to address this gap.
- Deep models and multi-stage temporal networks advanced data-driven surgical workflow analysis, while multimodal systems emphasized context-aware understanding.
- Ophthalmic surgery remains underrepresented despite benchmarks such as CATARACTS and OphNet, with narrated or context-rich vitreoretinal data still limited.
- Language as Context in Surgical Modeling: Prior work showed that speech and expert language can provide non-intrusive semantic supervision for phase recognition, skill assessment, and temporal alignment.
- Vision-language pretraining and contrastive learning enabled zero-shot or few-shot contextual transfer, while IMAGEBIND demonstrated alignment across diverse modalities in shared latent spaces.
3 Methodology
The methodology combines hierarchical vision-language pretraining, synthetic microscope-iOCT alignment, and dual-head temporal modeling for ERM peeling phase recognition. Macro-phases are classified from real videos, while synthetic metadata supervises continuous micro-phase regression.
- Phase Representation: ERM peeling is represented with five clinically defined macro-phases covering preparation, membrane handling, and retinal stabilization.
- Phase Representation: Micro-phases encode tool-to-retina distance and forceps jaw closure as continuous measures of instrument-tissue interaction.
- Datasets: The real dataset contains 33 online ERM peeling videos with abstract-, keystep-, and narration-level annotations, while synthetic data supplies ERM and subretinal-injection procedures with aligned iOCT.
- Multimodal Alignment: Stage A aligns trainable ResNet-50 microscope embeddings with frozen Bio_ClinicalBERT narration embeddings using hierarchical text-image pairs.
- Multimodal Alignment: Stage B freezes the microscope encoder and trains an iOCT encoder in a shared 512-dimensional space using hard triplets from temporally matched and phase-based unmatched pairs.
- Temporal Prediction: A dual-head MS-TCN++ models temporal structure while classifying five macro-phases and regressing synthetic forceps closure and tool-to-retina distance.
4 Experimental Results
The proposed temporal multimodal model improves real-video macro-phase recognition over the zero-shot baseline and predicts synthetic micro-phases quantitatively. Real-video micro-phase outputs remain exploratory because ground truth is unavailable.
- Macro-Phase Recognition: 53% mean F1, 50.3% mean precision, and 57.9% mean recall were achieved for macro-phase recognition with dual-head MS-TCN++ and label refinement.
- Macro-Phase Recognition: The model showed particularly strong gains for Anterior and Posterior Preparation, while Membrane Identification and Retina Stabilization remained harder to recognize.
- Micro-Phase Prediction: Micro-phase prediction on held-out synthetic ERM sequences achieved MAE 0.050 ± 0.001 for tool-to-retina distance and 0.167 ± 0.135 for forceps jaw closure.
- Micro-Phase Prediction: Forceps closure captured meaningful transition timing despite higher error, whereas distance predictions were stable but conservative and potentially over-smoothed.
- Limitations: Real-video micro-phase results were qualitative only, and the study’s macro-phase comparison does not isolate temporal modeling, label refinement, and multi-task learning.
5 Conclusion
The microscope image serves as a common anchor that transfers structural priors from synthetic iOCT into real surgical video without synchronized tri-modal recordings. Further progress depends on broader evaluation and more explicit cross-modal learning objectives.
- Conclusion: A microscope-based shared anchor bridges otherwise disjoint datasets and transfers synthetic iOCT structural priors to real surgical video without synchronized tri-modal recordings.The approach connects microscope–narration and microscope–iOCT data through a common representation.
- Future work: Larger, more diverse datasets with real videos annotated for forceps properties are needed to assess adaptability and determine modality-specific clinical benefits.Such evaluation would also help disentangle each modality’s contribution and inform training strategies.
- Future work: Loss functions that explicitly encourage cross-modal attention are proposed as a next step beyond merely pulling matched pairs together.The stated goal is richer and more robust multimodal representations for surgical AI.