Source-linked AI summary
From Multimodal Observation to Interpretable Suggestions: Counterfactual Time-Expanded Relational Modeling of Surgical Teams
Vincenzo Marco De Luca, Antonio Longa, Giovanna Varni, Andrea Passerini
TL;DR
Surgical AI has largely under-modeled team interactions and offered limited actionable support for improving teamwork. The paper introduces a Time-Expanded multimodal relational model with counterfactual suggestions that identify minimal behavioral and interaction changes. Experiments on simulated surgical procedures report improved predictive performance and interpretable insights into team dynamics.
Problem
Existing surgical AI solutions focus mainly on visual workflow and technical execution, leaving team interactions and actionable teamwork support under-modeled.
Method
The paper models multimodal surgical-team dynamics with Time-Expanded graphs and generates counterfactual suggestions grounded in minimal, structured behavioral and interaction changes.
Results
TE-ReNN improves performance across all OTAS teamwork dimensions, with an approximately 7% gain over the strongest baseline, MHA+GAT.
Takeaways & Limitations
The framework supports team-level prediction and actionable, human-interpretable guidance for improving teamwork in simulated surgical training.
Abstract
from arXiv · showhide
In surgery, patient safety is threatened not only by technical issues but also by poor teamwork. However, existing surgical AI-based solutions focus mainly on visual workflow and technical execution, neglecting the modeling of team interactions and missing opportunities to actively support clinicians in improving their teamwork skills. To address this gap, we propose a tempo-relational framework for modeling surgical team dynamics from multimodal observations. By leveraging Time-Expanded graphs, the approach captures both relational structure and temporal evolution, achieving strong expressivity while remaining robust in the low-data regime typical of surgical settings. Beyond prediction, such modeling enables the generation of efficient, interpretable, and actionable suggestions for clinicians. More specifically, we generate suggestions via a counterfactual procedure that identifies minimal yet structured changes in individual behaviors and interaction patterns associated with improvements in team performance. Experiments with simulated surgical procedures show that our approach improves predictive performance in diverse behavioral and interaction goals while offering meaningful insights into team dynamics. This work advances surgical AI beyond outcome-driven prediction towards a socially grounded, team-centric, and actionable paradigm to better understand and support the development of team skills in surgical settings.
1 Introduction
The paper addresses the limited modeling of intra-operative team dynamics by proposing a multimodal, graph-based framework for prediction and actionable support. Its Time-Expanded relational model and counterfactual procedure target interpretable improvements in teamwork, with validation in simulated surgical training.
- Existing surgical AI models procedural workflows and individual technical skills, but intra-operative team dynamics and actionable insights remain largely unmodeled.
- Time-Expanded graphs capture temporal evolution within relational structure through message passing across spatial and temporal dimensions.
- The proposed approach is designed to preserve expressive relational modeling while remaining robust in the low-data regimes typical of surgical settings.
- Its counterfactual procedure identifies minimal, interpretable behavioral changes associated with improved teamwork outcomes.
- The framework is validated through counterfactual insights from clinical teams performing procedures on high-fidelity simulators.
- TE-ReNN combines multimodal team-interaction modeling with graph-based counterfactual recommendations and outperforms existing approaches in surgical training simulation.
2 Related Work
Prior automated team-modeling methods are constrained by scarce multimodal multiparty data and tend to separate temporal modeling from relational modeling. Counterfactual explanations provide a basis for reasoning about hypothetical interventions, but are commonly used as post-hoc diagnostics for individual predictions.
- Automated team modeling is limited by the difficulty of collecting multimodal data in multiparty settings and the scarcity of large-scale datasets.
- Static approaches aggregate multimodal features across individuals but ignore temporal dynamics and interpersonal interactions.
- Temporal models capture behavioral evolution but typically treat team members independently and overlook relational structure.
- Relational graph-based models incorporate interactions among team members but often neglect temporal dynamics or focus on limited tasks.
- Counterfactual explanations formalize reasoning about hypothetical interventions and are widely used as post-hoc diagnostic tools for individual predictions.
3 Method
The method represents surgical teams as multimodal interaction graphs over aligned 15-second windows, then connects these snapshots into a Time-Expanded graph to model relational and temporal dynamics. It generates actionable counterfactual suggestions by retrieving desirable examples and identifying salient behavioral differences relative to the observed sequence.
- 3.1 Time-Expanded Relational Neural Networks: Multimodal audio-video features, annotations, and conversation transcripts are aggregated into aligned, non-overlapping 15-second windows.The features include acoustic, motion, human–object interaction, and textual information.
- 3.1 Time-Expanded Relational Neural Networks: Each snapshot graph represents team members as nodes and broadcast verbal communication as directed edges between members present in the same window.Node features include paralinguistic, text, motion, and human–tool interaction cues.
- 3.1 Time-Expanded Relational Neural Networks: Snapshot graphs are connected by identity-preserving temporal edges, allowing information to propagate across team members within windows and along each individual’s trajectory over time.Graphs span up to six minutes, corresponding to 24 temporal windows, to capture evolving coordination while remaining computationally tractable.
- 3.2 Actionable Counterfactuals: The counterfactual method avoids unrealistic single-member edits by retrieving a similar sequence with a desirable outcome and comparing its temporal prefix with the current sequence.This exemplar-based strategy accounts for cascading effects across team members and subsequent time steps.
- 3.2 Actionable Counterfactuals: An offline embedding repository and online retrieval procedure support efficient counterfactual generation for prefixes with matching temporal length and desired outcomes.Training prefixes are encoded with the trained GNN and indexed for inference-time retrieval using cosine similarity.
- 3.2 Actionable Counterfactuals: Saliency-ranked node and edge components are matched at role-consistent positions, and the first salient absent component becomes the counterfactual suggestion.Retrieval grounds candidate behaviors in observed sequences, while explanation-based refinement isolates components associated with the desirable outcome and identifies the earliest intervention point.
4 Experimental setting
The experiments use simulated surgical procedures with multimodal behavioral data, evaluated against models spanning static, temporal, relational, and tempo-relational modeling. Counterfactual analyses examine iteration efficiency and feature changes alongside predictive performance.
- Dataset: The MM-OR dataset contains 21 simulated knee replacement procedures, with experiments using 15 procedures, including nine full replacements.
- Dataset: Behavioral classes combine three paralinguistic features from the eGeMAPS set.
- Dataset: Audio-video recordings use five cameras and one environmental microphone, while team behaviors are manually annotated with five OTAS dimensions.
- Pre-processing: Automated diarization and transcription were manually corrected because of background noise, a single microphone, and frequent English–German code-switching.
- Counterfactual analysis: Counterfactual evaluation considers the average iterations needed to find an appropriate intervention and the average distance between original and counterfactual paralinguistic features.
- Baselines: Baselines span static, temporal, relational, and tempo-relational neural networks, with GCN, GAT, and GIN among the relational backbones.
5 Results
TE-ReNN improves low-data prediction of surgical teamwork across OTAS dimensions and supports counterfactual suggestions that target relational and behavioral changes. Human evaluations and qualitative examples indicate that these suggestions are realistic, timely, helpful, and interpretable, with a small minimality trade-off.
- TE-ReNN addresses research questions on low-data team modeling and interpretable, actionable counterfactuals for improving teamwork.
- Predictive performance: Approximately 7% improvement over the strongest baseline (MHA+GAT) was achieved across OTAS teamwork dimensions, with limited dependence on the GNN backbone.
- Counterfactual assessment: Counterfactuals modify either communication topology or behavioral classes, with paralinguistic changes particularly effective for Leadership and Cooperation and topological changes better for Situational Awareness and Communication.
- Counterfactual assessment: Communication, Coordination, and Cooperation require limited behavioral changes for large score improvements, whereas Leadership and Situational Awareness generally require more substantial modifications.
- Counterfactual assessment: Realism (2.3 vs. 1.1), punctuality (2.1 vs. 1.2), and helpfulness (2.2 vs. 0.9) exceeded CoDy, while minimality was slightly lower (2.0 vs. 2.3).Ratings used a 5-point scale, with 4 as the highest score.
- Counterfactual assessment: Qualitative counterfactuals recommend prompt responses, fewer off-topic interactions, calmer behavior, and reduced aggression to improve predicted teamwork.
6 Conclusions
The paper introduces TE-ReNN for multimodal modeling of small surgical teams in low-data settings and combines it with exemplar-based counterfactual reasoning. The approach captures temporal and interpersonal dynamics while producing interpretable behavioral and relational suggestions for improving teamwork.
- TE-ReNN embeds temporal evolution into graph structure to jointly capture interpersonal interactions and their dynamics over time.
- The model delivers consistent improvements over static, temporal, relational, and tempo-relational baselines across all OTAS teamwork dimensions.
- An exemplar-based counterfactual approach identifies minimal, interpretable behavioral changes grounded in real observations at relational and paralinguistic levels.
- Qualitative examples translate counterfactuals into concrete recommendations for improving teamwork in high-stakes team settings.