PaperlayerView exact revision on arXiv

Source-linked AI summary

An Integrated Video-AI Platform for Action-Level Microanastomosis Training and Performance Feedback

Yan Meng, Daniel A. Donoho

arXiv:2609.06380v1cs.CV

TL;DR

Repeated microanastomosis practice needs timely, action-specific feedback, but expert review of lengthy videos is difficult to sustain across frequent or geographically distributed training. This platform links action segmentation, instrument analysis and supervised skill classification with a grounded language interface; post-processing raised segmentation accuracy and F1, while the five classifiers achieved 76.0% mean accuracy.

Abstract

from arXiv · show

Developing microanastomosis skill requires repeated practice with timely, action-specific feedback, yet expert review of lengthy microscope videos does not scale to frequent or distributed training. We present an integrated video-AI platform that turns a complete simulated procedure into inspectable, interactive feedback through three connected modules. First, a proposed transformer segments the video into six surgical actions. Second, object detection and tracking localize instrument tips within each action; the resulting kinematic features and action statistics drive supervised classification of five NOMAT-aligned performance dimensions. Third, a grounded large language model (LLM) uses these structured outputs to answer user questions about the current scene, actions, motion, and predicted performance through a unified interface. In a two-site study, 17 participants completed 72 procedures comprising 576 suture placements. The action-segmentation module achieved 87.66\% accuracy and 82.86\% F1, increasing to 93.62\% and 88.32\% after workflow-aware refinement. The five performance classifiers achieved 76.0\% mean accuracy, with Cohen's $\kappa$ from 0.63 to 0.93. Although the language interface and educational benefit require prospective evaluation, these results establish the technical basis for an expert-supervised platform that can shorten review, expose the evidence behind performance estimates, and support scalable formative microsurgical training.

Introduction

Microanastomosis training depends on repeated practice and timely feedback, but reviewing lengthy videos is difficult to scale; the paper proposes an integrated platform connecting action recognition, performance assessment, and grounded interaction.

Introduction

  • Expert review of lengthy recordings is difficult to sustain for repeated or geographically distributed practice, while procedure-level scores alone provide limited guidance.The introduction argues that useful feedback should identify recognizable actions and connect predictions to interpretable evidence.Source: Introduction · p. 1
  • The platform links action segmentation, action-conditioned instrument tracking and performance classification, and an LLM interface grounded in structured outputs.The LLM provides natural-language access to scene- and performance-related information without replacing the underlying measurements.Source: Introduction · p. 1, Figure 1 · p. 2
  • The integrated interface connects video upload, action segmentation, instrument analysis, supervised performance classification, and grounded natural-language interaction.The workflow is presented as one review interface rather than a set of independent analysis outputs.Source: Introduction · p. 1
  • The proposed contributions include HTA and VWSSA action segmentation, interpretable features for five NOMAT-aligned classifiers, and a two-site evaluation.The evaluation included 17 participants, 72 procedures, 576 stitch-level clips, and a preliminary workflow user study.Source: Introduction · p. 2

Method

The method connects a standardized simulated procedure to action segmentation, instrument tracking, performance classification, and an LLM interface, with intermediate outputs linked to video intervals and measurements.

Method

  • Explicit intermediate representations let the frame-level action timeline condition instrument and feature analysis, which then conditions performance classification and LLM responses.The shared context keeps responses traceable to a video interval or computed measurement.Source: Method · p. 2

Practice Kit and Application Workflow

  • Participants perform an end-to-side anastomosis by placing eight sutures in a standardized order, while a microscope camera records video without instrument-mounted sensors.The kit includes a stereomicroscope, camera, monitor, synthetic cards, and a uniform instrument set.Source: Practice Kit and Application Workflow · p. 2, Figure 2 · p. 3

Transformer-Based Action Segmentation

  • HTA combines global attention across the input with local temporal windows, while VWSSA emphasizes changing spatial regions and post-processing removes short segments and corrects implausible transitions.The model predicts seven labels, including background; its temporal branches target procedural context, brief motions, and action boundaries.Source: Transformer-Based Action Segmentation · p. 2

Instrument Detection and Tip Tracking

  • Instrument tracking combines detection and Deep SORT identity tracking with convex-hull tip candidates, descriptor matching, and mapped trajectories for motion features.The resulting measurements include velocity, acceleration, jerk, inter-instrument distance, relative speed, and angular displacement.Source: Instrument Detection and Tip Tracking · pp. 3–4

NOMAT-Aligned Skill Classification

  • A 32-dimensional motion vector and per-action duration, repetition count, and cumulative time feed separate Gradient Boosting classifiers for five NOMAT-aligned dimensions.Each classifier predicts Poor, Moderate, or Good performance.Source: NOMAT-Aligned Skill Classification · p. 4

Grounded LLM Interaction and User Interface

  • The LLM answers questions from structured action, instrument, motion, statistics, and provisional-classification context, but should abstain when information is absent or confidence is insufficient.It does not analyze raw pixels or generate the performance score; video or faculty review is the fallback.Source: Grounded LLM Interaction and User… · p. 4

Browser Acquisition and Runtime Scheduling

  • The browser sends frames to persistent workers, updates action output once per second, and renders separately from inference; that schedule is not a measured latency.Separate workers retain models in their original environments, and serialized calls prevent interleaved responses.Source: Browser Acquisition and Runtime… · p. 4

Detection and Measurement in the Prototype

  • The prototype uses a lightweight tip localizer and retains at most one representative output per class, limiting that association assumption to standardized tool arrangements.Dashboard speed and acceleration use box-centroid displacement in pixels, not calibrated tip trajectories; cross-session comparison requires calibration and processing, and image-plane acceleration or jerk does not measure tissue force.Source: Detection and Measurement in the… · p. 5

Experiment

The study collected procedures and expert ratings at two sites, then evaluated action segmentation, performance classification, and participant perceptions using distinct quantitative and descriptive approaches.

Participants and Data Collection

  • Seventeen participants at two sites completed repeated procedures, yielding 72 recordings averaging about 26 minutes and 576 stitch-level clips.Twenty procedures with more than 1,120 action segments were annotated; 15 were used for training, three for validation, and two for testing, while the model segmented the remaining 52 procedures.Source: Participants and Data Collection · p. 5

Training and Evaluation

  • Two board-certified neurosurgeons rated procedures with NOMAT, reconciled disagreements by consensus, and grouped five-point scores into Poor, Moderate, and Good categories.Segmentation evaluation used frame accuracy and action-level precision, recall, Jaccard index, and F1; classifier accuracy and Cohen’s κ were evaluated against consensus categories.Source: Participants and Data Collection · p. 5, Training and Evaluation · p. 5

User Study

  • The language interface had no independent benchmark of answer correctness, grounding, latency, or usability, and follow-up survey responses were descriptive rather than paired changes.The follow-up survey was offered to 11 participants who completed at least five trials; three responded, and baseline and follow-up responses were not linked.Source: Training and Evaluation · p. 5, User Study · p. 5

Results

The results show strong action-segmentation performance and five NOMAT-aligned classifier results, while also exposing limitations in supplementary detector validation and feature interpretation. Interface integration is demonstrated, but user feedback is descriptive and the language interface was not independently benchmarked.

Action Segmentation

  • Combining HTA and VWSSA increased accuracy by 17.95 percentage points and F1 by 20.55 points; vessel cutting, needle–vessel contact, and knot cutting had the strongest per-action F1.Needle handling and withdrawal were harder because they were brief and visually similar to adjacent actions; Figure 5 shows reduced fragmentation and improved temporal consistency after post-processing.Source: Action Segmentation · p. 5, Table 2 · p. 6, Table 3 · p. 6
  • On action segmentation, the proposed model achieved 87.66% accuracy and 82.86% F1, outperforming all comparison methods; post-processing increased these metrics to 93.62% and 88.32%, respectively.Source: Action Segmentation · p. 5, Table 1 · p. 6

Instrument Tracking and Skill Assessment

  • The classifiers achieved 76.0% mean accuracy, with Cohen’s κ from 0.63 to 0.93, across five NOMAT-aligned performance dimensions.Individual accuracies were 84.8% for overall instrument handling, 73.8% for needle-handling quality, 63.4% for knot-tying quality, 74.0% for needle-action efficiency, and 84.0% for knot-tying efficiency. Instrument detections and tip-localized trajectories provide kinematics, complemented by action-duration, repetition, and cumulative-time statistics.Source: Instrument Tracking and Skill Assessment · p. 7, Figure 6 · p. 7, Figure 7 · p. 7

Supplementary Detector Validation

  • Supplementary detector logs report precision of 96.3%, recall of 96.6%, mAP@0.5 of 97.8%, and mAP@0.5:0.95 of 92.9%.These are internal statistics from a separate image inventory, not held-out results for the 72-procedure cohort; the validation run lists 7,121 images versus 7,122 in the inventory. Needle AP@0.5:0.95 was 78.9%, compared with 94.6–98.3% for instrument categories, and high detection AP alone does not validate identities, tip localization, or downstream assessment.Source: Supplementary Detector Validation · p. 8

Integrated Interface Demonstration

  • The prototype synchronizes outputs from all three modules in one workspace, linking selected video intervals to actions, tracked instruments, motion traces, statistics, and provisional performance information.The LLM can answer from this shared state, but response correctness and learning benefit require dedicated evaluation.Source: Integrated Interface Demonstration · p. 8

Interpreting the Feature Visualizations

  • Motion peaks mark intervals for inspection, but their meaning depends on the action; fragmented segmentation can inflate counts or misassign trajectories, while smoothing can erase brief actions.The plots are descriptive and do not establish that an individual motion measure causes or indicates surgical competence.Source: Interpreting the Feature Visualizations · pp. 8–9

User Study Results

  • Among three follow-up respondents, eight workflow items averaged 4.33/5; structured video feedback averaged 4.00 and time-burden acceptability 3.33.Confidence scores increased at group level, but these were descriptive trends rather than paired individual changes.Source: User Study Results · p. 9, Figure 9 · p. 9

Discussion

The platform links action segmentation, instrument-motion analysis, performance classification, and grounded language interaction so users can inspect the evidence behind assessments. Its results remain bounded by small, controlled evaluations, and prospective studies are needed to assess broader generalization and educational value.

Discussion

  • The integrated platform connects action timelines, tracked tool motion, performance classifications, and grounded LLM interaction, linking assessments to measurable evidence and source video.Action boundaries determine which trajectories, motion features, and temporal statistics feed downstream classifiers.Source: Discussion · pp. 9–10
  • Generalization remains unestablished beyond a small dataset, two manually annotated segmentation-test procedures, and controlled simulation recordings.The authors call for larger participant-independent external evaluations across institutions, equipment, participant backgrounds, and handedness.Source: Discussion · p. 10
  • Predictions should remain provisional rather than support credentialing or punitive monitoring without separate validation, governance, and consent.Prospective studies should compare AI-assisted review with conventional feedback and measure review time, feedback timeliness, retention, and transfer.Source: Discussion · p. 10

Conclusion

The paper presents an integrated video-AI platform for microanastomosis training and evaluates it across two sites. Refined segmentation and performance-classification results establish a technical basis, while language grounding, generalization, interaction, and educational benefit still need evaluation.

Conclusion

  • 93.62% segmentation accuracy after refinement and 76.0% mean accuracy across performance classifiers were achieved in the two-site evaluation.The study included 17 participants, 72 procedures, and 576 suture placements.Source: Conclusion · p. 10
  • The platform combines task-relevant video intervals, action-conditioned instrument kinematics, five NOMAT-aligned performance estimates, and grounded LLM interaction.The modules present synchronized analysis through a unified interface.Source: Conclusion · p. 10
  • Language grounding, cross-site generalization, user interaction, and educational benefit remain priorities for dedicated evaluation.The reported results establish the system's technical basis, not those further outcomes.Source: Conclusion · p. 10
Loading 2609.06380v1…