SurgStreamer: proactive, request-driven monitoring of surgical video streams with structured state and a reusable reasoning layer

1. Introduction

• Surgical teams need requested-event alerts and retrospective answers; events are dataset-visible occurrences, not assessments of action correctness or complication prevention.Source: §1 · p. 1
• Frame- and segment-recognition methods do not answer user requests or measure when an event is signalled, while tested streaming VLMs showed several alerting failures.The tested models sometimes alerted on mere visibility or absent objects, and rarely detected disappearances or phase changes.Source: §1 · p. 1
• SurgStreamer grounds monitoring in a probabilistic state over tools, actions and phases, then evaluates formal conditions over the state stream and event memory with causal alerts.Source: §1 · pp. 1–2
• The paper tests exact event semantics, name-free pointer control, and transfer boundaries; its claims include superiority to tested streaming VLMs and transfer limits for perception and thresholds.Perception is procedure-specific, and the introduction says the paper reports where its transfer claim fails.Source: §1 · p. 2
• The perception stack uses established techniques, while pre-registration and sealed test sets make the three claims testable.Source: §1 · p. 2

Surgical workflow and instrument recognition

• Surgical workflow recognition progressed from recurrent models to temporal convolutions and transformers, while fine-grained actions are often modelled as instrument–verb–target triplets.Multicentre and procedure-specific benchmarks document performance drops across centres and procedures.Source: §2 · p. 2

Online and event-level video understanding

• Online action detection and spotting assess causal predictions and event timing; this paper adds request-specific modes, phase restrictions, ordering, confirmation, and evidence accumulation.Source: §2 · p. 3

Streaming and surgical vision–language models

• Streaming VLMs respond as video unfolds, but surgical VLM work is mostly retrospective; SurgOnAir narrates workflow rather than answering requests or being evaluated on transition timing.Source: §2 · p. 3

Time-aware evaluation

• MedStreamBench scores first-alert responsiveness and post-alert stability, whereas this paper scores one alert edge per requested occurrence and uses hard negatives to test silence.Source: §2 · p. 3

Structured reasoning and runtime monitoring

• SurgStreamer combines programs over perception outputs with runtime verification, adding a learned controller to approximate exact semantics under perception noise.Source: §2 · p. 3

3.1. Problem formulation

• The system emits a causal response each second by factoring monitoring into procedure-specific perception, request compilation, and procedure-agnostic reasoning over state history.Reasoning receives the condition and referenced entity states, not pixels or entity names.Source: §3.1 · p. 3

3.2. Entities, state and events

• The state represents tools, actions, and phases; episodes bridge absences of at most 2 seconds, and candidate onsets require 2-second confirmation.Offsets are confirmed after an absence lasts G + 1 seconds, while phase entry is treated as an onset with G = 0; these decisions use only evidence available by confirmation.Source: §3.2 · pp. 3–4

3.3. Monitoring conditions

• Monitoring conditions specify an operator, entities, occurrence mode, and optional phase restriction; alerts occur only at confirmation for admissible selected events.Pending and confirmation labels are generated offline for controller training, but evaluation uses alert times; prefix checks on 2,000 truncations found no alert differences from complete streams.Source: §3.3 · p. 4, Table 2 · p. 4

3.4. Event memory

• The event memory is a causal append-only log of episode starts, confirmed offsets, and phase transitions that supports retrospective queries.It stores typed, time-stamped events rather than answers, allowing one log to serve retrospective requests about vocabulary entities.Source: §3.4 · p. 5

3.5. Request compiler (interface)

• The natural-language compiler maps requests to formal conditions, while the main evaluation uses gold conditions and therefore does not evaluate the compiler.Ontology-based lexical grounding marks unsupported entities so they never alert, avoiding mapping absent objects onto similar real instruments.Source: §3.5 · p. 5

3.6. Perception

• Perception encodes frames with DINOv3, fine-tunes visual blocks for tools, phases, and actions, and uses causal temporal heads to estimate presence and phase.Training uses label-source losses only where each source is available and out-of-fold encoders so downstream components encounter realistic noise.Source: §3.6 · p. 5

3.7. Episode and phase decoding

• Episode flags use causal per-class hysteresis, while an ordered hidden Markov model decodes phases with earlier-phase transitions down-weighted.Decoder parameters are selected on out-of-fold predictions from training procedures; phase changes require the leading filtered probability to exceed a threshold.Source: §3.7 · p. 5

3.8. Controllers

• The learned controller reads pointer-addressed entity states and temporal features, then predicts alerts; a frozen per-family hybrid can select among state and controller alternatives.Its training targets every base-event occurrence and uses out-of-fold predicted state with 30% ground-truth state; inference thresholds alert probabilities and structured decoding applies once, every, or n-th modes.Source: §3.8 · pp. 5–6

4.1. Data and splits

• The sealed test set contains 12 procedures; training, validation, and development-test splits are separate, and dataset overlaps are respected.The source data combine Cholec80 and CholecT50; their instrument-presence annotations are not mixed within a query.Source: §4.1 · p. 6

4.2. Benchmark

• The benchmark covers 20 query families and ≈20,000 full-procedure query instances, with 75-s windows used for VLM comparisons.Window evaluation includes positive and negative examples, including persistent-presence negatives that require silence.Source: §4.2 · p. 6

4.3. Baselines

• OneStreamer-4B baselines use 1-FPS video and a 16-update sliding context; tested variants include video, SurgStreamer state, and video with an adapted vision encoder.The VLM conclusions are limited to this configuration; fine-tuned baselines use the same 43 training procedures.Source: §4.3 · p. 6

4.4. Protocols and pre-registration

• Equal-context windows reset SurgStreamer’s temporal state at each start, while its state accumulates across the 75 s; the protocol timeline marks equal-context and V-VA comparisons as post hoc.Protocols were timestamped in a private repository before corresponding scoring; they were not registered publicly.Source: §4.4 · pp. 6–7, Table 4 · p. 7

4.5. Metrics and statistics

• Occurrence alerts are matched within a tolerance, while negatives use false-positive rate and retrospective requests use accuracy measures; uncertainty uses paired procedure-level bootstrap intervals.The bootstrap uses 1,000 resamples, and p-values are Holm-adjusted within comparison families.Source: §4.5 · p. 7

5.1. Sealed final test

• 0.598 macro event-F1 [0.550, 0.651] was reached by the hybrid with gold conditions on 12 sealed procedures.It exceeded C-rule by +0.028 on the 1024-s state and +0.036 on the 256-s state; C-learn’s nominal 0.607 was not significantly different from the hybrid.Source: §5.1 · p. 7, Table 5 · p. 8
• 0.84 tool-onset F1, 0.66 phase-entry F1, and 0.45 action-onset F1 accompanied event-memory balanced accuracy of at least 0.91 for has/order questions.Counts were the hardest retrospective task: exact tool-count accuracy was 0.60, and counts were within one episode in 85–86% of cases.Source: §5.1 · p. 8

5.2. Comparison with streaming vision–language models

• V-VA’s visual adaptation improved video-only performance by 0.096 over V-LoRA but left tool-offset F1 at 0.27 versus 0.69 and phase-entry F1 at 0.00 versus 0.65.SurgStreamer’s single-request updates took about 10 ms and 0.64 GB, versus 0.34–0.74 s and approximately 9 GB for tested OneStreamer configurations.Source: §5.2 · p. 8

5.5. Failure modes

• A false scissors-presence detection produced a decoded disappearance and an alert 42 s before the reference event, suppressing the real disappearance in once mode.This illustrates that an error in the decoded state can propagate to an incorrect alert.Source: §5.5 · p. 11, Figure 9 · p. 12

5.2. Comparison with streaming vision–language models

• 0.677 macro event-F1 [0.577, 0.759] on identical 75-s windows exceeded every tested OneStreamer configuration, with Holm-adjusted p ≤0.012 for all six baselines.The margin over C-LLM, the strongest baseline, was +0.137; SurgStreamer scored highest at every matching tolerance from 2 to 20 s.Source: §5.2 · p. 8, Table 9 · p. 13

5.3. External centres

• 0.470 performance on held-out HeiChole procedures followed perception-only retraining, up from 0.239, with controllers, thresholds, and hybrid choices unchanged.Without retraining, the full-procedure score on all 33 procedures was 0.231 [0.202, 0.258]; the cited adaptation gain was +0.231 [+0.190, +0.277].Source: §5.3 · p. 8, Table 10 · p. 13

5.4. A new procedure: controller transfer to PitVis

• 0.947 on oracle state for the transferred cholecystectomy controller was close to the in-domain controller’s 0.938, but predicted-state transfer missed the non-inferiority margin by −0.024 [−0.040, −0.008].With frozen controller weights and recalibrated alert thresholds, the transferred controller scored 0.397, the in-domain controller’s point estimate.Source: §5.4 · p. 11, Table 11 · p. 13

5.6. Supporting analyses

• 0.566 versus 0.546 cross-validation event-F1 favored C-learn over exact rules, with gains on temporal and relational operators; the posterior-reading P-rule scored 0.419.The compiler reproduced 91.5% of development requests exactly; compiled conditions lowered development macro event-F1 by 0.016, and the compiled system was not evaluated on the final test.Source: §5.6 · p. 13

Why does state grounding help?

• Given oracle state, exact rules answer every benchmark request correctly by construction, while the learned controller nearly does so; on predicted state, perception and event timing appear to be the main limits.The proposed explanation that explicit run lengths and times help the controller was not isolated experimentally.Source: §6 · p. 13

What transfers?

• On a new procedure, cholecystectomy-trained controller weights matched an in-domain controller on oracle state, but predicted-state transfer missed the margin unless alert thresholds were recalibrated.Rule semantics transfer by construction, while thresholds remain procedure-specific.Source: §6 · pp. 13–14

What limits performance on real input?

• At external centres, perception diagnostics implicate instrument appearance, episode fragmentation, and annotation conventions; a correctly evaluated condition can still produce a wrong alert from an incorrect decoded state.On HeiChole, equal-context performance was nominally higher than full-history performance (0.363 vs 0.324, n.s.); the proposed error-accumulation explanation was not tested.Source: §6 · p. 14

Limitations

• The transfer evidence is limited to one new procedure and one external dataset, with test sets of 7 and 9 procedures; requests came from templates, and the main results used gold conditions.No surgeons wrote requests or rated alerts, and no prospective evaluation was performed, so clinical utility remains to be shown.Source: §6 · p. 14

Future work

• Future work includes surgeon-written free-text requests, broader baselines, episode-parameter sensitivity analysis, and multi-procedure pretraining or label-free perception adaptation.The paper also proposes failure attribution across the compiler, perception, decoding, and control stages.Source: §6 · p. 14

7. Conclusion

• SurgStreamer evaluates compiled monitoring conditions over structured state and event memory, using exact rules or learned reasoning.The system answers natural-language monitoring requests over surgical video streams.Source: §7 · p. 14
• On a new procedure, controller weights trained on cholecystectomy matched an in-domain controller on oracle state, but predicted-state transfer matched only after alert-threshold recalibration.Perception required retraining for new centres and procedures.Source: §7 · p. 14
• The results support state-grounded modular designs for proactive surgical monitoring and point to perception under distribution shift as the main remaining obstacle.Source: §7 · p. 14

CRediT authorship contribution statement

• Bowen Xiang is credited with conceptualization, methodology, software, validation, formal analysis, investigation, and data curation.Source: CRediT authorship contribution statement · p. 14
• Bowen Xiang prepared the original draft and contributed to review, editing, and visualization.Source: CRediT authorship contribution statement · p. 14
• Jon S. Heiselman and Michael I. Miga contributed to writing review and editing, and to visualization.Source: CRediT authorship contribution statement · p. 14

Ethics statement

• The study used publicly available, de-identified videos from Cholec80, CholecT50, HeiChole, and PitVis under their respective licences.Source: Ethics statement · p. 14
• No new data were collected.Source: Ethics statement · p. 14
• The graphical abstract and Figs. 1, 2, 3, 6, 8, and 9 reproduce de-identified endoscopic frames from Cholec80 and PitVis with attribution.Source: Ethics statement · p. 14

• The research received no specific grant from funding agencies in the public, commercial, or not-for-profit sectors.Source: Funding · p. 14

• Fine-tuning raised the perception score from 0.39 to 0.47 and tool-onset F1 from 0.54 to 0.70; the four-head ensemble reached 0.54.The ordered phase HMM raised online phase-transition F1 from 0.29 to 0.40.Source: Appendix A · p. 17, Table A.1 · p. 17
• Cross-annotation episode-onset F1 at ±2 s ranged from 0.55 to 0.89 across instruments, providing a reference for timing disagreement between annotation conventions.The authors state that this is not an upper bound.Source: Appendix A · p. 17, Table A.2 · p. 17
• The compiler exactly reproduced 91.5% of development requests and mapped none of ten out-of-vocabulary requests to a real instrument; a plain prompt mapped eight.These are development estimates because the compiler was revised after its errors were inspected.Source: Appendix A · p. 17, Table A.3 · p. 17
• SurgStreamer scored highest at every matching tolerance from 2 to 20 s in the equal-context window comparison, while baseline ordering changed.Source: Appendix A · p. 17, Table A.4 · p. 18

Appendix B. Few-shot adaptation (exploratory)

• The exploratory design trained a light temporal head on a frozen Cholec80-tuned backbone using k = 1, 3, 5, or 10 target procedures, with C-learn weights unchanged.Five random draws were used per k; test procedures had been inspected in earlier transfer studies.Source: Appendix B · p. 18, Table B.1 · p. 18
• With ten annotated videos, few-shot performance remained below full adaptation on PitVis (0.304 vs 0.372) and HeiChole (0.327 vs 0.449).Performance increased with more annotated videos on both targets, but full adaptation also used more videos, so the comparison mixes video count and adaptation method.Source: Appendix B · p. 18, Table B.1 · p. 18
• The frozen learned controller exceeded exact rules at every k on both targets, while neither proposed add-on met its adoption rule.Adapter calibration lowered false alerts by 35–40% without raising macro event-F1; training a controller on simulated perception errors lost 0.035 on Cholec80 and did not improve transfer.Source: Appendix B · p. 19

Appendix D. Implementation details

• Implementation details specify encoder training with AdamW, 2 · 10^-5 backbone and 10^-3 head learning rates, and 6,000 steps.Temporal heads used chunked training; C-learn used 600-s windows, and the LoRA baselines used distinct training settings.Source: Appendix D · p. 19