PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Masks and Trajectories: Flow-Guided Latent Action Injection for Stable Surgical Video Generation

Tsz-Yui Qin, Siyu Zhou, Chi-Keung Tang, Yuxiang Nie, Shu Yang

arXiv:2610.09800v1cs.CV

TL;DR

Surgical video generation methods often rely on costly auxiliary conditions, while general video metrics do not directly assess surgical plausibility. FLAIR learns optical-flow motion priors and predicts them from text for reference-free generation, with reported improvements in motion stability, action consistency, and surgical-domain plausibility over baselines.

  • Problem

    Many surgical video generation methods rely on costly inference-time auxiliary conditions, and general-purpose metrics do not assess surgical-specific errors.

    Source: §1 · p. 2, §2.3 · p. 3

  • Method

    FLAIR learns flow-derived latent action priors from real surgical videos, predicts them from prompts, and injects them into a frozen diffusion model for text-only inference.

    Source: §1 · p. 2, §4.4 · p. 7

  • Results

    FLAIR consistently improves motion stability, action consistency, and surgical-domain plausibility over existing baselines in the reported experiments.

    Source: §7 · p. 11

  • Takeaways & Limitations

    SurgActionClip-30K and SurgMetrics provide data and evaluation resources alongside FLAIR for action-aware surgical video generation.

    Source: §7 · p. 11

    FLAIR predicts a plausible text-conditioned motion prior rather than enforcing a single user-specified action trajectory.

    Source: F Discussion on Limitations and… · p. 37

Abstract

from arXiv · show

Surgical video generation holds substantial potential for surgical education, simulation, and data augmentation, yet generating surgical videos with realistic and clinically plausible motion remains challenging. Most existing methods rely on auxiliary conditions, such as masks, trajectories, depth, or reference videos, to achieve visually plausible synthesis. Yet, these auxiliary conditions typically require additional manual annotation or specialized acquisition, making it difficult to scale such methods beyond small, curated datasets. This motivates the need for a reference-free architecture capable of generating high-quality surgical video without requiring auxiliary visual conditions at inference time. We propose FLAIR, a Flow-guided LatentAction Injection framework for Reference-free surgical video generation. FLAIR learns action priors from optical flow of real surgical videos, dynamically predicts corresponding latent action representation from an input prompt, and injects it into a frozen base model to generate surgical videos with improved action consistency. We further construct SurgActionClip-30K, the first large-scale surgical vision dataset comprising action-centric segmented clips and structured caption labels, addressing the persistent lack of fine-grained, action-centric surgical datasets. Lastly, we introduce SurgMetrics, the first surgical domain-specific evaluation metrics for quantifying the quality of generated surgical videos, addressing the persistent absence of clinically grounded evaluation standards in this domain. Extensive experiments demonstrate that FLAIR enables generating high-quality surgical videos using text-only inference without auxiliary conditions, and validation in SurgMetrics demonstrates its strength in alignment with human perception compared to traditional metrics.

1 Introduction

FLAIR targets text-only surgical video generation because existing controllability often depends on costly inference-time conditions. It learns motion priors from optical flow and predicts them from prompts, with reported gains in motion stability and action consistency over domain-adapted baselines.

1 Introduction

  • Surgical actions involve procedure-constrained instrument–tissue interactions where small motion deviations can alter clinical meaning, but existing recordings are mostly long and untrimmed.The paper situates surgical video use in education, skill assessment, and downstream surgical AI.Source: §1 · p. 1
  • Existing methods often rely on costly inference-time reference frames, masks, depth maps, or tool trajectories, motivating reference-free, text-only motion guidance.Such conditions can be costly to obtain and unavailable in low-resource educational or clinical settings.Source: §1 · p. 2
  • FLAIR learns optical-flow-derived action priors, predicts them from surgical prompts, and injects them into a frozen diffusion model to improve motion stability and action consistency over domain-adapted baselines.At inference, it requires no masks, trajectories, depth maps, or reference videos.Source: §1 · p. 2
  • SurgActionClip-30K provides action-centric segmented clips and structured captions, while SurgMetrics adds surgical-specific artifact scores and a domain-typicality measure.The paper presents both as contributions alongside FLAIR.Source: §1 · p. 2

2 Related Work

The related work identifies three gaps: surgical datasets lack precise action-boundary clips, generation methods commonly use auxiliary inference conditions, and general video metrics miss surgical-specific errors. The paper positions its dataset, reference-free generation approach, and evaluator as responses to these gaps.

2.1 Surgical Vision Datasets

  • Existing surgical datasets include long untrimmed recordings and shorter image-text or clip-caption collections, but none provide the precision and specificity of discrete action boundaries needed for fine-grained action-centric training.SurgActionClip-30K is presented as a unified collection of action-centric clips with standardized descriptions.Source: §2.1 · pp. 2–3

2.2 Surgical Video Generation Models

  • Surgical video generation methods use reference frames, RGB-D and flow cues, segmentation maps, or affordance masks and trajectories, while FLAIR uses optical-flow features as training-time supervision for text-only inference [3] [5] [21] [23] [29].The cited methods include SurgSora [5], HieraSurg [3], and SAW [29].Source: §2.2 · p. 3

2.3 Evaluation Metrics for Surgical Videos

  • General-purpose video metrics measure visual similarity or semantic correspondence but miss surgical-specific errors such as implausible instrument–tissue interactions, deformation, and domain shift.SurgMetrics is described as a multi-axis, scenario-specific evaluator with six dedicated quality dimensions.Source: §2.3 · p. 3

3 Dataset Preparation

SurgActionClip-30K combines action clips from diverse surgical sources and pairs them with structured captions. Its preparation uses procedural annotations and instrument presence to establish clip boundaries and verify detailed descriptions.

3.1 Cross-domain data sources

  • SurgActionClip-30K consolidates 11 datasets across laparoscopic, endoscopic, and microsurgical domains, covering diverse procedures and actions guided by the BSA-10 taxonomy [44].Representative actions include dissection, coagulation, clipping, retraction, and aspiration.Source: §3.1 · p. 3
  • The preparation process converts sources into a unified clip-metadata-caption schema.The passage points to Appendix B.1 for source-level details.Source: §3.1 · p. 4

3.2 Action-centric clip extraction and captioning

  • Clips are segmented or merged using procedural labels, filtered to remove non-operative or uninformative segments, and split on changes in visible tool configurations when instrument annotations are absent.Each clip receives an instrument-action-target caption enriched with procedure, anatomy, tissue response, and imaging context; initial descriptions are generated with Hulu-Med-4B and checked against metadata.Source: §3.2 · p. 4

4 Methods

FLAIR extracts motion tokens from real surgical optical flow, converts them into global and temporal conditions for a frozen video diffusion Transformer, and predicts the motion prior from text for inference. The predictor output is treated as a text-conditioned prior rather than an exact trajectory.

4.1 Framework Overview

  • At inference, FLAIR removes the Flow Action Encoder and uses a prompt-predicted motion prior alongside the backbone’s native text condition.During training, the encoder-derived latent sequences supervise text-to-action prediction and adapter optimization.Source: §4.1 · p. 4

4.2 Flow Action Modeling

  • The Flow Action Encoder maps adjacent-frame optical flow into ordered latent action tokens, trained with reconstruction and anti-collapse objectives to preserve motion information without RGB appearance cues.The encoder is trained on SurgActionClip-30K; its reconstruction, instance-discrimination, and variance-floor objectives support motion-preserving representations.Source: §4.2 · p. 5

4.3 Action Adapter and Optimization

  • The adapter pools flow-derived tokens and produces clip-level and temporal conditions, which are resized as needed and injected additively into the frozen generator’s diffusion-time and patch embeddings.Its optimization uses the generator’s native flow-matching objective and a ranking term comparing matched and mismatched motion sequences.Source: §4.3 · pp. 5–6

4.4 Text-to-Action Prediction

  • A text-to-action predictor uses a frozen Long-CLIP prompt feature and learnable queries to predict latent tokens; its output is a text-conditioned motion prior, not an exact trajectory.The predicted sequence replaces the FAE sequence as adapter input at inference while the diffusion backbone retains its own native text encoder.Source: §4.4 · pp. 6–7

5 SurgMetrics: Surgical Video Evaluation

SurgMetrics combines surgical-video features with motion statistics to score six artifact types and separately measures domain typicality. Its axis-specific supervision uses controlled degradations, while a distinct distance score captures deviation from clean real laparoscopic videos.

5.1 Lightweight feature fusion and multi-axis scoring

  • SurgMetrics fuses SurgViSTA surgical features with optical-flow motion statistics to predict six artifact scores, with higher scores indicating stronger evidence of each artifact.The six axes cover semantic implausibility, appearance artifacts, temporal or global-motion drift, instrument deformation, tissue deformation, and local tracklet instability.Source: §5.1 · p. 7, §5 · p. 7
  • LoRA-FLAIR surpasses LoRA on both general video metrics and SurgMetrics across the reported comparisons.The table caption notes that dashes indicate data that are not comparable or unavailable.Source: Table 1 · p. 8

5.2 Axis-specific artifact supervision

  • Training uses 21 controlled surgical degradation families, combines their axis contributions with a noisy-OR target, and applies axis-balanced Smooth L1 training with a clean-to-multiple-artifact curriculum.Clean clips receive zero-valued targets on all axes.Source: §5.2 · p. 8

5.3 Surgical domain distance

  • Under identical settings, Wan2.1 LoRA-FLAIR shows a more coherent motion trajectory and more sustained instrument–tissue interaction than Wan2.1 LoRA across seven synchronized frames.The comparison spans frame 0 to frame 80.Source: Figure 3 · p. 9
  • A separate domain-distance score uses PCA-projected SurgViSTA features and a shrinkage covariance model fitted on clean real training videos; values near zero indicate consistency with that reference distribution.This distinguishes domain deviation from artifact severity, which alone does not determine whether a clip belongs to the target domain.Source: §5.3 · pp. 8–9

6 Experiments

Across the reported comparisons, FLAIR retains competitive general-metric performance while improving action consistency and reducing evidence of several surgical artifacts. Human ratings align with SurgMetrics, and a next-frame probe indicates that FAE latents retain transition-specific motion information.

6.1 Quantitative and Qualitative Results

  • The shared evaluation uses 200 prompts spanning ten basic actions across eight procedures, comparing three backbones with base-model-only and converged LoRA baselines.The last 20 prompts extend evaluation to VATS lobectomy and hepatectomy for external generalization; some related methods lack comparable or available outputs for all metrics.Source: Inference Prompt Set · p. 9
  • FLAIR maintains competitive visual quality and prompt-specified execution with higher action consistency, while showing lower evidence across most SurgMetrics artifact axes than conventional domain-fine-tuned baselines.The strongest improvements are reported for drift, instrument deformation, and tracklet instability.Source: Inference Prompt Set · pp. 9–10
  • In the qualitative comparison, LoRA-only develops inconsistent instrument motion and structural artifacts, whereas Wan2.1 LoRA-FLAIR remains stable with sustained instrument–tissue interaction.The comparison uses identical generation settings.Source: Inference Prompt Set · p. 10

6.2 Human-Alignment Validation for SurgMetrics

  • 0.746–0.867 Spearman correlations link SurgMetrics artifact axes with human ratings, while FLAIR receives lower interaction-error ratings than LoRA across all three backbones.Validation is restricted to laparoscopic videos; after adjusting for model identity, the overall-quality correlation remains 0.731, down from 0.845.Source: §6.2 · p. 10

6.3 Ablation Study on FAE Motion Representation

  • More than 6.55% lower next-frame MSE is reported for FLAIR’s FAE latent than for no-latent, zero-latent, and shuffled-latent controls.The result supports that the latent preserves transition-specific motion information usable by an independently trained predictor.Source: §6.3 · p. 11

7 Conclusion

The paper presents FLAIR for reference-free surgical video generation, alongside SurgActionClip-30K and SurgMetrics as resources spanning modeling, data, and evaluation.

7 Conclusion

  • Table 2 reports next-frame prediction error across FAE latent conditions, with percentages relative to the no-latent baseline.The supplied passage gives the table’s scope but no cell values.Source: Table 2 · p. 11
  • FLAIR injects optical-flow-derived motion priors into a frozen video diffusion model for reference-free generation with improved motion stability and action consistency.Inference requires no auxiliary conditions.Source: §7 · p. 11
  • SurgActionClip-30K provides action-centric surgical clips to support scalable, action-aware video generation.The paper describes it as part of an integrated foundation spanning modeling, data, and evaluation.Source: §7 · p. 11
  • SurgMetrics provides domain-specific evaluation metrics for surgical video generation.The paper presents it alongside FLAIR and SurgActionClip-30K as part of an integrated research foundation.Source: §7 · p. 11

AI Use Statement

Generative AI was used to draft structured captions during dataset construction and to assist with manuscript language editing; the authors verified the scientific work.

AI Use Statement

  • Hulu-Med-4B generated initial structured captions for surgical clips using procedural and annotation metadata.The captions were subsequently constrained and checked for consistency with clip metadata.Source: AI Use Statement · p. 11
  • A large language model assisted with language editing and improving manuscript clarity.Source: AI Use Statement · p. 11
  • The authors carried out and verified the methodological design, experiments, data processing, analysis, interpretation, and scientific conclusions.Source: AI Use Statement · p. 11

Ethics Statement

This section describes FLAIR’s flow-based design and supporting dataset, evaluation, training, and ethical safeguards. It also outlines how surgical prompts map to flow-derived latent actions that are optimized through an adapter while the video diffusion backbone remains frozen.

Ethics Statement

  • The work uses anonymized public research videos without personally identifiable patient information; participating surgeons gave informed consent and received no compensation, and generated videos are restricted to research and education.Hulu-Med-4B ran locally, with no surgical video transmitted to external services; release of clips and metadata is subject to source licenses.Source: Ethics Statement · p. 11

A.1 Why the Surgical Domain Differs Fundamentally from General Video Domains

  • Surgical actions can change clinical meaning through tiny motion deviations, while mostly static backgrounds make optical flow useful for isolating instrument–tissue motion [27] [33].These domain properties motivate learning motion priors from optical flow rather than coarse action labels or trajectory sketches [12] [18].Source: §A.1 · p. 17

A.2 Why FLAIR Helps

  • Surgical motion occupies a constrained space, and FLAIR maps prompts into compact action-conditioned patterns learned from real clips to improve stability and action consistency.The flow action encoder learns these patterns, while the text-to-action predictor maps prompts into the same latent region rather than predicting frame dynamics from scratch.Source: §A.2 · p. 18

A.3 Why SurgMetrics is Helpful

  • SurgMetrics models recurring surgical-video failures with 21 degradation families and a multi-axis evaluator; its synthesized-artifact scores correlate strongly with human judgments [5] [6] [7] [21] [48].The approach is intended as a scalable alternative to general-purpose metrics [11] [13] and expert review of every comparison.Source: §A.3 · p. 18

B SurgActionClip-30K Dataset Construction

  • SurgActionClip-30K combines 11 source datasets into action-centric clips with structured captions, applying temporal and instrument-aware refinement, quality checks, and patient- or video-level split guidance.The caption template covers context, instruments, action, anatomy, tissue response, and imaging conditions; instrument mentions are constrained by available metadata or visible evidence.Source: Figure 4 · p. 19, B SurgActionClip-30K Dataset… · p. 19, §B.3 · pp. 20–21

C.1 Inference Prompt Set

  • The released 200-prompt benchmark includes a 6-procedure × 10-action × 3-variant grid of 180 prompts and 20 prompts for two external-validation procedures.Prompts use a fixed subject phrase and explicitly name the Basic Surgical Action; the full prompt list is released for reproducibility.Source: §C.1 · p. 22

C.2.1 Flow Action Encoder Objective

  • The flow action encoder objective combines flow reconstruction, instance discrimination, and a variance-floor regularizer to retain motion information and avoid latent collapse.The variance floor sets a minimum target standard deviation of γ = 0.07 and penalizes dimensions only when their batch variation falls below that floor.Source: §C.2.1 · p. 23

C.2 Detailed Complete Training Objectives

  • The text-to-action predictor aligns 80 normalized predicted tokens with frozen flow-derived targets, while the adapter learns through a frozen generator using real action sequences during training and text-predicted sequences at inference.The adapter objective compares matched and mismatched action sequences while regularizing temporal smoothness and injection magnitudes; the initial training stage learns a compact motion representation from real clips.Source: §C.2.2 · p. 24, §C.2.3 · pp. 24–25, §C.3 · p. 25

C.3 Detailed FLAIR Training Setup and Procedure

  • Stage II trains a text-to-action predictor against fixed latent action sequences; Stage III updates only the action adapter, using flow-derived actions in training and text-predicted actions at inference.During Stage III, the video diffusion backbone and surgical-domain LoRA remain unchanged, while gradients pass through denoising to update the adapter.Source: §C.3 · p. 25

C.4 More Qualitative Results

Additional qualitative results show FLAIR improving motion stability across video backbones. SurgMetrics is built from controlled artifact degradations and validated against external data and human judgments, while its domain-distance branch distinguishes degrees of surgical relatedness.

C.4 More Qualitative Results

  • Across HunyuanVideo-1.5 and CogVideoX-2B, LoRA-FLAIR keeps instruments visible and physically grounded as LoRA-only generations drift, detach, or become unstable.The examples extend the Wan-based comparison to two additional backbones.Source: §C.4 · pp. 25–26, Figure 6 · p. 26, Figure 5 · p. 26

D Additional Details on SurgMetrics

  • SurgMetrics validation combines controlled-artifact sensitivity, generalization beyond training data, and alignment with human perception.The validation uses clips from five surgical datasets and separates procedures across training and evaluation splits.Source: D Additional Details on SurgMetrics · p. 26, §D.2 · p. 27

D.1 Twenty-one Controlled Degradation Families

  • SurgMetrics trains six artifact heads using severity-routed labels from 21 controlled degradation families, avoiding expert annotation of generated videos.Drift supervision targets erratic or discontinuous speed changes rather than penalizing any absolute motion speed.Source: §D.1 · p. 27, Figure 7 · p. 28

D.2 Validation of SurgMetrics

  • On unseen cases, SurgMetrics distinguishes clean from multi-artifact clips and generalizes to external datasets, though unseen combinations remain more difficult.Low variation across seeds supports the reported consistency.Source: §D.2 · pp. 27–28

D.2.1 Human-Alignment Validation

  • Across six artifact axes, SurgMetrics–human Spearman correlations range from 0.746 to 0.867; FLAIR also receives lower interaction-error ratings than LoRA across all three backbones.The human ratings cover artifact dimensions, overall quality, and prompt-specific instrument–action–target correctness.Source: §D.2.1 · pp. 28–29

D.3 Domain Distance Reference Values

  • Domain-distance robust z-scores rise from 1.85 and 1.96 for PhaKIR and SLAM to 3.60 for OphNet and 4.99 and 5.82 for SurgVU24 and UCF101.The scores measure distance from the training distribution, not artifact severity; domain typicality and artifact scores are independent signals.Source: §D.3 · pp. 29–30, Table 6 · p. 30

D.4 Ablation Study on SurgMetrics

The SurgMetrics ablations show that combining visual and optical-flow inputs performs best overall, with flow features contributing temporal-motion evidence. The accompanying setup details describe FLAIR’s backbones, action representations, predictor, and adapters.

D.4 Ablation Study on SurgMetrics

  • Adding optical flow to SurgViSTA yields consistent aggregate-metric gains, including +0.1373 Spearman on the external single-family cohort; adding SurgViSTA to Optical-only yields +0.3209 held-out multi Spearman and +0.2768 held-out AUC.Full achieves the best score on every aggregate metric and adds 9,024 parameters over SurgViSTA-only.Source: §D.4 · p. 30

E.1 Detailed LoRA training Setup

  • The study adapts Wan2.1-T2V-1.3B, HunyuanVideo-1.5, and CogVideoX-2B with LoRA while keeping each pretrained video backbone, VAE, and text encoder frozen.LoRA rank is 32 and scaling parameter α is 1 for all three models.Source: §E.1 · pp. 31–32

E.2 Detailed FAE Training Setup

  • The Flow Action Encoder maps optical-flow inputs to normalized 128-dimensional action tokens, and its final encoder yields sequences of 80 tokens.The reported encoder has 7,399,103 parameters and is selected using downstream motion utility measured by a frozen world-model probe.Source: §E.2 · p. 32

E.3 Detailed Predictor Training Setup

  • A frozen Long-CLIP-L encoder supplies caption embeddings to a non-autoregressive Transformer predictor trained to regress the frozen FAE latent sequence.The predictor outputs all 80 positions in parallel and its best checkpoint is selected by validation frame-level cosine similarity.Source: §E.3 · pp. 32–33

E.4 Detailed Adapter Training Setup

  • Across all three backbones, adapters train on real flow-derived latents but use caption-predicted latents at inference, while the video backbone and surgical LoRA remain frozen.Adapters inject global and temporal conditions at the input level; the deployed configurations differ by backbone.Source: §E.4 · p. 33

E.5 Detailed SurgMetrics Training Setup

SurgMetrics is trained in stages on cached visual and motion features, with case-level data splits and a curriculum over degradation complexity. Inference settings are held consistent within each backbone comparison, and general metrics quantify semantic alignment, distributional similarity, and temporal consistency.

E.5 Detailed SurgMetrics Training Setup

  • SurgMetrics adapts and freezes SurgViSTA before training a fusion model on cached 768-dimensional visual and 25-dimensional Farneback flow features, producing six artifact scores.The fusion model has 354,502 trainable parameters; training uses a curriculum that introduces clean and single-degradation records before the full set.Source: §E.5 · pp. 33–35

E.6 Detailed Inference Setup

  • Across the three backbones, Base-only, LoRA-only, and LoRA-FLAIR share 81 frames, seed, and backbone-specific settings; LoRA-FLAIR adds the action adapter and predicted motion latent.All three use the same cached prompt-z produced by the shared Long-CLIP predictor.Source: §E.6 · pp. 35–36, Table 10 · p. 35

E.7 Detailed implementation of general metrics

  • The general metrics cover prompt alignment, frame- and video-distribution distances, and instrument temporal consistency, but abnormal CogVideoX LoRA area fluctuation despite an apparent surgical scene illustrates their inadequacy for surgical evaluation.Lower area fluctuation indicates smoother temporal evolution of instrument size.Source: §E.7 · p. 36

F Discussion on Limitations and Broader Impact

FLAIR supplies text-conditioned motion priors rather than user-specified trajectories, and its action coverage is bounded by the taxonomy and source datasets. The work is intended for research and education, not as a validated clinical resource.

Limitations in action control

  • FLAIR predicts a plausible motion prior rather than enforcing a user-specified trajectory, leaving trajectory- or keyframe-level steering as future work.A prompt can admit multiple valid executions, such as different grasp angles or interaction durations.Source: F Discussion on Limitations and… · p. 37

Dataset coverage

  • Motion priors may transfer less directly to action types or surgical modalities absent from the source data and BSA-10 taxonomy.SurgActionClip-30K consolidates 11 datasets and spans six core procedures plus two held-out procedures.Source: F Discussion on Limitations and… · p. 37

Broader impact

  • FLAIR, SurgActionClip-30K, and SurgMetrics are intended to support education, simulation, and data augmentation, not validated clinical use.Real-world deployment should include expert oversight and clear labeling of synthetic content.Source: F Discussion on Limitations and… · pp. 37–38
Loading 2610.09800v1…