PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniRAS: Standardizing Foundation Model Training and Evaluation in Robot-Assisted Surgery

Leonardo Borgioli, Neil Getty, Wenli Xiu, Jessica Cassiani, Alvaro Ducas, Carlos Agustin Orda, Hira Waris, Fangfang Xia, Rick Stevens, Pier Cristoforo Giulianotti, Milos Zefran

arXiv:2608.31048v1eess.IV

TL;DR

Robot-assisted surgery lacks broad in-domain foundation-model benchmarks and large robotic video resources. OmniRAS releases robotic-cholecystectomy datasets, continues pretraining V-JEPA-2.1 encoders on a large surgical catalog, and evaluates them across six robotic tasks. Adapted OmniRAS models achieve the strongest results across task families, while frozen comparisons are mixed and smaller.

  • Problem

    Robot-assisted surgery has few foundation models, difficult-to-assemble robotic video corpora, and evaluations dominated by laparoscopic benchmarks.

  • Method

    OmniRAS releases two robotic-cholecystectomy benchmarks, documents continued pretraining of 1B and 2B V-JEPA-2.1 encoders, and evaluates them across six robotic tasks.

  • Results

    Adapted OmniRAS encoders achieve the strongest measured results across every evaluated task family, whereas frozen comparisons are mixed and partial adaptation produces larger, more consistent gains.

  • Takeaways & Limitations

    Continued pretraining provides a strong surgical initialization, but evaluation should report both frozen probing and partial adaptation because task adaptation substantially affects transfer.

Abstract

from arXiv · show

Few foundation models exist for robot-assisted surgery, partly because large robotic-surgery video corpora are difficult to assemble and existing models are evaluated mostly on laparoscopic benchmarks. Further, most existing models are evaluated on a small set of public benchmarks, mostly focused on laparoscopic surgery. We present OmniRAS, a family of 1B- and 2B-parameter V-JEPA-2.1 encoders for robot-assisted surgery, and detail their training. First, we release two densely annotated robotic-cholecystectomy datasets: OmniRAS-PR and a multi-label YT-Chole tool-verb-target task, the first triplet-style annotation for robotic cholecystectomy, together with splits, probe protocols, and an inter-rater study validating the shared phase ontology. Second, we document continued pretraining at up to 256 compute nodes with global batch 6,144 over 19 sources totaling approximately 2,650 hours of surgical video, 51% robotic, and analyze compute and data composition. Third, we evaluate against raw V-JEPA-2.1 and specialized surgical models on six tasks spanning triplet, phase, and step recognition, action segmentation, and detection, under frozen-encoder and final-four-block fine-tuning regimes. Across three seeds, this yields 254 downstream runs, including 109 with partial backbone fine-tuning. The best OmniRAS models achieve the strongest adapted results across all task families, while frozen differences are smaller.

I. INTRODUCTION

OmniRAS addresses limited robotic-surgery data and offdomain evaluation by releasing robotic benchmarks, documenting large-scale continued pretraining, and evaluating shared encoders across robotic tasks. Adapted surgical encoders achieve the strongest results across evaluated task families, while frozen differences are smaller.

  • Motivation: Robotic-surgery benchmarks are limited, and many surgical foundation models are evaluated mainly on laparoscopic data or with changing downstream architectures.These practices make robotic-domain and representation-level comparisons difficult.
  • Contributions: OmniRAS releases OmniRAS-PR and YT-Chole Triplets for robotic cholecystectomy under complementary phase and fine-grained tool-verb-target annotations.OmniRAS-PR combines 51 procedures under an 11-class ontology; YT-Chole is the first triplet-style action annotation on robotic cholecystectomy video.
  • Contributions: Independent re-annotations show substantial agreement across all three rater pairs for the shared phase and triplet ontology.The released benchmarks include split definitions, probe protocols, and annotation validation.
  • Training: Continued pretraining uses up to 256 nodes and global batch 6,144 across three production runs, with the longest configuration yielding the strongest transfer.The campaign includes 9.22 M, 18.43 M, and 36.86 M samples, although budget and catalog effects are not fully disentangled.
  • Evaluation: Across 254 downstream runs, including 109 with partial backbone fine-tuning, adapted surgical encoders achieve the strongest results across the evaluated robotic task families.The common probe covers triplets, phases, steps, action recognition, segmentation, and detection under frozen and last-four-block regimes.

B. Annotation Protocol

The annotation protocol uses procedural references, consensus, sparse labels, and independent re-annotation to assess phase and triplet reliability. Agreement is generally strong, with residual disagreement concentrated in concurrent phase activities and action-boundary placement.

  • Annotation Protocol: The protocol anchors triplet definitions in a structured cholecystectomy reference and uses consensus before independent review and disagreement resolution.Two additional re-annotators checked correctness and consistency with the agreed ontology.
  • Phase reliability: Phase reliability is assessed on a 10% YT-Chole sample using two independent raters, sparse labeling, and three boundary-tolerance levels.Ambiguous intervals are left unlabeled, and frames near either annotation’s segment boundaries are discarded before rescoring.
  • Phase reliability: 82.1% of residual phase disagreement lies in the five-phase Calot’s-triangle cluster, where sub-activities occur concurrently rather than sequentially.Pooling that cluster raises mean agreement to 0.833; model comparisons still use the eleven-class reference.
  • Triplet reliability: Triplet reliability is evaluated separately because tool, verb, and target labels are independent multi-label axes.For scoring, each frame is reduced to its longest-active label on each axis.
  • Dataset structure: The custom corpora use an aligned eleven-class phase ontology and independent multi-label axes for YT-Chole Triplets.The dataset overview and label inventory summarize corpus sizes, probe definitions, and label structure.
  • Triplet reliability: Triplet agreement improves mainly with small temporal tolerances, indicating that residual differences largely reflect action-boundary placement rather than ontology interpretation.Random frame removal does not reproduce the agreement increase, while the dominant-label projection supports temporal consistency.

E. Public benchmarks

The evaluation uses robotic benchmarks spanning temporal granularity, spatial localization, and two procedures, with new cholecystectomy datasets complementing public prostatectomy tasks. Annotation studies support the phase and triplet label protocols.

  • Benchmark scope: The benchmark suite spans coarse temporal structure, fine temporal structure, and sparse object-level spatial structure in robotic surgery.These axes are operationalized through phase, step, action, and detection tasks.
  • Public benchmarks: Public robotic benchmarks include SAR-RARP50 action recognition, GraSP phase and step recognition, and SARAS-ESAD action detection.YT-Chole and OmniRAS-PR add cholecystectomy coverage, so shared results are not confined to one procedure.
  • Annotation validation: YT-Chole triplets are annotated independently along instrument, verb, and target axes, with reliability assessed separately from the phase ontology.Each frame is reduced to its longest-active label on each axis before scoring because triplets are multi-label.
  • Pretraining context: The pretrained 2B checkpoint uses sixteen-frame, four-frames-per-second clips from a multi-source corpus spanning robotic and laparoscopic surgical video.Source mixing uses square-root inverse-frequency temperature sampling, with filters for near-static and black clips.

III. MODELS AND PROTOCOLS

OmniRAS continues the V-JEPA-2.1 representation-space prediction objective on surgical video and evaluates matched encoders through frozen probes or limited final-block adaptation. Task-specific heads cover triplets, phases, temporal recognition, segmentation, and detection.

  • Encoders: V-JEPA predicts hidden video representations from visible context rather than reconstructing pixels, and V-JEPA-2.1 adds dense per-token and deep self-supervision.OmniRAS continues this label-free objective on surgical video.
  • Evaluation rationale: Pretraining ℓ1 loss alone does not establish representation quality, so downstream probes provide task-grounded evidence of transfer.The probes are external to the self-supervised update rule.
  • Encoders: The released ViT-g and ViT-G encoders contain approximately 1B and 2B parameters, with raw checkpoints compared against continued-pretraining variants.The 1B headline model is epoch 19 of continued pretraining.
  • Probe protocol: FT4 unfreezes the final four encoder blocks with depth matching rather than parameter matching, while task heads inherit each encoder’s token width D.Thus, trainable backbone and head parameter counts vary across architectures.
  • Task heads: YT-Chole recognition factorizes triplets into independent multi-label instrument, verb, and target branches with separate class-averaged mAP reporting.Token features are pooled into a clip-level descriptor before the shared multi-task head.
  • Task heads: Phase recognition uses attentive pooling and softmax classification, while GraSP and SAR-RARP50 use temporal segmentation heads with global attention and dilated temporal convolutions.The GraSP head is transferred from SAR-RARP50 without structural changes.
  • Task heads: SARAS-ESAD combines multi-label action-presence prediction with class-conditioned box regression, reporting recognition and localization diagnostics separately.Mean IoU and box mAP@50 isolate localization quality.

IV. CONTINUED PRETRAINING AT PRODUCTION SCALE

The production campaign continued V-JEPA-2.1 pretraining on a large, predominantly robotic surgical-video catalogue using a two-tier strategy of expensive production runs and smaller screening runs. The reported longest run produced the strongest transfer, but compute and catalogue effects remain confounded.

  • Interpretation boundary: The longest production run yielded the strongest transfer among tested configurations, but its gain cannot be attributed to compute alone because its catalogue changed.Budget-scoped null results and frozen-probe ties should not be generalized beyond the tested settings.
  • Data and scale: The reported catalogue contains approximately 2,650 hours across 19 datasets, with about 51% robotic video and temperature-weighted source sampling.The 2B production catalogue includes 14 entries representing 19 distinct datasets.
  • Data composition: Robotic da Vinci video is the majority of the mixture by hours in all three production runs and underlies the reported benchmark results.The headline encoder additionally restores Kinetics-400 and Open-H-Embodiment rehearsal shards.
  • Data hygiene: Contamination is screened before pretraining with a crop- and mask-augmented perceptual-hash gate that drops sources exceeding the leakage threshold.Candidate training videos are matched against evaluation frames before inclusion.
  • Evaluation scale: The campaign comprised more than 254 downstream probe and fine-tuning runs, including 109 runs that unfreeze final backbone blocks.The study uses these runs to quantify downstream variability across seeds.
  • Compute: Production runs use 256 compute nodes and global batch 6,144, with reported budgets of 9.22 M, 18.43 M, and 36.86 M samples.The first two runs provide the matched production budget comparison; the third also changes the catalogue.
  • Campaign design: Smaller 1B screening runs explore composition, objective, and sampling choices because single-variable comparisons were infeasible at production scale.This two-tier design limits how broadly screening results generalize.

B. Transfer across production-scale runs

Increasing production-run size produced task-dependent transfer: smaller matched runs showed non-monotonic or divergent probe trajectories, whereas the 36.86 M-sample run eventually improved both IVT and GraSP. Its strongest result remains confounded by a simultaneous catalogue change.

  • Adapted transfer: The final encoder reaches 85.34 GraSP phase mAP after last-eight-block fine-tuning, reported as +8.6 points over the best published system.This is the adapted result obtained from the 36.86 M-sample encoder.
  • Production runs: 3,038 node-hours span production runs of 9.22 M, 18.43 M, and 36.86 M samples, evaluated with IVT and GraSP checkpoint probes.The probes were selected because they respond differently to continued pretraining and provide complementary transfer views.
  • Matched runs: The 9.22 M-sample run shows IVT values of 34.1, 33.7, 34.4, and 33.3 mAP, while GraSP progresses from 72.2 to 73.1, 74.1, and 73.5.These trajectories illustrate checkpoint sensitivity and non-monotonic transfer.
  • Matched runs: The 18.43 M-sample run raises IVT from 33.4 to 34.6 before falling to 33.2 and 32.4, while GraSP ends at 75.0 after a more stable trajectory.Reducing the self-supervised objective does not uniquely determine downstream transfer.
  • Largest run: The 36.86 M-sample run reaches 35.7 IVT mAP and 84.0 GraSP phase macro-F1 at its final probe, improving both measures over the longer horizon.GraSP shows the strongest evidence of the transition, while IVT extends it to a distinct task.
  • Interpretation boundary: Under the matched catalogue and recipe, doubling from 9.22 M to 18.43 M samples modestly improves final GraSP from 73.5 to 75.0 while IVT decreases from 33.3 to 32.4.The clearest transfer gain coincides with the run that changes both budget and catalogue.

C. Lessons from a 256-node campaign

The campaign favors sustained production training with complementary probes, while cautioning that the clearest transfer gain cannot be attributed to compute alone because catalogue composition changed.

  • Monitoring training: Complementary probes capture different transfer behavior: YT-Chole IVT is checkpoint-sensitive, whereas GraSP phase recognition reveals gains emerging after sustained training.Their disagreement at intermediate checkpoints provides useful evidence about representation transfer beyond the self-supervised objective.
  • Budget allocation: Clear transfer gains emerged in the final production configuration, while smaller screening runs did not resolve consistent effects of catalogue composition, sampling temperature, or masking objective.The authors therefore favor extending a validated configuration to production scale before distributing compute across shallow variants.
  • Interpretation: The 36.86M-sample run changed its training catalogue, so the campaign supports practical guidance rather than a controlled scaling law attributing improvement uniquely to budget.The matched 9.22M-to-18.43M comparison under the same catalogue and recipe was approximately probe-neutral.
  • Diagnostics: Masked-prediction fidelity improved across independent surgical units, while the 36.86M checkpoint lost SSv2 accuracy relative to the raw Meta 2B initialization.The diagnostics pair evidence of surgical predictive change with evidence of reduced general-action retention.
  • Reproducibility: A reproducible campaign should fix checkpoint indexing, epoch definitions, data manifests, and probe locations before long distributed runs.The proposed workflow validates the harness, spans distinct representation requirements, trains sufficiently long, and evaluates fixed checkpoints throughout.

V. RESULTS

Across the reported recognition and segmentation results, OmniRAS benefits are most apparent when limited backbone adaptation is allowed, although frozen transfer is already strong on several tasks.

  • YT-Chole triplets: 31.18 ± 1.64 IVT mAP is the strongest frozen YT-Chole result, but matched frozen effects are mixed across OmniRAS scales.The longer-trained OmniRAS 2B checkpoint exceeds raw 2B and OmniRAS 1B, while matched changes are +1.07 at 1B and −0.44 at 2B.
  • YT-Chole triplets: 39.92 ± 0.91 IVT mAP is reached by the 37M OmniRAS 2B model under FT8, after FT4 reaches 37.68 ± 0.23.Under FT4, OmniRAS improves over the corresponding raw encoder at both 1B and 2B scales.
  • OmniRAS-PR: 47.92±.53 frame-F1, 61.11±.85 phase mAP, and 67.15±.78 F1@10 are achieved by OmniRAS 2B with last-four-block adaptation on OmniRAS-PR.The adapted configurations occupy the top of the reported frame-level ranking.
  • OmniRAS-PR: Frozen OmniRAS 2B improves phase recognition over raw Meta 2B from 23.76 ± 4.03 to 44.89 ± 2.18 frame-F1 and from 34.09 ± 3.75 to 58.63 ± .81 phase mAP.Under matched FT4, OmniRAS 2B reaches 47.92 ± .53 frame-F1 and 61.11 ± .85 phase mAP, exceeding Meta 2B FT4.
  • SAR-RARP50: 91.56 ± 0.25 F1@10 is the strongest SAR-RARP50 result, achieved by the 37M OmniRAS 2B checkpoint when fully unfreezed.Last-four-block adaptation also raises OmniRAS performance at both scales relative to corresponding Meta models.
  • SAR-RARP50: Adaptation narrows the discrepancy between segmental F1@10 and frame-macro accuracy, with the 37M OmniRAS 2B model leading both metrics after adaptation.Frozen F1@10 is tightly grouped, whereas frame-macro accuracy is less ordered across encoders.

D. SARAS-ESAD action detection

OmniRAS 2B achieves the strongest reported SARAS-ESAD detection performance after backbone adaptation and augmentation, but localization diagnostics move differently from detection AP.

  • Detection performance: 0.2319 ± 0.0199 APmean is achieved by the 37M OmniRAS 2B model with FT4 and augmentation, exceeding the best challenge submission’s 0.1928 APmean.The same configuration also reaches 0.3262 ± 0.0084 AP10 and 0.1127 ± 0.0303 AP50.
  • Detection performance: FT4 increases 37M OmniRAS 2B APmean from 0.1793 ± 0.0083 frozen to 0.2035 ± 0.0132, while augmentation adds +0.028 APmean.Augmentation consistently helps the reported OmniRAS configurations.
  • Localization diagnostics: Mean IoU decreases from 0.4664 ± 0.0128 to 0.4426 ± 0.0096 after FT4, and box mAP@50 decreases from 0.4810 ± 0.0284 to 0.4209 ± 0.0254.The frozen OmniRAS 2B checkpoint instead has the highest localization diagnostics despite lower detection AP.
  • Related workflow transfer: Adapted image baselines remain below the strongest video encoders on GraSP, while limited adaptation preserves stronger step recognition than aggressive adaptation.The 37M OmniRAS 2B FT4 model reaches 57.85 ± 1.29 step mAP and 83.41 ± 0.63 phase mAP.

F. Cross-task synthesis

Across 254 downstream runs, OmniRAS provides its clearest advantage under partial adaptation across surgical recognition, segmentation, detection, and workflow transfer, while objective-aligned diagnostics show broader predictive change than frozen probes alone.

  • Cross-task synthesis: OmniRAS reaches the strongest result in every evaluated surgical task family, spanning triplets, phases, action recognition, detection, and workflow transfer.Representative results include 39.92 ± 0.91 IVT mAP, 67.15 ± .78 F1@10, 90.88 ± 0.09 F1@10, and 0.2319 ± 0.0199 APmean.
  • Adaptation: Partial fine-tuning exposes OmniRAS benefits more consistently than frozen probing, improving matched raw comparisons and producing the strongest OmniRAS-PR and SARAS-ESAD models.Longer pretraining further improves several adapted settings, including YT-Chole 2B FT4 from 36.09 ± 0.25 to 37.68 ± 0.23 and FT8 to 39.92 ± 0.91.
  • Retention: With Kinetics-400 rehearsal, the 36.86M-sample checkpoint retains general-video capability but remains below raw Meta 2B on SSv2.It reaches 53.82 top-1 and 44.70 macro-F1, compared with 60.27 and 52.58 for the raw initialization.
  • Caveats: GraSP-step reuses the underlying corpus with a different sampling seed, making it a robustness check rather than an independent benchmark; SARAS-ESAD has only four independent source videos.These constraints limit how statistical evidence from those settings should be interpreted.
  • Objective-aligned diagnostic: The masked-prediction diagnostic compares each checkpoint with its own target encoder and predictor, so it measures checkpoint-specific objective fidelity rather than a shared representation-space ranking.Matched clips, token partitions, and random seeds provide a paired comparison under identical masking conditions.

VI. DISCUSSION

The discussion attributes OmniRAS’s strongest gains primarily to downstream adaptation, while showing that surgical continued pretraining improves predictive fidelity without proving higher frozen-probe performance. It also identifies benchmark, ontology, source-composition, and reproducibility boundaries that qualify interpretation.

  • Source of gains: Adapted surgical encoders achieve the highest score in every task family, while the decomposition attributes less advantage to pretraining alone.The discussion separates aggregate rankings from the contributions of continued pretraining and downstream adaptation.
  • Source of gains: 4.81–6.56 points of YT-Chole IVT mAP improvement follow last-four-block fine-tuning, and adaptation also improves APmean for every V-JEPA variant on SARAS-ESAD.Frozen probing measures immediate feature accessibility rather than total usable task capacity.
  • Task-dependent adaptation: Phase recognition on GraSP is strongest with full fine-tuning, whereas step recognition is strongest with last-four-block fine-tuning.OmniRAS exceeds the corresponding Meta initialization under matched partial fine-tuning for both tasks.
  • Label-free evidence: 60 of 60 paired unit-level comparisons favor the 2B surgical checkpoint’s masked-target prediction over its initialization, but this does not establish higher final downstream values.The label-free evaluation replicates the V-JEPA-2.1 masking objective without a task head.
  • Validity and interpretation: The custom benchmark has only ten YT-Chole batches, while private data are in-house, so public GraSP, SAR-RARP50, and SARAS-ESAD remain necessary external anchors.The phase ontology also shows mean κ = 0.807 at ±4 s, with 82.1% of disagreement touching the Calot’s-triangle cluster.
  • Practical guidance: The paper recommends verifying the training harness, using at least two readouts, fixing bookkeeping conventions, and treating small-budget catalog comparisons as exploratory.The authors report that extending the final configuration was more informative than branching additional small-budget screening runs.

VII. CONCLUSION

OmniRAS combines robotic-cholecystectomy benchmarks, large-scale surgical continued pretraining, and controlled evaluation across six surgical tasks. Its adapted encoders achieve the strongest results across task families, while gains depend on adaptation and trade against general-video performance.

  • VII. CONCLUSION: 254 seed-level downstream runs span six surgical task families, and OmniRAS reaches the strongest measured result in every family considered.The task families include triplet, phase, step, action-recognition, segmentation, and detection evaluations.
  • VII. CONCLUSION: OmniRAS exceeds the published SAR-RARP50 reference, the best SARAS-ESAD challenge submission, and TAPIS on both GraSP phase and step recognition.These comparisons extend beyond foundation-model baselines.
  • VII. CONCLUSION: The largest production configuration shows the clearest transfer gains, but simultaneous catalog changes prevent attributing them to compute alone.Catalog, sampling, and masking effects remain unresolved at the tested screening budget.
  • VII. CONCLUSION: Partial adaptation produces larger and more consistent gains than frozen comparisons, while the preferred regime depends on the downstream task.GraSP phase recognition is strongest under full fine-tuning, whereas step recognition is strongest with last-four-block fine-tuning.
  • VII. CONCLUSION: Continued surgical pretraining improves surgical transfer but trades some general-video performance, with the trade-off depending on configuration.The 36.86M-sample checkpoint scores 53.82 top-1 on SSv2, 6.45 points below the raw model.
  • VII. CONCLUSION: The authors will release OmniRAS checkpoints and both introduced datasets subject to institutional, privacy, and data-use requirements.The datasets were constructed under two separate IRB-approved protocols.

APPENDIX A CLAUDE OPUS 5 FEW SHOT TRIPLET ESTIMATION

Table IV compares a frontier multimodal model under few-shot prompting with a task-adapted video representation on the full YT-Chole triplet validation set. Both methods are evaluated using IVT mAP over the same supported triplets.

  • TABLE IV: The comparison uses the full YT-Chole triplet validation set containing 1,696 clips.
  • TABLE IV: The two methods are a frontier multimodal model under few-shot prompting and a task-adapted video representation.
  • TABLE IV: IVT mAP is computed over the same 278 supported triplets for both methods.
Loading 2608.31048v1…