PaperlayerView exact revision on arXiv

Source-linked AI summary

A Hierarchy-Aware Video-Language Model Evaluation and Hyperbolic Baseline for Surgery

Ana Manzano Rodríguez, Pascal Mettes, Marlies P. Schijven, Cees G. M. Snoek

arXiv:2609.27139v1cs.CVcs.AI

TL;DR

Surgical video-language models are evaluated with flat per-level metrics despite the phase–step hierarchy and the importance of cross-level coherence. The paper introduces SURGHIBENCH and HYPERSURG, finding that error severity differs even at comparable accuracy and that hyperbolic geometry shifts predictions toward the correct procedural neighborhood. The conclusion frames extension to deeper hierarchies as prospective as richer annotations become available.

  • Problem

    Surgical video-language models are evaluated with per-level accuracy or F1, without assessing cross-level consistency or whether errors respect procedural structure.

    Source: §1 · p. 1

  • Method

    The paper introduces SURGHIBENCH to assess recognition, consistency, and error severity, and HYPERSURG, which enforces phase–step containment with hyperbolic entailment cones.

    Source: §1 · p. 2, §3 · p. 3, §4.3 · p. 5

  • Results

    Models with comparable accuracy produce different error patterns, while HYPERSURG shifts predictions toward the correct procedural neighborhood and reduces cross-level contradictions by up to 13 pp.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The gains from hierarchy-aware geometry scale with the tree-likeness of dataset annotations, providing an indicator of when it helps.

    Source: §6 · p. 8

    Extension of the evaluation framework and geometric approach to deeper hierarchies is prospective and depends on richer annotations becoming available.

    Source: §6 · p. 9

Abstract

from arXiv · show

Surgical procedures follow a phase-to-step hierarchy, yet the video-language models used to recognize them are evaluated with flat per-level metrics that ignore cross-level coherence and error structure. In this paper we make two contributions to address this problem, (i) we introduce SurgHiBench, the first hierarchy-aware evaluation suite for surgical video understanding, with three tasks measuring recognition, consistency, and severity across granularity levels. We evaluate a general-purpose CLIP model, a Euclidean surgical model, and, as second contribution: (ii) HyperSurg, a new hyperbolic model that enforces phase-step containment via entailment cones, across four (existing) datasets spanning three procedure types. The suite reveals that two models with the same accuracy can produce predictions of very different error severity, ranging from sibling confusions within the correct phase to unrelated cross-phase predictions. Hyperbolic geometry shifts predictions toward the correct procedural neighborhood, and these gains scale with the tree-likeness of each dataset's annotation hierarchy, providing a principled indicator when hierarchy-aware geometry helps.

1. Introduction

Surgical activities form a hierarchy, so errors that preserve the phase context differ from predictions placed in unrelated phases. The paper introduces a hierarchy-aware evaluation suite and hyperbolic baseline to address gaps in both modeling and evaluation.

1. Introduction

  • Surgical workflow recognition spans phases and steps, and errors within the correct phase preserve more procedural context than predictions from another phase.Effective support requires recognizing both broad phase and specific step while keeping predictions consistent.Source: §1 · p. 1
  • Current surgical video-language models use flat embeddings and independent per-level accuracy or F1, leaving cross-level consistency and error severity unmeasured.Two models with the same score may make errors of very different severity, a disconnect identified as a barrier to surgical AI adoption.Source: §1 · pp. 1–2
  • SURGHIBENCH evaluates per-level and joint recognition, cross-level consistency, and error severity across four existing datasets spanning three procedure types.It includes CLIP and SurgCLIP∗ (β) baselines alongside the proposed HYPERSURG model.Source: §1 · p. 2
  • HYPERSURG shifts predictions toward the correct procedural neighborhood, reduces cross-level contradictions by up to 13 pp, and improves step recognition by +9.3 pp on average.The suite finds that models with comparable accuracy can have substantially different prediction coherence and error severity.Source: §1 · p. 2

2. Related Work

Prior surgical video-language models and hierarchy-aware evaluation methods leave a gap: representations remain Euclidean or assessments use flat metrics. HYPERSURG instead encodes procedural granularity in a shared hyperbolic space, while SURGHIBENCH evaluates cross-level consistency.

Surgical Vision-Language Models

  • Surgical vision-language methods add hierarchical annotations or architecture, but remain in Euclidean space; HYPERSURG instead uses one hyperbolic space to encode granularity.The contrast is a change in underlying geometry rather than added architectural complexity.Source: §2 · p. 2

Hierarchical Evaluation

  • Prior hierarchy-aware evaluation spans medical diagnosis and general action recognition, but the cited frameworks had not evaluated surgical video or cross-level prediction consistency.Existing hierarchy-aware training and reranking approaches can increase error severity when every node in a medical hierarchy is a valid label.Source: §2 · p. 2

Hyperbolic Representations

  • Hyperbolic vision-language work includes entailment-based models, while concurrent HyperVLP also applies hyperbolic entailment to surgical video; both leave hierarchy-aware evaluation open.The cited prior models use standard flat metrics.Source: §2 · pp. 2–3

3. Hierarchy-Aware Evaluation for Surgery

SURGHIBENCH evaluates surgical video-language models beyond flat per-level recognition, measuring consistency and error severity across datasets with differing hierarchy structures. The paper describes its tasks, dataset spectrum, and three evaluation models.

3. Hierarchy-Aware Evaluation for Surgery

  • SURGHIBENCH evaluates whether predictions are correct at each level, mutually consistent across levels, and related or unrelated in the surgical hierarchy.The model-agnostic suite covers four existing datasets spanning three procedure types and evaluates three models.Source: §3 · p. 3

3.1. Hierarchy-Aware Tasks

  • The three tasks assess phase and step recognition, joint cross-level correctness, and whether an incorrect step remains within the ground-truth phase or shares its surgical action.Task 1 reports video-averaged accuracy and macro-F1 at each level plus a multi-level geometric-mean score; Task 2 includes Both-wrong, Both-right, and Step|Phase metrics.Source: §3.1 · p. 3

3.2. Surgical Video Datasets

  • The four annotated datasets range from a clean tree, where each step belongs to one phase, to DAG-like structures; tree-likeness is the percentage of steps with one phase.MIPO is 100% tree-like, GraSP is 33%, and the bypass datasets are 54–74%; the datasets cover radius fixation, gastric bypass, and prostatectomy.Source: §3.2 · pp. 3–4, Table 1 · p. 4
  • SurgLaVi-β pretraining uses narrated clips at action, step, and phase levels, providing hierarchical caption–clip pairs for representation learning.The open-source subset contains 2,464 videos, and the three levels yield approximately 113K caption–clip pairs.Source: §3.2 · p. 4

3.3. Baseline Models

  • The evaluation compares general-purpose CLIP, reimplemented Euclidean SurgCLIP∗ (β), and HYPERSURG using the same architecture, training data, and parameter count as SurgCLIP∗ (β).This setup attributes differences with HYPERSURG to embedding geometry alone.Source: §3.3 · p. 4

4. Hyperbolic Baseline Model for Surgery

HYPERSURG models surgical video and hierarchical captions in a shared hyperbolic space, using geometry and training objectives to represent procedural hierarchy. Its design separates semantic content from granularity and encourages parent–child containment.

4.1. Background: Lorentz Hyperbolic Space

  • Hyperbolic space separates generality, encoded by distance from the origin, from semantic content, encoded by angular direction.Its exponentially growing volume provides capacity for hierarchy levels that flat geometry collapses.Source: §4.1 · p. 5

4.2. Architecture

  • HYPERSURG jointly encodes surgical clips and hierarchical captions with a video encoder and a text encoder in a shared hyperbolic embedding space.It uses a TimeSformer for video and BERT-base for text, then projects both representations onto a Lorentz hyperboloid.Source: §4.2 · p. 5

4.3. Training Objectives

  • The contrastive loss aligns video–text pairs via Lorentz distance, while the entailment loss enforces parent–child containment within a cone.The cone’s minimum width is controlled by rmin, and violations are penalized with slack η.Source: §4.3 · pp. 5–6
  • A hybrid sampler includes three forced triplets per batch, each covering one caption per hierarchy level from the same video.This guarantees approximately nine entailment pairs per batch while maintaining contrastive diversity.Source: §4.3 · p. 6
  • An ablation confirms that entailment cones and hierarchy-aware triplet sampling each contribute beyond Lorentz distance alone.The model combines these hierarchy-aware components with a contrastive loss that aligns video–text pairs via Lorentz distance.Source: §4.3 · p. 6

5. Results

Across three tasks, the results show that hierarchy-aware evaluation distinguishes recognition, cross-level consistency, and error severity. HYPERSURG improves joint recognition and shifts errors toward the correct procedural neighborhood, with gains varying by dataset hierarchy.

5.1. Task 1: Single- and Multi-Level Recognition

  • 16.2% vs 13.6% phase accuracy and 17.0% vs 8.3% step accuracy, but CLIP’s F1 is lower at both levels: 6.6 vs 7.9 and 4.9 vs 5.6.The authors attribute this pattern to CLIP collapsing predictions into frequent classes; its overall F1 is 4.1 phase and 1.8 step.Source: §5.1 · p. 6
  • 23.2% vs 18.1% overall multi-level score, while HYPERSURG reaches 23.5% step accuracy and 23.8% phase accuracy; gains increase with tree-likeness.The bypass datasets (54–74% tree-likeness) show strong multi-level improvements, while MIPO (100%) has the largest; on GraSP, SurgCLIP∗ (β) leads on step accuracy because 67% of steps recur across phases.Source: Table 2 · p. 7, §5.1 · p. 7

5.2. Task 2: Cross-Level Consistency

  • 82.6% to 68.3% to 60.2% Both Wrong rates across CLIP, SurgCLIP∗ (β), and HYPERSURG; Step|Phase is 30.6% vs 23.0% overall for HYPERSURG and SurgCLIP∗ (β).GraSP is an exception to the Both Wrong trend; the authors also caution that CLIP’s apparently strong conditional consistency on MIPO is misleading given its phase F1 of 6.6.Source: Table 3 · p. 7, §5.2 · pp. 7–8

5.1. Task 1: Single- and Multi-Level Recognition

  • On MIPO, HYPERSURG achieves the best accuracy and F1 at both levels despite MIPO being an orthopaedic procedure outside the domains covered by surgical pretraining.The authors present this as evidence that the hierarchical inductive bias generalises to out-of-distribution procedures.Source: §5.1 · p. 7

5.3. Task 3: Hierarchy-Aware Error Severity

  • 38.7% sibling accuracy for HYPERSURG versus 33.4% for SurgCLIP∗ (β) and 19.0% for CLIP; semantic step accuracy is 33.5%, 22.4%, and 12.4%, respectively.On BernBypass70, HYPERSURG’s dominant confusion is a sibling step in the same phase involving the same anatomical structure, whereas the Euclidean model’s confusion is from another phase and anatomical site.Source: Table 4 · p. 8, §5.3 · p. 8

6. Conclusion

The paper argues that surgical phases and steps should not be treated as independent recognition tasks. It concludes that the evaluation framework and geometric approach are expected to extend to deeper hierarchies as richer annotations become available.

6. Conclusion

  • Across four datasets and three procedure types, models with comparable accuracy can produce different error patterns; HYPERSURG aligns embedding geometry with procedural hierarchy and produces more accurate, consistent cross-level predictions.The reported gains scale with annotation tree-likeness, which the authors offer as an indicator of when hierarchy-aware representations will help.Source: §6 · p. 8
  • The authors expect both the evaluation framework and geometric approach to extend naturally to deeper hierarchies as richer annotations become available.They describe these as tools for evaluating surgical workflow systems in ways that reflect operating-room needs.Source: §6 · p. 9

A. SURGHIBENCH Evaluation Suite

The appendix describes semantic step groupings, zero-shot prompt construction, and a replication check for SurgCLIP(β). It also specifies which benchmarks support cross-level evaluation and how MIPO prompts were prepared.

A.1. Semantic Step Groupings

  • Semantic step groupings combine steps that share a surgical action across anatomical sites or are closely related variants; the groupings were defined manually from each dataset’s step vocabulary.These groupings are used to compute Task 3 semantic step accuracy, and Table 5 lists them by dataset.Source: §A.1 · p. 11, Table 5 · p. 11

A.2. Zero-Shot Evaluation and MIPO Prompts

  • Zero-shot evaluation uses text prompts without prompt engineering, ensembling, or templates; MIPO prompts were written because its annotations lack text descriptions.For GraSP and the bypass datasets, the prompts come from SurgLaVi [18]; MIPO prompts follow the same narration style.Source: §A.2 · p. 11

A.3. Replication of SurgCLIP(β)

  • The SurgCLIP∗ (β) reimplementation follows Perez et al.’s [18] zero-shot protocol on six benchmarks, while only GraSP has reported step-level results in that protocol.Cholec80, AutoLaparo, and HeiChole have phase annotations only, so they cannot be evaluated on SurgHiBench cross-level tasks; Table 8 places original and reimplemented results side by side for replication assessment.Source: §A.3 · p. 11

A.4. Additional Training Details

The appendix reports training dynamics and implementation settings for HYPERSURG. It also notes a discrepancy between MIPO’s repository documentation and its released annotation files.

A.4. Additional Training Details

  • At the end of training, contrastive loss is 0.032 for phase, 0.050 for step, and 0.109 for action, preserving a difficulty gradient across hierarchy levels.The authors report phases as easiest to discriminate and actions as hardest.Source: §A.4 · p. 11
  • MIPO documentation reports four phases and six steps, but the released annotation files contain five phase labels and eight step labels; the paper uses the released-file counts.Source: Table 6 · p. 12, Table 7 · p. 12
  • The OneCycleLR scheduler uses a 5-epoch linear warmup with initial learning rate 10^-6, followed by cosine annealing, and ends at a final learning rate of 10^-6.Bias terms, positional embeddings, and learnable scalar parameters (temperature, curvature, projection scales) are excluded from weight decay.Source: §A.4 · p. 12
  • Each GPU processes 39 samples, including 30 random samples and 9 samples from three forced triplets; the effective batch size across eight GPUs is 312.Source: §A.4 · p. 12
  • Lorentz distance computations use float32, with the acosh argument clamped to [1 + ϵ, ∞); temperature and curvature are stored in log-space and clamped to specified ranges.The reported converged values are τ=0.035, c=0.815, eαv=0.054, and eαt=0.060.Source: §A.4 · p. 12

B. HYPERSURG Ablation

The ablation finds complementary contributions from the triplet sampler and entailment cones, with the full model improving over Lorentz distance alone. The entailment-loss weight also trades off phase and step accuracy, with λ=0.1 giving the best overall balance.

B. HYPERSURG Ablation

  • The ablation compares Lorentz-only, cones with random batches, the triplet sampler without cones, and the full configuration.All four variants use Lorentz distance.Source: B. HYPERSURG Ablation · p. 12, Table 9 · p. 13
  • The triplet sampler improves phase and step on every multi-level dataset; cones and sampler together add up to +6.7 pp phase and +8.6 pp step on Bypass.On GraSP, cones add no phase benefit under its DAG-like hierarchy; on MIPO, the sampler alone edges out the full model on step accuracy, while cones improve phase by +4.1 pp.Source: B. HYPERSURG Ablation · p. 12
  • Table 8 compares original SurgCLIP(β) results with SurgCLIP∗(β), reimplemented on the same data using the same procedure.The comparison reports zero-shot downstream performance using video-averaged percentages.Source: Table 8 · p. 13
  • The full model improves over Lorentz-only by +6.0 pp on phase and +6.9 pp on step, raising multi-level score from 16.9% to 23.2%.The full model combines Lorentz distance, entailment cones, and the hierarchy-aware triplet batch sampler.Source: B. HYPERSURG Ablation · p. 12, Table 9 · p. 13
  • Among five tested λ values, λ=0.1 yields the highest overall multi-level score (23.2%) and best or near-best results on three of four datasets.Lower λ=0.05 favors phase accuracy, while higher λ=0.3 favors step accuracy; Table 10 varies λ with the full configuration fixed.Source: B. HYPERSURG Ablation · p. 14, Table 10 · p. 14

C. Qualitative Results

Figure 5 gives qualitative examples of successful and failed predictions across four hierarchical datasets. It illustrates that staying within the correct phase can limit error severity, while identifying the wrong phase subtree can trap predictions there.

C. Qualitative Results

  • On GraSP, HYPERSURG predicts Idle as a cross-phase error, consistent with the dataset’s DAG-like hierarchy.Source: Figure 5 · p. 15
  • In the subtree lock-in example, all five HYPERSURG predictions are from Jejunojejunal Anastomosis rather than the ground-truth Gastrojejunal Anastomosis.Source: Figure 5 · p. 15
  • SurgCLIP∗(β) predicts a cousin from a different phase in the example paired with HYPERSURG’s within-phase sibling error.Source: C. Qualitative Results · p. 15
  • When HYPERSURG identifies the correct subtree, hierarchical containment can keep an incorrect step prediction within the correct phase; visually similar sibling phases can instead trap it in the wrong subtree.In one example, HYPERSURG predicts a sibling step within the correct phase, while SurgCLIP∗(β) ranks the correct step first.Source: C. Qualitative Results · p. 15
Loading 2609.27139v1…