PaperlayerView exact revision on arXiv

Source-linked AI summary

HeiCo-FOCUS: A Clinically Grounded Dataset for Long-Context Video Understanding

Leon Mayer, Lucas Luttner, Patrick Godau, Kai Fritzsche, Annika Reinke, Leonie Boland, Jule Brandt, Janne Heinecke, Chloe K. Nobuhara, Niklas Holzwarth, Evangelia Christodoulou, Marcel Knopp, Dominik Michael, Pascale Piermarco, Saliq Neyaz, Korhan Derin Özarslan, Jakob Hennighausen, Carlos Aumente-Maestro, Tim Rädsch, Dheeraj Baji, Peter Maximilian Full, Finn Aichholz, Justus Veit Erpenbeck, Linus Finn Schott, Bastian Winkelhausen, Claas de Boer, Bianca Güttner, Anneli Hummel, Gregor Just, Max Kirchner, Chenyang Li, Rozenn Raffaut, Ariel Rodriguez, Danush Kumar Venkatesh, Kevin Wang, Jinjing Xu, Mona Sheikh Zeinoddin, Salman Khan, Thomas M. Pausch, Stefanie Speidel, Danail Stoyanov, Daniel A. Hashimoto, Fiona R. Kolbinger, Thomas G. Weiser, Lena Maier-Hein

arXiv:2610.10156v1cs.CV

TL;DR

Existing video evaluations often emphasize local reasoning rather than cumulative temporal consistency over extended horizons, a gap with particular relevance to surgical videos and foreign-object retrieval. HeiCo-FOCUS introduces a clinically grounded, multi-track benchmark built around persistent object tracking in surgery. Evaluations show current VLM limitations in long-context understanding, while the benchmark offers a testbed for assessing long-horizon reasoning.

  • Problem

    Many long-video benchmarks assess local reasoning within extended segments rather than persistent entity tracking and state aggregation, while surgical videos demand precise interpretation over procedures lasting tens of minutes to hours.

    Source: §1 · p. 2

  • Method

    HeiCo-FOCUS builds a clinically grounded foreign-object tracking benchmark with persistent instance annotations and FRAME, SEGMENT, and PROCEDURE tracks.

    Source: §1 · p. 3, §3 · p. 4

  • Results

    State-of-the-art VLMs show limitations in long-context understanding, temporal consistency, and safety-critical reasoning on HeiCo-FOCUS.

    Source: §1 · p. 3

  • Takeaways & Limitations

    HeiCo-FOCUS provides a testbed for assessing whether modern VLMs can move beyond short-term pattern recognition toward reliable long-horizon reasoning.

    Source: §6 · p. 10

    Because the dataset covers a limited number of colorectal procedures, performance should not be interpreted as evidence of safe generalization across procedures, institutions, or clinical settings.

    Source: §6 · p. 10

Abstract

from arXiv · show

Recent advances in Vision-Language Models (VLMs) have led to rapid progress in video understanding across a wide range of benchmark tasks. However, existing evaluations largely focus on short-term reasoning, failing to assess a critical capability: maintaining cumulative temporal consistency over extended time horizons. To close this evaluation gap, we introduce HeiCo-FOCUS, a clinically grounded dataset for evaluating long-context video understanding through the task of Foreign Object Contextual Understanding in Surgery. Built on a dataset of Heidelberg Colorectal surgeries, this task requires models to continuously track multiple objects as they are inserted, manipulated, occluded, and removed over procedures lasting up to hours. HeiCo-FOCUS comprises 30,000 visual question answering (VQA) pairs covering five core capabilities: object recognition, temporal grounding, aggregation, event and procedural understanding, and complex reasoning. The dataset was constructed through a rigorous multi-stage annotation pipeline involving large-scale crowd annotation and 39 surgical domain experts to ensure high quality and clinical relevance. To systematically probe model behavior, we introduce a multi-track evaluation framework that progressively increases temporal and contextual demands from single frames to full procedures. Experiments with ten frontier VLMs show that HeiCo-FOCUS tasks are far from solved: only around half of the models clearly outperform a text-only baseline. Across the video tracks, models perform best on event and procedural understanding (mean Accuracy: 56.5% across all models), while temporal grounding remains particularly challenging for all evaluated models (mean Accuracy: 19.7%). We therefore expect HeiCo-FOCUS to serve as a catalyst for the development of models capable of reliable, temporally consistent reasoning over hours-long videos.

1 INTRODUCTION

HeiCo-FOCUS addresses the lack of long-horizon evaluation requiring persistent object tracking and cumulative reasoning, using foreign-object tracking in surgery as its clinically grounded task. Its design emphasizes trusted expert annotation, persistent object identities, and colorectal procedures; evaluations of state-of-the-art VLMs reveal continued limitations.

1 INTRODUCTION

  • Existing long-video evaluations mainly test local reasoning within extended segments, not persistent entity tracking or state aggregation across long horizons.These omissions leave cumulative temporal consistency unassessed.Source: §1 · p. 2
  • Foreign-object tracking in surgery tests persistent memory, temporal consistency, and aggregation while addressing the patient-safety problem of retained objects.Unintentionally retained foreign objects are rare but can cause serious complications.Source: §1 · p. 2
  • HeiCo-FOCUS uses a clinically curated base dataset and human experts throughout annotation to prioritize reliable, clinically meaningful evaluation.The paper reports involving more than an order of magnitude more domain experts than the median among over 400 analyzed medical imaging AI benchmarks.Source: §1 · p. 3
  • A dedicated expert-driven pipeline annotates persistent object instances, enabling questions about clinically meaningful long-horizon reasoning that existing surgical VQA datasets do not cover.Existing resources lack instance-level annotations, while manual instance annotation is described as prohibitively expensive.Source: §1 · p. 3
  • HeiCo-FOCUS uses colorectal procedures to combine long-context technical demands with a clinically important problem, rather than extending the predominance of cholecystectomy datasets.The paper reports colorectal procedures have up to 15 times higher complication rates, are fivefold longer, and contain on average three times more foreign-object instances per video than cholecystectomies.Source: §1 · p. 3
  • State-of-the-art VLMs show limitations in long-context understanding, temporal consistency, and safety-critical reasoning on HeiCo-FOCUS.The benchmark is unlikely to be saturated in the near term.Source: §1 · p. 3

2 RELATED WORK

Prior long-video and surgical VQA benchmarks have advanced video understanding, but their temporal windows and question types leave persistent long-context reasoning insufficiently addressed. The paper identifies a need for clinically grounded question design and rigorous expert validation.

2 RELATED WORK

  • Long-video benchmarks extend reasoning beyond short clips but typically use temporal windows limited to a few minutes.The paper describes this as a remaining constraint despite their contributions to video understanding.Source: §2 · p. 3
  • Existing surgical VQA datasets often use frame-level or short clips, emphasize perception and event understanding, and rarely report systematic expert verification.The paper also notes that it is unaware of a surgical AI dataset addressing temporal grounding and long-context reasoning.Source: §2 · p. 4
  • These limitations motivate a benchmark combining long-context video understanding, clinically grounded questions, and rigorous expert validation.Source: §2 · p. 4

3 HEICO-FOCUS

HeiCo-FOCUS is a clinically grounded long-context benchmark built on all 30 Heidelberg Colorectal surgeries, with 30,000 VQA pairs. It combines capability- and track-based evaluation with expert involvement and bias-aware validation.

3 HEICO-FOCUS

  • HeiCo-FOCUS contributes 30,000 VQA pairs for Foreign Object Contextual Understanding in Surgery, based on all 30 Heidelberg Colorectal procedures.Source: §3 · p. 4
  • Each procedure contributes 1,000 questions across FRAME, SEGMENT, and PROCEDURE tracks, with sigmoid resections recommended as the held-out test set.The track allocations are 400 FRAME, 400 SEGMENT, and 200 PROCEDURE questions per procedure; the recommended test set contains 10,000 questions.Source: §3 · p. 4
  • The three tracks progress from single-image understanding to segments of up to 5 minutes and procedure-long contexts spanning 5–296 minutes.FRAME focuses on perception and aggregation; SEGMENT and PROCEDURE assess all capabilities.Source: §3 · p. 4
  • The quality-controlled pipeline involves 39 clinical and technical domain experts, including 7 expert surgeons, 18 medical students, and 14 surgical AI researchers.Source: §3 · p. 5
  • Text-only model filtering removes questions answerable without visual input, while stratified evaluation and hierarchical statistical analysis address task imbalance and clustered questions.The paper gives hierarchical aggregation and bootstrapping as examples of accounting for multiple questions associated with the same video or segment.Source: §3 · p. 5

4 VQA GENERATION PIPELINE

The VQA pipeline turns surgical videos into clinically meaningful questions through taxonomy-guided design, instance tracking, and expert review. It combines automated processing and crowd annotation with surgical expertise to create and verify the benchmark’s question-answer pairs.

4.1 TAXONOMY

  • The taxonomy assigns each question one primary capability based on its dominant reasoning requirement and permits secondary labels for fine-grained stratified evaluation.Examples include assigning “Where” questions to spatial localization, “How many” to aggregation, and “When” to temporal localization.Source: §4.1 · p. 5

4.2 ANNOTATION PIPELINE

  • The pipeline screened clips for foreign objects, annotated sampled frames at 1 fps, refined boxes automatically, linked object identities across time, and generated and verified VQA pairs.Stage 4 generated bounding boxes around foreign objects at 1 fps across more than 300 k frames; Stage 6 verified the pipeline by analyzing about 500 diverse sample questions.Source: §4.2 · p. 5

4.3 VQA DESIGN

  • Instance-level trajectories support automatic question generation, but clips are visibility-annotated only and excluded from questions requiring individual-object tracking.The paper reports 324,273 manually curated frames and 79,682 verified foreign-object boxes, of which 55,629 carry instance identities.Source: §4.3 · p. 6
  • Question design combines informative anchor moments, surgeon review, and balanced template sampling across capabilities, object classes, answer values, and time.Three text-only models answer candidate questions without visual input; a question is discarded if all three answer correctly.Source: §4.3 · pp. 6–7

5 EXPERIMENTS AND RESULTS

Ten frontier VLMs were evaluated across three tracks against a text-only baseline; event and procedural understanding scored best across the video tracks, while temporal grounding was especially difficult on full procedures.

5 EXPERIMENTS AND RESULTS

  • The evaluation used accuracy, ten frontier VLMs via cloud APIs, track-specific video sampling, and a no-image GPT-5.6 Sol baseline; open answers were LLM-judged for semantic equivalence.SEGMENT windows were sampled at 1 fps, while PROCEDURE inputs used 300 uniformly spaced frames.Source: §5 · pp. 7–8
  • 56.5% average scores for event and procedural understanding contrasted with 26.9% mean temporal-grounding performance on SEGMENT and 12.6% on PROCEDURE.Temporal grounding on PROCEDURE was less than half its SEGMENT value.Source: Figure 4 · p. 8, §5 · p. 9
  • 44.8% Accuracy was Gemini 3.8 Flash’s FRAME-track high, 24.1 pp above the 20.7% text-only baseline; only half the models clearly outperformed it.The next-highest FRAME scores were 43.2% for Gemini 3 Flash and 38.9% for GPT-5.6 Sol.Source: Figure 4 · p. 8, §5 · p. 8
  • 41.7% average Accuracy over ten models on SEGMENT exceeded the 30.4% text-only baseline by 11.3 pp.Gemini 3.8 Flash led SEGMENT at 59.4% Accuracy.Source: Figure 4 · p. 8, §5 · p. 9
  • 37.9% average Accuracy on PROCEDURE exceeded the 27.8% text-only baseline; Gemini 3.8 Flash led with 56.7%.DeepSeek V4 Flash Vision, MiMo-V 2.5, and Nova 2 Lite scored below the baseline.Source: Figure 4 · p. 8, §5 · p. 9
  • Human raters scored 62.0%, 69.8%, and 57.4% Accuracy on FRAME, SEGMENT, and PROCEDURE, respectively, while the best model on each track scored 12.0, 12.1, and 6.9 pp lower.The human comparison used the same 300 questions as the raters.Source: §5 · pp. 8–9

6 DISCUSSION

HeiCo-FOCUS tests persistent object tracking and retrieval across long surgical procedures as a clinically grounded benchmark. The discussion highlights both the value of curated trajectories and limits in annotation, evaluation, and dataset scope.

6 DISCUSSION

  • The benchmark links long-context VLM performance to foreign-object tracking and retrieval, an objective and clinically critical task in procedures lasting up to hours.It targets persistent representations of multiple objects through scene changes, where errors accumulate and cannot be corrected from local context.Source: §6 · p. 9
  • Human raters scored only 35% on event aggregation and 40% on duration estimation, supporting the use of curated instance trajectories rather than ad-hoc history reconstruction.Clinical expertise was retained for question types requiring higher-level medical reasoning.Source: §6 · p. 9
  • The benchmark includes full-procedure questions exceeding 20 minutes, but small or numerous objects, video artifacts, and taxonomy choices challenged annotation and evaluation.The authors acknowledge that some annotation and taxonomy decisions may remain debatable and provide a community feedback process.Source: §6 · p. 9
  • Gemini 3.8 Flash cost about a fifth of GPT-5.6 Sol per video-track question, while prompt sensitivity and variability across repeated runs were not systematically investigated.The authors also note substantial monetary and environmental costs for evaluating long-context VLMs on extended surgical videos.Source: §6 · pp. 9–10
  • Because the dataset covers a limited number of colorectal procedures, benchmark performance is not evidence of safe generalization across procedures, institutions, or clinical settings.The conclusion presents HeiCo-FOCUS as a testbed for assessing long-horizon reasoning and a foundation for developing and evaluating models in complex real-world settings.Source: §6 · p. 10

DISCLOSURES

The disclosures report a former program-director role and research support.

DISCLOSURES

  • T.W. was the former Program Director for the Wellcome Leap SAVE program; the funder had no additional input into the publication decision.A.R. disclosed Helmholtz Association support through the Helmholtz Imaging Incubator.Source: DISCLOSURES · p. 10
  • A.R. disclosed research support from the Helmholtz Association through the Helmholtz Imaging Incubator.The support was within the scope of the Helmholtz Imaging Incubator (HI).Source: DISCLOSURES · p. 10

A COMPARISON WITH EXISTING SURGICAL VQA DATASETS

The section introduces a table comparing publicly released surgical VQA datasets and clarifies two table conventions.

A COMPARISON WITH EXISTING SURGICAL VQA DATASETS

  • Table 1 overviews publicly released surgical VQA datasets; class-label VQAs are generated from classification labels in available vision datasets.The caption marks datasets whose full release is not yet available with an asterisk.Source: Table 1 · p. 14

B DATA ACCESS AND USAGE

The section describes access through the orena-focus package and defines a taxonomy for evaluating surgical VQA across object, temporal, aggregation, procedural, and complex reasoning capabilities.

B DATA ACCESS AND USAGE

  • The orena-focus package provides access to videos and question–answer pairs, with dataset loading and evaluation utilities.Videos can be downloaded once, annotations are fetched from Hugging Face, and the Evaluator parses answers and routes them to an LLM judge.Source: B DATA ACCESS AND USAGE · p. 14, Figure 5

C TAXONOMY

  • Object recognition and identity matching cover object classes, instance correspondence, attributes and states, and localization in the image or relative to anatomy.Source: C TAXONOMY · p. 15
  • Temporal grounding covers locating object occurrences or events on the timeline and estimating their duration.Source: C TAXONOMY · p. 15
  • Aggregation combines information across foreign-object instances or categories and across foreign-object events over time.Source: C TAXONOMY · p. 15
  • Event and procedural understanding includes recognizing object interactions and usage purpose, and ordering events or procedural steps.Source: C TAXONOMY · p. 15
  • Complex reasoning covers functional, causal and consequence, and multi-step compositional inference across objects, space, and time.Source: C TAXONOMY · p. 15

D VQA GENERATION PROCESS

The benchmark is built through a staged annotation and question-generation process, with operational object rules, constrained answer formats, and checks on automated grading.

D VQA GENERATION PROCESS

  • The six-stage pipeline transforms surgical videos into VQA pairs through object screening, localization, refinement, instance tracking, question generation, and verification.Table 2 summarizes stage inputs, outputs, annotation scale across 30 videos, and the use of crowd, expert, and automated methods.Source: Table 2 · p. 16
  • Stage 4 produced 324,273 curated frames and 79,682 bounding boxes, while later stages generated questions from object information and verified the pipeline.Stage 4 assigns instances and corrects boxes and classes; Stage 5 generates questions and answers, and Stage 6 verifies question generation.Source: D VQA GENERATION PROCESS · p. 17
  • The dataset contains 30,000 VQA pairs balanced across five capability groups, divided into 12,000 FRAME, 12,000 SEGMENT, and 6,000 PROCEDURE pairs.Source: D VQA GENERATION PROCESS · p. 17

E MODEL PROMPTS

  • Prompts combine surgical context, a foreign-object definition, a task-specific question, and structured-output instructions; the example asks about objects inserted after a sponge.The context prefix identifies the procedure and timepoint and directs answers to rely on visual evidence.Source: E MODEL PROMPTS · p. 17, §E.1 · p. 18, §E.5 · p. 22

E.2 FOREIGN OBJECT DEFINITIONS

  • Foreign objects are defined as items introduced into the body cavity that must be retrieved or accounted for, with a specified class list and rules for insertion, presence, retrieval, and identity.Source: §E.2 · p. 19

E.3 FORMAT INSTRUCTIONS

  • Answer instructions constrain output formats, including time, duration, class, number, percentage, binary, multiple-choice, and open-ended responses.Source: §E.3 · p. 20

E.4 SCORING

  • Answers are parsed by expected format and scored using the rule for that format; unparseable answers count as incorrect.Table 3 describes scoring by answer format and defines d as the input-window length in seconds.Source: §E.4 · p. 20, Table 3 · p. 20

F VALIDATION OF THE LLM JUDGE; G HUMAN BASELINE

  • The judge was compared with two raters on 164 answers, while Table 4 describes a separate comparison of human raters and models on 300 test questions.The judge agreed with the resident on 90.7% of answers (κ = 0.74) and with the second rater on 92.7% (κ = 0.78); the two raters agreed on 91.8% (κ = 0.78).Source: F VALIDATION OF THE LLM JUDGE · p. 22, Table 4 · p. 22

H FURTHER RESULTS

The further-results visuals show model performance by capability, sub-capability, and API cost; the supplied passages describe their encodings but do not report specific outcomes.

H FURTHER RESULTS

  • Figure 7 compares model Accuracy across FRAME, SEGMENT, and PROCEDURE tracks by capability group, with means and 95% confidence intervals.Means are per-video Accuracy; confidence intervals use a two-level bootstrap over videos and questions with 1,000 resamples.Source: Figure 7 · p. 23
  • Figure 8 reports per-model Accuracy for each sub-capability, with blank cells indicating sub-capabilities absent from a track and Blind Baseline denoting the text-only baseline.Source: H FURTHER RESULTS · p. 23, Figure 8 · p. 24
  • Figure 9 plots Accuracy with 95% confidence intervals against API cost per 1,000 test questions on a log scale for models that see video.Costs sum answer API charges and exclude judge calls.Source: H FURTHER RESULTS · p. 23, Figure 9 · p. 25

I CHALLENGING EXAMPLES

Sample images illustrate annotation challenges, including object-classification and identity decisions and uncertainty about whether sponges remain in the surgical site.

I CHALLENGING EXAMPLES

  • Specimen and attached clips are excluded from foreign-object annotation once placed in a specimen bag.This rule defines a boundary for annotation of objects associated with specimens.Source: I CHALLENGING EXAMPLES · p. 27
  • Poor lighting at image borders makes shadows difficult to distinguish from blood-soaked sponges.Source: I CHALLENGING EXAMPLES · p. 27
  • Prolonged blue-screen interruptions make it difficult to determine whether sponges were removed or remained in situ before reappearing.Source: I CHALLENGING EXAMPLES · p. 27
  • Dense clip arrangements and perspective shifts can make it ambiguous whether a clip is the same object as before.Source: I CHALLENGING EXAMPLES · p. 28
Loading 2610.10156v1…