PaperlayerView exact revision on arXiv

Source-linked AI summary

Colon-Bench: An Agentic Workflow for Scalable Dense Lesion Annotation in Full-Procedure Colonoscopy Videos

Abdullah Hamdi, Changchun Yang, Xin Gao

arXiv:2603.25645v3eess.IVcs.CVcs.HC

TL;DR

Colon-Bench addresses the shortage of densely annotated, long-sequence colonoscopy videos for evaluating medical AI and MLLMs. It constructs a verified multi-task benchmark through an agentic annotation workflow and evaluates MLLMs across classification, OV-VOS, and VQA, introducing colon-skill prompting that improves most MLLM performance by up to 9.7%.

  • Problem

    Dense, diverse, long-sequence colonoscopy datasets with spatial, temporal, and linguistic annotations are limited, constraining evaluation of modern MLLMs.

  • Method

    Colon-Bench uses agentic annotation with temporal proposals, box tracking, AI confirmation, human review, and multi-task evaluation across classification, OV-VOS, and VQA.

  • Results

    Most MLLMs improve by up to 9.7% with colon-skill prompting, while benchmark results show strong MLLM localization and reasoning across colonoscopy tasks.

  • Takeaways & Limitations

    Colon-Bench provides a broad, densely supervised benchmark for testing MLLM colonoscopy understanding and training-free domain-specific prompting.

Abstract

from arXiv · show

Early screening via colonoscopy is critical for colon cancer prevention, yet developing robust AI systems for this domain is hindered by the lack of densely annotated, long-sequence video datasets. Existing datasets predominantly focus on single-class polyp detection and lack the rich spatial, temporal, and linguistic annotations required to evaluate modern Multimodal Large Language Models (MLLMs). To address this critical gap, we introduce Colon-Bench, generated via a novel multi-stage agentic workflow. Our pipeline seamlessly integrates temporal proposals, bounding-box tracking, AI-driven visual confirmation, and human-in-the-loop review to scalably annotate full-procedure videos. The resulting verified benchmark is unprecedented in scope, encompassing 528 videos, 14 distinct lesion categories (including polyps, ulcers, and bleeding), over 300,000 bounding boxes, 213,000 segmentation masks, and 133,000 words of clinical descriptions. We utilize Colon-Bench to rigorously evaluate state-of-the-art MLLMs across lesion classification, Open-Vocabulary Video Object Segmentation (OV-VOS), and video Visual Question Answering (VQA). The MLLM results demonstrate surprisingly high localization performance in medical domains compared to SAM-3. Finally, we analyze common VQA errors from MLLMs to introduce a novel "colon-skill" prompting strategy, improving zero-shot MLLM performance by up to 9.7% across most MLLMs. The dataset and the code are available at https://abdullahamdi.com/colon-bench .

1 Introduction

Colon-Bench addresses the lack of dense, diverse, long-sequence colonoscopy annotations needed for AI research and MLLM evaluation. Its agentic workflow combines automated proposal, tracking, confirmation, and human review to produce a multi-task benchmark with rich spatial, temporal, and textual supervision.

  • Research gap: Existing colonoscopy datasets emphasize single-class polyp detection or narrower anatomical tasks rather than diverse lesions with dense boxes, masks, and clinical text.Colon-Bench is positioned as broader in lesion taxonomy, supervision, and supported evaluation tasks.
  • Research gap: Manual dense annotation is labor-intensive and inconsistent, while colonoscopy videos contain sparse lesions obscured by blur, occlusion, debris, fluids, and camera contact.These challenges motivate a scalable annotation pipeline for colonoscopy video analysis.
  • Approach: Colon-Bench uses a multi-stage agentic workflow combining temporal proposals, bounding-box tracking, AI visual confirmation, and human-in-the-loop review.The accepted annotations support VQA, binary lesion classification, and OV-VOS.
  • Benchmark: 14 lesion categories, over 300k bounding boxes, 213k segmentation masks, and 133k words of clinical descriptions provide dense, multi-task supervision for colonoscopy video understanding.The benchmark annotations were verified by humans and an experienced surgeon.
  • Evaluation and novelty: Colon-Bench evaluates frontier MLLMs on lesion reasoning and localization, and cross-model VQA errors motivate a colon-skill prompt that improves most MLLM performance by up to 9.7% without additional training.The benchmark is designed to test MLLMs on long, occluded, noisy medical sequences whose capabilities remain largely untested.

2 Methodology

Colon-Bench uses a staged annotation and evaluation workflow combining automated proposals, tracking, AI confirmation, and human review to produce dense colonoscopy-video benchmarks. It evaluates MLLMs across classification, segmentation, and VQA, with results comparing them against specialized baselines and SAM-3.

  • Agentic Workflow: 1,325 candidate lesion windows from 60 REAL-COLON videos were progressively refined through verification filtering, EdgeTAM tracking, AI confirmation, and human review.The pipeline initially covered 22.97 hours and produced over 314k initial bounding boxes during spatial annotation.
  • Agentic Workflow: Automated filters and human review progressively discard false-positive windows while increasing temporal precision, F1, and specificity on surrogate REAL-COLON polyp labels.These surrogate labels provide quality control rather than exhaustive recall targets because many Colon-Bench lesion categories are absent from REAL-COLON.
  • Colon-Bench Evaluation Benchmark: Colon-Bench comprises 1,597 clips from 60 patients and supports binary classification, detection, instance segmentation, and two VQA difficulty tiers.The benchmark contains 955,126 frames, with 272 detection clips and 264 instance-segmentation clips; its segmentation task establishes a colonoscopy OV-VOS benchmark.
  • Colon-Bench Evaluation Benchmark: VQA includes 1,485 prompted questions over 499 clips and 2,740 unprompted questions over 918 clips, covering lesion type, appearance, location, morphology, and temporal references.Temporal references are concentrated in prompted VQA, appearing in 59% of prompted questions versus 2% of unprompted questions.
  • Colon-Bench Results: Gemini 3 Pro and Gemini 3 Flash lead VQA and segmentation, while Gemini 3.1 Flash Lite leads classification F1 at 80.7%.Seed 1.6 is the strongest open-weight model overall, ranking third in VQA and achieving classification F1 of 69.4%.
  • Colon-Bench Results: GPT-5.4 with EdgeTAM beats SAM-3 by 32.0% mIoU on open-vocabulary video lesion segmentation, while MLLMs beat Endo-CLIP by 30% for lesion detection.The qualitative comparison likewise reports that SAM-3 underperforms modern MLLMs paired with a prompted EdgeTAM tracker.

3 Analysis and Insights

Colon-Skill distills shared MLLM VQA errors into textual guidance, while ablations show that richer temporal context improves segmentation and VQA performance.

  • Colon-Skill for MLLMs: Colon-Skill combines morphological cues, confusion traps, and a decision checklist synthesized from cross-model VQA errors.The guidance is tailored to colonoscopic VQA and added to prompts at inference time.
  • Colon-Skill for MLLMs: +9.7% is the maximum improvement from prepending Colon-Skill to VQA prompts across models.The evaluation uses identical conditions before and after prompt augmentation.
  • Frame Count Ablation: 54.4% mean IoU and 61.5% mDice are achieved by Gemini 3 Flash with 7 input frames, versus 43.1% and 48.8% with 1 frame.More frames improve segmentation but increase the cost of per-window detections.
  • Temporal Context in VQA: −7.0% is the largest single-frame VQA accuracy drop, observed for Qwen3.5 397B relative to full-video input.Single-frame input also reduces accuracy for Qwen-VL Max and Gemini 3 Flash.

4 Conclusions

Colon-Bench is presented as an agentic workflow and multi-task benchmark for dense, full-procedure colonoscopy video understanding. Its annotations support MLLM reasoning and spatial grounding across diverse lesion categories.

  • Conclusions: Colon-Bench provides dense annotations for 14 lesion categories using masks, bounding boxes, and clinical text.The benchmark targets full-procedure colonoscopy video understanding.
  • Conclusions: State-of-the-art MLLMs excel at high-level colonoscopy reasoning and spatial grounding with engineered context and agentic workflows.This conclusion is supported within the benchmark's evaluated scope.

A Additional Dataset Details

Colon-Bench was produced by filtering candidate lesion windows through verification, tracking, AI confirmation, and human review. The accepted set contains clinician-reviewed videos with dense spatial and textual annotations, while ablations quantify stage-specific trade-offs.

  • Dataset Construction: 1,325 candidate lesion windows from 60 sequences were reduced to 528 retained windows covering 464,035 frames across 59 sequences.The retained windows represent 39.8% of the initial candidates after successive verification stages and human review.
  • Dataset Statistics: 314,408 bounding boxes, 227,343 segmentation masks, and 145,515 words comprise the curated set.Curated windows also average 878.9 frames and 252.4 words, exceeding rejected-window averages.
  • Human Review: 597 lesion videos were clinician-reviewed, with 528 accepted at an 88.4% success rate and 464,035 total frames.The accepted videos constitute the verified portion of the agentic pipeline output.
  • Pipeline Ablation: +10.2% precision is the gain from tracking, while recall decreases by 17.2%.Verification filtering provides a separate +7.3% precision gain with minimal recall loss.

A.3 Colon-Bench Suite of Evaluation Benchmarks

Colon-Bench organizes colonoscopy video evaluation across classification, detection, segmentation, and two levels of VQA. Its clips, annotations, and question splits support both spatial localization and open-ended clinical reasoning.

  • Benchmark Composition: 1,597 unique clips from 60 patient sequences totaling 955,126 frames form the benchmark suite.The suite spans binary lesion classification, lesion detection, instance segmentation, and two VQA difficulty levels.
  • Benchmark Tasks: 790 clips support classification, 272 support detection, and 264 support instance segmentation.Detection includes 61,538 per-frame boxes, while segmentation includes 57,550 per-frame masks.
  • Dataset Statistics: Confirmed and rejected windows are compared in Table 4 to report curated Colon-Bench dataset statistics.The table distinguishes 528 confirmed windows from 69 rejected during human review.
  • VQA Splits: 1,485 prompted VQA questions cover 499 clips, while 2,740 unprompted questions cover 918 clips.The unprompted split fully encompasses the detection and segmentation videos for direct localization-to-reasoning comparison.

A.4 Benchmark Formation and Blind Tests

The benchmark uses prompted and unprompted five-choice VQA with debiasing checks, while a filtering funnel records successive reductions in windows, frames, and duration.

  • Blind Tests: 44.6% prompted and 37.1% unprompted blind-only accuracy exceeded the 20% random baseline after debiasing.Residual margins were attributed to skewed lesion-type distributions rather than surface-level cues.
  • Benchmark Formation: Each clip generates three five-way questions covering lesion identification, clinical characteristics, and temporal reasoning.Prompted videos include bounding-box and mask overlays on confirmed lesion windows, whereas unprompted videos use raw frames and include non-lesion windows.
  • Benchmark Formation: The filtering funnel reports windows, total frames, and video duration at successive pipeline stages, with dropped rows identifying removed windows.These quantities make the benchmark’s formation auditable across filtering stages.
  • Benchmark Formation: Stage-wise rejection statistics track window duration in frames at 10 fps and clinical-text description length, with only a subset of text fields available early.The statistics characterize how annotations change as windows pass through the pipeline.

B Additional Results

Additional results document the benchmark’s task statistics, evaluation metrics, qualitative examples, and visualization conventions for VQA, detection, segmentation, and classification.

  • Task Statistics: Colon-Bench task statistics report per-frame boxes and masks, five-choice VQA questions, and approximately three questions per video.Frame counts derive from per-clip start and end indices, while boxes and masks are per-frame annotations.
  • Segmentation: Segmentation uses lesion bounding-box detections to prompt a SAM-based tracker, with Mean IoU and Mean Dice averaged over evaluated frames.The benchmark contains 272 videos for segmentation evaluation according to the table caption.
  • Qualitative Results: Qualitative detection examples cover masses, polyps, and ulcers, while segmentation examples compare ground-truth and model mask overlays across lesion types.The segmentation comparison includes erythematous regions, lipoma, sessile polyp, erosion, and ulcer examples.
  • VQA: Prompted and unprompted VQA accuracy are computed over 1,485 and 2,740 questions respectively, treating unanswered questions as incorrect.Models are ordered by accuracy, with a 20% random-chance baseline shown in the figures.
  • Detection: Detection reports F1 Score, AP@50, and mAP@50-95, with models ordered by F1 and detections used in the segmentation benchmark.AP@50 uses IoU ≥0.50, while mAP@50-95 averages precision across IoU thresholds from 0.50 to 0.95.
  • Classification: Binary classification evaluates lesion-present versus lesion-absent predictions using accuracy, positive-class precision, recall, and F1.Accuracy is computed over all records, with unevaluated records counted as incorrect.

C.1 Ablation Study

Ablations examine pipeline metrics, detection-frame count, temporal context, bounding-box annotations, and the clinical aspects represented in VQA questions.

  • Segmentation Ablation: Increasing detection frames steadily improves downstream segmentation quality, with Gemini 3 Flash improving mIoU/mDice as frames increase.The study evaluates this trend through the detection-frame ablation and its corresponding figure.
  • Temporal Context: Temporal context is generally helpful for binary classification, especially for Seed 1.6, while Gemini 3 Flash is slightly stronger on single frames.Prompted VQA shows small, model-dependent shifts, with most models benefiting from video.
  • VQA Ablation: Bounding-box annotations produce only marginal changes in VQA performance, indicating limited dependence on explicit boxes.The comparison uses matched lesion windows with and without bounding-box annotations.
  • Question Aspects: VQA questions are multi-label and cover lesion type, morphology, size, color or surface appearance, anatomical location, temporal reasoning, and procedural action.The taxonomy is assigned through keyword matching over question text.
  • Question Aspects: Lesion type, color or surface appearance, and anatomical location dominate both VQA splits, while explicit temporal references distinguish prompted from unprompted questions.Temporal references occur in 59% of prompted questions and 2% of unprompted questions.

C.3 Colon-Skill for MLLMs

Colon-Skill augments MLLM VQA prompts with guidance synthesized from shared lesion-category errors and failure modes, improving performance unevenly across model capacities.

  • Skill Construction: A frontier LLM synthesizes shared MLLM failure cases into morphological cues, confusion traps, and a decision checklist.Questions that a majority of models answer incorrectly are retained for error analysis before skill generation.
  • Skill Construction: The resulting skill context is prepended to every VQA prompt at inference time.This provides structured domain knowledge without additional model training.
  • Performance: +9.7% prompted and +9.0% unprompted VQA gains are achieved across nine MLLMs with skill-augmented prompting.The effect is most pronounced for higher-capacity models.
  • Performance: Smaller models such as Molmo-2-8B show marginal or slightly negative skill-context deltas.The results suggest sufficient model capacity is needed to integrate the additional context.
  • Skill Content: The Colon-Skill addresses recurring errors including confusing sessile polyps with pedunculated lesions and misinterpreting NBI colors.Its rules also distinguish dark diverticular holes from masses and encourage conservative size estimates.
Loading 2603.25645v3…