Colon-Bench: An Agentic Workflow for Scalable Dense Lesion Annotation in Full-Procedure Colonoscopy Videos

• Colonoscopy lesions are sparse and often obscured by blur, occlusion, debris, stool, fluids, or camera contact, while dense manual annotation is labor-intensive and inconsistent.Source: §1 · pp. 1–2
• Compared with prior datasets focused on single-class polyp detection or anatomical segmentation, Colon-Bench combines long-sequence coverage with dense boxes, masks, and clinical text for multiple video tasks.Source: Table 1 · p. 2, §1 · p. 2
• Temporal proposals, box tracking, AI-driven visual confirmation, and clinician review reduce manual labor while producing spatial, temporal, and textual annotations.Source: §1 · p. 3, Figure 1 · p. 3
• Colon-Bench covers 14 lesion categories, over 464k frames, 300k bounding boxes, 213k masks, and 133k clinical-description words, verified by humans and an experienced surgeon, across binary classification, OV-VOS, and two VQA tiers.Source: §1 · p. 3
• Cross-model error analysis motivates textual colon-skill guidance, which improves most MLLMs by up to 9.7% without additional training.Source: §1 · p. 3

2.1 Agentic Workflow for Colonoscopy Annotation

• The annotation workflow combines VLM-proposed lesion windows, verification filtering, EdgeTAM box tracking, cued AI confirmation, and physician review to retain 528 curated windows from 597 reviewed windows.The dataset contains 464,035 frames, 300,132 bounding boxes, 213,067 segmentation masks, and over 133,000 words of verified clinical text; its 14 lesion categories are long-tailed, with sessile polyps most frequent at 411 windows. Automated stages improve precision, F1, and specificity on REAL-COLON surrogate polyp labels, which are quality-control rather than exhaustive recall targets because many Colon-Bench lesion categories are absent there.Source: §2.1 · pp. 3–4, Table 2 · p. 4

2.2 Colon-Bench Evaluation Benchmark

• Colon-Bench defines five evaluation tasks across 1,597 clips from 60 patients, including binary classification, detection, instance segmentation, and prompted and unprompted VQA.The VQA splits contain 1,485 questions over 499 clips and 2,740 questions over 918 clips, respectively; two-stage adversarial distractor generation and blind text-only testing yield post-debiasing accuracy of 44.6% prompted and 37.1% unprompted versus a 20% random baseline.Source: §2.2 · pp. 4–5

2.3 Colon-Bench Results

• GPT-5.4 with EdgeTAM beats SAM-3 by 32.0% mIoU on video lesion segmentation; Gemini 3 Pro and Flash lead VQA and segmentation, while Gemini 3.1 Flash Lite leads classification F1 at 80.7%.Gemini 3 Pro and Flash achieve segmentation IoU/Dice of 45-48%/51-55%.Source: Table 3 · p. 6, §2.3 · pp. 6–7, Figure 4 · p. 7
• Among open-weight models, Seed 1.6 ranks third in VQA with 69.4% classification F1; Qwen variants perform well on VQA but nearly collapse on classification, and segmentation drops sharply beyond the top three.Seed 1.6 has the highest precision among full-benchmark models at 85.0%; Colon-Bench MLLMs beat Endo-CLIP by 30%.Source: §2.3 · pp. 6–7

3.1 Colon-Skill for MLLMs

• The Colon-Skill distills shared model errors into morphological cues, confusion traps, and a decision checklist for colonoscopic VQA.Questions that a majority of models answer incorrectly, with their question/answer context and lesion-category metadata, are given to a frontier LLM to synthesize the skill.Source: §3.1 · p. 8
• Colon-Skill prompting yields consistent VQA improvements of up to +9.7% across models.The skill is synthesized from shared cross-model failure cases and prepended to every VQA prompt for re-evaluation under identical conditions.Source: §3.1 · p. 8

3.2 Ablation Study

• Using 7 rather than 1 frame raises Gemini 3 Flash mean IoU from 43.1% to 54.4% and mDice from 48.8% to 61.5%, while Qwen-VL Max mean IoU rises from 19.7% to 33.7%.Because detections are per-window, the study uses 3 equally spaced frames per window as a trade-off for segmentation results.Source: §3.2 · p. 8
• Single-frame input lowers prompted VQA accuracy versus full video for Qwen3.5 397B (−7.0%), Qwen-VL Max (−3.4%), and Gemini 3 Flash (−2.5%).The effect of temporal context is model-dependent; full video generally outperforms single-frame input.Source: §3.2 · p. 9

4 Conclusions

• Colon-Bench combines a novel agentic workflow with a comprehensive multitask benchmark for full-procedure colonoscopy video understanding.The benchmark densely annotates 14 lesion categories with masks, bounding boxes, and clinical text.Source: §4 · p. 9
• State-of-the-art MLLMs excel at high-level colonoscopy reasoning and spatial grounding with engineered context and agentic workflows.Source: §4 · p. 9

A.1 Annotation Pipeline

• From 1,325 candidate lesion windows across 22.97 hours, filtering retained 528 windows (39.8%), covering 464,035 frames across 59 sequences.The pipeline began with 60 REAL-colon video sequences [3]; verification filtering removed 422 windows, AI confirmation 306, and human review 69.Source: §A.1 · p. 12
• The curated set comprises 314,408 bounding boxes, 227,343 segmentation masks, and 145,515 words of clinical text; curated windows average 878.9 frames and 252.4 words, versus 414–509 frames and 145–190 words for rejected windows.Source: §A.1 · p. 12
• Curated windows averaged 878.9 frames and 252.4 annotation words, versus 414–509 frames and 145–190 words for rejected windows.Window-duration distributions across pipeline stages are shown in Fig. 7.Source: §A.1 · p. 12

A.2 Validating the Annotations

• Clinicians reviewed 597 lesion videos, accepting 528 for an 88.4% success rate and 464,035 total frames.Source: §A.2 · p. 12
• Verification filtering raised precision by 7.3%, while tracking raised precision by 10.2% at a recall cost of 17.2%.The ablation evaluated stages on full REAL-colon polyp box annotations; tracking restricts positive predictions to frames with actual bounding boxes.Source: §A.2 · p. 12
• AI confirmation removed 34% of remaining windows and improved precision by 6.7% and F1 by 2.0pp; human review added 2.4% precision and 1.0% F1.Spatial metrics remained nearly unchanged after human review, indicating rejected windows had minimal impact on localisation quality.Source: §A.2 · p. 12

A.3 Colon-Bench Suite of Evaluation Benchmarks

• The benchmark spans binary lesion classification, lesion detection, instance segmentation, and prompted and unprompted visual question answering across 1,597 clips from 60 patient sequences and 955,126 frames.Classification covers 790 clips: 518 lesion-free and 272 lesion-positive.Source: §A.3 · p. 12
• Detection and segmentation use 272 and 264 lesion-positive clips, respectively, providing 61,538 per-frame bounding boxes and 57,550 per-frame masks.Source: §A.3 · p. 12
• The prompted VQA split has 1,485 five-choice questions over 499 clips, while unprompted VQA has 2,740 questions over 918 clips and encompasses all detection and segmentation videos.This enables direct comparison of spatial localisation with open-ended clinical reasoning.Source: §A.3 · pp. 12–13

Blind Tests

• 44.6% prompted and 37.1% unprompted blind-only accuracy on Gemini-3 Flash exceeded the 20% random baseline, with residual margins attributed to skewed lesion-type distributions.The reported comparison follows two-stage debiasing and blind text-only review.Source: §A.4 · p. 13

Benchmark Formation

• Each clip receives three five-way questions covering lesion identification, clinical characteristics, and temporal reasoning, generated using Gemini 3 Flash.Prompted questions use confirmed lesion windows with bounding-box and mask overlays; unprompted questions use raw frames and also include non-lesion windows.Source: §A.4 · p. 13

Blind Tests

• Two-stage debiasing regenerates four length- and style-matched adversarial distractors per question, then reverts questions with newly detected text-only shortcuts to their original wording.The distractor-generation call receives only the question stem and correct answer, re-randomizes option positions, and is followed by a blind text-only stress test.Source: §A.4 · p. 13

Benchmark Formation

• Both question sets undergo structural validation and fixed-seed randomization for reproducibility.The prompted and unprompted sets differ in overlays and window inclusion.Source: §A.4 · p. 13

Pipeline ablation

• The pipeline ablation reports temporal frame-level precision, recall, and F1 across processing stages, while its spatial metrics are available after tracking and include box F1 at multiple IoU thresholds and mean IoU.The table caption identifies REAL-colon as the source of the polyp annotations.Source: Figure 6 · p. 15, Table 7 · p. 15

Task statistics

• The task-statistics table defines frame counts from clip start/end indices, boxes and masks as per-frame annotations, and VQA as five-choice questions at approximately three per video.Source: Table 9 · p. 16

Classification

• Binary lesion classification is evaluated on 790 records, with accuracy over all records and positive-class precision, recall, and F1; unevaluated records count as incorrect.The accuracy visualization marks 50% as the random-chance baseline.Source: Table 12 · p. 19, Figure 13 · p. 23, Figure 14 · p. 24

Qualitative comparisons

• Qualitative comparisons show predicted boxes and masks alongside ground truth across lesion examples, and the detection boxes also serve as prompts for EdgeTAM tracking in segmentation.The examples include masses, polyps, ulcers, erythematous regions, lipoma, and erosion.Source: Figure 8 · p. 21, B Additional Results · p. 21, Figure 9 · p. 22

VQA benchmark

• The benchmark reports prompted VQA accuracy on 1,485 questions and unprompted accuracy on 2,740, counting unanswered questions as incorrect against a 20% random-chance baseline.Models are ordered by accuracy in both visualizations.Source: Table 10 · p. 17, Figure 10 · p. 22, Figure 11 · p. 23

Detection and segmentation

• Detection and segmentation evaluations each cover 272 videos, with detection metrics including F1, AP@50, and mAP@50-95 and segmentation metrics including Mean IoU and Mean Dice.Segmentation first uses bounding-box prompting to detect a lesion, then a SAM-based tracker propagates the mask across the tracking window.Source: Table 11 · p. 18, Table 13 · p. 20, B Additional Results · p. 20

C.1 Ablation Study

• More detection frames steadily improve segmentation quality, with diminishing gains and Gemini 3 Flash improving in mIoU and mDice.The frame-count ablation evaluates segmentation using different numbers of detection frames.Source: §C.1 · p. 25, Figure 15 · p. 25, Table 14 · p. 25
• Temporal context generally helps binary lesion classification, especially for Seed 1.6, while Gemini 3 Flash is slightly stronger with a single frame.The comparison uses video versus single-frame inputs.Source: §C.1 · p. 25, Table 17 · p. 27
• Bounding boxes produce only marginal VQA changes, indicating limited dependence on explicit boxes in the 499 matched clip windows.The ablation compares annotated polyp-box videos with frame-generated videos without annotations.Source: §C.1 · p. 25, Table 15 · p. 26
• For prompted VQA, video benefits most models, but a few show minor gains with a single frame.The temporal-context comparison covers 1,485 questions.Source: §C.1 · p. 25, Table 16 · p. 26

C.2 VQA Question Aspect Distribution

• Questions use multi-label keyword matching across clinical and visual aspects, averaging 2.11 labels for prompted questions and 1.65 for unprompted questions.The prompted and unprompted splits contain 1,485 and 2,740 questions, respectively; questions matching no category are assigned to Others.Source: §C.2 · p. 26
• Lesion type, color or surface appearance, and anatomical location dominate both splits, while temporal references occur in 59% of prompted questions versus 2% of unprompted questions.Unprompted questions deliberately omit time cues; Others accounts for 1.5% of prompted and 6.7% of unprompted questions.Source: §C.2 · p. 26, Figure 16 · p. 27

C.3 Colon-Skill for MLLMs

• Colon-Skill is synthesized from shared failure cases that most models answer incorrectly, combining morphological cues, confusion traps, and a decision checklist prepended to every VQA prompt.The workflow stratifies per-model predictions by lesion category before synthesizing the skill context.Source: §C.3 · p. 27, Figure 17 · p. 29
• Smaller models such as Molmo-2-8B show marginal or slightly negative deltas, suggesting adequate capacity is needed to integrate the added skill context.The passage qualifies the gains by model capacity.Source: §C.3 · p. 27
• +9.7% prompted and +9.0% unprompted are the maximum VQA gains across nine MLLMs with skill-augmented prompting.The reported effect is most pronounced for higher-capacity models.Source: §C.3 · p. 27, Table 18 · p. 28, Figure 21 · p. 32