Colon-X: Advancing Intelligent Colonoscopy toward Clinical Reasoning

• Colonoscopy remains limited by operator variability and fatigue, while intelligence in the field trails general-domain progress, motivating clinically grounded multimodal reasoning.Prior work reported that (semi-)automated assistance reduced colorectal neoplasia miss rates by nearly 50% compared with conventional workflows.Source: §I · pp. 1–2
• COLONVQA contains 1,100,786 VQA queries across 212,742 images, 76 clinically meaningful findings, and 18 multimodal tasks.The queries amount to over 49.9 million textual tokens and are organized in a five-level taxonomy.Source: §I · p. 2
• COLONEVAL assesses the generalizability of 22 MLLMs, while COLONPERT tests leading models’ reliability under human-induced perturbations.The assessments examine diverse colonoscopy tasks and text-dominance biases arising from image overlays and textual instructions.Source: §I · p. 2
• COLONREASON uses a multi-agent debating pipeline to generate clinically grounded reasoning traces, which human verification found reasonably consistent with human judgment.The traces are intended as structured supervisory signals for reasoning-model development.Source: §I · p. 2
• COLONR1 addresses reward information collapse with task-adaptive rewards, negative sampling, and self-evolving memory for difficult cases.Trained on approximately 7.5K samples, it outperforms supervised fine-tuning counterparts by over 20% on COLONEVAL.Source: §I · p. 2
• COLON-X brings together dataset, evaluation, and methodology resources in a unified multimodal framework.Source: §I · p. 2

II. RELATED WORKS

• Existing colonoscopy benchmarks span low-level restoration and feature extraction, high-level clinical interpretation, and emerging vision-language tasks.High-level work has expanded both task breadth and finding granularity, including disease detection, safety monitoring, and dense temporal masks for video polyp segmentation.Source: §II · p. 2
• Prior multimodal resources include Kvasir-VQA, Kvasir-VQA-x1 [32], Liu et al., Gut-VLM, and colonoscopy-specific ColonINST.The described datasets differ in scale and scope; ColonINST includes over 450.7K VQA entries across four clinical tasks and 62 categories.Source: §II · p. 3
• COLON-X focuses on lower gastrointestinal findings and presents its database as the most extensive, category-rich, and task-diverse among the compared benchmarks.The work also extends beyond dataset construction to multimodal understanding evaluation and clinical reasoning.Source: §II · p. 3

III. SCALING COLONOSCOPY DATA TO MILLION SCALE

• The benchmarking crisis is attributed to scarce biomedical data and task-specific models trained on isolated benchmarks; COLONVQA consolidates public sources to support task-modality synergies.Source: §III · p. 3

• The study retrieved public medical data using colonoscopy-related keywords, including colon, polyp, colonoscopy, and gastrointestinal.The collection comprised 32 datasets and approximately 533K raw images covering pathological findings, anatomical landmarks, and surgical tools.Source: A. Imaging Data Preprocessing · p. 3
• Preprocessing standardized category names, removed duplicates and ambiguous or insufficiently labeled images, and downsampled frames to reduce temporal redundancy.Duplicate removal combined an automated tool with human inspection.Source: A. Imaging Data Preprocessing · p. 3
• Official train-validation-test splits were retained when available; otherwise, a random 6:1:3 video- or procedure-level split served as a proxy for patient-level isolation.The paper notes that patient identifiers are typically anonymized and inconsistent across datasets.Source: A. Imaging Data Preprocessing · p. 3
• The resulting collection contains 212,742 colonoscopy images across 76 clinically meaningful categories.Source: A. Imaging Data Preprocessing · p. 3

• A unified “colonoscopy image + task instruction → response” interface is designed to support controllability, transferability, and adaptability.The interface unifies image-label pairs with distinct clinical focuses or diagnostic priorities.Source: B. VQA Data Curation · p. 3
• Figure 2 presents representative VQA samples organized in a five-level taxonomy reflecting typical colonoscopy workflows.Task-category statistics are summarized in Table 2(b).Source: Figure 2 · p. 4
• The 18 image-to-text tasks are grouped into quality control, safety monitoring, lesion diagnosis, disease grading, and clinical documentation.Examples include bowel cleanliness scoring, instrument identification, lesion description, NICE and PARIS grading, polyp sizing, and image captioning.Source: B. VQA Data Curation · p. 4
• 212,742 image-label pairs yield over 1.1M VQA entries and 49.9M textual tokens.Five prompt templates are designed per task, with one randomly assigned per image to avoid inflating data through simple prompt rephrasing.Source: B. VQA Data Curation · p. 4

• The evaluation examines MLLM generalizability and reliability as two dimensions of multimodal understanding in colonoscopy.Source: §IV · p. 4

A. Generalizability

• COLONEVAL uses 4,568 reviewed VQA entries across 16 clinical tasks and compares 22 MLLMs using exact-match accuracy.Source: A. Generalizability · p. 4, Table III · p. 5
• In the reported comparison, top-ranked closed model Gemini 2.5 Flash (Q5) outperforms leading open model InternVL3-8B (⋆5) by 32.34%; the passage does not identify the compared metric.Source: A. Generalizability · p. 5
• Closed-source models generally lead, but three open models under 8B achieve over 87% accuracy in safety monitoring.Source: A. Generalizability · p. 5, Table III · p. 5
• Reasoning models do not consistently improve accuracy: Gemini 2.5 gains 12.67%, while Grok and Claude reasoning variants drop 15.04% and 16.14%.Source: A. Generalizability · p. 5

B. Reliability

• COLONPERT identifies text-dominance bias under human perturbations, arising from on-image text and textual prompts.Source: B. Reliability · pp. 5–6
• Zero-masking on-image text caused accuracy drops up to 90% for InternVL3-8B and MedGemma-4B, while three closed models dropped within 10%.Source: B. Reliability · p. 6, Table IV · p. 6
• Misleading image text reduced InternVL3-8B accuracy by 34% and Gemini 2.5 Flash accuracy by 45%.Source: B. Reliability · p. 6, Table IV · p. 6
• Contradictory prompts reduced HuatuoGPT-Vision-7B accuracy by 28.07%; emotional narratives reduced its accuracy by 12.50% and Gemini 2.5 Flash by 16.25%.Source: B. Reliability · p. 6, Table IV · p. 6
• The authors attribute text-dominance bias to modality imbalance and propose visual grounding for implicit bias and adversarial text augmentation for explicit bias.Source: B. Reliability · p. 6

V. COLONOSCOPY MEETS CLINICAL REASONING

• The paper presents COLONREASON and the COLONR1 baseline model to advance reinforced reasoning abilities for medical VQA.Source: §V · p. 6

A. Reasoning Trace Annotation

• Two role-playing agents generate initial reasoning traces for the same image-question-answer triplet, providing diverse impressions of the case.The pipeline is motivated by the cost and subjectivity of human-annotated reasoning traces.Source: A. Reasoning Trace Annotation · p. 6
• Each agent critiques the other's initial reasoning, after which an agent integrates its own initial thoughts with the peer critique.This update mirrors revisiting judgments after peer discussion, particularly in uncertain or hard cases.Source: A. Reasoning Trace Annotation · p. 6
• Each distilled essential viewpoint is paired with confidence score s_i ∈ [−10, +10], quantifying epistemic adjustment: −10 means total loss of confidence, +10 signals reinforced certainty, and zero denotes neutrality.An aggregator integrates all reasoning-confidence pairs into a unified reasoning trace.Source: A. Reasoning Trace Annotation · p. 7
• A panel of K judges accepts a synthesized trace when more than half vote that it supports the answer; otherwise, the process restarts.The workflow is presented as a looped, multi-agent process.Source: A. Reasoning Trace Annotation · p. 7, Figure 5 · p. 7

B. Incentivizing Reasoning in Colonoscopy

• Directly applying naive GRPO to colonoscopy causes rapid policy-gradient explosion, attributed to rewards that fail to discriminate across task structures and case distributions.These factors destabilize policy updates and motivate the proposed optimization changes.Source: B. Incentivizing Reasoning in… · p. 7
• Task-adaptive rewards score open questions continuously, yes-or-no questions binarily, and multiple-choice answers on a three-level scale.For multiple choice, the graded reward distinguishes incorrect, partially correct, and fully correct answers, including label-content mismatches.Source: B. Incentivizing Reasoning in… · p. 7
• Negative sampling replaces one response with an incorrect answer when a group has identical rewards, restoring reward contrast for more discriminative updates.This targets easy queries or late training, when all responses may be correct and relative advantages disappear.Source: B. Incentivizing Reasoning in… · p. 7
• Self-evolving memory stores hard queries whose response groups average below 0.8 reward, then refines prompts with previously incorrect responses when those queries recur.The mechanism is intended to help the policy learn from persistent failures on hard cases.Source: B. Incentivizing Reasoning in… · pp. 7–8
• 56.61% accuracy is achieved by COLONR1 on COLONEVAL, compared with standard SFT accuracy below 32%; naive GRPO suffers policy-gradient explosion.Combining negative sampling and self-evolving memory with task-adaptive rewards yields the highest performance among the reported variants.Source: Table VI · p. 8, B. Incentivizing Reasoning in… · p. 8
• With roughly 7.5K training cases, COLONR1 improves accuracy by 1.31% over its non-reasoning variant, while SFT and Med-R1 degrade on thinking traces.The authors note substantial room to improve the alignment of thinking and decision-making.Source: B. Incentivizing Reasoning in… · p. 8

VI. CONCLUSION & OUTLOOK

• COLON-X combines a colonoscopy dataset, multimodal understanding evaluations, and a reasoning-centered data-to-model framework.The paper frames this integrated framework as addressing a longstanding gap and paving a way toward clinical reasoning and broader medical applications.Source: §VI · p. 8
• The authors state that a large gap remains before generalized clinical intelligence is achieved.They identify data quality and granularity, including knowledge distillation, disease grading, and rare cases, as priorities for data-centric intelligence.Source: §VI · p. 8
• Reliable multimodal evaluation, especially of trustworthiness in real-world deployment, remains an open challenge.Source: §VI · p. 8