PaperlayerView exact revision on arXiv

Source-linked AI summary

ARISE: Adaptive Agentic Reasoning with Image-grounded Self-Evaluation for Interpretable IBD Assessment

Pronoma Banerjee, Anuva Shah, Jason Wu, Md. Masudur Rahman, Sanjay Mohanty, Satya Kurada, Juan P. Wachs

arXiv:2610.04777v1cs.CVcs.AI

TL;DR

IBD imaging assessment relies on specialist interpretation, while VLM use is constrained by domain-specific reasoning, hallucination, scarce annotated data, and limited interpretability. ARISE adapts a frozen VLM through five stages of hypothesis generation, image-grounded evidence use, refinement, symbolic verification, and diagnosis. Across WCE and ultrasound cohorts, the authors report improved diagnostic performance and stage-level scores that expose process–outcome gaps.

  • Problem

    IBD imaging interpretation depends on scarce subspecialty expertise, and VLMs face domain-reasoning, hallucination, annotated-data, and interpretability challenges.

    Source: Introduction · p. 1

  • Method

    ARISE adapts a frozen VLM to few-shot IBD assessment through five stages that generate, ground, refine, verify, and apply diagnostic hypotheses.

    Source: Introduction · pp. 1–2

  • Results

    Across two independent cohorts and three VLM families, ARISE improved accuracy, F1, precision, and recall over baselines on every model–task combination.

    Source: Conclusion · p. 6

  • Takeaways & Limitations

    Stage-level scoring exposes cases where strong intermediate reasoning does not fully carry through to final accuracy, a gap a single outcome metric would hide.

    Source: Agentic and Task-level scores across… · p. 5, Conclusion · p. 6

Abstract

from arXiv · show

Inflammatory bowel disease (IBD) requires frequent imaging-based assessment, yet interpretation of modalities such as wireless capsule endoscopy (WCE) and intestinal ultrasound remains heavily dependent on specialist expertise. Vision-Language Models (VLMs) demonstrate significant potential in multimodal medical image analysis, but their clinical adoption is hindered by their insufficient domain-specific reasoning, susceptibility to hallucination, scarcity of high quality training data in fine-grained diagnostics and limited interpretability. We introduce ARISE (Adaptive Agentic Reasoning with Image-grounded Self-Evaluation), an autonomous planning framework that models few-shot medical image understanding as a sequential agentic workflow. ARISE structures agent execution into a transparent 5-stage workflow: hypothesis generation, image-grounded evidence summarization, evidence-conditioned refinement, symbolic verification, and final diagnosis. We apply ARISE to IBD assessment across two independent patient cohorts: wireless capsule endoscopy (WCE) images for Crohn's disease and B-mode ultrasound data for ulcerative colitis. ARISE consistently improves diagnostic performance over baseline VLMs while exposing where reasoning succeeds or fails, providing a more interpretable basis for clinical decision support and realistic deployment.

Introduction

IBD imaging assessment depends on scarce subspecialists, while general-purpose VLMs face domain-reasoning, hallucination, data-scarcity, and interpretability challenges. ARISE addresses these gaps with a five-stage, image-grounded and solver-verified workflow, evaluated across Crohn’s disease WCE and ulcerative-colitis ultrasound cohorts.

Introduction

  • IBD monitoring requires repeated imaging assessment, but accurate interpretation of WCE and intestinal ultrasound depends heavily on scarce subspecialty-trained experts.This dependence is especially consequential in resource-limited and non-tertiary settings, where it delays diagnosis and monitoring.Source: Introduction · p. 1
  • VLMs face domain-reasoning gaps, hallucination and contradictory rationales, scarce annotated IBD images, and limited interpretability for fine-grained diagnosis.Inference-time adaptation methods avoid retraining but do not necessarily provide reasoning that can be inspected or verified.Source: Introduction · p. 1
  • Existing solver-backed neurosymbolic methods target text reasoning, while final-task-only evaluations obscure failures in multi-stage medical-agent reasoning.The stated gap is a few-shot approach with image-grounded, internally verified reasoning open to expert inspection before deployment.Source: Introduction · p. 1
  • ARISE adapts a frozen VLM through five stages: hypothesis generation, image-grounded evidence summarization, refinement, symbolic verification and repair, and final diagnosis.Each stage yields an inspectable artifact and score; the framework is evaluated on WCE for Crohn’s disease and B-mode ultrasound for ulcerative colitis.Source: Introduction · pp. 1–2
  • ARISE’s contributions include few-shot hypothesis-based adaptation, solver-backed verification, stage-level evaluation, and testing across independent cohorts and multiple VLM families.The authors report improved diagnostic performance across modalities and model backbones.Source: Introduction · p. 2

Methodology

ARISE adapts a frozen VLM by maintaining an auditable hypothesis state rather than updating model parameters. Its workflow turns task and clinical knowledge into hypotheses, grounds and refines them with support images, verifies them symbolically, and uses the verified state for diagnosis and workflow-aware scoring.

Methodology

  • ARISE adapts a frozen VLM to few-shot IBD assessment without fine-tuning, storing adaptation in an auditable hypothesis state and scoring process and outcome.The workflow carries explicit state across stages, invokes a symbolic tool, and emits an inspectable artifact at each stage.Source: Methodology · p. 2

Problem Formulation

  • The problem formulation pairs input images with diagnostic labels, uses a small labeled support set and disjoint query set, and defines task and clinical-cue descriptions.The agent state contains hypotheses, their logical premises, accumulated evidence, and five stage scores, aiming for verified hypotheses that support accurate query prediction.Source: Problem Formulation · p. 2

Stage S1: Hypothesis Generation

  • S1 prompts the VLM with task and clinical knowledge to generate concise, checkable diagnostic hypotheses linked to observable image findings.Sampling temperature and nucleus sampling can be varied to encourage hypothesis diversity.Source: Stage S1: Hypothesis Generation · p. 2

Stage S3: Evidence-Conditioned Refinement

  • S3 refines the initial hypotheses using accumulated image-grounded evidence, with its score measuring downstream prediction improvement on a held-out validation subset.The workflow overview places evidence summarization before refinement and symbolic consistency checking before diagnosis.Source: Stage S3: Evidence-Conditioned… · p. 3, Figure 1 · p. 3

Stage S4: Symbolic Verification

  • S4 translates refined hypotheses into first-order logic and uses an SMT solver to detect conflicts, return a minimal conflicting set, and support hypothesis repair.Its score evaluates translation fidelity through bidirectional entailment between original and reconstructed hypotheses.Source: Stage S4: Symbolic Verification · p. 3

Stage S5: Final Diagnosis

  • S5 classifies query images using the verified hypotheses and reports the hypothesis supporting each prediction; its stage score checks label validity and grounding.The task score averages accuracy, F1, precision, and recall; workflow stages are aggregated into an agentic score, with grounding weighted most heavily.Source: Stage S5: Final Diagnosis · p. 3, Workflow-Aware Scoring: Agentic + Task · p. 4

Experiments

The experiments assess ARISE on Crohn’s disease WCE classification and ulcerative-colitis ultrasound stratification, using multiple VLMs and evaluating both diagnostic performance and workflow stages.

Datasets and Tasks

  • The evaluation covers three Crohn’s disease WCE tasks and ulcerative-colitis bowel-wall stratification (preserved vs. lost).The Crohn’s tasks range from pathological-image classification to ulceration diagnosis and six-lesion classification; the ultrasound target is a clinically relevant inflammation marker.Source: Datasets and Tasks · p. 4

Models and Prompting

  • Four instruction-following VLMs from GPT, Gemini, and Qwen are evaluated on identical query splits, with Qwen2-VL serving as the openweight baseline.A reasoning-oriented language model generates hypotheses and logical premises, and Z3 performs symbolic contradiction detection.Source: Models and Prompting · p. 4
  • The experiments compare base VLMs with their ARISE versions on diagnostic metrics, then assess stage-level agentic scores and overall task scores across models.The Crohn’s WCE accuracy comparisons span three tasks; the ultrasound comparison evaluates colon-wall stratification.Source: Models and Prompting · p. 4, Table 1 · p. 4, Table 2 · p. 4
  • The authors frame the joint evaluation as testing verifiable reasoning while making process-level reasoning quality and its relation to diagnostic accuracy visible.This is the stated purpose of evaluating both workflow and task-level performance.Source: Models and Prompting · p. 4

Results

ARISE improves diagnostic performance over base VLMs on Crohn’s disease WCE tasks and ulcerative-colitis ultrasound, while stage-level results show that evidence-guided refinement is a key difficulty.

Diagnostic Performance: Base VLM vs. ARISE

  • ARISE improves accuracy for every model across the three Crohn’s WCE tasks, with larger gains on harder tasks and weaker baselines; it also improves over all baselines on noisy ulcerative-colitis ultrasound.The results suggest gains from structuring and verifying reasoning rather than relying on a particular backbone’s raw visual capability.Source: Diagnostic Performance: Base VLM vs.… · p. 4

Stage-Level Reasoning Quality

  • Hypothesis generation and symbolic verification are consistently strong, while evidence-guided refinement is the most challenging stage and varies most across models.Stronger models also perform better during refinement and final diagnosis.Source: Stage-Level Reasoning Quality · p. 5

Agentic and Task-level scores across models

  • Stronger agentic performance generally accompanies stronger diagnostic performance, but high scores on structured stages do not guarantee correct final classification.The differences between models are largest in evidence-guided refinement and final diagnosis, exposing a process–outcome gap.Source: Agentic and Task-level scores across… · p. 5

Qualitative Visual Grounding

  • In a representative ultrasound case, GPT-5.4 gives an incorrect diagnosis when queried directly, whereas ARISE grounds its interpretation in concrete observations and predicts the correct “lost” label.The example attributes this shift to evidence summarization and symbolic verification guiding the model away from ambiguous visual claims.Source: Qualitative Visual Grounding · p. 5

Path to Deployment

ARISE is framed as clinician-in-the-loop decision support, with interpretable outputs that clinicians can inspect or override. The paper outlines a staged path from retrospective validation to clinical pilots and possible broader deployment.

Path to Deployment

  • The current evaluation spans 3,498 Crohn’s disease WCE images and B-mode intestinal ultrasound data from C-TRUS, annotated by an IU School of Medicine expert.Clinical experts informed the visual cues, decision boundaries, and interpretation guidelines used by ARISE.Source: Path to Deployment · p. 5
  • The planned retrospective validation will use data from approximately 40 adults with Crohn’s disease, comparing predictions with expert annotations and analyzing stage-level errors across severity, patient variation, image quality, and hardware.For endoscopy cases, ARISE will also estimate SES-CD from AI-identified ulcer and narrowing features and compare scores with expert assessments.Source: Path to Deployment · p. 5
  • After retrospective validation, the plan moves from shadow-mode comparison with specialists to a clinician-in-the-loop pilot where specialists can accept, question, or override outputs.The pilot is intended to assess where explanations are clinically useful or insufficient.Source: Path to Deployment · p. 5
  • A small case study will test whether visual-cue explanations improve nurses’ and generalist clinicians’ interpretation and accuracy relative to unaided assessment; successful prospective evaluation would support multi-hospital testing.Figure 6 illustrates an ultrasound decision-support interface designed to make outputs accessible and interpretable to nurses.Source: Path to Deployment · p. 5, Figure 6 · p. 6

Conclusion

ARISE improved diagnostic metrics across two independent IBD cohorts and three VLM families, while stage-level scoring exposed gaps between intermediate reasoning and final accuracy.

Conclusion

  • Across two cohorts and three VLM families, ARISE improved accuracy, F1, precision, and recall over baselines on every model–task combination.The largest gains occurred on the hardest task and weakest baseline model.Source: Conclusion · p. 6
  • Stage-level scoring exposed cases where strong intermediate reasoning did not fully carry through to final accuracy.Source: Conclusion · p. 6
  • ARISE stores each reasoning stage as an inspectable artifact, supporting auditability beyond raw accuracy.The authors identify future work on using agentic scores to improve the model and on deployment in real-world clinical settings.Source: Conclusion · p. 6

A. Stage Scoring Definitions and Equations

The appendix defines stage scores for hypothesis coverage, image-grounded evidence, refinement, symbolic translation, and final diagnosis, alongside separate task and workflow-level scores.

Stage Scoring Definitions and Equations

  • The notation defines H(0) as the initial hypothesis set, E* as accumulated image-grounded evidence, H* as the final verified set, and V as the controlled clinical vocabulary.It also defines Γ as the natural-language-to-FOL parser, Φ = Γ(H) as logical premises, ŷ as the predicted label, and ĥ as its supporting hypothesis.Source: A. Stage Scoring Definitions and… · p. 8
  • S1 scores hypothesis generation with a three-item rubric, including having at least one hypothesis and coverage of every label; both conditions yield s1 = 1.The supplied passage identifies the rubric and its best-score conditions but does not include the rubric's other item.Source: A. Stage Scoring Definitions and… · p. 8
  • S2 compares agent-cited findings with expert-marked label-relevant findings, reporting precision and recall separately and weighting fidelity above completeness.Observability is the fraction of claims that reference a concrete vocabulary finding rather than an abstract inference.Source: A. Stage Scoring Definitions and… · p. 8
  • S3 measures the downstream prediction-accuracy improvement before versus after refinement against ground truth on a 50-observation validation subset.Source: A. Stage Scoring Definitions and… · p. 8
  • S4 evaluates parser translation fidelity with an NLI model by taking the minimum of the two mutual-entailment probabilities between the original and reconstructed hypotheses.The passage names DeBERTa-v3-large as the NLI model.Source: A. Stage Scoring Definitions and… · p. 8
  • The final process score checks label-space validity and support from a verified hypothesis, while task score averages accuracy, F1, precision, and recall.The five stage scores are combined using weights reported in the main manuscript, with the greatest weight assigned to image grounding.Source: A. Stage Scoring Definitions and… · pp. 8–9

B. Datasets and experimental protocol

The study uses expert-reviewed WCE images for Crohn’s disease and B-mode ultrasound images for ulcerative colitis, with a filtered subset for the colon-wall task.

Datasets and experimental protocol

Experimental Protocol

The protocol compares each backbone’s query-only baseline with the same model after ARISE, then evaluates task performance and reasoning-stage scores.

Experimental Protocol

  • Each backbone is evaluated first in a query-only setting on the held-out test split, then on the same query split after ARISE constructs verified hypotheses.Source: Experimental Protocol · p. 9
  • Task-level comparisons report accuracy, F1 score, precision, and recall for Base and ARISE across all four clinical tasks.Source: Experimental Protocol · p. 9
  • Stage scores S1–S5 are compared across four VLMs on the Crohn’s disease six-lesion classification task.S3 uses a separate 50-example validation subset.Source: Experimental Protocol · p. 9
  • Weighted AGENTIC scores are compared with TASK scores to identify cases where a well-formed reasoning process does not fully translate to correct final classification.Source: Experimental Protocol · p. 9

C. Results for Crohn’s disease

The Crohn’s disease results cover binary WCE classification and six-lesion diagnosis, with full task-level metrics reported across the evaluated settings.

C. Results for Crohn’s disease

  • Full task-level metrics for each backbone are reported for the query-only Base setting and within ARISE.The supplied passages do not provide the metric values.Source: C. Results for Crohn’s disease · p. 9

Prompt A: Normal vs. Pathological WCE Classification

  • Prompt A assigns each WCE frame to normal or pathological, labeling it pathological when a definite abnormality is visible.The ARISE prompt adds refined hypotheses as supporting rules and requests the predicted class with a supporting-hypothesis ID when available.Source: Prompt A: Normal vs. Pathological WCE… · p. 9, Table A2 · p. 10

Prompt B: Six-Lesion WCE Classification

Support and In-Context Examples

ARISE uses selected labeled image examples as multimodal demonstrations, while explanatory text stored with each example is used during final diagnostic evaluation.

Support and In-Context Examples

  • Support examples are selected for each task, and ARISE uses manually selected labeled image examples as multimodal in-context demonstrations.Each demonstration contains an example image and its class label; accompanying explanatory text is used in final diagnostic evaluation.Source: Support and In-Context Examples · p. 10
  • GPT and Gemini use four Prompt A examples, while Qwen2-VL-2B uses two; for Prompt B, GPT permits up to 12, Gemini uses four, and Qwen uses two.Qwen uses fewer examples to reduce GPU memory consumption on a Colab T4.Source: Support and In-Context Examples · p. 10
  • The Crohn’s disease WCE evaluation includes a six-lesion classification task.Source: Table A4 · p. 10

Related papers

Loading 2610.04777v1…