PaperlayerView exact revision on arXiv

Source-linked AI summary

IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation

Yankai Jiang, Qiaoru Li, Binlu Xu, Haoran Sun, Chao Ding, Junting Dong, Yuxiang Cai, Xuhong Zhang, Jianwei Yin

arXiv:2601.03054v4cs.CVcs.AI

TL;DR

Medical MLLMs have limited pixel-level comprehension, and existing segmentation methods often rely on implicit tokens or single-pass reasoning. IBISAgent interleaves reasoning and click actions with segmentation tools in a multi-step process, trained with SFT and fine-grained-reward RL. It reports substantially higher segmentation scores than existing medical-specific MLLMs, with added inference time from iterative refinement.

  • Problem

    Medical MLLMs are largely confined to image-level understanding, while biomedical segmentation needs fine-grained pixel-level reasoning and iterative refinement.

    Source: §1 · pp. 1–2

  • Method

    IBISAgent interleaves reasoning and text-based clicks with segmentation-tool calls in a multi-step process, trained with cold-start SFT and agentic RL.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    IBISAgent surpasses existing medical-specific MLLMs by at least 35.13% in IoU, 37.58% in DSC, and 29.79% in F1 score.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    IBISAgent achieves competitive performance across diverse segmentation tasks while supporting pixel-level reasoning without additional model components.

    Source: §5 · p. 8

    Multi-round mask refinement makes IBISAgent slower per case than existing MLLMs, trading additional computation time for improved performance.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Recent research on medical MLLMs has gradually shifted its focus from image-level understanding to fine-grained, pixel-level comprehension. Although segmentation serves as the foundation for pixel-level understanding, existing approaches face two major challenges. First, they introduce implicit segmentation tokens and require simultaneous fine-tuning of both the MLLM and external pixel decoders, which increases the risk of catastrophic forgetting and limits generalization to out-of-domain scenarios. Second, most methods rely on single-pass reasoning and lack the capability to iteratively refine segmentation results, leading to suboptimal performance. To overcome these limitations, we propose a novel agentic MLLM, named IBISAgent, that reformulates segmentation as a vision-centric, multi-step decision-making process. IBISAgent enables MLLMs to generate interleaved reasoning and text-based click actions, invoke segmentation tools, and produce high-quality masks without architectural modifications. By iteratively performing multi-step visual reasoning on masked image features, IBISAgent naturally supports mask refinement and promotes the development of pixel-level visual reasoning capabilities. We further design a two-stage training framework consisting of cold-start supervised fine-tuning and agentic reinforcement learning with tailored, fine-grained rewards, enhancing the model's robustness in complex medical referring and reasoning segmentation tasks. Extensive experiments demonstrate that IBISAgent consistently outperforms both closed-source and open-source SOTA methods.

1. Introduction

Medical MLLMs remain largely focused on image-level understanding, while biomedical segmentation requires fine-grained pixel-level reasoning. IBISAgent uses iterative tool-mediated reasoning and two-stage training to address this gap.

1. Introduction

  • Existing medical segmentation MLLMs use implicit segmentation tokens and jointly fine-tune pixel decoders, risking catastrophic forgetting and weak cross-domain generalization.The tokens can also disrupt natural text output and weaken reasoning.Source: §1 · p. 2
  • Biomedical images can contain subtle cues, making a single forward pass inadequate; human annotators instead refine masks iteratively with positive and negative clicks.The paper asks whether MLLMs can revisit images, reevaluate intermediate decisions, and adapt to feedback.Source: §1 · p. 2
  • IBISAgent treats segmentation as a multi-step decision process, interleaving reasoning and text-based clicks with segmentation-tool calls to refine masks.It preserves language representations and supports mask refinement through plug-and-play tools.Source: §1 · p. 2
  • A 400K+ sample dataset of step-wise reasoning, click trajectories, and masks supports cold-start SFT followed by RL with fine-grained rewards.The rewards include region-based click placement and progressive segmentation improvement.Source: §1 · p. 2
  • IBISAgent significantly exceeds state-of-the-art MLLMs across held-in and held-out biomedical object referring and segmentation tasks.The evaluation spans in-domain and zero-shot scenarios.Source: §1 · p. 2

2. Related Work

Prior medical segmentation MLLMs typically rely on implicit segmentation tokens and decoder fine-tuning, while tool-augmented approaches also tend to use single-turn reasoning. IBISAgent instead frames segmentation as iterative, multi-step refinement.

Pixel-Level Understanding in Medical MLLMs

  • LISA-style medical MLLMs project special-token embeddings through vision decoders, requiring task-specific tokens and joint fine-tuning that can disrupt text output and risk forgetting.These approaches also lack autonomous multi-step mask refinement.Source: §2 · pp. 2–3

MLLMs with Segmentation Tools

  • Tool-based methods prompt segmentation models with predicted boxes or points, but their single-turn grounding can struggle to localize targets in complex scenarios.The passage describes this limitation in prior work.Source: §2 · p. 3
  • IBISAgent formulates segmentation as a multi-step decision process with iterative mask refinement and step-wise segmentation-quality feedback.The passage states that this formulation mitigates error accumulation and improves performance, stability, and generalization.Source: §2 · p. 3

3. Method

IBISAgent is a multimodal agent for biomedical segmentation and mask refinement, trained to reason over pixel-space feedback and invoke segmentation tools. Its method combines iterative interaction, cold-start supervised learning, and agentic reinforcement learning.

3. Method

  • IBISAgent generates adaptive spatial prompts and invokes segmentation tools for biomedical segmentation and mask refinement.The paper attributes this capability to native visual grounding and action planning, enhanced through SFT and RL.Source: §3 · p. 3

3.1. Architecture Overview.

  • At each interaction step, the agent combines textual thinking, a click or final-answer action, and a segmentation observation fed back into its reasoning.Clicks specify a target, positive or negative attribute, and relative image coordinate; observations use prior masks and the original image.Source: §3.1 · p. 3

3.2. Cold-Start Supervised Fine-Tuning

  • Cold-start SFT uses simulated click trajectories from existing masks, then filters samples and adds generated questions, answers, reasoning, and self-correction examples.The resulting 456K-sample dataset includes gold-standard and error-induced self-correction trajectories; loss masking excludes executed segmentation outputs and designated erroneous actions.Source: §3.2 · pp. 4–5

3.3. Agentic Reinforcement Learning

  • RL trains on 888K VQA samples without click trajectories or reasoning annotations, encouraging the agent to explore actions beyond SFT imitation.Its reward combines format validity, final-answer quality, click placement, progressive segmentation improvement, and trajectory length.Source: §3.3 · p. 5

4. Experiment

IBISAgent is evaluated across in-domain, out-of-domain, and held-out segmentation benchmarks, with comparisons and ablations examining its performance and training design. Results favor IBISAgent over medical-specific and general-purpose MLLMs, while reward ablations show gains in mask quality and interaction efficiency.

4.1. Experimental Setup

  • Evaluation covers an in-domain test set, an out-of-domain benchmark, a held-out dataset from three medical centers, and medical VQA benchmarks.Segmentation metrics are mIOU, Dice score, and F1 for mask-to-entity correspondence; implementation uses Qwen2.5-VL-7B and MedSAM2.Source: §4.1 · p. 6

4.2. Comparison with Previous SOTA Methods

  • IBISAgent performs substantially better than general-domain MLLMs across benchmarks, including after those models receive additional fine-tuning on the same training data.The reported improvements are attributed primarily to its expert-like multi-round reasoning and refinement design, not simply to specialized data.Source: §4.2 · pp. 6–7
  • For the reasoning–segmentation example, IBISAgent produces better reasoning and segmentation than existing MLLMs, whose outputs show incorrect reasoning and low-quality masks.The comparison includes examples of incorrect localization, false positives, and hallucinated objects in other models.Source: Figure 3 · p. 7, §4.2 · p. 7
  • IBISAgent surpasses medical-specific MLLMs by at least 35.13% in IoU, 37.58% in DSC, and 29.79% in F1 score, on average.The comparisons include models trained on substantially larger datasets and a tool-augmented model using MedSAM.Source: §4.2 · p. 7
  • IBISAgent consistently outperforms tool-augmented MLLM agent systems that predict boxes or points before segmentation.Competing open-source methods use official code and weights and are further fine-tuned on Dcold and Drl for a fair comparison.Source: Table 3 · p. 7, §4.3 · p. 8

4.3. Ablation Studies

  • Progressively adding the proposed training stages improves on the prompt-only baseline, and the RL stage provides the largest reported performance gain.The compared variants include cold-start SFT, SFT with synthesized self-reflective trajectories, RL alone, and combined cold-start SFT plus RL.Source: §4.3 · p. 8, Table 4 · p. 8
  • Click-placement and progressive-segmentation rewards most improve mask quality, while progressive-improvement and trajectory-length rewards improve interaction efficiency.Combining all reward components balances segmentation performance with interaction efficiency; Table 5’s gray row uses only standard format and answer rewards.Source: §4.3 · p. 8, Table 5 · p. 8

5. Conclusion

IBISAgent is an MLLM-based agent for pixel-level visual reasoning in unified biomedical object referring and segmentation. The paper concludes that its multi-step, two-stage-trained approach performs competitively across diverse segmentation tasks and motivates further work on vision-centric decision-making agents for medical image analysis.

5. Conclusion

  • IBISAgent combines multi-step segmentation reasoning, two-stage training, fine-grained rewards, and a large-scale dataset of thinking and action trajectories.The authors report competitive performance across diverse segmentation tasks and position vision-centric, multi-step agents as a direction for holistic medical image analysis.Source: §5 · p. 8

A.1. Dataset for SFT

The appendices describe datasets spanning SFT, RL, and evaluation, alongside automated click-trajectory and instruction generation for interactive segmentation.

A.1. Dataset for SFT

  • Dcold contains 47, 146 samples and 456, 795 VQA pairs, with average IoU 94.27, median IoU 95.07, average Dice 0.9703, median Dice 97.47, and 8.69 steps per sample.The trajectories are curated for high fidelity.Source: §A.1 · p. 11

A.1.1. Modality Diversity; A.1.2. Task Diversity

  • Dcold spans 9 imaging modalities and 38 segmentation tasks, including organs and tumors from MSD and specialized datasets such as ACDC, LIDC-IDRI, and GlaS.Table 6 gives the modality breakdown; task-group statistics are in Sec. E.Source: §A.1.1 · p. 11, §A.1.2 · p. 11, Table 6 · p. 12

A.2.1. Pixel-Level Reasoning Corpus for RL.; A.2.2. Commonly-Used Medical VQA Datasets.

  • The RL corpus contains 60, 826 samples and 564, 385 QA pairs across 8 modalities and 39 segmentation tasks, with images, ground-truth masks, and fine-grained localization questions.RL training also mixes in common medical VQA datasets so the policy can selectively invoke pixel-space reasoning.Source: §A.2.1 · p. 11, §A.2.2 · p. 11

A.3.1. In-domain test set Dtest

  • The disjoint in-domain Dtest has 9, 902 samples and 156, 289 VQA pairs, emphasizing challenging fine-grained targets across 32 task groups and 8 modalities.Figure 4b shows sample counts and QA-pair counts by modality.Source: §A.3.1 · p. 11, Figure 4 · p. 12

A.3.2. Out-of-domain test set MeCOVQA-G+

  • MeCOVQA-G+ provides 3, 157 expert-reviewed text–segmentation pairs across X-ray, CT, MRI, ultrasound, and endoscopy, expanding the scale and modality diversity of MeCoVQA-G [13].It is described as an out-of-domain medical text-to-segmentation benchmark.Source: §A.3.2 · pp. 12–13

A.3.3. Held-out in-house test set

  • The held-out zero-shot test set contains 1k in-house CT, MRI, and histopathology images across 7 cancer types, with a VQA pair and reasoning trajectory for each case.The cases include 400 breast cancer histopathology images and 100 cases for each of six other cancer groups.Source: §A.3.3 · p. 13

B.1. Trajectory Generation

  • Trajectory generation iteratively uses MedSAM2 [27] and simulated clicks to correct prediction errors until IoU exceeds τiou or Tmax is reached.The next click targets the center of the largest false-negative or false-positive region, while the previous mask logits and cumulative click history guide refinement.Source: §B.1 · p. 13

B.2. QA Generation

  • Instruction generation conditions prompts on image, ground-truth mask, and verified captions, with 7 initialization categories, 6 refinement categories, and varied assistant responses.Templates are instantiated with target names and imaging modalities; the pipeline also includes five clinical-uncertainty query sub-templates.Source: §B.2 · pp. 13–14

B.3. Reasoning Generation for Our SFT Dataset

The SFT reasoning traces are generated through a teacher–student pipeline that gives the teacher oracle information but requires it to reason from the student’s visible perspective. Forbidden-concept constraints mitigate leakage, and the resulting traces are described as dense and anatomically grounded without scenario-specific manual templates.

B.3. Reasoning Generation for Our SFT Dataset

  • GPT-5 serves as a teacher that synthesizes step-by-step reasoning traces for the student agent, which sees only the current segmentation mask.The student’s next action may require reasoning about false-positive and false-negative regions that it cannot directly observe.Source: §B.3 · p. 14

1. Persona and Perspective Simulation.; 2. Privileged Information.

  • The teacher is prompted to adopt an expert-radiologist persona while explaining the next step from the junior agent’s limited visual perspective.It uses an oracle image showing true positives, false positives, and false negatives to determine the correct next action.Source: §1 · p. 14, §2 · pp. 14–15

3. Preventing Information Leakage.

  • The prompt forbids privileged labels such as FP, FN, and Ground Truth, and the approach yields dense, anatomically grounded traces without manual templates for every scenario.The passage attributes leakage mitigation to GPT-5’s instruction-following capability.Source: §3 · p. 15

C. Reward Functions

IBISAgent’s reward design combines format, answer, segmentation-quality, spatial-click, and efficiency feedback. Across the supplied evaluations, it improves VQA accuracy, remains robust to segmentation-tool choice, and outperforms the compared tools in segmentation.

C. Reward Functions

  • The reward design scores valid output structure and task answers, uses MedSAM2 mask IoU for segmentation feedback, and rewards spatially valid clicks, positive IoU gains, and efficient trajectories.Segmentation IoU receives piecewise rewards, while positive clicks must fall in under-segmented regions and negative clicks in over-segmented regions.Source: C. Reward Functions · pp. 16–17

D.1. VQA Performance.

  • At least 2.4% higher average accuracy than existing medical MLLMs trained on large-scale VQA datasets.The paper reports outperformance of open-source and proprietary MLLMs across three medical VQA benchmarks.Source: §D.1 · p. 18, Table 8 · p. 18

D.2. Impact of Segmentation Tool Types

  • IBISAgent retains segmentation accuracy across MedSAM, SAM, and SAM 2 replacements, with only minor differences in final accuracy.The paper attributes this robustness to precise click locations and multi-round refinement.Source: §D.2 · p. 18, Table 9 · p. 18

D.3. The Performance of the Segmentation Tools

  • IBISAgent consistently achieves the highest performance across both segmentation modes and surpasses competing tools when initialized with ground-truth boxes.The comparison includes text-driven segmentation and a setting using a ground-truth bounding box as the first step.Source: §D.3 · p. 18, Table 10 · p. 19
  • IBISAgent’s refinement results exceed the inherent upper bound of the underlying segmentation tools themselves.The paper frames precise localization and iterative mask refinement as the basis for this result.Source: §D.3 · p. 18

D.4. More Case Studies

  • In challenging cases involving low-contrast pancreatic and irregular lung tumors, existing MLLMs often hallucinate, ground incorrectly, or miss diagnoses, whereas IBISAgent segments correctly and refines masks away from adjacent healthy tissue.Source: §D.4 · p. 18

D.5. How IBISAgent Corrects Errors

IBISAgent can detect and correct inconsistent targets or masks, and can backtrack after an erroneous action. Its IoU reward rises steadily during RL training; the system and user prompts are also presented.

D.5. How IBISAgent Corrects Errors

  • IBISAgent rejects nonexistent targets, corrects initial masks inconsistent with instructions, and can backtrack and re-plan after an incorrect action.The paper attributes backtracking behavior in part to Reflective Behavior Synthesis examples during SFT.Source: §D.5 · p. 19

D.6. RL Training Dynamics

  • The IoU reward increases steadily during RL training, which the paper presents as evidence of stable training.The curve plots IoU reward against training steps.Source: §D.6 · p. 19

E. System and User Prompts

F. Future Works

The authors identify 2D operation and interaction efficiency as open challenges. Supplementary examples illustrate tumor segmentation, iterative refinement, error correction, and RL training dynamics.

F. Future Works

  • IBISAgent currently operates primarily in 2D, while extending it to 3D or unified 2D–3D settings remains a proposed direction.Source: F. Future Works · p. 19
  • Reducing the computational overhead of multi-step agentic interaction remains an open challenge.Source: F. Future Works · p. 19
  • The pancreatic and lung tumor examples show multi-step refinement correcting errors around subtle lesions, with reported IoUs of 97.14% and 95.76%, respectively.Source: Figure 9 · p. 20, Figure 10 · p. 21
  • Figure 12 illustrates responses to nonexistent targets and mismatched initial masks, as well as backtracking after an incorrect decision.Source: Figure 12 · p. 22
  • The IoU reward curve is presented as showing stable and consistent improvement during RL training.Source: Figure 13 · p. 23

Related papers

Loading 2601.03054v4…