PaperlayerView exact revision on arXiv

Source-linked AI summary

Ophiuchus: Incentivizing Tool-augmented "Think with Images" for Joint Medical Segmentation, Understanding and Reasoning

Yankai Jiang, Yujie Zhang, Peng Zhang, Wenjie Li, Yichen Li, Jintai Chen, Xiaoming Shi, Shihui Zhen

arXiv:2512.14157v2cs.AIcs.CV

TL;DR

Medical MLLMs still struggle to focus dynamically on fine-grained visual regions during reasoning. Ophiuchus interleaves tool-guided visual inspection with multimodal reasoning, and the authors report it exceeds SOTA MLLMs across tasks. Its reported average improvement over medical-specific and medical reasoning MLLMs is at least 10.8%.

  • Problem

    Medical MLLMs rely on static, global perception and text-only intermediate reasoning, often missing fine-grained visual evidence and lacking a look-again mechanism.

    Source: §1 · p. 1

  • Method

    Ophiuchus adaptively invokes external visual tools to inspect localized image regions and incorporate their evidence into interleaved multimodal reasoning.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Ophiuchus significantly exceeds SOTA MLLMs across all tasks; against medical-specific and medical reasoning MLLMs, it reports at least a 10.8% average improvement.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    The three-stage training framework successfully cultivates sophisticated reasoning patterns, including tool orchestration and multimodal chain-of-thought capabilities.

    Source: §5 · p. 9

    Agentic RL with multi-turn rollouts, tool execution, and verifiable rewards is compute-intensive, and generating reasoning trajectories consumes substantial API resources.

    Source: G. Limitations · p. 22

Abstract

from arXiv · show

Recent medical MLLMs have made significant progress in generating step-by-step textual reasoning chains. However, they still struggle with complex clinical tasks that necessitate dynamic and iterative focusing on fine-grained visual regions. To close this gap, we introduce Ophiuchus, a versatile, tool-augmented framework that equips an MLLM to (i) decide when fine-grained visual evidence is needed, (ii) determine where to probe and ground within the medical image, and (iii) seamlessly weave the relevant sub-image content back into an interleaved, multimodal chain of thought for precise segmentation and diagnosis. Ophiuchus moves beyond mere tool-calling by tightly fusing the MLLM's inherent grounding and reasoning capabilities with external tools, enabling more accurate and trustworthy decisions. The core of our method is a three-stage training strategy: cold-start SFT for basic tool selection; self-reflection fine-tuning to strengthen decision revision; and agentic tool reinforcement learning to elicit sophisticated, expert-like diagnostic behaviors. Extensive experiments show that Ophiuchus consistently outperforms both closed-source and open-source SOTA methods across diverse medical benchmarks, including VQA, detection, and reasoning-based segmentation. Our project code is available at https://github.com/SII-zyj/Ophiuchus.

1. Introduction

Medical MLLMs can miss fine-grained visual evidence while reasoning in text; Ophiuchus interleaves tool-guided local inspection with multimodal reasoning and reports state-of-the-art performance across medical benchmarks.

1. Introduction

  • Medical MLLMs often miss fine-grained evidence and lack an adaptive look-again mechanism, limiting their ability to revise predictions as new visual evidence emerges.Source: §1 · p. 1
  • A 64k-sample dataset with grounded-region annotations and tool-invocation trajectories supports cold-start SFT, self-reflection fine-tuning, and agentic tool reinforcement learning.Source: §1 · p. 2
  • Ophiuchus significantly exceeds SOTA MLLMs across all tasks and establishes new state-of-the-art performance across multiple medical benchmarks.Source: §1 · p. 2
  • Ophiuchus adaptively integrates localized visual evidence into an evolving reasoning chain through tool-augmented interleaved multimodal reasoning.Source: §1 · p. 2

2. Related Works

Related work spans medical MLLMs with region-level or pixel-level perception and agents that invoke external medical tools. Ophiuchus contrasts with approaches constrained by fixed behaviors or workflows.

2. Related Works

  • Medical MLLMs cover diverse imaging tasks, while ROI supervision and segmentation-based pixel understanding target fine-grained perception.Source: §2 · pp. 2–3
  • Medical MLLMs address region-level perception with ROI supervision and pixel understanding with segmentation, but these approaches can have limited grounding accuracy and mask precision.Source: §2 · p. 3
  • Prior medical agents invoke specialized tools, but fragmented or predefined workflows can constrain coherent planning, generalization, and recovery from tool failures.Source: §2 · p. 3
  • Ophiuchus uses three-stage training for tool orchestration and iterative reasoning over tool outputs, contrasting with fixed perception behaviors and predefined workflows.Source: §2 · p. 3

3. Methods

Ophiuchus interleaves medical-image reasoning with adaptive tool use, and trains this behavior through cold-start supervision, self-reflection, and reinforcement learning. Its curated data pairs fine-grained visual questions with tool-use trajectories and human-checked answers.

3.1. Ophiuchus

  • Ophiuchus interleaves thoughts, tool calls, and observations, allowing later calls to target earlier outputs; BiomedParse segments from text prompts, while zoom exposes crops and object contours.The agent can answer directly or invoke a tool after its current reasoning step.Source: §3.1 · p. 3

3.2. Training Framework

  • Training progresses from cold-start SFT on ground-truth reasoning and tool-call trajectories, through self-reflection fine-tuning on revised trajectories, to agentic tool reinforcement learning.Self-reflection cases pair trajectories with different tool sequences where answer correctness changes between checkpoints.Source: §3.2 · p. 4
  • ATRL combines format, answer-quality, and conditional tool-use rewards, then applies GRPO to encourage effective tool use and expert-like reasoning.The tool-use bonus requires both a correct answer and at least one external perception-tool call; GRPO uses normalized advantages to distinguish trajectory quality.Source: §3.2 · p. 5

3.3. Dataset Curation

  • The curated dataset draws on biomedical segmentation and lesion-report data, selecting questions where fine-grained localization is needed and filtering for format and answer correctness.The resulting 64k samples are human-cross-validated and split into 30k cold-start, 30k RL, and 4k in-domain test samples.Source: §3.3 · pp. 5–6

4. Experiments

Experiments evaluate Ophiuchus on medical VQA, segmentation, and tool-use behavior, using public benchmarks and a held-out in-house set. Results favor Ophiuchus across the reported comparisons, while ablations examine its training stages, reward design, and iterative tool use.

4.1. Experiment Settings

  • The evaluation covers public medical VQA and reasoning benchmarks, in-domain tests, and a held-out in-house dataset of 1k CT, MRI, and pathology images across seven cancer types.Multiple-choice VQA uses accuracy, while segmentation-answer subsets use DSC; training uses Qwen2.5-VL-7B and three stages.Source: §4.1 · p. 6

4.2. Main Results

  • Across eight medical VQA benchmarks, Ophiuchus consistently outperforms image-zoom MLLMs and tool-using agents, with at least a 10.8% average improvement over medical-specific and medical-reasoning MLLMs.The evaluation also reports substantially better performance than general-purpose SOTA MLLMs.Source: §4.2 · pp. 6–7
  • On In-house-Seg, Ophiuchus improves over prior work by at least 15.2% and achieves the best In-domain-Seg performance.Figure 3 compares performance across both segmentation datasets and includes task-specific SOTA BiomedParse for reference.Source: §4.2 · p. 7, Figure 3 · p. 7

4.3. Ablation Studies

  • The three-stage training recipe improves over the base model: RL yields the largest gains, and combining it with cold-start SFT and self-reflection maximizes benefits.Self-reflection improvements are attributed to fine-tuning on reflection data rather than merely increasing standard SFT epochs; training behavior expands early, then contracts as RL consolidates tool use.Source: §4.3 · p. 8
  • On cases the base model cannot answer without tools, Ophiuchus invokes tools and answers correctly on 93.7% on average; when tools are unnecessary, its average call rate is 3.3%.All benchmarks exceed 90% on necessary tool-use cases.Source: §4.3 · p. 8
  • The conditional tool-use reward yields the highest accuracy, while removing it causes a substantial performance drop.The reward is conditional on both a correct answer and tool use; unconditional tool-use rewards perform worse.Source: Figure 4 · p. 7, §4.3 · p. 8
  • Static segmentation or cropping workflows, including ground-truth crops, remain below Ophiuchus, whose gains are attributed to iterative evidence selection and integration.The discussion emphasizes tool quality and complementarity over raw tool count, noting that SAM 2, BioMedParse, and zoom-in provide broad fine-grained analysis functionality.Source: §4.3 · p. 9, §4.4 · p. 9

5. Conclusion

Ophiuchus is a medical MLLM agent that interleaves multimodal reasoning with external tools for fine-grained visual perception. Its three-stage training framework cultivates sophisticated reasoning patterns, and the authors propose it as a foundation for medical AI agents that think with images using tools.

5. Conclusion

  • Ophiuchus interleaves multimodal reasoning with external tools for fine-grained visual perception and expert-like diagnostic behavior.The authors describe advantages in tool orchestration, generalization, and multimodal chain-of-thought capabilities.Source: §5 · p. 9

Impact Statement

The study uses publicly available training datasets and reports a retrospective in-house test study approved by a hospital ethics committee, with informed consent waived. Personally identifiable information and private details were removed.

Impact Statement

  • The retrospective in-house test study was ethics-approved, consent-waived, and had personally identifiable information and private details removed.The work reports no human-subject or animal-testing experiments and uses publicly available training datasets.Source: Impact Statement · p. 10

A.1. Details of our Curated Datasets

The appendix describes how Ophiuchus’s curated data and evaluation sets are assembled and verified, and reports high tool-invocation accuracy. It also specifies the training objective and rewards used to supervise tool use.

A.1. Details of our Curated Datasets

  • The 64k-sample VQA dataset combines fine-grained mask and description sources, prioritizes tool-beneficial cases, and includes some questions answerable without tools.Samples are cross-validated against human annotations and span nine medical imaging modalities.Source: §A.1 · p. 13
  • The 64k VQA samples are split into 30k cold-start SFT, 30k reinforcement-learning, and 4k in-domain test examples; 3k cases form the self-reflection set.The dataset covers recognition, localization, lesion assessment, counting, and segmentation tasks.Source: §A.1 · pp. 13–14

A.2. Details of the In-House Testset

  • Evaluation includes an in-house held-out set of 1k CT, MRI, and histopathology images across seven cancer types, with questions divided into VQA and segmentation subsets.The paper also describes six public benchmarks, including VQA, classification-style QA, and higher-level reasoning datasets.Source: §A.2 · p. 14, §A.3 · p. 14

A.4. Filtering and Verification Pipeline for QA Pairs and Reasoning Traces

  • QA generation is constrained to convert image, ground-truth mask, and class-label information into questions without adding information.GPT-5 checks answer correctness and clinical wording, while target-region grounding and image answerability are also specified as verification criteria.Source: §A.4 · p. 14, §3 · pp. 14–15

5. Answer Correctness

  • Reasoning traces are checked for interleaved thought, tool-call, and observation steps, justified tool choices, observation integration, and clinical consistency.Tool calls are executed and their outputs checked against structured annotations; retained data are additionally cross-verified by human annotators.Source: §5 · p. 15

B. Cold-Start SFT Objective

  • Cold-start SFT minimizes average negative log-likelihood over reasoning and tool-call tokens, conditioned on the image, question, and prior trajectory.The passage identifies the ground-truth answer and total reasoning-and-tool-call token count in the objective description.Source: B. Cold-Start SFT Objective · p. 15

C. Formula Definition of Reward Signals

  • The strategic tool-use reward grants a bonus only when a trajectory invokes an external perception tool and produces a correct answer.The reward is intended to discourage arbitrary or redundant tool use.Source: C. Formula Definition of Reward Signals · p. 16

D.1. The accuracy rate of tool invocation

  • 97.9% average tool-use accuracy, versus 25.1% for the prompt-driven baseline, across VQA and segmentation settings.The measure jointly checks format adherence and correct tool invocation on in-domain and in-house data.Source: §D.1 · p. 16

D.2. Ablation on Tool Quantity and Type

The ablations report stronger performance with broader and complementary tool access, while Ophiuchus also generalizes to unseen segmentation tools. Its performance is not bounded by the standalone tools it uses.

D.2. Ablation on Tool Quantity and Type

  • 57.6 average with zoom-in alone exceeds zoom-in-only baselines near 53.8, indicating strong performance even when only local magnification is available.On VQA-RAD, the reported score is 68.0 versus 65.7–66.0 for prior work.Source: §D.2 · p. 17
  • 62.8 average with unseen MedSAM exceeds MMedAgent’s 50.3, showing training-free use of a tool not introduced during development.Ophiuchus scores 69.5 versus 64.0 on VQA-RAD and 63.9 versus 52.3 on In-House-Seg.Source: §D.2 · p. 17
  • 68.1 average is achieved with all three tools, the best overall setting; combining tool types also improves performance across VQA and segmentation.BiomedParse alone averages 64.6, while zoom-in plus BiomedParse averages 66.8 and the two segmentation tools together average 65.5.Source: §D.2 · p. 17

D.3. Impact of Changing Segmentation Tools on the Agent’s Performance.

  • Replacing SAM2 with other SAM-based tools causes only minor performance variations, and more capable segmentation tools yield better segmentation performance.The alternatives are used zero-shot because they share similar functionality and input parameter formats.Source: §D.3 · p. 17, Table 9 · p. 18

D.4. Do the Tool Performances Constitute an Upper Limit for Ophiuchus?

  • Ophiuchus surpasses the standalone tools and can switch tools when an output is unreliable or compose tools such as zoom-in and SAM2.The comparison includes individual tools and tool ensembles; SAM2 receives ground-truth boxes because it cannot process text queries directly.Source: §D.4 · p. 18, Table 10 · p. 18

D.5. Comparison Using the Same External Tools and Training Data

Under a comparison giving baselines the same tools and training data, existing methods still fall short of Ophiuchus. The analysis also reports that Ophiuchus rarely calls tools on questions the base model can answer unaided.

D.5. Comparison Using the Same External Tools and Training Data

  • Existing baselines remain significantly behind Ophiuchus when given the same external tools and, for open-source methods, the same three-stage training data.The authors attribute the gains primarily to integrating tool invocation and visual evidence into the model’s chain of thought, rather than tool access or externally constructed training data alone.Source: §D.5 · p. 18, Table 11 · p. 19

D.6. Analysis of Over-Calling of Tools.

  • Ophiuchus rarely uses tools on cases that Qwen2.5-VL-7B answers correctly without them, supporting selective tool invocation rather than routine over-calling.The analysis counts tool use among base-model-correct cases across benchmarks.Source: §D.6 · p. 19, Table 12 · p. 19

D.7. Stability Analysis of Training.

The reported training curves show stable, consistent SFT and RL improvements, and changes to the tested RL hyperparameters produce only minor performance fluctuations.

D.7. Stability Analysis of Training.

  • SFT loss, RL reward components, and the final-answer IoU curve show stable and consistent training improvements.The plotted curves include overall reward, reasoning-format reward, and strategic tool-use reward.Source: §D.7 · p. 19
  • Changes to learning rate and rollout count lead to only minor performance fluctuations, while the default settings achieve the best performance.The hyperparameter ablation evaluates eight VQA benchmarks.Source: §D.7 · p. 20, Table 13 · p. 20

D.8. Generalization to Report Generation Tasks.

On MIMIC-CXR report generation, Ophiuchus outperforms general-purpose and medical MLLMs on finding and impression generation; the authors attribute this to improved fine-grained visual understanding.

D.8. Generalization to Report Generation Tasks.

  • On MIMIC-CXR, Ophiuchus is evaluated for both finding and impression generation using BERTScore, BLEU, ROUGE-1, ROUGE-2, and ROUGE-L.The results are reported in Tables 14 and 15, respectively.Source: §D.8 · p. 20
  • Ophiuchus achieves superior performance on both report-generation tasks compared with general-purpose large-scale MLLMs and existing medical MLLMs.The passage does not provide metric values.Source: §D.8 · p. 20
  • The authors attribute this improvement to enhanced understanding and identification of fine-grained visual cues needed for clinically meaningful report generation.Source: §D.8 · p. 20

E.1. Successful Emergent Thinking Patterns

The section describes three successful tool-use patterns, a hard failure case, self-reflection behavior, and prompts used to construct and validate VQA data and reasoning trajectories.

E.1. Successful Emergent Thinking Patterns

  • Visual-cue search uses segmentation tools to propose regions and zooms in when global observation is insufficient; in the example, this confirms two polyps.Ophiuchus applies BiomedParse to propose candidate regions, then zooms into each for morphology and boundary inspection.Source: §E.1 · p. 20
  • Visual confirmation uses regional tool observations and zoomed details to resolve uncertainty; in the CT example, local thickening and luminal irregularity confirm the suspected finding.The model first down-weights pancreas, spleen, and renal options, then inspects a targeted region.Source: §E.1 · p. 20
  • Tool-guided re-examination can mitigate hallucination; in the liver-lesion example, zooming into a BiomedParse mask corrects an initial impression of heterogeneous attenuation.The close view shows a well-circumscribed round hypodense mass with homogeneous attenuation.Source: §E.1 · p. 21

E.2. Failure Cases

  • In an ultra-low-signal case, the tiny pancreatic lesion is missed by BiomedParse, and Ophiuchus rejects the tool result but still gives the wrong final answer.The trajectory shows uncertainty localization and tool-output checking, but neither the model nor tool succeeds in this case.Source: §E.2 · p. 21

E.3. Self-Reflection Examples

  • In the self-reflection example, Ophiuchus identifies a likely BiomedParse error, reexamines the CT, switches to SAM2 with a bounding box, and confirms a pancreatic cyst.Source: §E.3 · p. 21

F.1. Prompt for VQA Construction and Verification

  • VQA construction prompts generate and verify question-answer pairs, and the same template design is used for segmentation-task data.The prompts are described as supporting dataset diversity and reliability.Source: §F.1 · p. 21

F.2. Prompt for Reasoning Trajectory Construction

  • Reasoning trajectories are generated with ground-truth clinical metadata and tool observations, then verified for format, tool correctness, evidence integration, clinical soundness, and exact answer match.The prompt instructs GPT-5 to treat the answer as unknown during reasoning and reveal it only at the end.Source: §F.2 · p. 22

F.3. System Prompt and User Prompt

  • The experiments use separate system-prompt and user-prompt templates.Source: §F.3 · p. 22

G. Limitations

The authors identify training and inference cost and the system’s 2D-centric design as limitations, while proposing broader tools and volumetric and temporal modalities as future directions.

G. Limitations

  • Agentic RL with multi-turn rollouts, tool execution, and verifiable rewards is compute-intensive, and reasoning-trajectory generation consumes substantial API resources.Source: G. Limitations · p. 22
  • The present agent system is 2D-centric, limiting performance on tasks that depend on 3D topology and motion.Source: G. Limitations · p. 22

H. Future Work

  • The authors plan to broaden the medically oriented toolset and extend the framework to volumetric and temporal modalities, including MRI, CT, and video.Source: H. Future Work · p. 22
  • Ophiuchus can combine zoom-in with SAM 2 to improve segmentation over using SAM 2 alone.Source: Figure 14 · p. 29
  • Even when segmentation tools fail, Ophiuchus can answer correctly by using gathered visual cues to revise its decision.Source: Figure 15 · p. 30
Loading 2512.14157v2…