Source-linked AI summary
Meissa: Multi-modal Medical Agentic Intelligence
Yixiong Chen, Xinyi Bai, Yue Pan, Zongwei Zhou, Alan Yuille
TL;DR
Medical agent systems are difficult to deploy clinically because frontier-model APIs create privacy, cost, and latency constraints, while structured supervision for agentic behavior is lacking. Meissa distills unified, stratified, and prospective-retrospective trajectories into a lightweight offline model that matches or exceeds proprietary agents in 10 of 16 settings across 13 benchmarks.
Problem
Medical agent systems lack structured supervision for learning when external interaction is needed and how actions should evolve from observations.
Method
Meissa learns strategy selection and execution by distilling state-action-observation trajectories with three-tier stratification and prospective-retrospective supervision.
Results
10 of 16 evaluation settings show Meissa matching or exceeding proprietary frontier agents across 13 medical benchmarks with a 4B-parameter model.
Takeaways & Limitations
Meissa brings medical agentic capability to fully offline deployment while remaining competitive with frontier models across diverse benchmarks.
Takeaways & Limitations
Meissa remains limited on tasks requiring deep parametric knowledge or expert clinical reasoning.
Abstract
from arXiv · showhide
Multi-modal large language models (MM-LLMs) have shown strong performance in medical image understanding and clinical reasoning. Recent medical agent systems extend them with tool use and multi-agent collaboration, enabling complex decision-making. However, these systems rely almost entirely on frontier models (e.g., GPT), whose API-based deployment incurs high cost, high latency, and privacy risks that conflict with on-premise clinical requirements. We present Meissa, a lightweight 4B-parameter medical MM-LLM that brings agentic capability offline. Instead of imitating static answers, Meissa learns both when to engage external interaction (strategy selection) and how to execute multi-step interaction (strategy execution) by distilling structured trajectories from frontier models. Specifically, we propose: (1) Unified trajectory modeling: trajectories (reasoning and action traces) are represented within a single state-action-observation formalism, allowing one model to generalize across heterogeneous medical environments. (2) Three-tier stratified supervision: the model's own errors trigger progressive escalation from direct reasoning to tool-augmented and multi-agent interaction, explicitly learning difficulty-aware strategy selection. (3) Prospective-retrospective supervision: pairing exploratory forward traces with hindsight-rationalized execution traces enables stable learning of effective interaction policies. Trained on 40K curated trajectories, Meissa matches or exceeds proprietary frontier agents in 10 of 16 evaluation settings across 13 medical benchmarks spanning radiology, pathology, and clinical reasoning. Using over 25x fewer parameters than typical frontier models like Gemini-3, Meissa operates fully offline with 22x lower end-to-end latency compared to API-based deployment. Data, models, and environments are released at https://github.com/Schuture/Meissa.
1 Introduction
Meissa addresses the difficulty of deploying capable medical agents offline by distilling frontier-model action-taking behavior into a lightweight model with structured supervision. Its 4B-parameter system combines unified trajectories, error-driven escalation, and prospective-retrospective traces, achieving competitive results across medical benchmarks.
- Medical agents can perform complex clinical tasks through tools, intermediate evidence, and specialist coordination, but cloud deployment raises privacy, cost, and latency concerns.
- Meissa defines agentic behavior as selecting whether to act externally and choosing action sequences from observed outcomes across multiple interaction modes.
- The training pipeline unifies heterogeneous trajectories, uses model errors to escalate from direct reasoning to agentic interaction, and pairs prospective with retrospective traces.
- 4B parameters enable Meissa to remain competitive with frontier models across 13 medical benchmarks, matching or exceeding them in 10 of 16 evaluation settings.
- Meissa provides offline medical agentic capability and matches closed-source models on in-distribution and out-of-distribution benchmarks.
2 Related Work
Prior medical MM-LLM and agent research established strong medical vision-language modeling, tool use, simulation, debate, and routing, but these systems generally depend on frontier backends or isolated frameworks. Meissa instead distills heterogeneous interaction policies into one offline model.
- Medical MM-LLMs progressed from specialized architectures to general-purpose foundation-model adaptations, with scaling and proprietary data driving later systems.
- Trajectory-based methods demonstrate that agent capabilities can be transferred through fine-tuning, while visual and medical systems add image reasoning and specialized tools.
- Meissa is named after λ Orionis and functions as a unified core coordinating heterogeneous tools and sub-agents across medical environments.
- Multi-agent and clinical-simulation systems support expert debate and multi-turn diagnosis, but require frontier LLM backends during inference.
- Routing methods select between models or embed confidence-based decisions, whereas Meissa learns routing implicitly through stratified trajectory supervision.
3 Method
Meissa models medical agent behavior as state-action-observation trajectories organized by interaction depth. Stratified supervision teaches when to act, while diverse environments and behavioral cloning teach how to execute actions across tools, images, agents, and clinical simulations.
- 3 Method: Meissa frames agentic behavior distillation as learning a policy over interaction depth, the number of external actions taken before answering.
- 3.1 Agentic Behavior Modeling: A unified trajectory formalism represents direct reasoning and complex collaboration as sequences of states, actions, and observations.
- 3.1 Agentic Behavior Modeling: Interaction depth unifies direct reasoning at T=0 with tool calling, visual reasoning, multi-agent collaboration, and clinical simulation, whose maximum depths range from 4 to 12.
- 3.1 Agentic Behavior Modeling: Behavioral cloning predicts each action from preceding observations, separating policy learning from environment-specific details and supporting cross-environment generalization.
- 3.2 Behavior Distillation via Stratified Trajectory Supervision: Stratified supervision escalates from student-correct direct trajectories to stronger teacher reasoning and finally agentic trajectories for residual errors.
- 3.2 Behavior Distillation via Stratified Trajectory Supervision: The final dataset contains ∼40K trajectories: 8.2K direct, 9.8K enhanced, and 23.9K agentic.
- 3.3 Agent Environments: Four environments provide distinct trajectories: vision-tool chains, image-text feedback loops, expert debate, and multi-turn clinical information gathering.
4 Experiments
Across 13 benchmarks and four evaluation frameworks, Meissa is competitive with frontier models while learning adaptive routing and observation-dependent tool use. Ablations show that all three supervision tiers and both prospective and retrospective traces contribute to performance.
- Benchmark Results: Meissa ranks 1st or 2nd in 10 of 16 evaluation settings across 13 medical benchmarks.The evaluation spans four agent frameworks, including five strict out-of-distribution benchmarks.
- Benchmark Results: Meissa matches or exceeds Ophiuchus-7B on 3 of 5 Framework II benchmarks despite having nearly half as many parameters.It leads on PathVQA at 78.2 versus 74.3 and OmniMedVQA at 82.8 versus 78.6, while trailing on SLAKE and VQA-RAD.
- Strategy Selection: 72% of queries are routed to direct reasoning and 28% to agentic interaction, with learned routing approaching oracle strategy selection.Always-agentic inference can reduce accuracy on easy queries because of overlong contexts and misleading tool outputs.
- Strategy Selection: Accuracy rises from 55.6% at Tmax=0 to 60.4% at Tmax=3, with only 0.9 additional points at Tmax=∞.Three interaction rounds capture most available gains, after which deeper interaction has rapidly diminishing returns.
- Ablations: All three supervision tiers are necessary, with Tier 2 providing a 5.5-point gain over Tier 1+3 and prospective-retrospective supervision performing best together.Prospective-only supervision excels on out-of-distribution tasks, whereas retrospective-only supervision excels on in-distribution tasks.
- Policy Robustness: Replacing visual feedback with blank or random images reduces accuracy by 6.8 and 13.9 points, respectively, while disabling tools reduces agentic-query performance by 2.8 points.Paraphrasing outputs causes only a 0.8-point drop, and prompt-format changes cause a 2.3-point drop.
5 Discussion and Conclusion
Meissa distills agentic behavior into a compact offline medical agent through unified trajectories, stratified supervision, and prospective-retrospective training. The approach is competitive across diverse benchmarks, but deployment still requires safeguards for uncertainty and abstention.
- Discussion and Conclusion: Meissa uses a unified trajectory formalism, stratified supervision, and prospective-retrospective supervision to learn agentic behavior across heterogeneous medical environments.The framework covers tool calling, visual reasoning, multi-agent debate, and clinical simulation.
- Discussion and Conclusion: Trained on approximately 40K trajectories, Meissa is competitive with frontier models across 13 benchmarks while operating fully offline.The paper presents stratified SFT as deterministic and compute-efficient, with near-oracle routing and approximately 12 GPU-hours of training.
- Limitations: Clinical deployment requires human-in-the-loop safeguards because Meissa lacks calibrated uncertainty estimation and an abstention mechanism for out-of-distribution queries.The paper identifies selective prediction for principled deferral as an important future direction.
A Supplementary Material
The supplementary material provides deferred methodological, dataset, environment, supervision, validation, experimental, training, prompting, and generation details.
- Supplementary Contents: The supplement includes the complete trajectory-generation algorithm, benchmark and dataset details, environment tool specifications, and supervision comparisons.It also contains trajectory validation, extended results, training details, and framework prompts and data-generation details.
A.1 Algorithm: Stratified Trajectory Data Generation
Algorithm 1 generates stratified trajectory data with retries, environment-specific evaluation, validation filters, and paired prospective-retrospective traces. Failed samples are discarded rather than retained as incorrect supervision.
- Retry Logic: Each residual sample receives up to K=8 agent-environment attempts before persistent failures are discarded.The single API-call success rate on residual samples is 89%.
- Environment Selection: Samples are assigned to predefined environments according to the dataset-to-environment evaluation setting.Examples include MIMIC-CXR-VQA mapped to tool calling and MedQA mapped to multi-agent collaboration.
- Validation: A trajectory is accepted only after structural and behavioral validation filters are applied.The procedure applies Valid(τ) before including a trajectory in the training data.
- Prospective-Retrospective Pairing: Every accepted agentic trajectory is paired with its original prospective trace and a retrospective re-narration, doubling the agentic supervision signal.The pairing supplies forward exploratory and hindsight-rationalized execution traces.
A.2 Dataset and Benchmark Details
The training set contains approximately 40K unique trajectories distributed across three supervision tiers, with agentic data forming the largest share.
- Training data composition: 8.2K direct trajectories comprise Tier 1, representing 20% of the unique training trajectories.
- Training data composition: 9.8K enhanced trajectories comprise Tier 2, representing 23% of the unique training trajectories.
- Training data composition: 23.9K agentic trajectories comprise Tier 3, representing 57% of the unique training trajectories.
- Training data composition: 41.9K effective training samples result because each agentic trajectory appears twice as prospective and retrospective traces.
Training Data Sources
Algorithm 1 generates stratified trajectory data by filtering student and teacher outputs before constructing agentic trajectories from residual examples.
- Stratified trajectory generation: Algorithm 1 takes a training set, student model, teacher model, four agent environments, and a maximum of eight retries.
- Tier 1: Direct reasoning: Tier 1 retains examples that the student model answers correctly using direct reasoning.
- Tier 2: Enhanced reasoning: Tier 2 applies teacher enhanced reasoning to remaining examples and retains cases answered correctly by the teacher.
- Tier 3: Agentic trajectories: Tier 3 processes residual examples with teacher-generated agentic trajectories and retry-based interaction.
- Data separation: ChestAgentBench and NEJM were excluded from all trajectory generation and passed n-gram de-contamination with n=8.
Evaluation Benchmark Details
The evaluation details specify how benchmark data are organized and how accuracy is computed for free-form clinical simulation tasks.
- Benchmark and data reporting: Table 6 records trajectory counts by training-data source, tier, and environment, including samples discarded during Tier 3 filtering.
- Clinical simulation evaluation: Clinical simulation benchmarks NEJM and MIMIC-IV use exact match, substring match, and token overlap in a multi-stage accuracy pipeline.
- Clinical simulation evaluation: All benchmarks ultimately report accuracy, including free-form diagnosis evaluations in clinical simulation.
A.3 Agent Environment Tool Details
The tool-calling environment contains specialized medical imaging tools with separate inference and trace-generation versions.
- Tool environment: The tool-calling environment provides eight specialized medical imaging tools.
- Tool versions: Each tool has a Normal version for inference and a Think version for Gemini trace generation.
- Tool versions: Think versions add a thought field requiring the agent to articulate reasoning before each tool call.
Continuous Tool Calling (MedRAX)
The Thinking with Image environment uses vision-manipulation tools whose edited outputs are fed back into the conversation context.
- Vision-manipulation tools produce edited images that are injected back into the conversation context.
Interleaved Thinking with Image (Ophiuchus)
Meissa learns difficulty-aware routing and efficient multi-step interaction, with stable training and strong performance across medical-image tasks. The analyses also identify failure modes involving unnecessary tool use, conflicting or false-positive tool outputs, and repetitive invocation.
- Training stability: Training is stable under LoRA fine-tuning, with standard deviations below 2.5 on every benchmark.The reported variability is small across the evaluated benchmarks.
- Medical-image performance: On ChestAgentBench, gains over the base model are largest for characterization (+22.1 points) and diagnosis (+19.0 points), followed by classification (+16.9) and detection (+15.8).Tool-generated segmentation masks and report summaries help ground fine-grained clinical attributes.
- Learned routing: 96% of easy queries use direct reasoning, while 97% of hard queries trigger agentic interaction; medium queries escalate to tool use approximately 38% of the time.The intermediate routing rate indicates fine-grained calibration rather than a binary easy/hard policy.
- Efficiency: 89% of oracle accuracy is achieved at 2.0× cost, compared with 4.3× for always-agentic inference, reducing cost by 53%.The learned router preserves most oracle accuracy while avoiding unnecessary interaction on simpler queries.
- Interaction depth: Accuracy improves monotonically with interaction depth, but 4.8 of the total 5.7-point gain occurs by Tmax=3.Most agentic queries are resolved within three actions, with diminishing returns thereafter.
- Failure modes and recovery: Failure analyses show that Meissa can over-invoke tools, loop without progress, or let conflicting and false-positive tool outputs override correct visual judgment.Other cases demonstrate graceful recovery after tool failures or crashes through fallback reasoning and strategy switching.