Source-linked AI summary
ClinSeekAgent: Automating Multimodal Evidence Seeking for Agentic Clinical Reasoning
Juncheng Wu, Letian Zhang, Yuhan Wang, Haoqin Tu, Hardy Chen, Zijun Wang, Cihang Xie, Yuyin Zhou
TL;DR
Clinical decision-support agents often rely on fixed, curated evidence rather than actively retrieving heterogeneous patient and medical information. ClinSeekAgent enables iterative multimodal evidence seeking from raw sources and improves performance across text-only EHR and multimodal tasks, including a 15.1-point gain for Claude Opus 4.6 on the multimodal benchmark.
Problem
Many clinical-agent settings rely on curated vignettes or general knowledge instead of actively seeking longitudinal EHR, external medical, and imaging evidence.
Method
ClinSeekAgent lets models iteratively gather and synthesize evidence through web search, raw EHR retrieval, and medical imaging tools.
Results
Claude Opus 4.6 improved from 47.5 to 62.6 overall F1 on multimodal tasks, while ClinSeekAgent improved five of six evaluated models.
Takeaways & Limitations
The results support active evidence acquisition as a promising direction for more flexible, grounded clinical AI agents.
Takeaways & Limitations
Decision-making subtasks show less consistent improvements and can degrade because the pipeline may fail to find critical information.
Abstract
from arXiv · showhide
Large language models (LLMs) and agentic systems have shown promise for clinical decision support, but existing works largely assume that evidence has already been curated and handed to the model. Real-world clinical workflows instead require agents to actively seek, iteratively plan, and synthesize multimodal evidence from heterogeneous sources. In this paper, we introduce ClinSeekAgent, an automated agentic framework for dynamic multimodal evidence seeking that shifts the paradigm from passive evidence consumption to active evidence acquisition. Given only a clinical query and access to raw data sources, ClinSeekAgent gathers evidence by querying medical knowledge bases, navigating raw EHRs, and invoking medical imaging tools; refines its hypotheses as new information emerges; and integrates the collected evidence into grounded clinical decisions. ClinSeekAgent serves both as an inference-time agent for frontier LLMs and as a training-time pipeline for distilling high-quality agent trajectories into compact open-source models. To validate its inference-time effectiveness, we construct ClinSeek-Bench, which pairs Curated Input reasoning from fixed pre-selected evidence with Automated Evidence-Seeking over raw clinical data. On text-only EHR tasks, ClinSeekAgent improves Claude Opus 4.6 from 60.0 to 63.2 overall F1 and MiniMax M2.5 from 43.1 to 47.3, with positive risk-prediction gains in 7 out of 9 evaluated host models. On multimodal tasks, ClinSeekAgent improves Claude Opus 4.6 from 47.5 to 62.6 (+15.1); all evaluated models improve across the three CXR-related task groups. We further validate ClinSeekAgent as a training pipeline by distilling agentic evidence-seeking trajectories into ClinSeek-35B-A3B, which achieves 34.0 average F1 on existing AgentEHR-Bench, improving over its Qwen3.5-35B-A3B baseline by +11.9 points and approaching Claude Opus 4.6.
1 Introduction
ClinSeekAgent addresses the gap between simplified medical-agent settings and real-world clinical workflows by actively acquiring and integrating multimodal evidence from heterogeneous clinical sources. ClinSeek-Bench evaluates this shift against fixed curated inputs, with inference-time gains reported on text-only EHR tasks.
- Motivation: Real-world clinical decision support requires actively seeking evidence from varied sources rather than reasoning only over short, organized patient vignettes.The agent must determine where to retrieve evidence, what to retrieve, and how to integrate evidence into a grounded decision.
- Related Work: Existing EHR-specific pipelines increasingly use structured records and multimodal clinical data, but often operate on pre-extracted patient information.These approaches convert structured tables into text, retrieve task-related entities, and synthesize reasoning data from prepared contexts.
- ClinSeekAgent: ClinSeekAgent actively gathers evidence from heterogeneous clinical data sources instead of passively consuming a fixed evidence package prepared before inference.The framework is introduced as an automated pipeline for dynamic multimodal evidence seeking in clinical reasoning.
- ClinSeek-Bench: ClinSeek-Bench pairs Curated Input, where models answer from provided patient context, with an agentic setting that reformulates existing EHR and multimodal tasks.This paired design evaluates inference-time evidence seeking against the original task-specific evidence package.
2 ClinSeekAgent: Multimodal Evidence-Seeking Pipeline
ClinSeekAgent frames each clinical task as an inference-time interaction in which the model receives a task instance and accesses raw evidence through a unified multimodal tool space. It records open-ended tool-use trajectories while allowing the agentic model to determine which evidence sources to query and in what order.
- Each task instance includes a patient identifier, reference timestamp, clinical instruction, optional modality metadata, and an answer schema or candidate label space when available.
- At inference, the model receives the task and tool access rather than curated patient context, then retrieves the evidence needed to produce the prediction.For EHR tasks, it first loads the patient database, and queries are restricted to records available before the reference timestamp.
- The unified tool space contains 20 tools spanning EHR retrieval, web search, and medical image analysis.These comprise 11 EHR tools, 3 browser tools, and 6 image tools for retrieval, external knowledge acquisition, and visual evidence extraction.
- Each run records an open-ended trajectory of tool actions, observations, and the final answer that produced the prediction.
- Unlike rule-based pipelines, ClinSeekAgent imposes no fixed evidence-source ordering, allowing tools to be selected and interleaved across multiple turns.The agentic model induces the evidence-seeking policy while the framework standardizes the environment and tool interface.
3 Inference-time Validation: Curated Input vs Automated Evidence Seeking
ClinSeek-Bench compares fixed curated evidence with agent-driven retrieval from raw clinical data under matched tasks and labels. ClinSeekAgent delivers its clearest gains on multimodal and risk-prediction evaluations, while decision-making improvements are less consistent.
- Benchmark design: ClinSeek-Bench pairs each task under Curated Input, using benchmark-provided evidence, and Automated Evidence-Seeking, retrieving evidence from raw clinical data with ClinSeekAgent tools.The paired settings retain the same task definition and answer label while removing curated context for automated seeking.
- Text-only results: 63.2 versus 60.0 overall F1: Claude Opus 4.6 gains +3.2 points on text-only EHR tasks with ClinSeekAgent.MiniMax M2.5 rises from 43.1 to 47.3, a +4.2-point gain.
- Multimodal results: +15.1 points: Claude Opus 4.6 achieves the largest reported multimodal improvement, while ClinSeekAgent improves five out of six evaluated models overall.CXR-related benchmarks improve consistently across evaluated models, including Qwen3.5-35B-A3B and Gemma-4-26B-A4B-it.
- Text-only results: 7 out of 9 evaluated models achieve positive average risk-prediction gains with ClinSeekAgent, especially on long-horizon hospital-event prediction tasks.Risk prediction benefits from actively searching sparse, decisive signals distributed across patient records.
- Limitations: Decision-making subtasks show less consistent improvements and often degrade, despite ClinSeekAgent reaching 84.4 versus 67.1 (+17.3 points) for Qwen3.5-35B-A3B on risk prediction.Qwen3.5-35B-A3B still trails the domain expert by 23.2 points on the cited decision-making comparison, 22.0 versus 45.2.
4 Training-time Validation: Teaching Open Models to Use ClinSeekAgent
ClinSeekAgent trajectories can supervise smaller open models to learn long-horizon clinical evidence-seeking, improving AgentEHR-Bench performance while changing retrieval behavior beyond final-answer imitation.
- Training-time validation: ClinSeekAgent is validated as a training pipeline for transferring long-horizon clinical search behavior, including the evidence-seeking process rather than only final-answer prediction.The experiment tests whether a student model can learn the evidence-seeking process induced by ClinSeekAgent.
- Training setup: Claude Opus 4.6 generates training trajectories, which are used to supervised-fine-tune Qwen3.5-35B-A3B in native tool-call format.Training uses a maximum sequence length of 52K tokens.
- AgentEHR-Bench results: 34.0 average F1 is achieved by ClinSeek-35B-A3B versus 22.1 for the Qwen3.5-35B-A3B base model, a +11.9-point gain on AgentEHR-Bench.Task-level gains are Diagnoses (+18.8), Laboratory Events (+20.8), Microbiology Events (+11.4), and Procedures (+9.8), while Transfers drops by -1.4.
- AgentEHR-Bench results: 34.0 average F1 makes ClinSeek-35B-A3B the strongest open-source model in the evaluation, outperforming Kimi K2.5 by +4.1, MiniMax-M2.5 by +6.3, and GLM-4.7 by +6.4.The distilled model also closes most of the gap to Claude Opus 4.6.
- Learned tool-use behavior: 31,446 tool calls versus 33,043 for the base model show that distillation does not substantially shorten search, but teaches a more diverse and flexible EHR retrieval policy.The student treats the EHR as a programmable database rather than relying only on fixed retrieval templates, indicating procedural evidence-seeking transfer.
5 Related Work
Prior medical LLM work largely evaluates reasoning over curated clinical evidence, whereas real-world decision support requires retrieving task-relevant information from longitudinal patient records. Emerging medical agents address this gap through tool use, search, adaptive multi-agent organization, and multi-step evidence interaction.
- Medical Reasoning with Curated Evidence: Medical LLMs demonstrate strong medical question-answering and diagnostic-reasoning performance over provided clinical scenarios.These settings primarily test reasoning over curated evidence rather than evidence retrieval from longitudinal patient records.
- Medical Reasoning with Curated Evidence: Real-world clinical decision support requires models to identify and retrieve task-relevant evidence from longitudinal patient records.This distinguishes clinical workflows from reasoning solely over provided patient vignettes.
- Agentic Evidence Seeking over Clinical Data: Recent medical agent systems extend single-pass reasoning with tool use, search, multi-agent collaboration, and multi-step evidence interaction.Examples include MDAgents, DeepMed, and Meissa, which support adaptive organization, evidence search, or interaction for medical reasoning.
6 Conclusion · Technical Appendix
ClinSeekAgent enables clinical agents to actively gather, refine, and synthesize multimodal evidence from diverse sources instead of relying on user-curated inputs. As both an inference-time and training-time pipeline, it improves strong agentic models and transfers long-horizon evidence-seeking behavior to an open-source student model.
- 6 Conclusion: ClinSeekAgent is an automated framework for dynamic multimodal evidence seeking in clinical decision support.It supports proactive evidence gathering, refinement, and synthesis from diverse sources.
- 6 Conclusion: The framework shifts clinical AI from passive evidence consumption toward active evidence acquisition.This enables agents to seek evidence rather than merely rely on user-curated inputs.
- 6 Conclusion: ClinSeekAgent reformulates text-only and multimodal clinical tasks into an agentic setting for inference-time evaluation.The evaluation tests whether agents can acquire evidence dynamically during reasoning.
- 6 Conclusion: The inference-time pipeline improves the performance of strong agentic models.The supplied conclusion summarizes gains across the reformulated clinical tasks.
- 6 Conclusion: The training pipeline distills long-horizon evidence-seeking trajectories into an open-source student model.Distillation also improves the student model’s tool-use behavior.
- 6 Conclusion: The distilled student achieves open-source state-of-the-art performance on AgentEHR-Bench.These results support trajectory distillation as a way to transfer agentic evidence-seeking capabilities.
A Limitations and Discussion
ClinSeekAgent’s evaluation and training pipeline retain important limitations: current multimodal tasks may not adequately test long-horizon cross-modal evidence seeking, while teacher-generated trajectories can contain inefficient tool use that teaches suboptimal behavior.
- Evaluation limitations: Current multimodal tasks are often solvable with few tool calls or limited cross-modal interaction, so they do not fully stress-test long-horizon evidence seeking.Although tasks combine EHR and imaging evidence, many require limited interaction between modalities.
- Training limitations: Teacher-generated trajectories are not always tool-efficient, containing redundant or low-value calls that pollute context and teach suboptimal evidence-seeking behavior.The training pipeline primarily relies on supervised fine-tuning over Claude Opus 4.6 trajectories; refinement, filtering, or compression could improve them.
B Uncertainty Estimation · C ClinSeekAgent Tool Space
ClinSeekAgent’s uncertainty analysis reports per-sample 95% Student-t confidence intervals and confirms gains on text-only EHR, multimodal, and AgentEHR evaluations. Its unified tool space combines longitudinal EHR retrieval, external medical-knowledge search, and medical-image analysis.
- B Uncertainty Estimation: 95% two-sided Student-t confidence intervals are computed over per-sample F1-acc scores for each model, task, and evaluation setting.Values are reported in percentage points as mean ± CI radius; N is the number of evaluated questions per cell.
- B Uncertainty Estimation: N = 1800 pooled text-only samples yield relatively stable overall estimates with CI radii around two points.The overall results remain consistent with ClinSeekAgent’s improvements on strong agentic models.
- B Uncertainty Estimation: 60.0 ± 2.11 to 63.2 ± 2.09: ClinSeekAgent improves Claude Opus 4.6 on text-only EHR tasks.MiniMax M2.5 also improves from 43.1 ± 2.17 to 47.3 ± 2.24, while weaker-agent gains are more task- and model-dependent.
- B Uncertainty Estimation: 47.5 ± 2.89 to 62.6 ± 2.65: ClinSeekAgent improves Claude Opus 4.6 on pooled multimodal tasks.Pooled multimodal results use N = 989 samples with CI radii around three points, and five of six evaluated models improve overall.
- B Uncertainty Estimation: 22.1 ± 2.00 to 34.0 ± 1.98: ClinSeek-35B-A3B improves over the Qwen3.5-35B-A3B base model on the AgentEHR five-task evaluation.The result is computed over N = 500 samples, exceeds Kimi K2.5 at 29.9 ± 1.93, and approaches Claude Opus 4.6 at 36.0 ± 2.05.
- C ClinSeekAgent Tool Space: ClinSeekAgent offers a unified interface for seeking clinical evidence across multiple sources.Its tool space includes patient-specific longitudinal EHR retrieval, external medical-knowledge search, and visual-evidence extraction from medical images.
- C ClinSeekAgent Tool Space: Table 4 reports text-based EHR mean F1-acc values with 95% CI radii computed over per-sample scores.The table expresses values in percentage points and omits delta columns for compactness.
D Evaluation and Inference Settings
Evaluation uses sample-wise F1 as the primary metric, with scores averaged within task groups and across all evaluated examples. Each model is run once per question, while agents interact with available tools until finishing or reaching the interaction limit.
- Evaluation Metric: Sample-wise F1 is computed per example from normalized predictions and ground-truth answers, then averaged within task groups and overall.Overall scores average all evaluated examples.
- Evaluation Protocol: Each model is evaluated with one run per question.
- Agentic Evaluation: Agents interact with available tools until calling the finish tool or reaching the maximum interaction budget.
E Training Settings for ClinSeek-35B-A3B · F More Case Study · F.1 Failure mode analysis
The paper specifies ClinSeek-35B-A3B’s long-horizon supervised fine-tuning setup and reports supporting evaluation and inference configurations. A failure analysis also shows that ClinSeekAgent can miss critical patient information in decision-making prediction.
- E Training Settings for ClinSeek-35B-A3B: Table 9 summarizes the training configuration used for ClinSeek-35B-A3B.
- E Training Settings for ClinSeek-35B-A3B: The multimodal EHR evaluation reports mean F1-acc with 95% confidence-interval radii and task-specific sample sizes.These results are summarized in Table 5.
- E Training Settings for ClinSeek-35B-A3B: The AgentEHR evaluation reports mean F1 scores with 95% confidence-interval radii across five tasks, including an average over the subtasks.These results are summarized in Table 6.
- E Training Settings for ClinSeek-35B-A3B: ClinSeekAgent’s tool space includes patient-specific EHR retrieval, external medical knowledge search, and medical image analysis.The available tools are summarized in Table 7.
- E Training Settings for ClinSeek-35B-A3B: Default inference settings are kept consistent across models whenever supported by their corresponding backends.These defaults are summarized in Table 8.
- E Training Settings for ClinSeek-35B-A3B: ClinSeek-35B-A3B is fine-tuned on long-horizon ClinSeekAgent trajectories in native tool-call format with a 52K-token maximum sequence length.This configuration is summarized in Table 9.
- F.1 Failure mode analysis: ClinSeekAgent fails to locate critical patient information in a decision-making prediction task.Figure 6 compares this failure with the Curated Input baseline.
F.2 More successful cases
The successful cases show ClinSeekAgent combining chest X-ray findings with longitudinal laboratory evidence to distinguish severe but potentially reversible respiratory illness from terminal or multisystem disease. It also uses multimodal findings to assign a phenotype cluster rather than forcing a single diagnosis.
- Medmod Decompensation: Multiple CXR abnormalities were detected, including lung opacity 0.700, cardiomegaly 0.659, and effusion 0.587, without unequivocal terminal pathology.The model interpreted the findings as chronic cardiopulmonary disease rather than an acute catastrophe.
- Medmod Decompensation: ABG values showed worsening respiratory acidosis, with pH declining from 7.30 to 7.21, while lactate remained normal at 1.2.The model characterized the case as isolated respiratory failure rather than multi-organ failure; pO₂ 142 on FiO₂ 100% yielded P/F = 142.
- Medmod Phenotyping: The second CXR showed lung opacity 0.829, effusion 0.715, atelectasis 0.686, and consolidation 0.620, indicating a strong multi-finding image.The model interpreted the combined abnormalities as a phenotype cluster involving pleurisy/effusion/collapse, pneumonia, and respiratory failure.