PaperlayerView exact revision on arXiv

Source-linked AI summary

ActiveMedAgent: Cost-Aware Trajectory Learning for Multimodal Medical Diagnosis

Weiwei Ma, Xiaobing Yu, Peijie Qiu, Jin Yang, Zhaoqi An, Xuanzhao Dong, Xiaoqi Zhao, Xiaofeng Liu

arXiv:2610.11140v1cs.LGcs.AIcs.CLcs.CVcs.MM

TL;DR

Medical multimodal diagnosis is commonly evaluated after all evidence is available, despite clinical escalation of tests according to uncertainty and cost. ActiveMedAgent learns acquisition and stopping decisions from diagnostic belief trajectories while keeping the VLM frozen. Across three datasets, it improves the quality–cost frontier, and selective acquisition is correct where full-modality input fails in 175 cases.

  • Problem

    Common medical multimodal evaluations assume evidence is available upfront, leaving selective, cost-aware diagnostic escalation unaddressed.

    Source: §1 · p. 1

  • Method

    ActiveMedAgent trains a lightweight controller offline on diagnostic belief trajectories to choose between acquiring another channel and committing, without changing the backbone VLM.

    Source: §1 · p. 1, §3 · p. 3, §3.2 · p. 5

  • Results

    Across three datasets, ActiveMedAgent improves the quality–cost frontier; in 175 cases, selective acquisition is correct while Full Modality fails.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Frozen VLMs benefit from explicit control over when evidence is exposed, and named clinical channels make requests, stops, and errors easier to inspect.

    Source: §5 · p. 8

    Experiments are retrospective and use protocol-level rather than institution-specific costs, turnaround times, or patient-burden estimates.

    Source: §6 · p. 9

Abstract

from arXiv · show

Clinical diagnosis is inherently sequential: clinicians escalate from cheap to costly tests only when additional evidence is expected to resolve diagnostic uncertainty. We present ActiveMedAgent, a framework that brings this cost-aware sequential logic to multimodal medical AI. Given a frozen, API-accessed vision-language model, ActiveMedAgent tracks probability distributions over candidate diagnoses and scores each acquisition by its per-step diagnostic utility minus cost. A lightweight MLP controller is then trained offline on these scored trajectories, learning when to request additional evidence and when to commit. Across three commonly used benchmarks, trajectory-based policy learning consistently outperforms both unguided acquisition and full-modality baselines. Notably, we identify an information overload effect. In 175 cases, the agent produces a correct diagnosis with fewer channels while the full-modality baseline fails, showing that learning what to omit can be as important as learning what to acquire.

1 Introduction

Medical multimodal diagnosis is often treated as static prediction, although clinical practice escalates evidence selectively as uncertainty and test costs warrant. ActiveMedAgent learns this escalation from diagnostic belief trajectories, and the introduction highlights cases where selective acquisition succeeds while full-modality input fails.

1 Introduction

  • Clinical diagnosis proceeds through selective escalation, so the relevant objective is finding the minimum evidence needed for a correct diagnosis while accounting for acquisition cost.The system maintains beliefs over candidate diagnoses and decides whether further evidence is worth its cost.Source: §1 · p. 1
  • Trajectory learning supervises how belief states evolve as modalities resolve ambiguity, add redundant evidence, or introduce distraction.Unlike static multimodal learning, it uses intermediate reasoning signals rather than only the final input and label.Source: §1 · p. 1
  • ActiveMedAgent lets a model acquire one additional information channel or commit to a ranked differential, using a learned controller while leaving the VLM unchanged.The controller learns from previously collected trajectories and compact diagnostic states.Source: §1 · p. 2
  • In 175 cases, selective acquisition yields a correct diagnosis while the full-information baseline is wrong, highlighting information overload.The introduction also reports an example accuracy increase from 18.6% to 31.2% on NEJM with GPT-4o-mini, where naive acquisition fails.Source: §1 · p. 2

2 Related Work

Prior medical VLM benchmarks commonly assume modalities are available upfront, while newer work studies sequential diagnosis with test costs. ActiveMedAgent focuses on auditable acquisition over a fixed hierarchy of clinical evidence channels.

2 Related Work

3 Method

ActiveMedAgent frames diagnosis as cost-aware sequential evidence acquisition, learning from belief trajectories to choose when to request a channel or commit while keeping the VLM backbone frozen.

3 Method

  • ActiveMedAgent learns a cost-aware acquisition policy from diagnostic belief trajectories, distinguishing ambiguity-resolving evidence from redundant or harmful acquisitions without modifying the VLM backbone.The controller operates over compact diagnostic states and can be retrained or replaced independently of the backbone.Source: §3 · p. 3

3.1 Problem Formulation

  • The formulation maximizes final reciprocal rank minus cumulative normalized acquisition costs, with λ controlling the quality-cost trade-off.The agent chooses between an unobserved channel and COMMIT.Source: §3.1 · p. 3

3.2 Belief Extraction from VLM Outputs

  • Structured prompting elicits ranked diagnoses with confidence scores, while the controller state summarizes acquired channels, belief uncertainty, confidence, and cumulative cost.The full state also includes dataset identity and a frozen presentation-text embedding.Source: §3.2 · pp. 3–5
  • Offline trajectories score each acquisition by reciprocal-rank improvement minus cost, labeling steps beneficial, redundant, or harmful for reward-weighted policy learning.High-utility transitions receive greater training weight, while harmful acquisitions are suppressed.Source: §3.2 · pp. 4–5
  • At inference, the agent requests a channel only when the policy selects it and its estimated net value is positive; otherwise it commits, producing adaptive-length trajectories.Numerical information-gain estimation uses an extra VLM call per remaining channel; the cheaper keyword fallback underperforms and is not the default.Source: §3.2 · pp. 5–6

4 Experiments

Across the three diagnostic pathway datasets, PolicyNet matches or exceeds Full Modality in all six settings while usually acquiring fewer channels. Experiments also examine training, acquisition costs, and the limits of these comparisons.

4.1 Datasets, protocol, and evaluation setup

  • The evaluation uses 947 NEJM, 635 MIDAS, and 1,268 OLIVES cases, split 60/40 into training and test sets, with top-1 accuracy, MRR, and cost metrics.At test time, the agent observes free presentation channels before requesting another channel or committing.Source: §4.1 · p. 6

4.2 Training Details

  • A separate PolicyNet is trained for each backbone on its scored trajectories; training takes under two minutes on CPU, while VLM trajectory collection dominates cost.The authors do not claim transfer between GPT-4o and GPT-4o-mini because their uncertainty and acquisition profiles differ.Source: §4.2 · p. 6

4.3 Main Results

  • PolicyNet matches or exceeds Full Modality in all six settings while usually using fewer channels; Figure 3 reports savings up to $1,768 per correct diagnosis.The $1,768 saving is for NEJM/GPT-4o-mini.Source: §4.3 · p. 6, Figure 3 · p. 6, Table 1 · p. 7
  • In 175 cases, selective acquisition yields a correct diagnosis where Full Modality fails, but these results do not establish that less evidence is clinically better.The authors caution that prompt-order sensitivity is not ruled out.Source: §4.3 · p. 8
  • On NEJM+2dist, removing case-text embedding drops accuracy from 0.625 to 0.585, while removing distractor-aware training drops it to 0.575.The vanilla MLP reaches 0.520; increasing the cost penalty also reduces mean cost, with λ = 5 cutting cost by 55% for a 1.4-point accuracy drop.Source: §4.3 · p. 8, Table 4 · p. 8

5 Conclusion

ActiveMedAgent learns diagnostic escalation around a frozen VLM, separating model reasoning from priced evidence acquisition. The conclusion emphasizes evidence control, auditability, and further testing before clinical use.

5 Conclusion

  • ActiveMedAgent learns diagnostic escalation around a frozen VLM, separating VLM reasoning from priced evidence acquisition.The controller restricts actions to named clinical channels and trains on trajectory utilities.Source: §5 · p. 8
  • Across three datasets, ActiveMedAgent improves the quality–cost frontier, while exhaustive evidence can distract VLMs.The paper argues that the evidence path, not only final correctness, matters for auditable multimodal diagnosis.Source: §5 · p. 8
  • The authors call for prospective testing, local clinical cost estimates, and study of how clinicians use or override learned recommendations.Source: §5 · p. 8

6 Limitations

The authors identify limits in the retrospective evaluation, its cost and probability assumptions, and the scope of the evidence across datasets. They also note risks from the learned policy and its backbone-generated training trajectories.

6 Limitations

  • Experiments are retrospective and use protocol-level costs rather than institution-specific costs, turnaround times, or patient-burden estimates.Source: §6 · p. 9
  • VLM-reported probabilities are imperfectly calibrated, so the controller relies mainly on rank changes and compact uncertainty features rather than treating scores as clinical probabilities.Numerical EIG uses additional VLM calls for candidate-channel replay, while reported clinical acquisition cost excludes API overhead.Source: §6 · p. 9
  • OLIVES is domain replication rather than deployment evidence because its label structure and ophthalmology-specific channel hierarchy differ from NEJM and MIDAS.The dataset contains 1,268 cases.Source: §6 · p. 9
  • The policy could under-request costly evidence for rare or ambiguous cases, and it can inherit systematic diagnostic blind spots from its backbone.Source: §6 · p. 9

7 Ethics Statement

ActiveMedAgent is presented as a research framework for studying evidence acquisition, not as an autonomous diagnostic system. The authors identify safety and bias risks and specify safeguards for any clinical use.

7 Ethics Statement

  • ActiveMedAgent is intended to study evidence acquisition, not to operate as an autonomous diagnostic system.Source: §7 · p. 9
  • Cost savings could create ethical risks if they outweigh patient safety, while biased training data could produce uneven acquisition behavior across groups.Source: §7 · p. 9
  • Any clinical use would require prospective validation, local cost calibration, human override, audit logs, and subgroup monitoring.Source: §7 · p. 9

A.1 Protocol and Metric Notes

The appendix clarifies how acquisition policies and metrics are compared, then tests whether learned acquisition’s benefits persist across benchmark and distractor checks. Across these checks, controlling when evidence is exposed matters more than simply consuming more evidence.

A.1 Protocol and Metric Notes

  • Appendix evaluations use the same frozen backbone and sequential interface, comparing acquisition controllers rather than different diagnostic models.Cases start with presentation-time evidence; policies can request one remaining channel at a time or commit. Metrics separate ranking correctness, evidence use, and cost.Source: §A.1 · p. 13

A.2 Why These Results Stay in the Appendix

  • The appendix reuses benchmark slices to make focused checks of whether active acquisition beats Passive and whether Zero-shot acquisition is reliable.These checks complement rather than duplicate the main benchmark tables and figures.Source: §A.2 · p. 13, §B.1 · p. 14

B Benchmark and Stress-Test Evidence

  • The benchmark and stress-test checks ask whether gains reflect better control over evidence exposure rather than simply consuming more evidence.They test robustness to changes in table format, dataset, backbone, and baseline family.Source: B Benchmark and Stress-Test Evidence · p. 14

B.1 Compact Claim Checks

  • Learned acquisition improves over Passive, while Zero-shot is inconsistent and can underperform on weaker models.Across the checks, PolicyNet is the acquisition method that remains consistently positive across domains.Source: Table 5 · p. 14, §B.1 · p. 14

B.2 Distractor Stress Test

  • With distractors available, exhaustive or weakly controlled evidence access is fragile, while PolicyNet has the smallest clean-to-distractor drop among high-accuracy methods.The stress test is adversarial to the assumption that more information is always better; learned stopping can avoid some distraction from late channels.Source: §B.2 · p. 14, Table 7 · p. 14

B.3 Backbone and Dataset Replication

Replication checks examine whether acquisition results extend across backbones and datasets, and whether comparisons with sequential systems remain measurable under matched protocols. The findings are bounded by differing channel structures and by the controller’s more restricted action space.

B.3 Backbone and Dataset Replication

  • The replication results show the relative pattern for open-weight models as well as frontier backbones, while OLIVES remains a distinct domain-replication setting.Its ophthalmology labels and OCT-derived channels differ sharply from NEJM and MIDAS; the dataset replication tests channel structure, not universal clinical validity.Source: §B.3 · p. 15, Table 8 · p. 15, Table 9 · p. 15
  • A fixed acquisition schedule can be inefficient across domains; the learned policy varies its depth using the current belief state and remaining channel mask.NEJM, MIDAS, and OLIVES differ in their evidence pathways and modalities.Source: §B.3 · p. 15

B.4 Prior Systems and Statistical Checks

  • Against matched NEJM sequential-diagnosis systems, the improvement remains measurable after Holm–Bonferroni correction across pairwise comparisons.The comparison supports a better quality-cost operating point, but does not claim dominance over every interactive diagnostic assistant.Source: §B.4 · p. 15, Table 10 · p. 15
  • ActiveMedAgent restricts actions to auditable evidence channels, unlike systems that may ask richer free-form questions, yet remains competitive in the matched protocol.The constrained controller is easier to inspect, price, and reproduce.Source: §B.4 · p. 15

C Design Robustness and Ablations

The robustness checks probe cost sensitivity, prompt wording, belief-state features, and transfer or routing behavior. Together, they support probability-derived state features and case-conditioned routing while making clear that VLM confidence is not treated as clinically calibrated.

C Design Robustness and Ablations

C.1 Prompt Robustness

  • Across three prompt templates on NEJM gpt-4o with N=200, method ranking is invariant, and PolicyNet stays within 1.5 pp of its best result on every template.The paraphrase check indicates the policy advantage is not tied to a single prompt template.Source: Table 12 · p. 16, §C.1 · p. 16

C.2 Belief Features and Calibration

  • Probability-derived features outperform pure ranking by 1.5 pp and substantially beat rationale parsing, but the policy objective uses rank changes rather than calibrated probabilities.Confidence, entropy, and confidence gaps are compact signals about diagnostic state; the paper does not treat stated confidence as clinically calibrated.Source: Table 13 · p. 16, §C.2 · p. 16

C.3 Transfer and Routing Behavior

  • Zero-shot transfer loses about 5 pp yet remains well above random, while dynamic case-conditioned routing beats static strategies at lower cost than exhaustive acquisition.The dynamic policy is more expensive than the cheapest static patterns, using that cost to buy accuracy; no fixed ordering matches it.Source: Table 14 · p. 16, Table 15 · p. 16, §C.3 · pp. 16–17

C.4 Robustness Visualizations

The robustness visualizations show how cost penalties and prompt paraphrases affect operating points, while case-level analyses examine information overload and its observable correlate. The evidence supports a cautious interpretation of why selective acquisition can preserve correct diagnoses.

C.4 Robustness Visualizations

  • The λ sweep shows a tunable cost–accuracy trade-off; the supplementary view tracks acquisition cost and cost per correct diagnosis, and prompt-paraphrase ordering remains stable.Figure 5 explains that cost per correct can improve even when raw cost rises relative to Zero-shot, if premature stopping is avoided.Source: §C.4 · p. 17, Figure 4 · p. 17, Figure 5 · p. 17

D Mechanism and Safety Analysis

  • Distractor attention is an observable correlate of errors in open-weight backbones, supporting the information-overload analysis without establishing a universal causal mechanism.The analysis connects case-level failures to deployment cautions.Source: §D.1 · pp. 17–18, Table 17 · p. 18

D.1 Case-Level Information Overload

  • Case-level win/loss counts show repeated instances where PolicyNet is correct and a more exhaustive baseline is wrong, rather than only an isolated example.The attention analysis on accessible open-weight backbones finds distractor attention predictive of case-level errors; it is supporting evidence, not universal causal proof.Source: §D.1 · pp. 17–18, Table 16 · p. 17, Table 17 · p. 18

D.2 Residual Failures and Subgroup Behavior

The appendix supports evaluating ActiveMedAgent as a sequential evidence-control system, while making residual errors, subgroup variation, and deployment constraints explicit. Its analyses connect acquisition behavior and belief features to the costs and risks of selective diagnosis.

D.2 Residual Failures and Subgroup Behavior

  • Subgroup performance and calibration vary; the tables are diagnostic, not evidence of clinical fairness, and motivate calibration and monitoring before clinical use.The authors distinguish premature commitment, wrong acquisition order, and distractor latching as separate engineering problems.Source: §D.2 · p. 18, Table 19 · p. 18
  • ∼30% of errors in 48 wrong-top-1 cases involve distractor latching or image-only-over-acquisition.The taxonomy motivates distractor-aware training.Source: Table 18 · p. 18, §D.2 · p. 18

D.3 Beyond Monetary Cost

  • The experiments use a simplified cost hierarchy; time, invasiveness, radiation, and patient burden could extend it, but richer cost weighting requires local clinical input.The expanded cost profile is not used as the reward in the present experiments.Source: §D.3 · pp. 18–19, Table 20 · p. 18

D.4 Mechanism Visualizations

  • The mechanism figures depict belief movement, distractor attention, case-varying acquisition depth, and reliability of belief-derived features.Together, these visualizations connect trajectory behavior, a plausible overload pathway, policy behavior, and the calibration caveat.Source: §D.4 · p. 19, Figure 7 · p. 19, Figure 8 · p. 19

E Appendix Summary

  • Appendix analyses show learned acquisition improves the quality–cost frontier while remaining competitive with prior sequential-diagnosis systems.The authors frame value not only in final accuracy but in whether the diagnostic path is necessary, auditable, and cost-aware.Source: E Appendix Summary · pp. 19–20
Loading 2610.11140v1…