PaperlayerView exact revision on arXiv

Source-linked AI summary

Reliability Stress Tests and Decision-Time Routing for Chest X-ray Vision-Language Models

Xinye Yang, Zhusi Zhong, Scott Collins, Grayson Baird, Xuyu Wang, Zhicheng Jiao

arXiv:2610.02270v1eess.IVcs.CV

TL;DR

Evaluation of medical VLMs can vary with prompting, workflow, and benchmark construction, yet these factors are often studied separately. This paper tests chest X-ray models across datasets, prompts, and workflows, then evaluates selective escalation. Reliability varies by configuration, and routing improves the cost–quality trade-off over fixed workflows.

  • Problem

    Medical VLM evaluation is sensitive to prompting, workflow design, and benchmark construction, but studies often treat these factors in isolation.

    Source: §I · p. 1

  • Method

    The paper conducts a controlled chest X-ray stress test across model families, prompt styles, and workflows, then tests decision-time routing from single-VLM to multi-agent inference.

    Source: §I · p. 1, D. Decision-Time Routing… · p. 5

  • Results

    Reliability depends on model family, scale, and workflow, and selective escalation improves cost–quality trade-offs over fixed workflows.

    Source: §I · p. 1, §V · p. 5

  • Takeaways & Limitations

    The conclusion argues for moving beyond fixed workflows toward dynamic, cost-aware routing for clinical deployment.

    Source: §VI · p. 6

    Both cohorts contain only 50 cases, limiting statistical power and the learned router’s training signal; the label space omits rare diagnoses, and frozen weights with greedy decoding may affect configuration rankings.

    Source: §V · p. 6

Abstract

from arXiv · show

Medical vision-language model (VLM) evaluation is sensitive to workflow design, prompting strategy, and benchmark construction, yet most studies treat these factors in isolation. We introduce a reliability stress test for chest X-ray interpretation built on two balanced datasets (a private report-backed set and a curated MIMIC subset). Three medical VLMs (CheXagent, MedGemma-4B, and MedGemma-27B) are evaluated across three prompt styles and two workflows (single-VLM and multi-agent), producing 36 configurations. We show that exact-match accuracy alone can overstate the effectiveness of conservative models that default to "Normal" predictions. Diagnostic reliability also depends heavily on model family and scale: multi-agent reasoning helps some configurations but hurts others. Building on these observations, we propose a decision-time routing framework that selectively escalates to multi-agent inference only when beneficial, improving the cost-quality trade-off over fixed workflows. Our results highlight the need for evaluation protocols that jointly consider prompt sensitivity, failure-mode diversity, and workflow choice before clinical deployment.

I. INTRODUCTION

The introduction frames prompt and workflow choices as sources of variation in clinical error patterns and asks how they interact with model family and scale. It presents a controlled evaluation and decision-time routing as a practical response to that variability.

I. INTRODUCTION

  • The study evaluates CheXagent and two MedGemma scales on two balanced datasets using three prompt styles and single-VLM or multi-agent workflows.Its stated contribution combines a controlled, multidimensional evaluation with a deployable routing strategy.Source: §I · p. 1
  • Exact-match evaluation alone can mislead because conservative models may hide weaknesses by over-predicting Normal.The introduction also notes that prompt and workflow changes can alter the distribution of clinical errors.Source: §I · p. 1
  • System reliability depends strongly on model family and scale, not just prompt wording.This is presented alongside the finding that multi-agent gains are configuration-dependent.Source: §I · p. 1
  • Multi-agent reasoning provides conditional, not universal, performance gains across configurations.The evaluation examines how prompt style and workflow interact with model family and scale.Source: §I · p. 1
  • Selective decision-time escalation consistently improves cost–quality trade-offs over fixed workflows.The proposed routing strategy treats workflow instability as a routing problem, rather than an evaluation artifact.Source: §I · p. 1

II. RELATED WORK

The related work spans medical VLM evaluation, broad visual-language benchmarks, reliability testing, and inference routing. This paper extends static prompt-robustness evaluation to workflow behavior and frames routing as a choice between two workflows using frozen model parameters.

II. RELATED WORK

  • Prior medical VLM evaluations typically use fixed, single-pass inference, leaving workflow–scale interactions open.The cited models include LLaVA-Med, CheXagent, and MedGemma [4]–.Source: §II · p. 1
  • Broad frameworks cover visual-language grounding but can miss interactive, multi-step diagnosis; this study trades breadth for focused workflow-reliability testing.The passage names MultiMedEval and XBench [7].Source: §II · p. 2
  • This work extends paraphrase-sensitivity and hallucination-rate benchmarks to dynamic workflow behavior, where the same weights encounter different instructions or multi-agent pipelines.The passage cites MediConfusion and PSF-Med [8].Source: §II · p. 2
  • Unlike routing between distinct models or human experts, this paper studies a binary choice between fast single-VLM inference and multi-agent processing with frozen parameters.The passage situates this framing alongside FrugalGPT [9], AT-CXR [10], VL-RouterBench [11], and routing frameworks [12]–[15].Source: §II · p. 2

III. EXPERIMENTAL SETUP

The evaluation uses two balanced 50-case chest X-ray cohorts, with matched reports available for both. The public MIMIC subset provides a reproducible baseline for comparing workflows.

A. Datasets

  • Each cohort contains 50 studies evenly split between normal and abnormal cases, and the private dataset uses labels from original clinical reports.The curated public MIMIC-CXR subset includes a matched report for every case.Source: A. Datasets · p. 2
  • The MIMIC subset’s matched reports provide a reproducible public baseline and reduce the risk of label-source mismatches distorting workflow comparisons.Source: A. Datasets · p. 2

B. Models, Prompts, and Workflows

The setup varies model family and scale, prompt style, and inference workflow, then assesses diagnostic performance alongside computational cost. The multi-agent workflow uses sequential stages with the same frozen model weights.

B. Models, Prompts, and Workflows

C. Evaluation Metrics

  • The evaluation reports six complementary diagnostic metrics alongside latency and token count to quantify computational cost.Source: C. Evaluation Metrics · p. 2
  • Exact match requires predicted labels to equal ground truth, whereas partial match requires any non-empty label overlap.Macro F1 averages per-class F1 and penalizes ignorance of rare pathologies.Source: C. Evaluation Metrics · p. 2
  • Hallucination FP measures false positives for absent labels, Normal Accuracy measures accuracy on normal cases, and Abnormal Partial measures pathology sensitivity.Source: C. Evaluation Metrics · p. 2

A. Primary Findings

Performance depends on the interaction of model scale and prompt style, and multi-agent reasoning helps some configurations while hurting others. The supplied figures also frame decision-time routing as a choice between macro F1 and latency.

A. Primary Findings

  • MedGemma-27B with structured multi-agent inference achieved the highest MIMIC macro F1 (0.335), while MedGemma-4B outperformed it in single-VLM macro F1 with structured (0.239 vs. 0.138) and labels-only (0.252 vs. 0.171) prompts.These comparisons show that performance depends on the interaction between parameter scale and prompt style, not model size alone.Source: A. Primary Findings · p. 2
  • The decision-time framework uses initial single-VLM predictions and routing features to return a low-cost result or escalate to multi-agent processing.The multi-agent path includes extraction, formulation, and synthesis stages.Source: Figure 1 · p. 3
  • Figure 2 presents macro F1 against average latency for different routing strategies.The caption identifies the trade-off dimensions but does not report strategy outcomes.Source: Figure 2 · p. 3
  • In the MIMIC labels-only single-VLM setting, MedGemma-4B reached 52% exact match and 70% partial match at 0.20s average latency, yet macro F1 was just 0.252, indicating poor label coverage across pathologies.High exact-match scores alone can mask poor clinical coverage, because conservative models achieve them by defaulting to “Normal” while producing low macro F1.Source: A. Primary Findings · p. 3, Figure 3 · p. 3
  • Multi-agent reasoning substantially improved MedGemma-27B on MIMIC, had little effect on MedGemma-4B, and hurt CheXagent on the private cohort.The direction and magnitude of workflow differences depend on the base model and dataset.Source: A. Primary Findings · p. 3

B. Typology of Failure Modes

The failure patterns reveal two opposing effects: multi-agent stages can recover missed findings, but consensus can also suppress genuine abnormalities. Which effect dominates depends on model characteristics and the dataset.

B. Typology of Failure Modes

  • On MIMIC, MedGemma-27B’s abnormal-to-normal collapse rate fell from 30% with structured single-VLM inference to 10% with the multi-agent pipeline.The additional extraction and formulation stages recovered findings missed by the single pass.Source: B. Typology of Failure Modes · pp. 3–4
  • For CheXagent on the private labels-only setting, multi-agent processing raised abnormal-to-normal collapse from 10% to 44% as synthesis defaulted to “Normal” when stages disagreed.Sparse extraction evidence from shorter, less structured outputs led downstream stages to discount legitimate but weakly expressed findings.Source: B. Typology of Failure Modes · p. 4
  • The mechanisms act in opposite directions: verbose MedGemma-27B outputs can help recover missed findings, while the same pipeline can strip genuine detections from smaller or more terse models.MedGemma-4B already had low collapse rates under single-VLM inference, so multi-agent overhead introduced noise without compensating gains.Source: B. Typology of Failure Modes · p. 4

C. Qualitative Case Examples

Two cases illustrate that multi-agent inference can either recover missed findings or suppress a correct abnormal prediction. The reported outcomes differ by model and dataset.

C. Qualitative Case Examples

  • Figure 4 shows that the direction and magnitude of the macro F1 difference between multi-agent and single-VLM workflows depend on the base model and dataset.The two cases provide contrasting examples of those workflow effects.Source: C. Qualitative Case Examples · p. 4, Figure 4 · p. 5
  • On MIMIC, MedGemma-27B’s single-VLM prediction missed cardiomegaly, edema, and pleural effusion, while multi-agent inference identified cardiomegaly and edema.The workflow turned a complete miss into a useful partial match.Source: C. Qualitative Case Examples · pp. 4–5
  • On the private cohort, CheXagent correctly predicted cardiomegaly for a case with stable cardiomegaly and clear lungs, but the multi-agent workflow changed the prediction to Normal.This illustrates abnormal-to-normal suppression.Source: C. Qualitative Case Examples · pp. 4–5

D. Decision-Time Routing Proof-of-Concept

Decision-time routing selectively escalates from single-VLM inference to multi-agent processing, with results varying by dataset and heuristic; broader discussion links reliability to model family, scale, and workflow design. The study also notes small-cohort and evaluation-setting limits, while identifying adaptive routing as a direction for future work.

D. Decision-Time Routing Proof-of-Concept

  • Routing uses initial single-VLM outputs to decide whether to escalate, with heuristics based on complex Normal predictions or zero partial-match overlap and a learned logistic-regression router.The learned router uses predicted-label count, normality, token count, severity, and partial-match indicators; its target is whether multi-agent macro F1 exceeds single-VLM macro F1 for a case.Source: D. Decision-Time Routing… · p. 5
  • Heuristic A escalates no MIMIC cases because its Normal-and-high-complexity trigger is rarely met, while output-based Heuristic B avoids this issue and approaches oracle performance.With only 5–10 positive routing labels per fold, the learned router cannot reliably separate classes; the authors expect the gap to narrow with larger cohorts.Source: D. Decision-Time Routing… · p. 5

V. DISCUSSION AND FUTURE DIRECTIONS

  • Reliability depends jointly on model family, scale, and workflow: multi-agent processing reduces CheXagent’s MIMIC hallucination false-positive rate from above 0.35 to below 0.13, but MedGemma models can lose true positives after filtering.MedGemma hallucination rates were already below 0.06, and no fixed inference strategy is optimal across configurations.Source: §V · p. 5
  • The authors frame decision-time routing as a first step toward adaptive deployment and propose future extension to federated, continuously learning routers and multimodal inputs.They note that both cohorts contain only 50 cases, the label space omits rare diagnoses, and frozen weights with greedy decoding leave other settings untested.Source: §V · pp. 5–6

D. Decision-Time Routing Proof-of-Concept

  • On the private dataset, simple heuristics and the oracle reach macro F1 0.488 with selective escalation, while always-single scores 0.457 and always-multi 0.118.On MIMIC, the oracle matches always-multi at 0.338 versus 0.335 while escalating only 20% of cases.Source: D. Decision-Time Routing… · p. 5, Table V · p. 6

VI. CONCLUSION

The conclusion reports that chest X-ray VLM performance varies across models and workflows, so blanket multi-agent deployment is not always best. It proposes selective decision-time routing as a way to approach oracle-level performance while keeping latency low.

VI. CONCLUSION

  • Across 36 workflow configurations, blanket multi-agent deployment is not always best.Multi-agent reasoning substantially improved MedGemma-27B on public data, while direct-prompting CheXagent outperformed it on the private cohort and MedGemma-4B remained competitive under single-VLM inference.Source: §VI · p. 6
  • Aggressive prompt chaining can over-regularize predictions and suppress real pathology.Source: §VI · p. 6
  • The proposed decision-time heuristic selectively escalates to multi-agent inference, approaching oracle-level performance while keeping latency low.Source: §VI · p. 6
  • The conclusion recommends moving beyond fixed workflows toward dynamic, cost-aware routing to reduce unnecessary token costs while maintaining diagnostic integrity.Source: §VI · p. 6
Loading 2610.02270v1…