PaperlayerView exact revision on arXiv

Source-linked AI summary

MC-CXR: A Multi-Context Chest X-ray Benchmark for Context-Induced Disruption in Vision-Language Models

Junhyeok Lee, Songsoo Kim, Kyu Sung Choi

arXiv:2608.24118v1cs.CL

TL;DR

Clinical VLM evaluation has largely measured image or context use without testing whether models preserve a correct image-grounded decision when plausible auxiliary evidence conflicts. MC-CXR addresses this gap with paired reliable-versus-misleading perturbations across multiple context modalities, finding stronger directional switching from misleading text than visual context under its standardized protocol.

  • Problem

    Existing benchmarks do not test whether VLMs preserve a correct current-image decision when plausible clinical context conflicts with the image.

  • Method

    MC-CXR fixes the current CXR and target finding, pairs reliable and misleading context variants, and evaluates text, prior-image, and visual-overlay conditions with paired disruption metrics.

  • Results

    Misleading text produces stronger directional switching than visual context: 74.6% of switched textual predictions align with the misleading label versus 17.6% for visual context, a 57.0-point gap.

  • Takeaways & Limitations

    Image-only accuracy is necessary but insufficient; context-robust evaluation must measure whether models arbitrate between current-image evidence and conflicting auxiliary context.

  • Takeaways & Limitations

    Findings are bounded by low image-only accuracy, single deterministic runs under a constrained direct-answer prompt, single-rater curation, unauditable MIMIC-CXR overlap, and no abstention option.

Abstract

from arXiv · show

Vision-language models (VLMs) are increasingly used in clinical pipelines where a chest X-ray is interpreted alongside retrieved reports, preliminary notes, or prior imaging. Existing benchmarks measure whether models answer correctly in isolation, but not whether they preserve a correct image-only decision when plausible context conflicts with the image. We introduce Multi-Context Chest X-ray (MC-CXR), a benchmark of 240 cases expanded into 2,522 instances that isolates context-induced disruption through paired perturbation. Each case fixes the current image and target finding while presenting matched reliable and misleading context across text and prior CXR, with visual overlays where available. MC-CXR defines three task families and two paired metrics, the switch-to-wrong rate and the context-aligned error rate. We evaluate ten VLMs spanning open-source general, medical-domain, and closed-source systems. Image-only accuracy is necessary but insufficient. Mean switch rates range from 45.6-78.1% across misleading textual sources and 35.7-61.7% across misleading visual sources. Among switched predictions, 74.6% align with the misleading label for text versus 17.6% for visual context, a 57.0-point gap (95% CI 50.9-62.8). This text-visual asymmetry is observed under the standardized direct-answer protocol. The dataset is available on PhysioNet.

1 Introduction

MC-CXR addresses whether VLMs can use clinical context conditionally: supporting reliable evidence while resisting plausible misleading context that conflicts with a correct image-only decision. It introduces a paired benchmark and evaluates context arbitration across text, prior images, and visual overlays.

  • Motivation: Clinical CXR interpretation combines current-image evidence with indications, prior imaging, and prior reports, while VLM pipelines may also use preliminary notes.Context reliability can vary because auxiliary evidence may be stale, mismatched, inaccurately summarized, preliminary, or over-weighted.
  • Evaluation framework: The benchmark formalizes context-robust reasoning into Image-only Recognition, Reliable Context Stability, and Misleading Context Resistance across textual, prior-image, and visual-overlay perturbations.Its evaluation target is deciding when context should support current-image evidence and when misleading context should be resisted.
  • Research gap: Existing CXR benchmarks do not directly test whether models preserve correct current-image decisions when added context is reliable or misleading.Image-only accuracy primarily measures recognition, leaving context parsing, conflict detection, and evidence arbitration untested.
  • Benchmark design: MC-CXR fixes each current CXR and target finding while pairing reliable and misleading variants of the same auxiliary context source.This paired-perturbation design tests conditional context use rather than penalizing context use itself.
  • Contribution: MC-CXR contains 240 radiologist-curated cases expanded into 2,522 instances pairing reliable and misleading clinical and imaging contexts.The paired inputs include indications, prior reports, preliminary notes, prior CXRs, and visual overlays.
  • Metrics: Switch-to-wrong and context-aligned error rates isolate disruption on image-only-correct cases and quantify whether misleading context directs model errors.The benchmark evaluates this behavior across ten VLMs.

2 Related Work

Prior medical VQA and longitudinal CXR benchmarks generally evaluate recognition, single-context answering, report generation, or reliable temporal context exploitation. MC-CXR instead varies context reliability while holding the current image fixed, measuring arbitration under agreement and conflict.

  • Existing benchmarks: Existing medical VQA and CXR benchmarks evaluate image recognition, single-context VQA, and report generation largely as separate capabilities.These settings typically provide one context source and assess performance without paired context-induced disruption.
  • Longitudinal reasoning: Longitudinal benchmarks study prior studies and reports over time, generally assuming that prior context is reliable and measuring how well models exploit it.Examples include temporal, anatomy-centered, longitudinal, and sequential interpretation benchmarks.
  • Benchmark coverage: Compared with the cited benchmarks, MC-CXR combines indication, prior image, prior report, preliminary note, and overlay inputs with explicit paired-disruption metrics.Table 1 frames this combination as the benchmark’s distinguishing coverage.
  • MC-CXR distinction: MC-CXR treats context reliability as the variable by contrasting reliable conditions with clinically plausible misleading counterparts.This shifts evaluation from context exploitation toward evidence arbitration: deciding when context should or should not override the image-only decision.
  • Methodological connection: MC-CXR adapts case-internal contrast protocols from NLP to multimodal clinical reasoning, varying context reliability across modalities rather than only text surface form.The benchmark holds the current image fixed and pairs five clinically realistic context sources across reliable and misleading conditions.

3 MC-CXR Benchmark

MC-CXR evaluates whether VLMs preserve correct image-grounded decisions when reliable or misleading auxiliary context is added. Its paired design fixes the current image and target while measuring stability, switching, and directional alignment across text, prior-image, and visual-overlay contexts.

  • Task Formulation: MC-CXR fixes a current CXR image and target finding while varying auxiliary context reliability and modality.Contexts span clinical text, prior CXRs, and PACS-style visual overlays.
  • Task Formulation: The benchmark defines image-only recognition, reliable-context stability, and misleading-context resistance as three task families.Reliable contexts test preservation of an image-only correct decision; misleading contexts test resistance to plausible context implying Y ≠ X.
  • Design Principles and Metrics: Paired perturbations keep the current image and target fixed while contrasting reliable and misleading conditions, separating context use from context-induced disruption.Misleading contexts imply a specific Y, enabling directional attribution of errors.
  • Design Principles and Metrics: The switch-to-wrong rate measures flips among image-only-correct cases, while context-aligned error rate measures how often switched errors match the misleading label Y.Together, the metrics distinguish random instability from directional context pull.
  • Dataset Construction: MC-CXR contains 240 cases from 233 patients, expanded into 2,522 model-evaluable instances across 11 context conditions.Cases are drawn from MIMIC-CXR through eligibility filtering and radiologist review; radiologists also selected plausible misleading labels.
  • Dataset Construction: Table 3 summarizes directional Y-alignment among switches across ten VLMs, with lower values indicating better resistance to misleading context.The table contrasts systematic textual pull with lower and more scattered visual alignment.

4 Experiments

The experiments evaluate ten VLMs using image-only-correct conditioning to isolate context-induced switching. Results show substantial disruption, with misleading preliminary notes producing the strongest text-based effect and current-report-derived notes preserving stability most effectively.

  • Evaluation Protocol: Ten VLMs are evaluated across general-purpose open-source, medical-domain open-source, and closed-source categories.The listed models include five general-purpose open-source systems and two medical-domain open-source systems, with the remaining models in the closed-source category.
  • Evaluation Protocol: The main analysis conditions on each model’s image-only-correct subset, preventing baseline recognition failures from being counted as context-induced disruption.Subset size depends on image-only recognition accuracy on the 240-case benchmark.
  • Evaluation Protocol: The rewritten preliminary-note condition produces the strongest misleading-text disruption, while the current-report-derived preliminary-note condition is the most stability-preserving reliable context.Figure 3 reports per-model switch rates and across-model means, with lower switch-to-wrong rates preferred.
  • Aggregate Results: 66.2% is the across-model mean for the current-report-derived preliminary-note condition, while all five misleading-context conditions have means below 20% in absolute accuracy.Misleading preliminary notes are lowest at 9.3%, and cross-patient prior images are highest at 19.4% among those conditions.
  • Aggregate Results: The image-only to rewritten preliminary-note accuracy gap is 12.5–17.5 percentage points, largest for the medical-domain category.Under misleading preliminary notes, across-model means are 5.5% for general open-source, 16.3% for medical-domain, and 11.0% for closed-source models.

5 Failure Patterns in Context Use

MC-CXR reveals that misleading context disrupts otherwise correct image-only decisions, with textual context producing strongly label-aligned overrides and visual context producing more scattered errors. Even reliable context causes substantial switching, showing that context use can be unstable beyond explicit misinformation.

  • 5.1 Instability under Reliable Context: 28.8% of image-only-correct decisions switch under reliable context across the five conditions.The on-finding overlay and reliable-indication conditions are most disruptive, while the current-report-derived preliminary note is most stability-preserving at 13.3%.
  • 5.2 Textual Context Over-Reliance: 78.1% of image-only-correct decisions flip under misleading preliminary notes, with 85.3% landing on the context-implied label.Rewritten prior reports and clinical indications also produce systematic overrides, with flip rates of 60.9% and 45.6%, respectively.
  • 5.3 Visual Confusion vs. Language-Prior Bias: 61.7% of decisions switch under off-finding overlays, but only 16.5% align with the misleading label.Cross-patient prior images show the same pattern at 35.7% switching and 18.0% label alignment.
  • 5.3 Visual Confusion vs. Language-Prior Bias: 74.6% of switched errors align with misleading text versus 17.6% for misleading visual context, a 57.0-point gap.The gap persists after excluding rewritten preliminary notes, falling to 47.7 points, and every evaluated model shows the same direction.
  • Cross-pattern: Models do not always preserve image-only-correct decisions when misleading context is present under the standardized direct-answer protocol.Switch-based metrics condition on image-only-correct cases to isolate context-induced changes from baseline recognition error.

6 Conclusion

MC-CXR evaluates whether VLMs preserve image-grounded decisions when clinical context is added. Across ten models and five context sources, misleading text drives strongly label-aligned switching, whereas misleading visual cues produce scattered predictions; the benchmark is released for evaluating context robustness and interventions.

  • MC-CXR measures whether VLMs preserve image-grounded decisions when clinical context is added.
  • Misleading text produces directional switching, reaching 85% alignment for rewritten preliminary notes, while visually misleading cues produce scattered predictions.
  • MC-CXR is released as a paired benchmark for evaluating context robustness and corresponding interventions in medical VLMs.

Limitations

MC-CXR’s findings are bounded by conditioning, protocol, curation, contamination, output-format, and visual-cue limitations. The benchmark’s reliability controls and evaluation scope therefore constrain how its results should be interpreted.

  • IOR-correct conditioning isolates context-induced change but does not rule out chance-correct cases because IOR accuracy is only 8.3–35.8%.Cohen’s κ against gold X peaks at 0.282.
  • Results come from single deterministic runs under a constrained-letter direct-answer prompt, so the pattern is protocol-dependent rather than prompt-invariant.A GPT-5.5 conflict-aware pilot reduces misleading-text switches.
  • Single-rater curation and a 51-case second-reader audit do not establish benchmark-wide validation or κ.The audit supports directional Y-implication for three condition families.
  • MIMIC-CXR overlap cannot be audited, so MC-CXR makes no contamination-free claim.
  • The 12-label constrained-letter output offers no abstention and is not equivalent to free-text reporting.Misleading visual conditions were not source-only calibrated for Y recognizability, so low visual alignment may partly reflect weaker cue specificity.

Ethical Considerations

MC-CXR is restricted to research on multimodal model evaluation and should not support autonomous clinical interpretation. Its curation was performed by a single board-certified radiologist, and underlying MIMIC-CXR data remain access-controlled.

  • MC-CXR is intended solely for research on multimodal model evaluation and should not support autonomous clinical interpretation.
  • The benchmark was curated by a single board-certified radiologist who is an author, without external annotators.
  • Original MIMIC-CXR radiographs are not redistributed, and underlying data remain subject to the PhysioNet Credentialed Health Data License.
  • Intentionally misleading clinical context should be reported only for robustness evaluation, and benchmark performance alone should not establish clinical safety or unsafety.

A.1 Source Data and Filtering

MC-CXR uses a curated 240-case cohort from MIMIC-CXR, requiring matched current and prior frontal CXRs plus parseable reports. Radiologists verify target findings and construct clinically plausible misleading alternatives under a 12-label evaluation scheme.

  • Source Data and Filtering: The cohort contains 240 study-id-disjoint MIMIC-CXR cases with radiologist-verified labels, current and prior frontal studies, and parseable Findings or Impression sections.Cases with Pleural Other are excluded before evaluation.
  • Source Data and Filtering: Radiologists verify each target finding X and override incorrect source labels in 26 cases, or 10.8% of the benchmark.The misleading label Y is clinically plausible, distinct from X, and non-co-occurring with X in the current report.
  • Source Data and Filtering: Reliable context uses original indications, same-patient prior reports, same-patient prior CXRs, and finalized preliminary notes, while misleading text is rewritten from X toward Y.Rewrites preserve non-target clinical content such as location, laterality, severity, temporal wording, and negated findings; misleading prior images are cross-patient and view- and demographics-matched.
  • Source Data and Filtering: The evaluation presents 12 diagnostic labels, excluding Support Devices and Pleural Other from the gold pool.The labels include common findings such as Pneumonia, Edema, Atelectasis, Pneumothorax, Pleural Effusion, Fracture, and No Finding.

C Evaluation Configuration and Compute

The evaluation uses standardized single-letter answers across ten locally served or API-accessed VLMs, with deterministic decoding and native image preprocessing. A blinded audit evaluates sampled misleading-text cases, but does not establish benchmark-wide validity.

  • Evaluation Configuration and Compute: Seven open-source and three closed-source VLMs are evaluated, with closed-source models queried through provider APIs in May 2026.The closed-source systems are Claude Opus 4.7, GPT-5.5, and Gemini 3.5 Flash; parameter counts are undisclosed.
  • Evaluation Configuration and Compute: Open-source models use greedy decoding with bfloat16 weights and max_new_tokens=8, while closed-source models use temperature=0 and max_tokens=1024.Each model uses its native processor at default resolution caps.
  • Evaluation Configuration and Compute: Open-source inference runs on a single node with 4×NVIDIA RTX A6000 GPUs, requiring roughly 12–125 minutes per model and about ten GPU-hours overall.Closed-source evaluation incurs API cost only.
  • Evaluation Configuration and Compute: A blinded independent reader audits a stratified 51-case subset covering the three misleading textual conditions.Sampling targets four cases per target-label class where available, with redistribution for sparse classes.
  • Evaluation Configuration and Compute: The audit does not establish benchmark-wide interrater validity, prior-image or overlay validation, or confirmed-only performance.Cohen’s κ is not computed because paired ratings on the same categorical scale were unavailable.

D.2 Prompt Ablation Pilot (GPT-5.5)

The GPT-5.5 pilot tests whether misleading-text switches change under image-first or conflict-aware instructions, while documenting closed-API provenance and validation boundaries. Conflict-aware prompting reduces observed switches, but the pilot is not a full re-estimation.

  • D.2 Prompt Ablation Pilot (GPT-5.5): The pilot uses the fixed 73-case cohort that GPT-5.5 answered correctly under image-only input.Only the system instruction varies across baseline, image-first, and conflict-aware prompts.
  • D.2 Prompt Ablation Pilot (GPT-5.5): The baseline prompt produced d2 switches in 40/73 cases (54.8%) and e2 switches in 56/73 cases (76.7%).These are contemporaneous baseline switch rates for the pilot comparison.
  • D.2 Prompt Ablation Pilot (GPT-5.5): The image-first instruction showed no clear paired change, with d2 at 57.5% and e2 at 75.3%.The pilot compares these conditions on the same fixed original cohort.
  • D.2 Prompt Ablation Pilot (GPT-5.5): The conflict-aware instruction reduced switches to 37.0% for d2 and 41.1% for e2.Paired McNemar reductions were −17.8 pp for d2 and −35.6 pp for e2.
  • D.2 Prompt Ablation Pilot (GPT-5.5): The pilot is not a contemporaneous switch-rate re-estimation because matched reliable and visual conditions were not rerun.The closed-API alias was also not version-pinned, and some predictions differed between stored and contemporaneous baseline runs.
  • D.3 Closed-API Provenance: Closed-source models were queried through service aliases without immutable backend snapshot identifiers, so exact backend stability is not assumed.Table 4 records provider aliases, SDKs, decoding parameters, and pinning availability.
  • E Validation Protocol: Overlay placements were verified for 163 on-finding and 199 off-finding cases, but overlay conditions are undefined for the full 240-case cohort.Text and prior-image misleading contexts are revised or replaced when they violate the controlled construction criteria.
Loading 2608.24118v1…