PaperlayerView exact revision on arXiv

Source-linked AI summary

SUP-MIMIC: A Multi-Task Clinical Diagnosis Benchmark for Evaluating LLMs' Robustness to Contradictory Evidence

Yi Yu, Bo Wang, Chong Feng, Ge Shi, Xia Liu, Ziyi Yang, Xuewen Shi

arXiv:2608.29582v1cs.CLcs.AI

TL;DR

Existing LLM benchmarks provide limited assessment of reasoning over non-bijective and contradictory clinical evidence. SUP-MIMIC constructs adversarial patient pairs from MIMIC-IV ICU records and evaluates Basic Assessment, Diagnostic Divergence, and Diagnostic Convergence. Across experiments, pairwise evaluation exposes inconsistencies and healthy prediction bias that pointwise accuracy can conceal.

  • Problem

    Existing evaluations focus largely on factual recall or isolated diagnosis and lack systematic assessment of causal reasoning under divergent and convergent clinical mappings.

  • Method

    SUP-MIMIC mines real ICU adversarial patient pairs and evaluates single-case diagnosis, divergent diagnoses among similar patients, and convergent diagnoses across heterogeneous presentations.

  • Results

    Pairwise evaluation exposes reasoning inconsistency and healthy prediction bias, while increasing model capability shifts errors toward multi-evidence integration failures.

  • Takeaways & Limitations

    Reliable clinical diagnosis requires advances in combinatorial reasoning and atypical case recognition beyond current scaling and medical pretraining strategies.

  • Takeaways & Limitations

    The benchmark excludes diagnoses with fewer than 100 ICU admissions and evaluates only structured EHR features, omitting imaging and waveform data.

Abstract

from arXiv · show

Current evaluations of large language models (LLMs) primarily focus on factual knowledge retrieval, overlooking the fundamental challenge of navigating the complex, non-bijective mappings between clinical indicators and diagnoses. Existing benchmarks fail to assess whether large language models truly possess the reasoning capability required for diagnostic ambiguity scenarios, where identical clinical presentations may correspond to different etiologies, and diagnostic convergence scenarios, where heterogeneous symptoms ultimately indicate the same disease. To address this issue, we propose SUP-MIMIC, a multi-task framework utilizing MIMIC-IV-v3.1 that comprises Basic Assessment (BA), Diagnostic Divergence Task (DDT), and Diagnostic Convergence Task (DCT). Specifically, DDT is designed to evaluate the model's "one-to-many" disambiguation capability among phenotypically similar cases, while DCT assesses the model's ability to identify "many-to-one" diagnostic patterns across different pathophysiological pathways. Comprehensive evaluation of state-of-the-art LLMs reveals substantial performance degradation on DDT and DCT compared to baseline tasks, exposing a systemic reliance on statistical shortcuts over genuine causal reasoning. Our findings further highlight a conservative bias toward "healthy" predictions, implying non-trivial risks for missed diagnoses in realistic medical settings. This work establishes a rigorous methodology for quantifying clinical reasoning robustness and provides a roadmap for enhancing the safety of language models in clinical medicine.

1 Introduction

Existing LLM evaluations provide limited evidence about robustness when clinical evidence is ambiguous, conflicting, or misleading. SUP-MIMIC addresses this gap with contrastive diagnostic tasks, finding that models’ strong single-case performance can coexist with pairwise inconsistency and healthy prediction bias.

  • Existing medical benchmarks predominantly test knowledge recall or isolated single-case diagnosis, offering limited insight into reasoning under ambiguous or conflicting evidence.
  • SUP-MIMIC uses adversarial patient pairs to evaluate robustness to contradictory evidence across Basic Assessment, Diagnostic Divergence, and Diagnostic Convergence tasks.
  • Models achieve substantially higher accuracy on BA than on DDT and DCT, showing that single-case evaluation can obscure failures under contrastive clinical evidence.
  • Pair-aware metrics reveal substantial pairwise inconsistency and a pervasive healthy prediction bias that increases the risk of missed diagnoses in challenging cases.
  • DDT tests whether models distinguish different diagnoses among clinically similar patients, whereas DCT tests whether they recognize the same diagnosis across heterogeneous presentations.

2 Related work

Prior clinical benchmarks and consistency methods often emphasize scenario-specific evaluation, surface patterns, predefined rules, or similarity matching. The paper identifies a gap in systematic assessment of causal and deep feature-diagnosis reasoning and proposes adversarial dataset construction to address it.

  • Clinical datasets and benchmarks support prediction, prognosis, treatment, text comprehension, and medical question answering, but evaluation frameworks remain scenario-specific.
  • Existing benchmarks primarily test surface-level pattern matching rather than pathophysiological understanding or causal reasoning.
  • Rule-based consistency checks and similarity-matching approaches are constrained by predefined rules or superficial features and fail to capture deeper commonalities.
  • The paper addresses this gap by formalizing non-bijective clinical reasoning, constructing adversarial evaluation datasets, and assessing implicit pathophysiological relationships.

3 Method

SUP-MIMIC formulates diagnosis as binary verification over structured ICU records and mines diagnosis-specific adversarial patient pairs using feature-based similarity. Expert validation then assesses the clinical informativeness of features and the diagnostic validity of sampled cases.

  • 3 Method: The method formulates multi-label patient records as binary patient-diagnosis verification, using the full 448-dimensional feature vector as model input.
  • 3 Method: The benchmark uses MIMIC-IV-v3.1 ICU records, restricts measurements to the first 24 hours, and imputes retained features with limited missingness.
  • 3 Method: Basic Assessment selects 200 representative diseases after excluding broad frequent labels and diagnoses with fewer than 100 positive cases.
  • 3 Method: Random Forest feature importance identifies diagnosis-specific top-Kf indicators used for patient similarity estimation and adversarial pair mining.
  • 3.3 Case Similarity and Adversarial Pair Construction: Similarity is computed within each diagnosis-specific feature space, with distances normalized across continuous, ordinal, binary, and categorical variables.
  • 3.3 Case Similarity and Adversarial Pair Construction: DDT mines clinically similar patient pairs with different diagnosis labels, while DCT mines heterogeneous pairs sharing the anchor diagnosis.
  • 3.3 Case Similarity and Adversarial Pair Construction: Two-stage expert validation reviews disease-specific feature sets and judges sampled cases using only their structured clinical indicators.

4 Experiments

SUP-MIMIC evaluates single-case accuracy alongside pairwise diagnostic robustness, revealing failures in distinguishing similar cases, recognizing heterogeneous presentations, and integrating cross-system evidence. Results indicate that medical pretraining improves consistency, while parameter scaling mainly benefits baseline diagnosis rather than adversarial robustness.

  • Evaluation Metrics: SUP-MIMIC evaluates BA, DDT, and DCT using binary diagnostic verification, reporting accuracy, Sick Recall, and Healthy Recall.Pairwise robustness is additionally measured by requiring correct decisions for both patients in each adversarial pair.
  • Main Results: PDRADDT drops from 0.68 to 0.37 for GPT-4o and from 0.58 to 0.20 for Llama3.3-70B despite strong pointwise performance.Balanced DDT construction removes class-imbalance advantages, exposing inconsistency across adversarial pairs.
  • Main Results: DCT accuracy remains below 0.53 for all models, while Healthy Recall exceeds Sick Recall by 30 to 50 absolute points.The pattern indicates difficulty accepting the anchor diagnosis when clinically dissimilar patients have atypical presentations.
  • Failure Mode Analysis: Combinatorial Neglect is the dominant failure mode at 34.0%, followed by Feature Misweighting at 23.1%, Comorbidity Conflation at 22.7%, and Biomarker Omission at 20.3%.Combinatorial Neglect is especially damaging in DDT, whereas Comorbidity Conflation is particularly relevant to DCT.
  • Adversarial Degradation: DDT and DCT degradation scores are nearly uncorrelated, so robustness to divergence does not predict robustness to convergence.Medical models cluster near zero degradation on both tasks, whereas general-purpose models are brittle along different axes.
  • Scaling Analysis: Scaling Qwen2.5 from 7B to 32B raises BA from 0.30 to 0.56 but DCT only from 0.31 to 0.35.The reported gains are non-uniform across tasks, with DCT showing a transient dip at 14B.
  • Disease Category Stratification: Cross-system disease categories show selective DCT collapse, whereas distinctive single-system biomarker categories maintain stable accuracy across tasks.ANOVA rejects uniformity for all three tasks, with significant DCT effects for metabolic and respiratory categories at p < 0.001.

5 Conclusion

SUP-MIMIC evaluates LLM robustness to contradictory evidence through adversarial patient pairs and multi-task diagnosis testing. Experiments show that pairwise evaluation exposes inconsistencies and that scaling or medical pretraining does not fully resolve adversarial robustness.

  • SUP-MIMIC evaluates LLM robustness to contradictory evidence using adversarial patient pairs mined from MIMIC-IV ICU records.
  • Pairwise evaluation exposes reasoning inconsistency that pointwise accuracy conceals.
  • As model capability increases, error profiles shift from basic omission toward integration failure.
  • Parameter scaling preferentially improves baseline performance while leaving adversarial robustness largely unaddressed.
  • Medical pretraining provides more uniform robustness across tasks but does not resolve the dominant bottleneck of multi-evidence integration.

Limitations

The study limits disease coverage by excluding diagnoses with fewer than 100 ICU admissions and evaluates only structured EHR features. It identifies rare-disease coverage and multimodal data integration as future expansion areas.

  • Diagnoses with fewer than 100 ICU admissions were excluded, limiting coverage of rare diseases.The authors propose manually annotated rare-disease cases to mitigate selection bias and expand diagnostic coverage.
  • The evaluation relies solely on structured EHR features and excludes medical imaging and waveform data.Future work could integrate multimodal data to more comprehensively simulate real-world clinical decision-making.

A Data Access and Licensing

SUP-MIMIC uses the de-identified MIMIC-IV-v3.1 database under approved credentialed access and releases its artifacts and evaluation frameworks for non-commercial academic research.

  • MIMIC-IV-v3.1 underwent HIPAA Safe Harbor anonymization, and access followed required safe research training and the PhysioNet Credentialed Health Data License.
  • SUP-MIMIC artifacts and evaluation frameworks will be released under the MIT License for non-commercial academic research.
  • AI assistants were used only for language polishing, grammar correction, and proofreading, not for scientific conclusions, dataset creation, or primary experimental code.

B.1 Feature Selection and Pair-Construction Framework

The pair-construction framework selects disease-specific important features to balance feature coverage, diagnostic accuracy, and cross-disease stability. K = 20 is adopted because it achieves the strongest reported balance without changing the full feature representation supplied to the LLM.

  • Feature Selection and Pair-Construction Framework: The framework varies K from 5 to 50 to assess feature utilization, diagnostic accuracy, and stability across diseases.
  • Feature Selection and Pair-Construction Framework: A random forest ranks features per disease, whose top-K unions determine the variables used to construct similar and dissimilar patient pairs.LLMs still receive all 448 features as input.
  • Feature Selection and Pair-Construction Framework: Increasing K from 5 to 20 raises feature utilization from 1.78% (8/448) to 51.33% (230/448), while accuracy rises from 0.2862 to 0.3563.
  • Feature Selection and Pair-Construction Framework: When K > 20, feature utilization reaches 57.36%–64.51%, but accuracy falls to about 0.31–0.32 and variance increases.Standard deviation rises from 0.0638 at K = 20 to 0.1177 at K = 25 and 0.1626 at K = 30.
  • Feature Selection and Pair-Construction Framework: K = 20 is selected as the default because it covers over half the original variables, achieves the highest overall accuracy, and maintains moderate cross-disease variance.This choice affects pair construction only; each patient remains represented by the full 448-dimensional feature vector.
  • Feature Selection and Pair-Construction Framework: Table 2 compares model performance and feature utilization across different numbers of important features using overall accuracy, variance, and standard deviation.

B.2 Confusion Pattern & Healthy Bias

Evaluated LLMs show a consistent discrepancy between disease and healthy-case recall, indicating a systematic preference for healthy predictions under diagnostic uncertainty. GPT-4o and HuatuoGPT-o1 provide the most balanced performance, while smaller open-source models are less robust.

  • Nearly all models achieve substantially higher recall on healthy samples than on disease samples.This discrepancy indicates a systematic healthy prediction bias under diagnostic uncertainty.
  • Figure 9 compares disease recall with healthy recall and annotates each model’s absolute performance gap.Models are ordered by descending disease-recall proficiency.
  • GPT-4o and HuatuoGPT-o1 achieve the highest disease recall while maintaining relatively strong healthy-case recognition.These models demonstrate a more balanced diagnostic capability among the evaluated systems.
  • Smaller open-source models exhibit lower disease sensitivity and larger performance variance in challenging diagnostic scenarios.

B.3 ROC/PR & Threshold Sensitivity

Confidence remains similarly discriminative across tasks, but default thresholds favor healthy predictions. Lower EER-optimized thresholds improve sick recall and reduce missed-diagnosis risk.

  • AUC changes by less than 0.025 from BA to CDT/DCT, while GPT-4o reaches approximately 0.83 uniformly across tasks.This indicates that confidence retains discriminative information despite biased default decisions.
  • At the default threshold of 0.5, all models exhibit healthy prediction bias.
  • EER thresholds cluster between 0.43 and 0.48, suggesting that lower thresholds better balance recall.
  • Sick Recall improves by 10–15 percentage points at EER thresholds, including GPT-4o’s increase from 52% to 65%.
  • MDR decreases by 20–25% at EER relative to the default threshold, while models assign higher confidence to healthy than sick predictions.

B.4 Benchmark Quality Assessment

Expert review supports the clinical relevance of SUP-MIMIC’s structured features and adversarial-pair construction. The benchmark uses standardized prompts and textualized structured ICU inputs for inference.

  • Benchmark Quality Assessment: Three practicing physicians with postgraduate degrees and at least eight years of clinical experience assessed the benchmark dataset.The panel included one intensive-care associate chief physician and two attending physicians from independent hospitals.
  • Benchmark Quality Assessment: Experts reached consensus that the selected top 20 disease features are tightly coupled with core pathophysiological mechanisms.These features were judged appropriate for constructing clinically meaningful adversarial patient pairs.
  • Benchmark Quality Assessment: Physicians evaluated the 448-dimensional structured representation using clinical indicators without free-text notes or imaging reports.They independently determined diagnoses and identified features supporting their decisions.
  • Benchmark Quality Assessment: The inference pipeline converts heterogeneous structured ICU variables into a unified natural-language representation for model reasoning.The request format integrates demographics, laboratory findings, and vital signs.
  • Benchmark Quality Assessment: Representative examples show the standardized system instruction and the feature presentation format used during inference.

D More error cases

The error cases illustrate missed or misinterpreted diagnostic evidence across diverse clinical scenarios. Examples include overlooked coagulation abnormalities, omitted biomarkers, and failure to integrate contradictory findings.

  • More error cases: The table categorizes representative LLM errors and marks reasonable evidence, misused evidence, and missing analytical support differently.
  • More error cases: The altered-mental-status assessment treats GCS 15 as strong negative evidence despite sepsis, hypoxemia, hypotension, and metabolic abnormalities.
  • More error cases: A sepsis-and-cancer case contains thrombocytopenia, prolonged coagulation times, and prior hypofibrinogenemia consistent with DIC or acquired coagulation defects.Later improvement in fibrinogen and platelet count does not negate the prior sustained abnormalities.
  • More error cases: A lung-cancer assessment relies on age, smoking, COPD, respiratory abnormalities, and infection despite lacking direct imaging evidence.
  • More error cases: Severe hyponatremia resembles adrenal crisis, but profound hypokalemia provides strong negative evidence against primary adrenal insufficiency.
  • More error cases: A young patient’s coma, hypercapnia, rhabdomyolysis, kidney injury, and extreme liver-enzyme elevation support overdose, immobilization, or toxin-related injury patterns.
Loading 2608.29582v1…