Source-linked AI summary
LiveMedBench: A Contamination-Free Medical Benchmark for LLMs with Automated Rubric Evaluation
Zhiling Yan, Dingjie Song, Zhe Fang, Yisheng Ji, Xiang Li, Quanzheng Li, Lichao Sun
TL;DR
Existing medical benchmarks are static, vulnerable to contamination and temporal misalignment, and poorly suited to verifying open-ended clinical reasoning. LiveMedBench continuously curates real-world cases, validates them through a multi-agent framework, and evaluates responses with automated case-specific rubrics. Across 38 LLMs, performance degrades substantially on post-cutoff cases, with GPT-5.2 reaching 39.2% and 84% of models showing significant drops.
Problem
Static benchmarks risk temporal misalignment and contamination, while lexical-overlap metrics and LLM-as-a-Judge provide inadequate objective verification for open-ended clinical reasoning.
Method
LiveMedBench weekly harvests real-world clinical cases, validates them against authoritative evidence with multiple agents, and converts physician responses into granular weighted rubrics for grading model outputs.
Results
39.2% is achieved by GPT-5.2, while 84% of evaluated models show significant performance drops on post-cutoff cases; contextual application accounts for 35–48% of errors.
Takeaways & Limitations
LiveMedBench provides a continually updated benchmark for assessing whether LLMs can apply medical knowledge to patient-specific constraints under changing clinical conditions.
Abstract
from arXiv · showhide
The deployment of Large Language Models (LLMs) in high-stakes clinical settings demands rigorous and reliable evaluation. However, existing medical benchmarks remain static, suffering from two critical limitations: (1) data contamination, where test sets inadvertently leak into training corpora, leading to inflated performance estimates; and (2) temporal misalignment, failing to capture the rapid evolution of medical knowledge. Furthermore, current evaluation metrics for open-ended clinical reasoning often rely on either shallow lexical overlap (e.g., ROUGE) or subjective LLM-as-a-Judge scoring, both inadequate for verifying clinical correctness. To bridge these gaps, we introduce LiveMedBench, a continuously updated, contamination-free, and rubric-based benchmark that weekly harvests real-world clinical cases from online medical communities, ensuring strict temporal separation from model training data. We propose a Multi-Agent Clinical Curation Framework that filters raw data noise and validates clinical integrity against evidence-based medical principles. For evaluation, we develop an Automated Rubric-based Evaluation Framework that decomposes physician responses into granular, case-specific criteria, achieving substantially stronger alignment with expert physicians than LLM-as-a-Judge. To date, LiveMedBench comprises 2,756 real-world cases spanning 38 medical specialties and multiple languages, paired with 16,702 unique evaluation criteria. Extensive evaluation of 38 LLMs reveals that even the best-performing model achieves only 39.2%, and 84% of models exhibit performance degradation on post-cutoff cases, confirming pervasive data contamination risks. Error analysis further identifies contextual application-not factual knowledge-as the dominant bottleneck, with 35-48% of failures stemming from the inability to tailor medical knowledge to patient-specific constraints.
1 Introduction
LiveMedBench addresses static medical benchmarks’ temporal misalignment, contamination risk, and inadequate open-ended evaluation by continuously curating real-world cases and grading responses with case-specific rubrics. Its evaluation of 38 LLMs reveals substantial post-cutoff degradation and contextual application failures.
- Motivation: Static benchmarks become misaligned with evolving medical knowledge and vulnerable to contamination from training corpora.These weaknesses can make strong scores reflect memorization rather than genuine clinical reasoning.
- Motivation: Lexical-overlap metrics, multiple-choice formats, and LLM-as-a-Judge lack adequate verification of open-ended clinical correctness.The paper identifies missing semantic, clinical, and fine-grained objective criteria.
- Findings: 39.2% is achieved by GPT-5.2, while 84% of evaluated models show significant performance drops on post-cutoff cases.Error analysis attributes 35–48% of failures to difficulty synthesizing medical facts with patient-specific constraints.
- Contributions: 2,756 real-world cases span 38 specialties and multiple languages, with weekly updates designed to mitigate contamination and knowledge obsolescence.The benchmark is built from continuously harvested clinical data rather than fixed seed cases.
- Contributions: 16,702 unique, case-specific criteria enable scalable, fine-grained assessment of open-ended model outputs against physician responses.The Automated Rubric-based Evaluation Framework converts expert responses into objective rubrics and aligns evaluation with human experts.
2 Related Work
Related work has moved medical evaluation toward open-ended, multilingual, multimodal, and rubric-based assessment, while live benchmarks address contamination through fresh data. LiveMedBench adapts this live paradigm to clinical ambiguity with multi-agent validation and case-specific evaluation.
- Evolution of medical evaluation: Medical LLM evaluation progressed from static knowledge tests toward free-response, multimodal, multilingual, and structured rubric-based assessment.This evolution reflects the need to evaluate clinical safety, communication, diagnostic reasoning, and multi-step planning.
- Live benchmarking: LiveMedBench extends contamination-mitigation approaches based on freshly harvested content to high-stakes medicine.Unlike perturbation-based methods, it uses newly collected clinical cases and a medical curation process.
- Research gap: Existing live or perturbed benchmarks often rely on objective metrics that are unsuitable for clinical ambiguity.LiveMedBench instead combines multi-agent clinical validation with rubric-based assessment.
3 LiveMedBench
LiveMedBench constructs clinically grounded cases from verified online discussions and evaluates model responses with weighted, case-specific rubrics. Its pipeline combines source filtering, multi-agent validation, rubric generation, automated grading, and human quality assurance.
- Pipeline overview: The end-to-end pipeline collects and filters online medical discussions, curates structured cases, generates rubrics, grades model outputs, and summarizes dataset statistics.Figure 2 additionally identifies rigorous human quality assurance as the final phase for clinical alignment.
- Data collection: Source platforms are selected for weekly activity, institutional endorsement, and professionally verified physician participation.Posts must also meet recency, language, clinical relevance, duplication, and verified-response requirements.
- Clinical curation: The Multi-Agent Clinical Curation Framework uses Screener, Validator, and Controller agents to transform noisy threads into structurally complete, evidence-aligned cases.The Screener extracts patient narrative, query, and physician advice; the Validator checks validity, sufficiency, and evidence alignment; the Controller audits support from the original thread.
- Rubric generation: The rubric generator filters physician advice by behavioral theme and converts relevant facts into weighted positive and negative criteria.Criteria cover accuracy, completeness, communication quality, context awareness, and safety, with extreme weights representing life-critical details.
- Automated grading: The rubric grader sums weights for satisfied criteria, normalizes by the maximum positive score, clips the result to [0, 1], and averages case-level scores.Negative weights penalize hallucinations or contradictions, while the indicator function records whether each criterion is met.
- Dataset statistics: 2,756 cases and 16,702 criteria span 38 specialties, two languages, four platforms, five behavioral themes, and five evaluation axes.Accuracy accounts for 51.2% of criteria and Completeness for 24.7%, with 6.06 criteria per case on average.
4 Experiments
LiveMedBench evaluates 38 LLMs across proprietary, open-source, general-purpose, and medical-specific models using zero-shot inference and automated rubric scoring. Performance remains low and frequently declines on post-cutoff cases, indicating sensitivity to temporal separation and possible contamination.
- Experimental Setup: 38 LLMs were evaluated across proprietary, open-source, general-purpose, and medical-specific categories in a zero-shot setting.The evaluation used temperature 0; detailed specifications were provided separately.
- Evaluation Metric: Each response receives a normalized score from binary criterion judgments, with satisfied rubric criteria contributing to the final clinical-performance score.Scores are clipped to [0, 1], and higher values indicate better performance.
- Overall Model Performance: 39.2%: GPT-5.2 achieved the highest LiveMedBench score, followed by GPT-5.1 at 38.5%.Grok-4.1 and Baichuan M3 led non-GPT models at 28.3% and 25.6%, respectively.
- Proprietary vs. Open-Source Models: 25.0%: GPT-OSS 120B and 22.5%: GLM-4.5 matched or exceeded several higher-tier proprietary models.Gemini 3 Pro scored 18.3%, while Claude 3.7 Sonnet scored 17.0%.
- Data Contamination and Knowledge Obsolescence: 84% of models showed performance degradation on cases published after their knowledge cutoffs.The post-cutoff comparison was used to examine temporal misalignment, data contamination, and knowledge obsolescence.
5 Analysis
Human evaluation supports the clinical coherence of LiveMedBench and the reliability of its rubric grader. Analysis shows substantial cross-model weaknesses in completeness, communication, and patient-specific application, while retrieval improves performance on fresh cases.
- Human Evaluation and Reliability Analysis: 50 cases and 292 criteria were assessed by two bilingual physicians to validate data quality, rubric validity, and grader reliability.The protocol included ratings of patient narratives, queries, physician advice, generated criteria, and model responses.
- Data Quality and Consensus: 0.9792 and 0.9566: physician agreement reached near-perfect Gwet’s AC1 values for patient narratives and queries, respectively.Generated rubric criteria also achieved high consensus with AC1 = 0.8914.
- Reliability of the Rubric-based Grader: 0.76 Macro F1: the rubric grader aligned with human criterion-level judgments, below the 0.89 human inter-rater upper bound.The grader was evaluated against physician assessments.
- Superiority over LLM-as-a-Judge: 0.54 versus 0.26: the rubric grader’s Pearson correlation with human scores exceeded the non-significant LLM-as-a-Judge baseline.The baseline had p = 0.07, whereas the rubric grader had p < 10^-4.
- Qualitative Root Causes: 35–48%: Contextual Neglect and Integration Failure constituted the dominant classified error type, followed by Guideline Overgeneralization at 22–32%.Hallucinations and knowledge gaps were near 0–8% among leading models.
- Knowledge Injection: All model architectures improved when fresh January 2026 cases were augmented with external knowledge retrieval.The authors interpret this pattern as evidence that retrieval failures partially contribute to unseen-case failures.
- Benchmark Comparison: LiveMedBench integrates free-response generation, multilingual support, granular case-specific rubrics, and live case updates in one benchmark.Compared with HealthBench, tested models scored lower on LiveMedBench; Gemini 2.5 Pro scored 0.16 versus 0.19 on HealthBench Hard.
6 Conclusion
LiveMedBench combines continual case updates, evidence-backed curation, and rubric-based grading to evaluate clinical performance. Its construction draws on professional medical communities and adapts an existing clinical-evaluation taxonomy to the benchmark’s data.
- Conclusion: LiveMedBench is a continually updated, contamination-free benchmark built from noisy online discussions transformed into evidence-backed clinical cases.Its automated evaluation framework provides granular assessment aligned with physician experts.
- Data Collection: The benchmark collects publicly visible discussion content from DXY, MedLive, iCliniq, and SDN for academic purposes without training on the collected problems.The collection procedure excludes paywalled or login-required content.
- Evaluation Taxonomy: LiveMedBench retains five HealthBench themes and replaces Instruction Following with Safety to capture contraindication and harm-prevention failures.The dropped themes were Global Health and Health Data Tasks because they were less represented in the targeted online communities.
- Evaluation Taxonomy: Each case receives exactly one theme, while individual rubric criteria may span multiple evaluation axes.Themes represent high-level clinical-interaction challenges; axes describe behavioral dimensions evaluated by criteria.
A.4 Details of Data Statistics
LiveMedBench contains broad specialty and linguistic coverage, multiple behavioral themes, and granular case-specific evaluation criteria. Its distributions are intentionally varied, with the largest shares concentrated in common specialties, communication, and accuracy.
- Dataset Overview: 2,756 cases and 16,702 unique rubric criteria form the benchmark’s evaluation foundation.The dataset distribution is summarized in Figure 3.
- Clinical Specialties: 38 specialties are represented with a long-tailed distribution, led by GI/Hepatology with 375 cases, Urology with 210, and Psychiatry with 205.The coverage ranges from general practice to specialized fields such as Neuro-Oncology.
- Linguistic Diversity: 1,568 English and 1,188 Chinese cases support bilingual evaluation across the benchmark’s source platforms.English cases come from iCliniq and SDN; Chinese cases come from DXY and Medlive.
- Health Contexts: 47.4% of cases center on Expertise-Tailored Communication, followed by Responding under Uncertainty at 27.2% and Response Depth at 17.6%.Context-Seeking and Emergency Referrals account for 4.5% and 3.3%, respectively.
- Behavioral Dimensions: 51.2% of rubric criteria assess Accuracy and 24.7% assess Completeness, with Communication Quality, Context Awareness, and Safety comprising the remainder.The five-axis structure extends evaluation beyond factual recall.
- Granularity of Evaluation: Cases contain 2–19 criteria, averaging 6.06 criteria per case.The variation is intended to reflect differing complexity across medical scenarios.
B Experimental Setup
The study establishes human and automated consensus protocols for validating LiveMedBench data and rubric grading. It uses physician agreement measures, Macro F1, and correlation with physician consensus to assess reliability.
- Inter-rater Reliability: Gwet’s AC1 is preferred over Cohen’s Kappa because it remains stable under highly skewed category distributions.The choice addresses the Kappa Paradox in medical dataset evaluation.
- Criterion-level Consensus: Macro F1 averages positive- and negative-class F1 scores to prevent majority-class dominance in binary criterion grading.Criteria are classified as Met or Not Met, with accuracy potentially misleading under class imbalance.
- Consensus Protocols: Human-Human Macro F1 measures intrinsic grading difficulty, while Model-Human Macro F1 measures automated-grader alignment with physicians.Physician A is treated as ground truth for each comparison, and model alignment is averaged across physicians.
- Data Preparation: The screener extracts patient narratives and primary queries from raw dialogues for subsequent clinical validation.The narrative corresponds to SOAP Subjective and Objective sections, while the query identifies the medical request.
C.4 Case-level Consensus
Case-level consensus compares automated and physician scores using Pearson correlation, with the Rubric-based Grader evaluated against an LLM-as-a-Judge baseline. Error analysis categorizes failures by their clinical and instruction-following causes.
- Case-level Consensus: Pearson correlation compares automated case scores with physician consensus to assess whether scoring reflects professional judgment.The automated methods include the Rubric-based score and the LLM-as-a-Judge score.
- Case-level Consensus: The human ground-truth score is the arithmetic mean of two physicians’ case scores.This provides the reference for evaluating automated aggregate scoring.
- Case-level Consensus: The LLM-as-a-Judge baseline uses GPT-4.1 to assign normalized [0, 1] quality scores directly from model responses.The baseline follows a standard general-quality judging protocol rather than criterion-level rubric evaluation.
- Error Analysis: Error analysis distinguishes contextual neglect, knowledge gaps, safety overestimation, hallucination, guideline rigidity, reasoning deficits, and instruction misinterpretation.These dimensions cover failures ranging from ignored patient constraints to nonadherence to explicit query requirements.
D.1.2 Results.
Results show substantial weaknesses across evaluation axes and a shift in leading-model failures away from basic factual errors toward contextual application. Model failures also differ substantially across cases, supporting model-specific capability gaps rather than uniformly bad samples.
- Performance by Evaluation Axis: Most state-of-the-art models exceed 50% error rates across most evaluation axes.The result indicates substantial remaining weaknesses in current medical-domain architectures.
- Performance by Evaluation Axis: Performance is strongest in Accuracy but weaker in Communication Quality, Completeness, and Context Awareness.The disparity suggests factual correctness is better optimized than contextual and structural thoroughness.
- Failure Landscape: Leading models show low hallucination and knowledge-gap rates, with MHME at 0% for GPT-5.2 and GLM-4.5 and KGOC at 0-8%.Clinical Reasoning Deficits are also relatively low at 10-13%.
- Contextual Application: Contextual Neglect and Integration Failure accounts for 35-48% of failures, while Guideline Overgeneralization accounts for 22-32%.These categories reflect difficulty applying medical knowledge to patient-specific constraints.
- Error Correlation and Data Quality: The mean Jaccard similarity of bottom-100 cases is 0.24, indicating that models fail on different subsets rather than a common set of low-quality cases.Pairwise correlations remain below 0.4 across model pairs.
D.3 Correlation Analysis
Correlation analyses examine relationships among clinical specialties and behavioral themes, while the benchmark’s source-data safeguards address privacy, safety, and representation boundaries.
- Specialty Correlations: Cardiology, Infectious Disease, Hematology, and Pulmonary & Critical Care form a specialty cluster with correlations above 0.9.The authors suggest these fields may be frequently co-represented in general medical training corpora.
- Specialty Correlations: Pathology and Radiation Oncology correlate less with the broader specialty cluster, indicating limited transfer from general diagnostic reasoning.These specialized domains require distinct knowledge representations.
- Ethical Considerations: LiveMedBench de-identifies public data, excludes cases with re-identification risk, and discards content violating safety protocols.The pipeline retains real-world noise while applying privacy and content-safety filters.
- Ethical Considerations: The data may reflect demographic biases among internet users in the US and China and may underrepresent underserved populations.Future iterations are intended to diversify sources.
F Versioning and Reproducibility
LiveMedBench addresses the moving-target problem with weekly updates while preserving reproducible evaluation through frozen, timestamped snapshots and a stable grading environment.
- Weekly updates capture emerging clinical trends, while frozen snapshots preserve timestamped, checksum-verified benchmark versions.The snapshot strategy is designed to keep reported results reproducible as the benchmark evolves.
- Pinned API versions prevent automated grading from drifting with changing endpoint implementations.The Rubric-based Grader uses specific API versions instead of volatile latest endpoints.
- Releasing evaluation code and exact case rubrics enables offline auditing and consistent re-grading.