LiveMedBench: A Contamination-Free Medical Benchmark for LLMs with Automated Rubric Evaluation

Motivation

• Static benchmarks [25] risk temporal misalignment as guidelines and policies change and practice evolves, while public test sets may contaminate training and inflate scores through memorization [19].Source: §1 · pp. 1–2

Limitations of existing approaches

• Lexical metrics such as BLEU miss clinical correctness, multiple-choice formats poorly fit open-ended workflows [10], and LLM-as-a-Judge lacks explicit criteria [41] [1].Prompt-based dynamic benchmarks cannot update seed cases and risk model-generated bias [26] [39], whereas MedArena and HealthBench require costly, extensive human annotation [1].Source: §1 · p. 2

LiveMedBench

• LiveMedBench contains 2,756 real-world cases across 38 specialties and multiple languages, harvested weekly from active online communities and checked against authoritative medical evidence.The curation pipeline uses multiple agents with Retrieval-Augmented Generation to validate clinical plausibility and consistency with established standards.Source: §1 · p. 2

Evaluation framework

• The Automated Rubric-based Evaluation Framework decomposes physician responses into 16,702 unique, case-specific objective criteria for scalable grading of open-ended outputs aligned with human experts.Source: §1 · p. 2

Results

• GPT-5.2 achieves only 39.2%, while 84% of evaluated models show significant performance drops on post-cutoff cases.The evaluation spans 38 proprietary and open-source models across general-purpose and medically specialized architectures.Source: §1 · p. 2
• Contextual application accounts for 35–48% of errors, and retrieval-based knowledge injection recovers much performance loss, implicating knowledge gaps rather than innate reasoning deficits.Source: §1 · p. 2

2.1 Medical Benchmarks

• Evaluation shifted from static knowledge tests such as MedQA [10] and PubMedQA [11] toward free-response, multimodal, multilingual, and rubric-based clinical reasoning.HealthBench [1] pioneered case-specific rubrics; LiveMedBench builds on these trends with automated, scalable rubric generation and continuously refreshed real-world cases.Source: §2.1 · p. 2

2.2 Dynamic and Live Benchmarks

• LiveMedBench adapts live evaluation to medicine, addressing contamination and temporal misalignment with multi-agent validation of clinical integrity, safety, and empathy.LiveBench [33] and LiveCodeBench [9] harvest fresh news or coding content, whereas DyReMe [39] and MedPerturb [7] perturb seed cases; these approaches often rely on objective metrics unsuited to clinical ambiguity.Source: §2.2 · p. 2

3.1 Data Collection

• The benchmark harvests discussions from four professionally verified medical communities and filters for recent, clinically relevant, text-only, nonduplicate posts with physician responses.Posts must be from January 1, 2023 or later, in English or Chinese, contain an ICD, ICF, or ICHI keyword, and have fewer than 50% non-alphanumeric characters; the protocol follows [25].Source: §3.1 · p. 3

3.2.1 The Screener.

• The Screener converts each thread into a SOAP-structured narrative, query, and physician-advice triplet, forwarding it only when all three components are present.It aggregates patient facts, extracts the request for guidance, and identifies actionable recommendations from verified physician replies.Source: §3.2.1 · pp. 3–4

3.2.2 The Validator.

• The Validator checks whether the query is a meaningful chief complaint, the narrative supports more than half of required information, and advice aligns with evidence.Any contradicted advice element vetoes the case; evidence alignment and narrative sufficiency must each exceed 0.5.Source: §3.2.2 · p. 4

3.2.3 The Controller.

• The Controller verifies every narrative and advice element against the original thread, discards unsupported or fabricated content, and synthesizes passing cases into canonical narrative, query, and advice.Source: §3.2.3 · p. 4

3.3.1 Automated Rubric Generator.

• The Rubric Generator uses a case’s dominant behavioral theme to select relevant advice facts, formulate positive and negative criteria, and assign evaluation axes and weights.Axes are Accuracy, Completeness, Communication Quality, Context Awareness, and Safety; weights range from −10 to 10, with extreme weights reserved for life-critical details.Source: §3.3.1 · p. 4

3.3.2 Rubric-based Grader.

• The Rubric-based Grader checks whether each atomic criterion is met, sums satisfied weights, normalizes by the maximum positive score, clips case scores to [0, 1], and averages them.Source: §3.3.2 · p. 4

3.4 Data Statistics

• LiveMedBench contains 2,756 cases and 16,702 unique rubric criteria across 38 specialties, with 1,568 English and 1,188 Chinese cases.Expertise-Tailored Communication is the leading behavioral theme (47.4%); Accuracy and Completeness account for 51.2% and 24.7% of criteria, respectively. Cases average 6.06 criteria (range: 2–19).Source: §3.4 · p. 4

4.1 Experimental Setup

• The evaluation covered 38 proprietary, open-source, and medical-specific LLMs in zero-shot settings, scoring responses by rubric criteria met and normalizing satisfied points to [0, 1].The grader used binary Met/Not Met judgments; higher normalized scores indicate stronger clinical performance.Source: §4.1 · pp. 4–5
• Two English- and Chinese-proficient physicians validated 50 cases (292 criteria) for data quality and criterion validity, and independently assessed responses to evaluate grader reliability.The study used Gwet’s AC1 for physician inter-rater reliability, Macro F1 for criterion-level grader agreement, and Pearson correlation for case-level scoring alignment.Source: §4.1 · p. 5

4.2 Overall Model Performance

• GPT-OSS 120B (25.0%) and GLM-4.5 (22.5%) matched or exceeded Gemini 3 Pro (18.3%) and Claude 3.7 Sonnet (17.0%), while smaller Qwen3-14B (15.4%) was competitive with Gemini 2.5 Pro (16.1%).The paper notes that open-source models can serve privacy-sensitive local and resource-limited edge deployments.Source: §4.2 · pp. 5–6
• 39.2% on LiveMedBench gives GPT-5.2 the top score, ahead of GPT-5.1 at 38.5%; Grok-4.1 and Baichuan M3 lead non-GPT models at 28.3% and 25.6%.Figure 4 compares all 38 models, with full-dataset results shown as solid bars and post-cutoff results as hatched bars.Source: §4.2 · p. 5, Figure 4 · p. 6
• General-purpose GPT-5.2 (39.2%) outperformed the top specialized model, while Med-Gemma 27B (5.9%) nearly matched Gemini 2.5 Flash (6.4%).The authors suggest general models benefit from scale and diverse training, while the comparison indicates specialized tuning can remain effective at smaller scales.Source: §4.2 · p. 6
• 84% of models (32/38) scored lower on post-cutoff cases, and Kimi-K2 had the largest reported drop, 3.99%.Average performance also trended downward on contemporary cases, including those published after January 2025; the authors attribute the pattern to contamination inflating seen-case scores and knowledge obsolescence on unseen cases.Source: §4.2 · p. 6

5.1 Human Evaluation and Reliability Analysis

• Physicians’ agreement was ≥0.9795 for patient narratives, queries, and physician advice, with AC1 values of 0.9792 and 0.9566, respectively; rubric-criteria AC1 was 0.8914.The study randomly sampled 50 cases and 292 corresponding criteria; the authors interpret these results as evidence of coherent, clinically valid cases and precise criteria.Source: §5.1 · pp. 6–7, Table 1 · p. 6

5.2 Performance across Specialties and Themes

• Proprietary models led across specialties and behavioral themes, but performance was stronger in common GI/Hep and EM than in niche specialties, while Context-Seeking remained a safety bottleneck.Models generally aligned strongly on Expertise-Tailored Communication and Responding under Uncertainty; the specialty analysis also reports low correlation between general internal medicine and niche fields such as Pathology.Source: Figure 5 · p. 7, §5.2 · p. 7

5.1 Human Evaluation and Reliability Analysis

• The Rubric-based Grader achieved Macro F1 0.76 versus Human Inter-Rater F1 0.89, and Pearson correlation 0.54 (p<10−4) versus 0.26 (p=0.07) for LLM-as-a-Judge.The paper attributes the baseline’s weak performance to higher-score skew, length bias, and low sensitivity to safety hazards; the rubric grader assigned zero for critical errors such as missed contraindications.Source: Table 2 · p. 7, §5.1 · p. 7

5.3 Error Analysis

• 35–48% of leading-model errors were CNIF, versus 22–32% GOPR and near 0–8% each for MHME and KGOC, pointing to difficulty tailoring knowledge to patient-specific constraints.Bottom-100-case overlap was low across models (mean Jaccard similarity=0.24), suggesting model-specific capability gaps rather than data-quality artifacts.Source: §5.3 · p. 8

5.4 Impact of Knowledge Injection

• External knowledge retrieval improved performance across all model architectures on January 2026 cases, while the Baseline-to-Closed-Book drop indicates effects of contamination and knowledge obsolescence.The comparison tested baseline performance on the full dataset against closed-book and retrieval-augmented open-book inference on fresh cases.Source: Table 4 · p. 8, §5.4 · p. 8

5.5 Medical Benchmark Comparison

• LiveMedBench uniquely combines free-response generation, multilingual support, granular case-specific rubrics, and live case updates among the compared benchmarks.The authors frame this combination as addressing the need for clinically rigorous, temporally adaptive evaluation.Source: Table 5 · p. 8, §5.5 · p. 8
• Gemini 2.5 Pro scored 0.19 on HealthBench Hard versus 0.16 on LiveMedBench, whose scores were significantly lower than on standard HealthBench.The authors interpret the gap as evidence of LiveMedBench’s complexity and room for future improvement.Source: Table 6 · p. 8, §5.5 · p. 8

6 Conclusion

• Evaluation of 38 LLMs showed consistent performance degradation on post-cutoff cases, supporting the need for live benchmarking.The benchmark combines a Multi-Agent Clinical Curation Framework, which turns noisy online discussions into evidence-backed cases, with an Automated Rubric-based Evaluation Framework for granular assessment aligned with physician experts.Source: §6 · p. 8

A.1 Online Medical Communities

• DXY is described as China’s largest online academic healthcare community, with physician-verified Q&A spanning clinical specialties.Its strict verification process is presented as supporting professional, reliable advice grounded in clinical practice.Source: §A.1.1 · p. 10

A.1.1 DXY.

• MedLive provides physician-focused clinical decision-making and academic exchange through technically dense discussions, expert consensus interpretations, and complex cases.The passage characterizes its terminology and clinical depth as useful for evaluating specialized medical-context handling.Source: §A.1.2 · p. 10

A.1.2 MedLive.

• iCliniq connects patients with verified doctors across more than 80 specialties and uses a second-opinion format for detailed, evidence-based responses.Its structured patient-query and physician-response interactions provide English-language training and evaluation data.Source: §A.1.3 · p. 10

A.1.3 iCliniq.

• SDN clinical forums provide clinician-to-clinician discussions, including collaborative reasoning, case breakdowns, and debates on treatment protocols.The passage distinguishes these exchanges from standard doctor-patient interactions and places them in a North American educational community.Source: §A.1.4 · p. 10

A.2 License

• The collection scrapes only public post discussions from DXY, MedLive, iCliniq, and SDN; problems are for academic use and are not used for training.The passage says the approach follows [9] and invokes Fair Use §107, excluding paywalled content and material requiring login or interaction.Source: §A.2 · p. 10

A.3 Theme and Axis Definitions

• LiveMedBench adapts HealthBench [1], retaining five of its seven themes and replacing Instruction Following with Safety to address contraindication and harm-prevention failures.Each case receives exactly one theme; the themes assess emergency referral, context-seeking, expertise-tailored communication, uncertainty, and response depth.Source: §A.3 · p. 10, §A.3.1 · p. 10

A.3.2 Axes.

• Rubric axes assess Accuracy, Completeness, Communication Quality, Context Awareness, and Safety; a case may have criteria spanning multiple axes.They cover factual correctness, clinically important information, clear and audience-matched communication, contextual cues and clarification, and avoidance of harm with appropriate warnings.Source: §A.3.2 · pp. 10–11

Dataset size

• 2,756 clinical cases and 16,702 unique rubric criteria make up LiveMedBench’s current dataset.Figure 3 illustrates the dataset’s detailed distribution.Source: §A.4 · p. 11

Clinical Specialties

• 38 clinical specialties are represented, with Gastroenterology and Hepatology at 375 cases, Urology at 210, and Psychiatry at 205.The specialty distribution is long-tailed and ranges from general practice to highly specialized fields.Source: §A.4 · p. 11

Linguistic Diversity

• 1,568 English cases and 1,188 Chinese cases support bilingual evaluation.English cases come from iCliniq and SDN; Chinese cases come from DXY and Medlive.Source: §A.4 · p. 11

Health Contexts

• 47.4% of cases concern Expertise-Tailored Communication, followed by Responding under Uncertainty (27.2%) and Response Depth (17.6%).Context-Seeking accounts for 4.5% and Emergency Referrals for 3.3%.Source: §A.4 · p. 11

Behavioral Dimensions

• 51.2% of rubric criteria assess Accuracy, while Completeness accounts for 24.7%, Communication Quality 9.4%, Context Awareness 7.8%, and Safety 6.9%.The rubric spans five evaluation axes.Source: §A.4 · p. 11

Granularity of Evaluation

• Cases receive 2–19 criteria each, averaging 6.06, with criterion counts following a normallike distribution.The number of checks varies with the complexity of each medical scenario.Source: §A.4 · p. 11

B.1 Models

• Table 7 lists the models considered in the study.The model details are provided in Table 7.Source: §B.1 · p. 11, Table 7 · p. 12

B.2 Details in Curation Framework

• The curation framework documents prompts for the screener, validator, and controller.The screener prompts are shown in Figures 6–8, the validator prompts in Figures 9–11, and the controller prompt in Figure 12.Source: §B.2 · p. 11

B.3 Details in Evaluation Framework

• The evaluation framework provides prompts for the Rubric Generator and Rubric-based Grader.Their prompts appear in Figures 13 and 14, respectively.Source: §B.3 · p. 11

C.1 Rating Scale

• Tables 8–10 detail the rating criteria physicians received for each evaluation task.Source: §C.1 · p. 11

C.2 Inter-rater Reliability

• Gwet’s AC1 addresses the Kappa Paradox by estimating chance agreement from average category marginals, remaining stable under skewed category distributions.The study uses AC1 instead of Cohen’s Kappa because high agreement can produce paradoxically low Kappa scores when trait prevalence is unbalanced.Source: §C.2 · p. 11

C.3 Criterion-level Consensus

• Following HealthBench [1], criterion-level Macro F1 evaluation uses physician and grader ratings of 50 randomly selected cases based on unique criteria.Source: §C.3 · pp. 11–12
• Macro F1 treats criterion grading as binary, with “Met” positive and “Not Met” negative, and equally averages class F1 scores to limit majority-class dominance.Source: §C.3 · p. 12
• Consensus evaluation measures physician–physician Macro F1 to gauge task difficulty and averages automated-grader Macro F1 against both physicians because no single gold standard exists.The model–human average is intended to indicate whether the grader aligns with medical professionals’ consensus view.Source: §C.3 · pp. 12–13

C.4 Case-level Consensus

• The Rubric-based Grader and a GPT-4.1 LLM-as-a-Judge baseline were compared with two-physician consensus using Pearson correlation.Consensus was defined as the arithmetic mean of the physicians’ scores; higher correlation indicates better reflection of professional judgment’s nuance and variance. The judge followed the standard protocol in [41].Source: §C.4 · pp. 13–14

D.1.1 Setup.

• Contextual neglect covers failure to use patient history, vitals, and laboratory results, producing generic responses or overlooking contraindications.Related failures include applying guidelines too rigidly without adapting them to the individual case.Source: §D.1.1 · pp. 14–15
• Clinical reasoning deficits include failing to synthesize clinical data into a coherent diagnosis or plan, prioritize problems, or infer secondary conclusions.The taxonomy also includes misinterpreting the query or ignoring explicit constraints such as requested format or word count.Source: §D.1.1 · p. 15
• Knowledge-gap errors include incorrect or outdated medical information and incomplete answers reflecting insufficient domain knowledge.The passages note that outdated protocols can result from training-data cutoffs.Source: §D.1.1 · p. 16
• Safety overestimation includes false refusals of benign queries and unnecessary emergency escalation for minor, self-limiting conditions.These errors are attributed to miscalibrated safety guardrails.Source: §D.1.1 · pp. 16–17
• The taxonomy also covers fabricated medical facts or citations and moderation filters that block legitimate clinical discussion.These are grouped as misclassification, hallucination, and moderation errors.Source: §D.1.1 · p. 17

D.1 Error Analysis

• Error analysis classified model failures into seven dimensions and was conducted on leading proprietary and open-source models.The prompt used for the analysis is shown in Figure 15.Source: §D.1.1 · p. 14, §D.1.2 · p. 17

D.1.2 Results.

• Error Rate counts unmet positive criteria and triggered negative criteria as errors.Source: §D.1.2 · p. 17
• Models generally had error rates above 50% across most evaluation axes, while leading models scored well in Accuracy but struggled with Community Quality, Completeness, and Context Awareness.This disparity suggests factual correctness is better optimized than contextual sensitivity and structural thoroughness.Source: §D.1.2 · pp. 17–18
• 35–48% of errors were Contextual Neglect and Integration Failure (CNIF), versus 22–32% from Guideline Overgeneralization (GOPR), making contextual application the predominant bottleneck.The analysis also reports Misclassification/Hallucination (MHME) at 0% for GPT-5.2 and GLM-4.5, Knowledge Gaps (KGOC) at 0–8%, and Clinical Reasoning Deficits (CRSD) at 10–13%; patient constraints such as comorbidities, allergies, and history can make theoretically correct advice unsuitable.Source: Table 11 · p. 18, §D.1.2 · pp. 18–19
• A qualitative audit of the bottom-100 cases for each of six representative models used GPT-4.1-generated error descriptors, semantically clustered into seven error types.The types include contextual neglect and integration failure, guideline overgeneralization and rigidity, clinical reasoning and synthesis deficit, instruction misinterpretation and nonadherence, knowledge gaps, safety overestimation, and misclassification or hallucination.Source: §D.1.2 · p. 18
• 0.24 was the mean Jaccard similarity of model pairs’ bottom-100 cases, with all pairwise correlations below 0.4, indicating models failed on different cases rather than a common set of poor-quality samples.The authors interpret this as evidence that low scores reflect model-specific capability gaps, not dataset-construction artifacts.Source: Figure 16 · p. 19, §D.1.2 · p. 19

D.2 Theme Correlation

• The section investigates transferability of model alignment skills by examining Pearson correlations among the five evaluation themes in a correlation matrix.Source: §D.2 · p. 19

D.3 Correlation Analysis

• Cardiology, Infectious Disease, Hematology, and Pulmonary & Critical Care form a specialty cluster with pairwise performance correlations of r>0.9.The correlations are Pearson coefficients across 38 clinical specialties.Source: §D.3 · p. 19, Figure 18 · p. 20
• The cluster’s strong inter-dependency suggests these specialties are frequently co-represented in general medical training corpora such as PubMed and MIMIC.Source: §D.3 · p. 19
• Pathology and Radiation Oncology show lower correlations with the broader cluster, indicating that general diagnostic reasoning may not transfer automatically to these specialized domains.The paper attributes this isolation to the distinct knowledge representations these domains require.Source: §D.3 · p. 20, Figure 18 · p. 20

E Ethical Considerations

• The benchmark redacts names, specific locations, and dates of birth, excludes cases with re-identification risk, and discards threads containing hate speech, discriminatory language, or medically unsafe advice.It retains real-world noise to test model robustness.Source: E Ethical Considerations · p. 20
• LiveMedBench is intended only for research evaluation, not medical advice, self-diagnosis, treatment planning, or clinical-deployment-model training without regulatory validation.Source: E Ethical Considerations · p. 20
• The internet sources may reflect demographic biases among users in the US and China and underrepresent underserved populations; future iterations aim to diversify sources.Source: E Ethical Considerations · p. 20

• Timestamped, checksummed Frozen Snapshots (e.g., LiveMedBench-v2026.01) preserve reproducibility despite weekly updates.Source: F Versioning and Reproducibility · p. 20
• Pinned grader API versions, released evaluation code, and case-specific rubrics prevent grading drift and enable offline auditing and consistent re-grading.Source: F Versioning and Reproducibility · p. 20