Retrieval-Augmented Agentic Rubric Generation for Reliable Medical Response Evaluation

1 Introduction

• Instance-specific medical rubrics address the gap between costly expert assessment and coarse automated evaluation criteria.Generic metrics miss clinical reasoning, while expert assessment and fine-grained annotation are difficult to scale.Source: §1 · pp. 1–2
• The framework retrieves authoritative evidence, decomposes it into atomic facts, models interaction constraints, and audits the resulting rubric.Figure 1 organizes the process into retrieval and evidence preparation, dual-track constraint construction, and audit and refinement.Source: §1 · p. 2, Figure 1 · p. 2
• 22.9% and 11.0% relative improvements over GPT-4o were achieved on HealthBench and LLMEval-Med CIA, reaching 50.20% and 31.90%.These results support stronger clinical-intent alignment across the two medical benchmarks.Source: §1 · p. 2
• +7.8% higher win rate than GPT-4o and 0.977 AUROC on HealthBench indicate stronger sensitivity to subtle clinical errors.The reported discriminative evaluation compares minimally flawed alternatives using rubric-guided assessment.Source: §1 · p. 2
• Rubric-guided refinement improves downstream response quality by 9.2%, extending the framework beyond evaluation.The generated criteria provide actionable feedback for controlled response edits.Source: §1 · p. 2

2 Related Work

• Closed-form benchmarks and text-similarity metrics inadequately capture semantic nuance, context, and clinically important errors.Human assessment improves realism but remains labor-intensive and costly.Source: §2 · pp. 2–3
• Generic LLM judges improve scalability but often rely on fixed or latent standards and remain sensitive to phrasing and framing.Prior reliability improvements include structured protocols, retrieval verification, debate, self-consistency, and ensemble aggregation.Source: §2 · p. 3
• Existing rubric-based medical judges improve transparency, but expert-authored criteria and human annotation limit scalability as medical knowledge evolves.HealthBench and LLMEval-Med use dialogue-specific criteria, while SedarEval still requires substantial annotation.Source: §2 · p. 3
• Clinical queries encode interaction constraints such as urgency, uncertainty, incomplete history, and communication style alongside information needs.These requirements motivate evaluation criteria tailored to both medical facts and conversational context.Source: §2 · p. 3
• The proposed framework aims to bridge expert-rubric validity and generic-judge scalability through authoritative evidence and explicit interaction modeling.Its positioning combines clinical relevance, interpretability, and scalability in one evaluation framework.Source: §2 · p. 3

3 Method

• The framework formalizes rubric generation as a multi-stage mapping from user query Q and knowledge base K to structured rubric R.Criteria c_j, evaluation axes a_j, and clinical weights w_j define the rubric representation.Source: §3 · p. 3

3.1 Stage 1: Retrieval & Evidence Preparation

• The Routing Agent transforms complex queries into optimized searches, while the Evidence Synthesis Agent consolidates authoritative candidates into evidence E.Cross-checking and deduplication resolve source conflicts and extract contraindications and red-flag warnings.Source: §3.1 · pp. 3–4

3.2 Stage 2: Dual-Track Constraint Construction

• The Dual-Track scheme separately decomposes evidence into atomic medical facts and extracts explicit or implicit interaction constraints.The Medical Fact Agent creates a Reference Board, while the Interaction Intent Agent identifies missing contextual variables needed for context-aware evaluation.Source: §3.2 · p. 4

3.3 Stage 3: Audit & Refinement

• The Rubric Synthesis Agent integrates verified facts, interaction constraints, and the query into an initial rubric R_init.This stage compiles both factual and conversational requirements before auditing.Source: §3.3 · p. 4
• The Auditing Agent performs gap analysis, removes unsupported or irrelevant constraints, and merges the result into a final clinically auditable rubric.The audit specializes Reflexion-style refinement toward medical safety and factual coverage, balancing rigor with communication quality.Source: §3.3 · p. 4

4.1 Datasets

• 254 HealthBench dialogues and LLMEval-Med clinical questions provide English and Chinese evaluation settings with expert-authored criteria as CIA gold standards.The HealthBench subset contains instances with 8–10 gold criteria across at least three evaluation axes, yielding approximately 2.5k rubric items.Source: §4.1 · pp. 4–5

4.2 Implementation Detail

• Each query uses a reference response and a near-miss candidate containing exactly one clinically meaningful perturbation, such as omitted warnings or altered urgency.GPT-4.1 generates candidates while preserving the remaining content, enabling tests of subtle clinical-error detection.Source: §4.2 · p. 5

4.3 Baselines

• The study compares generic, Llama Direct, GPT-4o, and no-rubric baselines that differ in query specificity, retrieval, decomposition, and explicit guidance.The no-rubric setting directly compares responses and serves as a reference for rubric-based evaluation.Source: §4.3 · p. 5

4.4 Evaluation Metrics

• Generated rubrics are evaluated for clinical coverage and discriminative sensitivity using CIA, win rate, mean score delta, and AUROC.CIA measures entailment of atomic expert constraints, while discriminative metrics assess ranking of reference responses over perturbed candidates.Source: §4.4 · p. 5, §4.4.2 · p. 6, §4.4.3 · p. 6

4.4.1 Scoring and Bias Mitigation

• The scoring function sums criterion satisfaction indicators weighted by integer clinical-severity values from −10 to 10.Each criterion contributes according to whether response X satisfies it and the weight assigned by the Rubric Synthesis Agent.Source: §4.4.1 · pp. 5–6
• Position bias is controlled by swapping response order across trials and computing the average score delta over those trials.The protocol defines score comparisons for each response position before aggregating the deltas.Source: §4.4.1 · p. 6

4.4.2 Clinical Intent Alignment (CIA)

• CIA decomposes expert rubrics into atomic constraints and tests whether the generated rubric entails each constraint, with precision and F1 reported alongside recall.The complementary metrics address CIA’s recall-oriented design.Source: §4.4.2 · p. 6

4.4.3 Discriminative Sensitivity

• Discriminative sensitivity reports win rate, mean score delta, and AUROC, with metric variability estimated using 1,000 nonparametric bootstrap resamples and 95% percentile intervals.A positive mean score delta indicates wider separation between reference and perturbed responses.Source: §4.4.3 · p. 6, §4.4.4 · p. 6

5 Results and Analysis

• The evaluation covers clinical intent alignment against gold standards and discriminative ability under near-miss conditions.These are the two reported aspects of rubric quality.Source: §5 · p. 6

5.1 Clinical Intent Alignment (CIA)

• Our method achieves the highest Clinical Intent Alignment across HealthBench and LLMEval-Med, improving over GPT-4o by +9.36% and +3.15%, respectively.CIA measures coverage against physician-authored rubrics, and the comparison is statistically significant by McNemar’s test.Source: §5.1 · p. 7, Table 1 · p. 7
• 50.20% recall, 29.24% precision, and 36.96% F1 make the framework strongest overall on HealthBench’s complementary precision–recall evaluation.The gain primarily reflects increased coverage rather than unconstrained rubric expansion, although moderate precision indicates some unsupported criteria remain.Source: §5.1 · p. 7, Table 2 · p. 7

5.2 Discriminative sensitivity under Near-Miss Conditions

• Rubrics achieve the highest win rate, largest mean score difference, and highest AUROC among evaluated settings, improving judge discrimination under near-miss conditions.Without rubrics, judges show high tie rates and limited score separation.Source: §5.2 · p. 7, Table 3 · p. 7
• Rubric guidance mainly amplifies subtle clinical score differences rather than forcing binary decisions, while absolute win rates remain below 0.4 for near-identical response pairs.Figure 2 summarizes mean score differences, win/tie/lose outcomes, and AUROC.Source: §5.2 · p. 7, Figure 2 · p. 7

6 Ablation Study

• Component contributions vary across datasets, but interaction-intent modeling provides the most consistent contribution to clinical-criterion coverage.The ablation study evaluates components on both HealthBench and LLMEval-Med.Source: §6 · p. 8
• Removing the Interaction Intent Agent causes the largest CIA degradation on HealthBench (−14.60) and LLMEval-Med (−23.90).This highlights the importance of modeling user-specific communication needs and contextual constraints alongside medical facts.Source: §6 · p. 8
• Removing the Routing Agent and Auditing Agent consistently reduces CIA and Win Rate across both datasets.These effects support targeted evidence retrieval and closed-loop gap analysis within the framework.Source: §6 · p. 8
• The Medical Fact Agent has the smallest CIA effect, with changes of −0.40 on HealthBench and −5.18 on LLMEval-Med.Atomic decomposition primarily makes evidence-to-criterion mapping explicit and auditable rather than driving coverage directly.Source: §6 · p. 8

7 Rubric-Guided Response Refinement

• The study asks whether instance-specific rubrics improve response quality through refinement without re-generation.This models a deployment setting in which an initial response is edited rather than regenerated.Source: §7 · p. 8

7.1 Task Setup and Baselines

• The experiment uses 254 HealthBench queries with frozen Llama-3.1-8B-Instant base responses, so improvements arise solely from refinement.Temperature is 0.7 and top_p is 0.9; no resampling or regeneration is performed.Source: §7.1 · p. 8
• Self-Critique provides a no-rubric lower-bound control, while physician-authored Reference Rubrics provide an oracle upper-bound control.The controls distinguish intrinsic self-correction from refinement under ideal expert guidance.Source: §7.1 · p. 8

7.2 Refinement Mechanism

• The Critique-then-Refine protocol converts rubric checks into a structured JSON Edit Plan containing prioritized actions.An evaluator reviews the base response against the rubric before an editor applies the resulting plan.Source: §7.2 · p. 8
• Constraint-guided editing applies only the Edit Plan and prohibits new medical facts or definitive diagnoses absent from the original context.These constraints are intended to reduce refinement-induced hallucinations.Source: §7.2 · p. 8

7.3 Evaluation Protocol

• Refinement is evaluated independently by an external judge using HealthBench’s physician-authored criteria rather than the automatically generated rubrics.This separates rubric use for editing from the criteria used to assess the resulting responses.Source: §7.3 · p. 9

7.4 Response Refinement Results

• Rubric-guided refinement consistently outperforms self-critique, and the proposed rubrics produce the largest automatic improvement while substantially narrowing the gap to the oracle.Reference rubrics remain the physician-authored upper bound.Source: §7.4 · p. 9, Table 5 · p. 9
• Improvements are strongest for factual accuracy and completeness, while communication gains are more modest; the proposed rubrics better balance factual improvement and readability than reference rubrics.These dimension-wise effects are illustrated in Appendix Figure 3.Source: §7.4 · p. 9

• The framework grounds instance-specific rubric generation in authoritative medical evidence while separating clinical constraints from interaction-level requirements.This produces structured, interpretable rubrics intended to reflect case-specific clinical priorities.Source: §8 · p. 9
• Empirical results show consistent improvements across HealthBench and LLMEval-Med in English and Chinese medical evaluation settings.The generated rubrics improve clinical coverage of gold key points and distinguish high-quality responses from minimally flawed alternatives.Source: §8 · p. 9
• Interaction-intent modeling provides the most consistent improvement in clinical-criterion coverage across datasets.Ablation results indicate that component contributions vary between datasets.Source: §8 · p. 9
• Instance-specific rubrics also serve as actionable feedback for controlled response refinement without requiring re-generation.This extends the framework beyond evaluation toward response improvement.Source: §8 · p. 9
• Automatic rubric generation offers a scalable and transparent foundation connecting fine-grained clinical assessment with large-scale automated judging.The conclusion positions rubric-centered evaluation as relevant to both assessing and improving medical language models.Source: §8 · p. 9

• The framework is evaluated with a single Llama-3.3-70B-Instruct backbone, leaving performance with smaller or other proprietary models unexamined.The authors also note that automated evaluators cannot fully replace nuanced clinical judgment from human medical experts.Source: Limitations · p. 10
• The multi-agent architecture averages approximately 43 seconds per query, limiting immediate real-time applicability without optimization.The rubric weights are clinically motivated but heuristic and lack systematic sensitivity analysis.Source: Limitations · p. 10
• Experiments primarily use fixed query-response pairs and do not fully capture informal, multi-turn consultations with follow-up questions and longitudinal context.Tracking historical context in multi-turn consultations remains future work.Source: Limitations · p. 10
• Downstream refinement is demonstrated only in a controlled, single-step setting, leaving more flexible or interactive strategies unexplored.The authors identify interactive refinement as future work.Source: Limitations · p. 10

Prompt Templates

• Figure 3 analyzes dimension-wise downstream refinement and trade-offs across evaluation dimensions, while Figure 4 illustrates grading against physician-written rubrics tailored to a conversation.The supplied descriptions identify the visual scopes but do not report specific cell values or a single winning setting.Source: Figure 3 · p. 15, Figure 4 · p. 15
• The appendix includes a taxonomy of authoritative medical sources selected to improve clinical reliability and reduce hallucination during rubric generation.It also summarizes the mathematical information states and functional mappings used by the pipeline.Source: Table 6 · p. 16, Table 7 · p. 16, Table 8 · p. 17
• Judge-robustness analysis repeats CIA evaluation with three judge models, while supplementary tables show instance-specific and generic rubric examples.The computational-cost table reports per-agent efficiency metrics using Llama-3.3-70B-Instruct.Source: Table 10 · p. 17, Table 9 · p. 17, Table 11 · p. 18
• The appendix catalogs prompts for routing, evidence synthesis, atomic fact extraction, interaction-intent inference, rubric synthesis, auditing, and pairwise judging.Together, these templates correspond to the framework’s staged multi-agent pipeline.Source: Table 13 · p. 19, Table 14 · p. 20, Table 15 · p. 20
• The Medical Fact Agent decomposes retrieved evidence into atomic facts and filters them for query relevance and safety-preserving constraints.This supports the construction of clinically grounded evaluation criteria.Source: Table 15 · p. 20, Table 16 · p. 21
• The Interaction Intent Agent infers user persona, missing clinical context, and appropriate response tone for safe dialogue grounding.These interaction requirements complement the medical evidence used in rubric synthesis.Source: Table 17 · p. 21, Table 18 · p. 22