RubricsTree: Scalable and Evolving Open-Ended Evaluation of Personal Health Agents across Health Memory and Medical Skills

1. Introduction

• Static multiple-choice benchmarks cannot assess open-ended generation or multi-step agent actions, while expert annotation is costly and generalized auto-judges can be inconsistent and only partly expert-aligned.Real-world health queries may require longitudinal context and multi-turn, tool-augmented reasoning.Source: §1 · p. 2
• RubricsTree organizes macro-level capabilities into auto-weighted clinical leaf rubrics, each implemented as a binary verification function grounded in a concrete clinical reference.The tree checks for the presence or absence of clinically necessary data points.Source: §1 · p. 2
• Overall ICC3 0.876 and Cohen’s κ 0.787 versus 0.291 and 0.431 for the principle baseline; HealthBench gains reach +66.4% for Gemini/GPT and +66.7% for Qwen.The introduction also reports detection rates above 93% for inappropriate instructions and inaccurate user data.Source: §1 · p. 3
• An expert panel led by a physician iteratively curated the tree from 4,000 real PHA queries, while a context-aware router activates only the relevant rubric subset per query.The contribution summary describes a resource of 100+ atomic, clinically verifiable Boolean rubrics.Source: §1 · p. 3
• The meta-evaluation measures expert alignment, robustness to contextual perturbations, invariance across judge settings, and downstream optimization on expert-annotated datasets.The reported downstream gains use RubricsTree as structured instructions or optimization rewards.Source: §1 · p. 3

2. RubricsTree

• RubricsTree organizes over 100 clinical criteria as Boolean checks in a hierarchy and routes context-relevant rubrics for evaluation.The leaf criteria are grounded in medical literature or physician expertise, and the architecture aggregates active rubrics with auto weights.Source: §2 · p. 3, Figure 2 · p. 4
• RubricsTree is described as producing a stable evaluation signal, with high expert ICC and low variance across runs for each evaluation item.The stated stability applies to evaluation items in different runs.Source: §2 · p. 4

2.1. Human-in-the-Loop Taxonomic Curation and Evolution

• The curation DAG grounds its macro-level and intermediate strata in clinical knowledge, with terminal leaves verifying a context-response pair as binary criteria.Clinical knowledge is synthesized with incoming user queries to construct and refine the taxonomy.Source: §2.1 · p. 4
• Experts evolve the leaf set to minimize criteria and context that existing rules cannot verify, while a complexity penalty discourages over-segmentation.The process is grounded in medical knowledge and matures the taxonomy into verified atomic rules.Source: §2.1 · p. 4

2.2. Auto-Weighting: From Macro-level Domains to Micro-verifiable Leaf Nodes

• RubricsTree distributes weight uniformly down the hierarchy from a root weight of 1, keeping each atomic check proportionally anchored to its macro-level domain.Leaf weights depend recursively on the number of direct children at each parent node.Source: §2.2 · p. 5

2.3. Context-Aware Adaptive Routing Mechanism

• Because each query concerns only a subset of health issues, adaptive routing selects relevant rubrics using semantic overlap and a soft threshold.The routing design avoids brittle keyword-based constraints.Source: §2.3 · p. 5
• Hierarchical traversal prunes irrelevant subtrees, while rubric-triggered thresholds vary by query; emergency queries use lower thresholds for recall.Binary leaf-level judges and pure embedding similarity yield worse routing quality and latency in the reported ablations.Source: §2.3 · p. 5
• Active Boolean checks form a normalized weighted dimension score, and failure of a highly weighted relevant criterion proportionally degrades it.Each active rubric's weight reflects its depth and ancestral significance in the tree.Source: §2.3 · p. 6

3. Experiments and Meta-Evaluation

• The meta-evaluation tests expert alignment, sensitivity to contextual perturbations, consistency, and downstream utility; DR measures detection frequency and ΔMP measures penalty magnitude.Expert agreement is measured against a separate six-member panel using ICC3 for continuous scores and Cohen’s κ for criterion-level categorical agreement.Source: §3 · p. 6, §3.1 · p. 6

3.1. Human Expert Agreement

• 0.876 vs. 0.291 Overall ICC3 and 0.787 vs. 0.431 Overall κ for RubricsTree versus Principle Baseline.RubricsTree also has higher κ in all four scenarios; the largest gains occur where holistic baseline scoring is weakest, consistent with the proposed atomic-rubric approach.Source: Figure 3 · p. 6, §3.1 · pp. 6–7

3.2. Oracle Evaluation on Contextual Perturbations

• 62.9%-100% DR across tested cells; RubricsTree dominates Principle Baseline in all 16, while the baseline has negative ΔMP in 9.Across three personas, RubricsTree reaches DR=100% in most cells; on Persona 3 Symptoms, it retains DR≥50% and positive ΔMP. For 20 sampled inaccurate-data queries, its score differences stay positive while baseline differences swing roughly −0.20 to +0.20.Source: Table 2 · p. 7, §3.2 · pp. 7–8, Figure 4 · p. 8

3.3. Consistency and Stability of Automated Evaluation

• RubricsTree exceeds Principle Baseline on ICC3 at all 19 axis points and retains lower run-to-run variance across four variation axes.The stability gap persists at T=0.1, and the ICC3 gap is essentially flat across four Gemini evaluation backbones.Source: §3.3 · pp. 8–9, Figure 5 · p. 9

3.4. Downstream Optimization on HealthBench

• +18.6% to +66.4% relative gains over base across eight models with Prompt and Response Optimization, concentrated on Completeness and Context Awareness.Prompt Optimization supplies a structured clinical handbook; Response Optimization feeds routed leaf-level pass/fail feedback into a targeted revision.Source: §3.4 · pp. 9–10, Figure 6 · p. 10
• +66.7%, +55.3%, and +40.3% testing-score improvements for Qwen 0.6B, 1.7B, and 4B, respectively, using RubricsTree as an RL reward.The smallest model has the highest relative improvement; the rubric aggregate becomes a dense reward that guides policy updates and penalizes clinical or agentic reasoning errors.Source: Figure 7 · p. 10, §3.4 · p. 10

4. Conclusion

• RubricsTree pairs an expert-curated hierarchy of Boolean rubrics with context-aware adaptive routing to address open-ended personal-health-agent evaluation.Source: §4 · p. 10
• RubricsTree exceeds the user-validated principle baseline in expert alignment, penalizes degraded contexts, stays stable across judges and temperatures, and improves HealthBench-Hard outcomes for Gemini and GPT.Source: §4 · p. 11

4.1. Limitations

• Transfer beyond the consented cohort needs further curation; rare safety rubrics may be under-routed, and one physician on the panel leaves possible specialty bias.Source: §4.1 · p. 11

A. Related Work

• Prior work has shifted toward open-ended, sensor-aware health agents, but their evaluation remains costly or relies on subjective judges [2].Source: A. Related Work · pp. 14–15

B.2. Expert Panel Composition

• A nine-member panel curated the taxonomy from about 4,000 queries, while a mostly separate six-member panel evaluated expert agreement.Source: §B.2 · p. 16

B.3. Internal Evaluation Dataset

• The internal evaluation set contains 532 real-world queries across four clinical scenarios and is not publicly released because it includes proprietary clinical content and user-contextual health information.Source: §B.3 · p. 16
• Robustness tests apply four contextual perturbations and stratify analysis across three patient personas to assess sensitivity to clinically meaningful context degradation.Source: §B.3 · p. 16

B.4. Judge Backbones, Sampling, and Prompt Roles

• Reported numbers generally average three runs; consistency tests vary four judge backbones, five temperatures, and five instruction roles.Source: §B.4 · p. 16

B.5. HealthBench-Hard Subset and Downstream Model Suite

• Downstream optimization uses 362 user-facing HealthBench-Hard queries with Gemini and GPT models; prompt optimization supplies a static rubric handbook, while response optimization returns targeted feedback after one evaluator pass.Source: §B.5 · p. 17

B.6. HealthBench-Consensus Subset for RL Training

• Qwen reinforcement learning uses user-facing HealthBench-Consensus criteria validated by physician agreement, with routed rubric scores aggregated as scalar rewards.Source: §B.6 · p. 17

C.1. The Autonomous ReAct Pipeline

• The agent loops through query triage, tool calls, observations, and response synthesis, retrieving more data when the observations do not yet support a clinically sound answer.The pipeline allows at most two parallel tool calls per reasoning step.Source: C. Personal Health Agent Pipeline and… · p. 17, §C.1 · pp. 17–18

C.2. The Clinical and Health-Analysis Tools

• Eight deterministic Python-based tools separate data retrieval from clinical-index calculations, leaving rigid medical mathematics to verifiable code.Source: §C.2 · p. 18

C.3. Representative ReAct Trace: 11-Step Autonomous Workflow

• The hypertension example traces 11 autonomous steps, with tool invocations chosen dynamically as the agent identifies knowledge gaps.Source: §C.3 · p. 18

D. Adaptive Routing Engine Ablation

• The router traverses the curated taxonomy by expanding relevant nodes’ children, with a query-specific activation threshold; alternatives use embeddings or independent leaf-level judgments.Source: D. Adaptive Routing Engine Ablation · p. 18
• 80.61% overall routing accuracy and 5.6 seconds average latency are reported for hierarchical traversal, which the authors adopt as the primary router.Source: Table 4 · p. 19, D. Adaptive Routing Engine Ablation · p. 21
• Embedding similarity runs in 3.2s but is brittle to clinical paraphrases; binary per-leaf judging reaches 76.14% semantic precision at 64.2 seconds latency.Source: D. Adaptive Routing Engine Ablation · p. 21

• Verification-based annotation achieved higher accuracy, precision, recall, and F1 agreement than annotation from scratch.Physicians’ from-scratch rubric selection averaged over 80% alignment across samples.Source: E. Adaptive Process Annotation:… · p. 21, Table 6 · p. 21, Table 5 · p. 21
• The authors conclude that verification-based review scales manual review without sacrificing clinical rigor.The reported comparison concerns adaptively generated rubric selections reviewed, verified, and corrected by annotators.Source: E. Adaptive Process Annotation:… · p. 21

• At T=0.1, the variance gap persists; Principle Baseline is most unstable on the Symptoms scenario, at 0.018.Variance measures absolute score spread across repeated runs, whereas ICC3 measures the proportion of total variance attributable to between-item differences rather than within-item noise.Source: F. Per-Item Run-to-Run Variance Analysis · p. 22
• 3 to 9× lower variance: RubricsTree ranges [0.002, 0.005] versus Principle Baseline [0.005, 0.018], independent of judge backbone or temperature.Figure 9 reports per-item run-to-run variance across sampling temperatures, clinical scenarios, prompt roles, and judge backbones; lower is better.Source: Figure 9 · p. 22, F. Per-Item Run-to-Run Variance Analysis · p. 22

• RubricsTree maintains DR=100% across the majority of persona-by-category-by-perturbation cells, with strictly positive ΔMP.The passage contrasts this pattern with the Principle Baseline failure on Persona 3 Symptoms.Source: G. Per-Persona Oracle Perturbation… · p. 22, Table 7 · p. 23
• The authors identify atomic Boolean verification with semantic routing as most consequential in the Persona 3 Symptoms regime, which holistic scalar evaluators miss.Source: G. Per-Persona Oracle Perturbation… · p. 23
• ΔMP<−95% on every perturbation regime for Principle Baseline on Persona 3 Symptoms, where its score increases on degraded responses.Table 7 stratifies mean penalty and detection rate by persona, clinical category, and perturbation regime.Source: G. Per-Persona Oracle Perturbation… · pp. 22–23, Table 7 · p. 23

H. Sample-Level Oracle Perturbation Cases

• Figures 10–12 show per-query mean score differences between clean baselines and corrupted responses with instructions, missing partial user data, or inappropriate instructions.The captions specify standard-error whiskers across runs and settings but provide no numerical outcomes.Source: Figure 10 · p. 24, Figure 11 · p. 24, Figure 12 · p. 25

I.1. HealthBench-Hard Subset Description

• The evaluation uses 362 user-facing queries from HealthBench-Hard, a high-complexity benchmark whose open-ended responses are graded against physician-adjudicated criteria.The criteria span all 21 ICD-10 chapters and 26 specialties, prioritizing clinical accuracy, patient safety, and communication quality.Source: §I.1 · pp. 23, 25

I.2. Response Optimization Pipeline

• The response-optimization pipeline has three phases: adaptive rubric selection, criteria-specific evaluation, and feedback-guided refinement.It uses an actor-evaluator setup in which a generated response is evaluated and then refined using targeted feedback.Source: §I.2 · p. 25

I.2.1. Phase 1: Adaptive Rubric Selection (Triage and Classification)

• Adaptive selection chooses query-relevant rubrics using clinical intent, response utility, and trigger-condition matching.The framework avoids evaluating every response against the entire rubric catalog, which is described as computationally inefficient and prone to noise.Source: §I.2.1 · pp. 25–26

I.2.2. Phase 2: Criteria-Specific Base Evaluation (The Clinical Auditor)

• The clinical auditor independently checks each selected rubric for omissions, missing data, or hallucinated clinical trends and returns reasoning with a binary pass/fail score.A score of 1 indicates satisfaction of the rubric; 0 indicates failure.Source: §I.2.2 · p. 26

I.2.3. Phase 3: Feedback-Guided Response Refinement

• Feedback-guided refinement targets failed criteria, adding missing information or follow-up questions while changing original statements only when flagged as incorrect or unsafe.The resulting output is the final optimized response.Source: §I.2.3 · p. 26

I.3. Per-Axis Optimization Results

• Per-axis optimization on Gemini-2.5-Flash is decomposed into Overall Score, Completeness, and Context Awareness, with results shown in Figures 13–15.The supplied passages identify the axes and figures but do not report their result values.Source: §I.3 · pp. 26–27, Figure 13 · p. 27, Figure 14 · p. 27
• Supplementary RubricsTree-driven optimization evaluates GPT-5-series models for completeness, with overall scores reported in Figure 16.The supplied passages do not give the figure's numerical results.Source: §I.3 · p. 27, Figure 16 · p. 28

I.4. Full Prompt Templates

• Rubric selection uses query relevance, expected improvement, and each rubric’s trigger condition, with user-health-memory selection focused on roughly 5–8 key metrics unless broader coverage is requested.The prompt directs the selector to identify all rubrics meeting those relevance criteria.Source: §I.4 · p. 29
• The auditor prompt adopts an adversarial stance, directing evaluators to look for omissions and assess the specified criterion rather than presume the response is safe.It also cautions against treating a safety disclaimer as sufficient for a high score.Source: §I.4 · p. 30
• The evaluation prompt instructs the auditor to judge a specific rubric against the query and response using a binary pass/fail output.The base protocol uses 1 for pass and 0 for not pass; the alternate format requests one short reason before the binary result.Source: §I.4 · p. 30
• Feedback-guided revision is instructed to preserve the original response while adding needed clarifications, except where feedback flags content as incorrect, unsafe, or necessary to avoid.The revised response is returned to address the original query.Source: §I.4 · pp. 31–32