RareDx: Controlled Knowledge Integration and Graph-Grounded Policy Optimization for Rare-Disease Diagnosis

• Rare diseases affect fewer than 1 in 2,000 people individually but more than 300 million collectively, and diagnosis commonly takes 4-5 years amid incomplete, heterogeneous, overlapping phenotypes.Source: §1 · p. 1
• Prior methods include general-LLM benchmarks, agentic systems combining retrieval or tools, and structured systems using phenotype, genotype, and disease knowledge for prioritization.Source: §1 · p. 2
• Monolithic evaluation obscures which pipeline components yield gains, while exact-match rewards are sparse and unconstrained semantic rewards can favor fabricated names or indiscriminate lists.Source: §1 · p. 2
• RareDx couples controlled evidence use with graph-grounded post-training; its complete 9B system reaches 38.34 macro Hit@10, 1.60 points above GPT-5.5 under the archived protocol.Source: §1 · p. 2

2.1 Problem formulation

• For k = 1, 5, or 10, Hit@k measures whether any accepted diagnosis appears among the first k canonicalized predictions; Hit@10 is primary for differential diagnosis.Source: §2.1 · p. 2

2.2 RareDx-Harness: controlled diagnostic evaluation

• The harness normalizes cases and output format across direct inference, static retrieval, adaptive ReAct tools, structured gene-to-disease reasoning, and reciprocal-rank fusion.Source: §2.2 · p. 2, Figure 1 · p. 3
• 30.54 to 37.34 macro Hit@10 for 9B and 29.83 to 38.13 for 27B are the reported gains under routing selected on development data and evaluated on the remaining 80%.Source: §2.2 · p. 3

2.3 RareDx-KGPO: knowledge-graph-grounded policy optimization

• The two-stage training teaches a valid Top-10 output contract through SFT, then uses group-relative advantages from graph-grounded clinical rewards, resampling zero-variance groups and regularizing policy shifts.Source: §2.3 · p. 3, Figure 2 · p. 4

2.4 Projecting generations into a medical knowledge graph

• Generated strings are projected into canonical disease entities, then scored using curated graded relevance, ontology paths, biomedical entity similarity, and phenotype-profile overlap.Source: §2.4 · pp. 3–4
• The reward adds a list-quality term that distinguishes differential rankings sharing a top candidate; its ideal ordering uses the same curated graded relevance scale.Source: §2.4 · p. 4

2.5 Evaluation setup

• Evaluation covers eight benchmark sources with ten-disease outputs scored by one deterministic normalizer, equal-weight macro scores, and no shared patient identifiers or normalized case text with training.Source: §2.5 · p. 5

3.1 Overall diagnostic performance

• 38.34 macro Hit@10 for RareDx-Harness 9B exceeds GPT-5.5 by 1.60 points across eight benchmarks.With disjoint development selection, 9B and 27B reach 37.34 and 38.13 Hit@10, versus 30.54 and 29.83 for Direct on matched held-out cases.Source: §3.1 · p. 5, Table 1 · p. 6

3.3 Reward-component ablation and adversarial audit

• On 120 held-out diagnoses, phenotype credit gives the nearest non-reference HPO neighbor 0.316 mean reward versus 0.008 without it, while exact diagnoses score 1.0.The vocabulary gate rejects every constructed pseudo-disease, whereas removing it rewards 82.5%; with rank 1 fixed, list-aware scores for coherent, reversed, and unrelated lists are 1.000/0.997/0.911.Source: §3.3 · p. 6

3.2 RL algorithm comparison

• RareDx-KGPO reaches approximately 0.58 reward without late collapse, while GRPO reaches 0.40 before collapsing after step 210.KGPO ties positive credit to canonical entities and graph-supported relations, keeping reward aligned with bounded, patient-specific differentials rather than generic disease lists.Source: §3.2 · p. 5, Figure 3 · p. 7

3.4 Harness comparison and component ablation

• Harness gains are 3.21/6.00/6.99 points at Hit@1/5/10 for 9B and 3.71/7.41/10.30 for 27B.Gains concentrate on RAMEDIS and HMS, with an additional large 27B gain on RDS; the small LIRICAL regression and dataset dependence support case-aware rather than uniform evidence acquisition.Source: Figure 4 · p. 7, §3.4 · p. 7
• Static RAG underperforms Direct, while HPO-Resnik raises Phenopackets gene recall@8 from 15.3% to 69.4% and Hit@10 from 27.5% to 36.3%.No acquisition strategy dominates across heterogeneous records; ReAct depth has a non-monotonic association with Hit@10, with one and two calls scoring 57.96 and 59.31 versus 43.32 at three calls, a descriptive rather than causal comparison.Source: §3.4 · pp. 7–8

3.5 Case study: from recognition to a usable differential

• On the fixed 50-case RAMEDIS audit, the 9B Harness recovers 16 diagnoses missed by direct 27B, loses seven, and nets nine Hit@10 cases.Thirteen recoveries enter the Top-5; under the same token budget, direct 27B averages 462 words and completes no required answer block, while post-trained 9B averages 91 words and completes all 50.Source: §3.5 · p. 8, Figure 6 · p. 9

3.6 Efficiency analysis

• On the same eight-A800 setup and 6,249 records, direct 9B generation runs at 45.5 cases/s versus 18.4 for 27B, a 2.48-fold throughput advantage.Timings cover direct batched generation and exclude loading, scoring, retrieval, and tool latency.Source: §3.6 · p. 9

4 Discussion

• Controlled evidence acquisition, normalization, and ranking—not greater parametric medical knowledge—are presented as the basis of RareDx-9B’s advantage.The discussion says structured resources can rival backbone scale particularly at Hit@5 and Hit@10, where coherent coverage matters alongside first-choice confidence.Source: §4 · p. 9
• Dataset heterogeneity, a small MME split, and automatic rather than clinical adjudication constrain the evaluation; benchmark accuracy does not establish clinical safety.The authors call for prospective assessment of calibration, harmful omission, evidence faithfulness, care-setting robustness, subgroup performance, and clinician revision rates.Source: §4 · p. 9

Use of Large Language Models

• The authors report using a language-model coding assistant for code review, refactoring, figure generation, and manuscript presentation, while retaining responsibility for research and results.They state that reported results came from the described experiments and citations were checked against identifiable publications.Source: Use of Large Language Models · p. 10

Ethics Statement

• RareDx is intended for academic research, not clinical advice or decisions without independent expert review and appropriate validation.The statement also assigns users responsibility for privacy, data governance, consent, and responsible disclosure.Source: Ethics Statement · p. 10

Reproducibility Statement

• The authors archive evaluation artifacts and generate numerical results and plots programmatically with fixed scripts; code and weights are slated for release after peer review.Training and implementation details are provided in the appendix.Source: Reproducibility Statement · p. 10

A Benchmark composition

• The macro score gives each of eight benchmark sources equal weight despite different sample counts, with RareBench coverage and RAMEDIS, MME, HMS, and LIRICAL reported separately.Table 4 records the test-record composition used throughout the main results.Source: A Benchmark composition · p. 14, Table 4 · p. 14

B Harness router specification and leakage-controlled audit

• The archived router chooses between frozen RareDx Top-10 Direct predictions and a deterministic Qwen3.6-35B Diagnostic Audit, not among all Harness strategies.The audit receives patient evidence and the Direct list, returns canonical disease names, and uses no retrieval or external tools.Source: B Harness router specification and… · p. 14
• A SHA-256 hash of seed, dataset, and case ID assigns about 20% of cases to development and the remainder to test for each model-dataset pair.The stated seed is 20260919.Source: B Harness router specification and… · p. 14
• Route selection prioritizes development Hit@10, then Hit@5 and Hit@1; exact ties choose cheaper Direct, and selection occurs once per model and benchmark source.No test score enters the selection equation; Audit is considered only where a complete frozen output exists.Source: B Harness router specification and… · p. 14
• On eight test partitions, the validation-only router reaches 18.81/30.21/37.34 for 9B and 21.71/34.69/38.13 for 27B.Across ten fixed hash seeds, route selection varies for some smaller or benchmark-specific sets, motivating reports of every candidate.Source: B Harness router specification and… · p. 14
• The Table 1 headline route was selected during exploratory development using benchmark-level results, so Table 5 is the appropriate estimate when validation-only selection is required.Frozen Direct rows are unaffected by this distinction.Source: B Harness router specification and… · p. 14

C Exact reward implementation

• A candidate maps to the nearest canonical vocabulary entry only when normalized Levenshtein distance is at most 0.20; unmapped strings receive no dense medical credit unless curated as graded gold.The accepted diagnosis and clinically related neighbors form the curated set, with graded relevance, ontology proximity, embedding similarity, and phenotype consistency contributing to mapped-candidate reward.Source: C Exact reward implementation · pp. 14–15
• Exact or synonymous ontology matches receive 1, while one- and two-edge relations receive 0.5 and 0.25; the base score discounts lower ranks by log2(k + 1).The base score is the maximum rank-adjusted candidate score across the Top-10 list.Source: C Exact reward implementation · p. 15
• The anti-hacking transform penalizes contaminated lines and lists exceeding ten raw candidates, while globally invalid responses without a valid anchor receive −1.Contamination includes unresolved degenerate lines; a valid anchor requires ontology distance at most one or embedding cosine at least 0.80. The deployed cleanliness multiplier is floored at 1 for a verified exact rank-1 answer.Source: C Exact reward implementation · p. 15
• For the reported single-turn run, the hedge term is 0.91 for lists longer than ten and the cleanliness term is q.The passage specifies w = 0, Tturn = 1, and Hhedge = 0.91[L>10].Source: C Exact reward implementation · p. 15
• The list-level nDCG extension has zero weight in the reported training run and is used only in the fixed-output audit.This distinction prevents attributing the audit-only term to the trained checkpoint.Source: C Exact reward implementation · p. 16

D Training provenance and overlap audit

• SFT creates ordered Top-10 targets, while the RL mixture contains 61,918 prompts; publication dates are available for only 22,403 of 42,977 RareArena rows.Train and validation are separate materialized files, but the missing randomization manifest prevents reconstructing a stronger chronological split claim.Source: D Training provenance and overlap audit · p. 16, Table 6 · p. 16
• The RL training set has one exact disease-plus-phenotype profile match with LIRICAL, while validation shares 290 profiles with Phenopackets; removing Phenopackets leaves 37.47 Hit@10 for RareDx-9B versus 35.41 for GPT-5.5.These profile-level matches have no shared patient identifier or case text; the archived preparation code also lacks the original randomization manifest.Source: D Training provenance and overlap audit · p. 16
• The retrieval collection intentionally overlaps evaluation entities, but its unrecorded snapshot dates prevent a claim of temporal disjointness.It includes 27,554 disease-level documents and an optional 2,299 gene documents, without benchmark patient reports.Source: D Training provenance and overlap audit · p. 16

E Proprietary baseline protocol

• Proprietary baseline prompts require ten ranked disease names without tools or retrieval, but archived records omit immutable provider snapshots, timestamps, and provider defaults.Consequently, exact serving revisions and numeric decoding defaults cannot be recovered or inferred.Source: E Proprietary baseline protocol · pp. 16–17, Table 7 · p. 17

F Post-training configurations

• Table 8 separates optimization rule, supervision signal, and online-teacher use; the authors identify the medical graph reward, not a new group-normalization update, as the contribution.The frozen 50-case audit rescored direct 27B outputs alongside frozen direct 9B and Harness outputs using the same parser and name matcher.Source: F Post-training configurations · p. 17, Table 8 · p. 17, G Case-study details · p. 17

G Case-study details

• The RAMEDIS case study uses row 12, giving both models the same six phenotypes, instruction, ranked-differential template, greedy decoding, and 1,024-token generation limit.The observed outputs are recorded in Table 9; the supplied passage does not report a comparative outcome.Source: G Case-study details · p. 17, Table 9 · p. 18

H Reward-audit construction

• The reward audit probes phenotype-neighbor relevance, pseudo-disease strings, and a delayed exact match while keeping graph and phenotype settings controlled.The pseudo-disease probe uses modifiers absent from the canonical vocabulary; the over-budget probe places ten real distractors before the reference. Table 11 reports the percentage of constructed outputs with positive reward.Source: H Reward-audit construction · pp. 17–18, Table 11 · p. 18

H.1 List-quality discrimination audit

• At w = 0.3, the unrelated-list mean reward is 0.911 (95% CI 0.907-0.914), versus 1.000 for the coherent list.The audit corrected the DCG numerator to use the curated relevance set used by IDCG; headline w = 0 checkpoints are unaffected.Source: §H.1 · p. 18

I Adaptive retrieval-depth audit

• Figure 7 removes the phenotype-disease graph signal or the vocabulary gate and Top-10 budget, while Figure 8 examines list quality; the supplied passages do not state their audit outcomes.The retrieval-depth audit pools four evaluation sources for behavioral analysis and computes Wilson intervals independently within each depth stratum.Source: I Adaptive retrieval-depth audit · p. 18, Figure 7 · p. 19, Figure 8 · p. 19

H.1 List-quality discrimination audit

• The list-quality audit compares coherent, reversed, and unrelated differentials whose exact diagnosis is always rank 1, and sweeps nDCG weight with case-level bootstrap intervals.The unrelated lists use deterministic real-disease distractors; the experiment reports within-case reward variation across list types.Source: §H.1 · p. 18, Figure 8 · p. 19

J Efficiency measurement details

• The archived 9B and 27B throughput comparison uses the same 6,249-record generation suite, with node throughput based on the slowest shard.Reported throughput excludes initialization, result merging, disease normalization, and metric computation.Source: J Efficiency measurement details · p. 19

K Training and implementation details

• The 9B RL run uses two 8-GPU A800 nodes, batches of 512, 16 rollouts per prompt, and at most 300 training steps.The reported checkpoint is selected on held-out validation evaluation rather than training reward alone.Source: K Training and implementation details · p. 19
• Table 12 stratifies ReAct diagnostic performance by retrieval-call count, while Table 13 profiles computation by sampled-list count M and ReAct turns T.The supplied passages describe these table scopes but do not provide their numerical values.Source: Table 12 · p. 20, Table 13 · p. 20

L Supplementary model scale ablation

• The experiment uses separate fixed round-60, rank-64 LoRA checkpoints on 128 development cases, not a new held-out test set or the main benchmark protocol.Scoring uses normalized exact matches to the gold diagnosis and supplied aliases, without vocabulary snapping or an LLM judge.Source: L Supplementary model scale ablation · p. 20
• Aggregation preserves greedy Top-1 by construction and fills nine remaining positions using reciprocal-rank voting over additional lists without gold labels.Intermediate sampling budgets were examined post hoc, while the k = 8 aggregation rule was fixed before the expanded evaluation.Source: L Supplementary model scale ablation · p. 20
• At k = 8, Hit@10 increases by 11.72 percentage points for 27B and 9.38 points for 35B relative to greedy decoding, at approximately nine times the generated-token budget.For 35B, k = 4 and k = 8 have the same aggregate Hit@10 on this subset, but their correct cases differ; this does not establish general saturation.Source: L Supplementary model scale ablation · p. 20
• Table 14 reports sampling-budget ablation on 128 development cases, with mean output tokens per case across generations excluding inputs, retrieval, and runtime overhead.The supplied passage gives the table scope, not its numerical values.Source: Table 14 · p. 21