PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Fuse LLMs with Ontology Rankers for Rare-Disease Diagnosis

Zhaoyang Jiang, Zhizhong Fu, Yunsoo Kim, Zicheng Li, Xuanqi Peng, Fei Teng, Jiacong Mi, Honghan Wu

arXiv:2609.02473v1cs.CL

TL;DR

Rare-disease ontology rankers provide inspectable phenotype evidence, but LLM differentials lack an equally clear trail. The paper corrects shared-publication leakage and learns case-level fusion of ontology and LLM rankings, improving ranking performance while preserving evidence for most correct predictions.

  • Problem

    LLM differentials can extend beyond fixed resources but lack the inspectable evidence trail provided by ontology rankers, while benchmark provenance can create publication-source overlap.

  • Method

    The paper removes phenotype relationships supported only by a test case’s source publication and learns case-specific fusion weights from the two systems’ ranking behavior and ontology support.

  • Results

    Fusion improves Phenomizer Recall@1 across Phenopacket Store and RAMEDIS, and transfer experiments show gains without retraining on the target LLM family.

  • Takeaways & Limitations

    LLMs can strengthen an ontology diagnostic tool while preserving candidate-level ontology evidence and permitting entry of a newly introduced model without target-model labels.

  • Takeaways & Limitations

    The experiments evaluate phenotype-only diagnostic ranking on public research corpora and do not establish clinical validity or support autonomous diagnosis.

Abstract

from arXiv · show

Ontology rankers remain useful for rare-disease diagnosis because each candidate can be traced to matched patient phenotypes. Large language models (LLMs) can generate differential diagnoses from the same patient description, but their predictions lack an equally clear evidence trail. Rather than asking which system should replace the other, we ask whether an LLM can improve the ranker without giving up its evidence. Our behavior-based fusion model examines the two ranked lists, their agreement, and the ontology support behind each candidate, and learns how much to rely on each system for the individual case. Before comparison, we remove a documented test-set leakage pathway caused by benchmark cases and ontology annotations being derived from the same publications. Across eight open LLMs, fusion improves Phenomizer Recall@1 by 7.86 percentage points on Phenopacket Store and 20.18 points on RAMEDIS. When paired with DeepSeek-V4-Flash through an API, a fusion model trained only on the other LLMs improves Recall@1 from 0.1657 to 0.2176, a 5.19-point gain, without retraining. For 90.8% of correct fused diagnoses, the disease retains candidate-level ontology evidence that can be inspected. These results show that LLMs can strengthen an established diagnostic tool without discarding the structured evidence that makes it useful.

1 Introduction

Rare-disease phenotype rankers provide inspectable evidence, while LLMs offer complementary diagnostic generation but raise reliability concerns. The paper therefore studies behavior-based fusion after correcting publication-source overlap.

  • Phenotype tools compare patient findings with curated disease profiles and expose HPO evidence supporting ranked candidates.For Phenomizer, clinicians can also inspect the statistical strength of each match.
  • LLMs can generate differentials from clinical text or HPO terms while drawing on knowledge beyond fixed curated resources.
  • Publication-source overlap occurs when benchmark cases and HPO annotations derive phenotype information from the same paper.This can connect a test patient’s findings with knowledge-base evidence for its confirmed disease.
  • Removing phenotype relationships supported only by the test case’s paper changes the comparison from a simple winner toward complementary ranker and LLM strengths.The ontology ranker is rerun unchanged after this provenance correction.
  • The learned gate assigns case-specific fusion weights from ranking behavior, agreement, patient-candidate fit, and available ontology knowledge.Training excludes the target LLM and other models from its backbone family, enabling application to a new model without target-model labels.
  • Fusion aims to strengthen the diagnostic model while retaining candidate-level ontology evidence.

2 Related Work

Related work establishes curated ontology ranking, rank aggregation, input-dependent routing, and broader benchmark-contamination concerns as foundations for this study.

  • Ontology systems ground rare-disease diagnosis in curated disease profiles and inspectable HPO matches.Phenomizer uses semantic similarity, Exomiser adds variant prioritization, and LIRICAL expresses phenotype evidence through likelihood ratios.
  • Information-retrieval research provides score-, rank-, and probability-based fusion methods for combining ranked outputs.
  • Routing research replaces one fixed rule with input-dependent selection between systems.This includes confidence-based cascades, clinical deferral, and supervised or label-free LLM routing.
  • Benchmark contamination research considers pretraining exposure, newly collected cases, and inference-time retrieval of benchmark answers.

3 Correcting Publication-Source Overlap

The paper defines publication-source overlap as shared provenance between a test case and ontology annotations, then removes the overlap before reranking.

  • Publication-source overlap arises when a paper supplies both a test case for disease D and HPOA associations between D and the paper’s findings.HPOA publication references make this reuse directly observable.
  • LOPO removes each disease–phenotype entry whose only supporting reference is the test case’s publication.Entries with corroborating publications or no recorded provenance remain, after which disease profiles are rebuilt and the unchanged ranker rerun.
  • Phenopacket Store case sources and HPOA annotation sources enable the publication-level join required for this correction.

4 Ontology and LLM Fusion

The fusion pipeline independently ranks diseases with an ontology tool and an LLM, converts their outputs to comparable rank scores, learns case-specific weights, and ranks their candidate union.

  • 4.1 Independent Rankings and a Shared Representation: Ontology and LLM components independently construct disease rankings before a shared scorer assigns case-specific weights.
  • 4.1 Independent Rankings and a Shared Representation: The fused candidate set is CT ∪ CL, allowing LLM-only diagnoses to enter but excluding diseases absent from both lists.The ontology ranker contributes its first 100 diseases, while the LLM contributes at most 10 mapped diagnoses.
  • 4.1 Independent Rankings and a Shared Representation: Rank information is used instead of native scores because the ontology ranker and LLM do not provide comparable confidence scales.
  • 4.1 Independent Rankings and a Shared Representation: The rank transformation uses κ = 0 for Phenopacket Store and κ = 60 for RAMEDIS, with larger κ values flattening score decay.This places both lists on a common scale without treating native outputs as calibrated probabilities.
  • 4.1 Independent Rankings and a Shared Representation: Each system’s 39-dimensional representation combines patient context, list shape, ontology support, and agreement signals.The same HPO-based similarity, likelihood-ratio, and annotation-coverage features evaluate leading diseases in both lists.
  • 4.2 Learning to Combine the Rankings: A shared scorer produces one scalar per component, whose difference is converted into complementary weights recomputed for every patient.Equal scores yield weights of 1/2, and the higher-scoring component receives more influence.
  • 4.2 Learning to Combine the Rankings: Weighted rank scores sort the candidate union, with absent candidates receiving zero contribution; listwise cross-entropy directly supervises the fused ranking.No label specifies which component to trust.
  • 4.3 Transfer Across LLM Families: Family-held-out training removes the target LLM family before sampling, standardization, and early stopping, then applies the unchanged map at test time.Model identity is not used as a feature, supporting transfer through observable ranking behavior.

5 Experimental Setup

The experiments evaluate ontology ranking and LLM components under controlled phenotype-only inputs, leakage correction, multiple datasets, and several fusion baselines.

  • Datasets: 10,377 Phenopacket Store cases and 624 RAMEDIS cases provide the evaluation corpora.Phenopacket Store includes 780 gold diseases; RAMEDIS is evaluated through RareBench.
  • Datasets: Phenopacket Store splits keep cases from the same publication together to reduce publication-level overlap across partitions.The splits contain 7,029/1,102/2,246 cases over 1,187/181/365 publications.
  • Systems: LLMs receive bounded phenotype-oriented inputs, while Phenomizer receives observed HPO terms and LLMs additionally receive explicit exclusions and available demographics.Disease labels, genes, variants, source identifiers, titles, and archive paths are excluded and checked with a planted-answer negative control.
  • Systems: Phenomizer ranks diseases using mean best-match information content and empirical p-values from its own null model.The evaluation also includes a bare semantic-similarity equation, three editable ontology rankers, and stock LIRICAL under ordinary and exact-LOPO profiles.
  • Fusion evaluation: Fusion is compared with component systems, rank-fusion baselines, a target-LLM fixed weight, and routers that select one component ranking per case.The baselines include RRF, Borda-fuse, Bayes-fuse, ProbFuse, and CombMNZ; RAMEDIS predictions are evaluated out of fold.

6 Results

Publication-level leakage substantially inflates ontology-ranker performance, but behavior-based fusion consistently improves diagnosis across datasets, controls, and an unseen LLM. The method’s strongest signals are ontology support and cross-list agreement, while transfer depends on usable LLM rankings.

  • Publication-Source Overlap: 74.6% of 10,348 eligible cases have at least one gold-disease annotation citing the case’s source publication, and 19.6% have profiles supported entirely by it.On average, 33.6% of each gold profile is supported only by the case publication.
  • Publication-Source Overlap: 32.64 points: Phenomizer Recall@1 falls from 0.4481 to 0.1217 after publication-level LOPO.The underlying semantic-similarity score shows the same direction and similar magnitude under every tie policy.
  • Publication-Source Overlap: 31.43 points: LIRICAL Recall@1 falls from 0.4868 to 0.1724 under the same provenance filtering, ruling out an implementation-specific explanation.All 739 cases unaffected by filtering retain byte-identical rankings.
  • Publication-Source Overlap: 22.95 points versus -0.67 points: deleting case-source annotations reduces Recall@1 from 0.4238 to 0.1943, whereas matched random deletion has negligible effect.The contrast indicates that provenance and case-specific semantic alignment jointly characterize the shortcut mechanism.
  • Diagnostic Accuracy: +4.88 points: family-held-out fusion reaches macro Recall@1 0.2002 versus 0.1515 for the strongest published rank-fusion baseline, CombMNZ.Fusion has the highest Recall@1 point estimate across all eight target LLMs and four held-out families, and also improves Recall@5 and MRR.
  • Robustness and Controls: +10.19 points: on cases retaining independently supported gold profiles, the frozen LOPO gate raises Recall@1 from 0.1578 to 0.2597.The gain persists when the ontology component receives the same negative-phenotype information as the LLM, increasing Recall@1 from 0.1361 to 0.2115.
  • Diagnostic Accuracy: +5.19 points: fusion improves DeepSeek-V4-Flash Recall@1 from 0.1657 to 0.2176 without DeepSeek-labelled training examples or retraining.The gate is trained on eight open LLMs and applied unchanged to the unseen model.
  • Diagnostic Accuracy: +20.18 points: across eight LLMs, fusion raises Phenomizer’s macro Recall@1 from 0.1554 to 0.3573, while RAMEDIS shows no significant difference from RRF.On RAMEDIS, fusion reaches 0.3573 versus 0.3562 for RRF, consistent with aggregation sufficing under sparse inputs.

7 Discussion and Conclusion

The provenance-corrected evaluation supports using LLMs and ontology rankers together: LLMs broaden candidates while rankers retain structured evidence. Behavior-based fusion transfers across model families, though sparse inputs can make learned weighting no better than rank aggregation.

  • Provenance auditing and complementary diagnostic evidence support treating structured tools and LLMs as collaborators rather than competing alternatives.The corrected evaluation changes the LLM’s role from replacement to candidate-set expansion alongside ontology-based evidence.
  • Family-held-out training and a test-only DeepSeek arm show that the gate can use complementarity without target-model-family labels.The target LLM contributes no training or validation labels in the test-only setting.
  • RAMEDIS extends the fusion result to an independently assembled disease cohort while exposing a boundary of the learned gate.With observed phenotypes but no exclusions or demographic context, learned weighting provides no detectable gain beyond combining rankings.
  • In sparse-input RAMEDIS cases, combining the rankings matters, but learning their weights provides no detectable additional gain.

Limitations

The provenance correction is limited to Phenopacket Store and HPOA, and the experiments assess phenotype-only ranking on public research corpora rather than clinical deployment.

  • The provenance analysis covers Phenopacket Store and HPOA, where cases and annotations record source publications.LOPO removes recorded overlap from the ontology side but does not audit LLM pretraining data or establish that the overlap is widespread elsewhere.
  • The experiments evaluate phenotype-only disease ranking rather than Exomiser’s variant-aware clinical mode.
  • The fusion model requires disease names that map to a shared ontology and has not been established for every future model or clinical setting.
  • The public-corpus experiments assess diagnostic ranking and candidate-level evidence, not clinical validity or autonomous diagnosis.

Ethics Statement

The study uses de-identified public research cases, controlled phenotype-only inputs, audited disease-name outputs, and retrospective computational evaluation without new patient data or clinical diagnosis.

  • The study uses de-identified case descriptions from public research corpora and collects no new patient data.
  • Clinical experts reviewed de-identified model outputs for the disease-name audit but did not diagnose or recommend treatment for real patients.
  • The evaluation reports retrospective disease ranking rather than clinical validity, and none of the systems should be used for autonomous diagnosis.
  • Phenotype-only inputs contain canonical observed and explicitly excluded HPO labels plus permitted demographic fields, excluding other case text.
  • A verifier checks outputs for disease, gene, variant, publication, and archive identifiers and tests detection with an inserted gold disease name.
  • LLMs receive the same phenotype-only diagnostic prompt and return ranked disease names, which are mapped to OMIM identifiers using a fixed lexicon.

A.3 Ontology Rankers and Provenance Controls

The evaluation removes publication-source overlap and examines how Phenomizer’s ranking behavior, statistical layer, ties, and fusion controls affect diagnosis.

  • Ontology ranking: The reference implementation ranks 8,553 diseases using 100,000 Monte Carlo samples, with four complete runs reported because sampling is stochastic.Three cases lacking canonical HPO terms are excluded from ranking.
  • Ontology ranking: The native Phenomizer procedure caps null distributions at 10 query terms, while longer queries retain all terms but use the size-10 null distribution.A diagnostic uncapped variant is evaluated separately.
  • Ranking ties: 14.1% of 4,454 test-case and protocol combinations have rank-one predictions inside unresolved tie blocks, with a maximum block size of 22.The fusion experiments use native ordering, alongside tie-averaged and worst-case sensitivity analyses.
  • Limitations: Profile-richness correction may attenuate genuine signal together with annotation-density bias in the statistical layer.The median gold disease has 52 annotations before LOPO and 34 afterward, compared with 12 for a full-space candidate.
  • Fusion controls: The learned gate uses ranking geometry, ontology support, and cross-list features, while routing controls select or mix complete rankings based on case behavior.The shared feature representation contains 39 features, and trained baselines use no held-out-family labels.

A.5 Additional Evaluation

Additional evaluation measures source overlap, tests the fusion principle across corpora, and examines robustness through sensitivity analyses and computational reporting.

  • Source overlap: The source-overlap dose analysis includes 7,717 ranking-eligible cases and uses 2,000 bootstrap samples clustered by source publication.It tests whether the LOPO effect grows with the gold profile fraction supported only by the case publication.
  • Source overlap: Cross-corpus overlap is estimated with PMID joins after excluding publications shared with Phenopacket Store, with disease identifiers mapped through OMIM and Orphanet.A mapping control confirms that the strong overlap signal remains detectable.
  • RAMEDIS evaluation: RAMEDIS contains 624 inborn-error-of-metabolism patients evaluated with five-fold StratifiedGroupKFold cross-validation and five seeds.Identical observed-HPO profiles are grouped so duplicates cannot cross folds.
  • RAMEDIS evaluation: RAMEDIS gate training uses patient-fold-separated rankings from all eight LLMs, while each held-out test patient is paired with its corresponding target-LLM ranking.This evaluates the second corpus without transferring parameters from Phenopacket Store.
  • Additional analyses: The study reports complementary ranking metrics, ablations, candidate-depth sensitivity, and comparisons with published fusion rules.All ontology rankers, provenance interventions, and fusion models run on CPU; local LLM inference uses an RTX 6000 Ada GPU.

B.2 Fusion Robustness

Fusion robustness is assessed through macro performance, feature ablations, initialization variability, candidate cutoffs, and multiple rank-fusion baselines.

  • Macro performance: Phenopacket Store macro ranking performance is averaged over five gate seeds and eight target LLMs.The corresponding results are organized in Table 12.
  • Ablations: Feature ablations and learned controls are evaluated under family holdout, with seed standard deviations and 95% confidence intervals reported.Decreases are measured relative to the full model.
  • Initialization robustness: Paired robustness results are reported over 30 initializations with seed-level 95% confidence intervals.This tests variability across repeated model initializations.
  • Candidate depth: Gold-disease coverage and Recall@1 are compared across ontology candidate cutoffs, including expanded inference for a gate fitted at K = 100.The table directly evaluates sensitivity to candidate depth.
  • Fusion baselines: Published rank-fusion rules are compared across native, average-tied, and least-favorable tie rankings, with results averaged over eight target LLMs.The evaluated methods include RRF, Borda-fuse, Bayes-fuse, ProbFuse, and CombMNZ.
Loading 2609.02473v1…