Source-linked AI summary
OntologyBench: Can Dense Retrieval Satisfy Structured Biomedical Constraints?
Xiao Yu Cindy Zhang, Wyeth Wasserman, Jian Zhu
TL;DR
OntologyBench asks whether dense biomedical retrieval can recover compatibility encoded by structured ontology relations and phenotype combinations. It benchmarks concept grounding, relational retrieval, and compositional phenotype retrieval, comparing embedding systems, supervision regimes, rerankers, and ontology-aware references. Performance is generally weaker on relational and compositional tasks, while ontology-derived supervision helps several tasks but reranking provides little or no end-to-end improvement.
Problem
Existing retrieval benchmarks provide limited evidence about whether models recover compatibility across multiple structured biomedical inputs rather than unstructured semantic similarity.
Method
OntologyBench evaluates concept grounding, relational retrieval, and compositional phenotype retrieval using curated biomedical ontologies and compares embedding, reranking, candidate-scoring, and ontology-aware reference methods.
Results
Performance is generally lower on relational and compositional tasks than on several grounding tasks; ontology-derived supervision improves several relational and compositional tasks, while evaluated rerankers provide little or no end-to-end improvement.
Takeaways & Limitations
The evaluated retrieval configurations do not reliably recover ontology-defined relevance for joint phenotype sets, motivating systems that integrate learned representations with structured biomedical knowledge.
Takeaways & Limitations
Cross-tier score differences are descriptive because task characteristics differ, and the benchmark is diagnostic rather than a direct proxy for end-to-end clinical performance.
Abstract
from arXiv · showhide
We introduce OntologyBench, a tiered biomedical retrieval benchmark comprising 471,854 training and 125,744 evaluation query-document relevance pairs across concept grounding, relational retrieval, and compositional phenotype-based retrieval. Although these tasks can be tractable using ontology-aware reference methods, across task tiers, embedding performance is generally lower on relational and compositional tasks than on concept-grounding tasks. Fine-tuning on ontology-derived supervision improves performance on several relational and compositional tasks, whereas the evaluated reranking and LLM-based candidate-scoring methods provide little or no end-to-end improvement. Errors frequently reflect diseases matching only subsets of the phenotype evidence. These findings indicate that the evaluated embedding and reranking configurations do not reliably recover the compatibility encoded by the selected ontology relations and phenotype combinations and motivate retrieval systems that better integrate learned representations with structured biomedical knowledge.
1 Introduction
OntologyBench addresses whether dense retrieval can recover compatibility across structured biomedical constraints, extending evaluation beyond unstructured semantic similarity. It spans concept grounding, relational retrieval, and compositional phenotype retrieval, where evaluated systems show persistent limitations.
- Dense retrieval supports flexible natural-language matching, but existing benchmarks provide limited evidence about compatibility across multiple structured inputs.
- OntologyBench evaluates retrieval across concept grounding, ontology relations, and compositional phenotype sets derived from curated biomedical structure.
- The benchmark uses MONDO diseases, HPO phenotypes, and HGNC/NCBI genes to construct a structured biomedical knowledge space.
- The benchmark compares retrieval behavior across task families whose candidate spaces, relevance structures, and query ambiguity differ.
- Tier 3 isolates compositional retrieval by combining multiple phenotype constraints while controlling for temporal progression, demographics, prevalence, and other diagnostic factors.
- The unified setting evaluates held-out phenotype-set compatibilities within a shared ontology vocabulary rather than inductive generalization to entirely unseen entities or relations.
4 Method
The method evaluates dense retrieval, reranking, and generative candidate scoring under ontology-derived tasks, separating candidate availability from downstream ranking. It also compares text-based systems with ontology-aware reference methods under different information regimes.
- Fine-tuning compares concept-grounding supervision with unified supervision over grounding, entity relations, and compositional phenotype queries.
- Two-stage evaluation separates candidate-generation failure from downstream-ranking performance by measuring both stages independently.
- The initial embedding retriever generates top-50 candidates, after which rerankers score the fixed candidate set for relevant diseases.
- At least one relevant disease appears in the initial top-50 candidates for 82.1% of queries, setting the maximum recoverable proportion for downstream reranking.
- Evaluated systems include BM25, general-purpose and biomedical dense encoders, late-interaction reranking, and LLM-based candidate scoring.
- Ontology-aware references use information-content similarity for phenotype tasks and graph traversal for Phen→Gene, providing diagnostic comparisons rather than directly comparable baselines.
- Retrieval is measured with nDCG, MRR, and Hit@k over complete candidate sets, with ontology structure entering through task construction and relevance labels.
5 Results
Ontology-aware references achieve strong retrieval when curated biomedical structure is directly available, while evaluated embeddings perform better on grounding than relational and compositional tasks. Multi-tier supervision improves several difficult tasks, but reranking provides little end-to-end benefit and phenotype errors often match only subsets of evidence.
- Cross-tier performance: Embedding scores were generally highest on concept-grounding tasks and lower on relational and compositional tasks.These cross-tier differences are descriptive because candidate spaces, relevance structures, and query ambiguity vary.
- Ontology-aware retrieval: 0.739 nDCG@10 was achieved by the ontology-aware PhenTriplet→Dis reference, compared with 0.313 for the best evaluated embedding model.The reference directly uses ontology structure, whereas embeddings infer compatibility from textual representations.
- Model comparisons: 0.151 nDCG@10 for BioLORD-0.1B exceeded Qwen3-Embed-0.6B at 0.109 and Qwen3-Embed-4B at 0.080 on Phenotype-Triplet→Disease.Domain specialization helped in this comparison but did not improve all relational tasks uniformly.
- Supervision effects: +0.204 nDCG@10 was the Qwen3-Embed-0.6B gain from multi-tier supervision on Phenotype-Triplet→Disease.Gains across four Tier 2 tasks ranged from +0.047 to +0.163, while some model–task combinations declined.
- Reranking and candidates: 82.1% of queries had a relevant disease in the first-stage top-50 candidates, leaving 17.9% unavailable for recovery through reranking.The complete two-stage metrics retained unavailable queries with zero retrieval credit.
- Failure modes: 58.5% of analyzed failures were aggregation failures in which retrieved diseases matched only a subset of query phenotypes.Unified supervision raised Tier 3 Hit@10 from 0.170 to 0.572 and nDCG@10 from 0.109 to 0.313, but remained below the ontology-aware reference.
6 Discussion
OntologyBench shows that evaluated text-based retrieval configurations do not reliably recover ontology-defined compatibility when multiple biomedical constraints must be combined. Ontology-derived supervision helps, but the remaining gap and candidate omissions motivate systems integrating learned representations with structured knowledge.
- Discussion: Within the fixed Tier-3 task, evaluated embeddings remained substantially below the ontology-aware reference when combining multiple phenotype constraints.Cross-tier differences remain descriptive because task characteristics differ.
- Discussion: Unified multi-tier supervision improved several relational and compositional tasks, indicating that ontology-derived supervision can better support those retrieval relationships.The remaining gap cannot be attributed solely to embedding objectives because reference methods directly access curated ontology relations.
- Discussion: End-to-end evaluation combines candidate-generation and downstream-ranking limitations because rerankers cannot recover relevant diseases omitted from the initial candidate set.The largest rule-selected failure category matched only part of the phenotype combination.
- Broader relevance: OntologyBench may apply beyond biomedicine to retrieval-augmented generation, agentic systems, knowledge-graph retrieval, and other knowledge-intensive applications requiring structured constraints.The benchmark organizes evaluation across grounding, relational, and compositional retrieval.
7 Conclusion
OntologyBench shows lower scores on relational and compositional retrieval than on several grounding tasks, with embedding configurations below ontology-aware references on the fixed phenotype-triplet task. The findings motivate combining flexible textual representations with explicit structured knowledge without establishing a universal ordering between embedding and symbolic approaches.
- Scores were generally lower on relational and compositional tasks than on several grounding tasks.
- On the fixed PhenTriplet→Dis task, evaluated embedding configurations remained below ontology-aware reference methods.
- Evaluated rerankers provided little or no end-to-end improvement over the first-stage retriever.
- The findings motivate retrieval systems that combine flexible textual representation with explicit structured knowledge.
- The results do not establish a universal ordering between embedding and symbolic approaches.
8 Data and Code Availability
OntologyBench is publicly available under open licenses, but its controlled ontology-based setting limits what the benchmark represents. The study evaluates structured retrieval behaviour rather than ontology geometry or end-to-end clinical performance, and does not exhaustively compare all emerging systems.
- Data and Code Availability: OntologyBench is publicly available on GitHub under the MIT license and through Hugging Face under the CC BY 4.0 License.
- Limitations: The benchmark does not capture temporal progression, causal mechanisms, incomplete observations, or patient-specific context in real-world diagnosis.
- Limitations: The unified setting evaluates transductive multi-task transfer within a shared ontology vocabulary rather than generalization to entirely unseen entities or relations.
- Limitations: The experiments cover representative retrieval architectures and model scales, not an exhaustive leaderboard-style comparison of emerging foundation systems.
- Limitations: Standard retrieval metrics do not explicitly quantify preservation of hierarchical distance or geometric alignment with ontology topology.
- Limitations: The study does not fully isolate which components of ontology-aligned supervision contribute to observed gains.
- Limitations: The benchmark is a diagnostic framework for structured retrieval behaviour rather than a direct proxy for end-to-end clinical performance.
- Data and Code Availability: OntologyBench uses publicly available ontology definitions rather than patient-level data and contains no personally identifiable information or sensitive health records.
A Dataset Examples
The appendix provides representative examples from OntologyBench’s three task tiers using gene, phenotype, and disease concepts. The examples illustrate the biomedical entities and descriptions used as retrieval documents.
- Dataset Examples: The appendix presents representative examples from the three OntologyBench task tiers.
- Dataset Examples: The examples include gene documents describing molecular functions, cellular roles, and disease-associated mutations.
- Dataset Examples: Phenotype documents include congenital tooth absence and elevated blood glycine concentration.
- Dataset Examples: Disease documents include congenital myasthenic syndrome, Johanson-Blizzard syndrome, and sporadic infantile bilateral striatal necrosis.
A.3 Tier 3: Compositional Phenotype-Based Disease Retrieval
Tier 3 studies compositional disease retrieval from phenotype combinations, using phenotype triplets as a minimal non-trivial regime. The appendix reports dataset scale and compatibility statistics that clarify the task’s lexical and hypothesis-space properties.
- Dataset Examples: Tier 3 includes disease documents describing striatal degeneration, ketone-body metabolism deficiency, and yellow fever progression.
- Tier 1 Context: 6,611 canonical disease concepts are associated with 48,854 alias queries, averaging 7.39 aliases per concept.
- Tier 1 Context: 1.0% of surface forms map to more than one concept, so Tier 1 primarily evaluates lexical normalization under moderate alias diversity.
- Tier 3 Design: Tier-3 phenotype cardinality is fixed at three to study the minimal non-trivial regime of compositional retrieval.
- Tier 3 Design: Single phenotypes have a mean of 328.9 compatible diseases, compared with 26.7 for pairs and 5.0 for triplets.
- Tier 3 Design: Phenotype quadruplets and quintuplets have means of 1.95 and 1.31 compatible diseases, respectively.
- Tier 3 Design: Triplets reduce the hypothesis space substantially while avoiding the near-deterministic collapse observed at higher cardinalities.
C Experimental Details
The evaluation ranks all documents within each task-specific corpus and reports macro-averaged retrieval metrics at multiple cutoffs. Experimental details specify token limits, training settings, and binary-relevance scoring.
- Implementation details: Bi-encoder models used 256-token maximum sequences, whereas ModernColBERT used maximum query and document lengths of 48 and 300 tokens.Fine-tuning used two epochs on one NVIDIA H100 MIG device, with single runs and seed 42.
- Evaluation protocol: All queries are ranked against every document in their task-specific corpus, without heuristic filtering or ontology pruning.
- Relevance definition: Relevance is binary, with Gq denoting the non-empty relevant-document set after duplicate document identifiers are removed.
- Metric computation: The ideal binary-relevance ranking contains min(k, |Gq|) relevant documents, while reciprocal-rank and Hit@k depend on the first relevant rank.
F Prompt Used for Generative Candidate Scoring
The candidate-scoring prompt asks a model to judge textual compatibility between a candidate disorder description and a complete patient phenotype set. It restricts evidence to the supplied descriptions and requires a single integer score in JSON format.
- Prompt design: Each candidate disease is evaluated independently using the same semantic prompt through the model’s native chat template.
- Prompt design: The system instruction defines the task as evaluating compatibility between a candidate disorder description and a set of patient phenotypes.
- Prompt constraints: Models must use only the phenotype list and candidate disorder description, without external medical knowledge.
- Scoring: The task assigns an integer compatibility score from 0 to 100 for how well the candidate supports the complete phenotype set.
- Scoring: Scores of 80–100 are defined as near-complete matches, and output must contain exactly one JSON object containing only the score.
G Additional Evaluation Metrics
Additional evaluation covers standard ranking metrics, rule-based Tier-3 failure categories, and representative embedding failures. The protocol distinguishes candidate availability from reranking and interprets error categories operationally rather than causally.
- Tier 1 metrics: Tier 1 grounding results are reported with nDCG@k, MRR@k, and Hit@k across Gene→Def, Phen→Def, and Dis→Def tasks.
- Tier 2 metrics: Tier-2 ontology-relation results are reported with nDCG@k, MRR@k, and Hit@k for relational retrieval tasks.
- Tier 3 metrics: Table 12 reports nDCG, MRR, and Hit at k ∈ {1, 5, 10} for PhenTriplet→Dis, distinguishing Tier 1-only from all-tier fine-tuning.
- Failure analysis: Failure analysis uses predicted-disease phenotype overlap, the ground-truth rank under a Phenomizer-style Resnik reference, and mean query phenotype information content.
- Failure analysis: Failures are assigned mutually exclusively to generic-bias, related-disease confusion, aggregation failure, or semantic drift using operational thresholds.Low-information is a query-specificity category, not a causal explanation.
- Failure analysis: Representative failures show embeddings retrieving partially compatible disorders despite full ground-truth phenotype coverage, while Phenomizer often resolves the queries through phenotype aggregation.