Source-linked AI summary
Structured prompt interrogation and recursive extraction of semantics (SPIRES): A method for populating knowledge bases using zero-shot learning
J. Harry Caufield, Harshad Hegde, Vincent Emonet, Nomi L. Harris, Marcin P. Joachimiak, Nicolas Matentzoglu, HyeongSik Kim, Sierra A. T. Moxon, Justin T. Reese, Melissa A. Haendel, Peter N. Robinson, Christopher J. Mungall
TL;DR
Knowledge-base construction requires laborious manual curation, while existing approaches struggle with arbitrary nested schemas and often require training data. SPIRES uses zero-shot, schema-driven recursive prompting with ontology-backed grounding, achieving relation-extraction performance near the mid-range while supporting new tasks without task-specific training. Its use remains bounded by hallucination risks, closed-model opacity, and potentially prohibitive API costs.
Problem
Existing knowledge-base population is time-consuming, and current AI/NLP approaches require extensive training data or cannot handle arbitrary complex nested schemas.
Method
SPIRES recursively prompts GPT-3+ using user-defined schemas, parses nested outputs, and grounds named entities with identifiers from existing ontologies and vocabularies.
Results
SPIRES achieved an F-score of 43.80 on BC5CDR without task-specific training or fine-tuning, while ontology grounding returned 98 of 100 correct GO identifiers with GPT-3.5-turbo.
Takeaways & Limitations
SPIRES offers customizable zero-shot knowledge extraction across varied domains while supporting schema-shaped data and external validation during knowledge-base construction.
Takeaways & Limitations
Using OpenAI models may be prohibitively expensive, and closed APIs have inscrutable training data that may contain biases; LLM hallucinations also remain a reliability concern.
Abstract
from arXiv · showhide
Creating knowledge bases and ontologies is a time consuming task that relies on a manual curation. AI/NLP approaches can assist expert curators in populating these knowledge bases, but current approaches rely on extensive training data, and are not able to populate arbitrary complex nested knowledge schemas. Here we present Structured Prompt Interrogation and Recursive Extraction of Semantics (SPIRES), a Knowledge Extraction approach that relies on the ability of Large Language Models (LLMs) to perform zero-shot learning (ZSL) and general-purpose query answering from flexible prompts and return information conforming to a specified schema. Given a detailed, user-defined knowledge schema and an input text, SPIRES recursively performs prompt interrogation against GPT-3+ to obtain a set of responses matching the provided schema. SPIRES uses existing ontologies and vocabularies to provide identifiers for all matched elements. We present examples of use of SPIRES in different domains, including extraction of food recipes, multi-species cellular signaling pathways, disease treatments, multi-step drug mechanisms, and chemical to disease causation graphs. Current SPIRES accuracy is comparable to the mid-range of existing Relation Extraction (RE) methods, but has the advantage of easy customization, flexibility, and, crucially, the ability to perform new tasks in the absence of any training data. This method supports a general strategy of leveraging the language interpreting capabilities of LLMs to assemble knowledge bases, assisting manual knowledge curation and acquisition while supporting validation with publicly-available databases and ontologies external to the LLM. SPIRES is available as part of the open source OntoGPT package: https://github.com/ monarch-initiative/ontogpt.
1 Introduction
SPIRES addresses the difficulty of populating detailed, nested knowledge schemas by using LLMs for flexible extraction and external ontologies for grounding. It is designed to assist manual curation while avoiding task-specific training data.
- Motivation and scope: The approach is intended to assist manual knowledge curation by extracting and grounding information before insertion into knowledge bases.This separates extraction from direct use of unfiltered LLM results and supports validation against external resources.
- The knowledge-base construction problem: Detailed knowledge schemas are difficult to populate because they encode nested structures such as recipe steps, inputs, outputs, actions, and states.Adapting existing pipelines to custom schemas requires considerable engineering.
- SPIRES approach: SPIRES recursively prompts GPT-3 or newer models to extract instances matching a user-defined schema, including nested attribute-value structures.Its outputs can contain primitives or further inlined instances.
- SPIRES approach: SPIRES grounds extracted named entities against publicly available ontologies and vocabularies, providing identifiers more consistently than LLM querying alone.The approach can leverage over a thousand OntoPortal ontologies and biomedical lexical grounders.
2 System and Methods
SPIRES represents schemas as constrained classes and attributes, then recursively parses model outputs and grounds terminal entities. Its workflow combines schema-driven prompting, structured result parsing, and ontology-backed identifiers.
- Schema representation: A SPIRES schema defines classes, attributes, allowable ranges, cardinality, nesting, metadata, and identifier constraints for domain instances.Attribute ranges may be primitives, classes, or enumerated value sets.
- Schema representation: Schemas can model nested recipe structures in which classes such as steps and ingredients contain further classes, quantities, and food items.Figure 1 represents classes as boxes, compound attributes as arrows, and multivalued attributes with crow’s feet.
- Prompt generation: SPIRES uses custom attribute prompts, automatically generating prompts from attribute names and marking multivalued outputs as semicolon-separated lists when needed.Attributes may point to primitive values, classes, or ontology-backed value sets.
- End-to-end workflow: OntoGPT processes schemas and text into GPT queries, parses the responses, grounds extracted instances using databases and ontologies, and returns schema-shaped structured data.Grounding can map named-entity leaves to identifiers from vocabularies such as FOODON and Wikidata.
- Evaluation design: The evaluation queried GPT-3.5-turbo and GPT-4-turbo on ontology-term grounding and tested SPIRES on the BC5CDR chemical-disease relation task.The grounding experiment sampled 100 terms from each of GO, EMAPA, and MONDO, while BC5CDR used 500 abstracts and 1,066 chemical-induced-disease triples.
3 Algorithm
SPIRES takes a schema, entry-point class, and text, then uses prompted LLM completion, parsing, recursion, and grounding to produce a structured instance that can be translated to OWL.
- 3 Algorithm: SPIRES takes a schema, entry-point class, and text, then returns a structured instance conforming to the schema.The procedure uses an LLM capable of prompt completion.
- 3.1 Step 1: Generate Prompt: The generated prompt combines instructions, an attribute template, the input text, and delimiters, with prompts customizable for attributes and multivalued fields.Attribute templates are pseudo-YAML structures, and multivalued attributes request semicolon-separated lists.
- 3.3 Step 3: Completion Result Parsing and Recursive Extraction: The LLM returns a pseudo-YAML completion that SPIRES parses heuristically by matching attribute names case-insensitively and normalizing whitespace to underscores.Each response line is split at its first colon before attribute matching.
- 3.3 Step 3: Completion Result Parsing and Recursive Extraction: Parsed values follow their attribute ranges and cardinalities: primitives are retained, references are grounded, and inlined classes trigger recursive SPIRES calls.Multivalued attributes are split using a delimiter before these rules are applied.
- 3.3 Step 3: Completion Result Parsing and Recursive Extraction: For nested attributes, SPIRES recursively generates prompts until non-inlined classes are reached, grounding named entities against configured vocabularies or ontologies.The resulting instance tree can be represented in JSON or YAML and optionally translated to OWL for reasoning and consistency checks.
4 Implementation
SPIRES is implemented through reusable schemas, ontology-grounding services, and optional OWL conversion, then evaluated across entity grounding and chemical-disease relation extraction tasks.
- Implementation: SPIRES uses LinkML schemas and OAKlib-backed normalization to support reusable structured extraction with identifiers from multiple ontologies.Ready-made schemas cover life-science applications and food recipes; extracted instances can optionally be converted to OWL for reasoning and consistency checks.
- Entity Grounding: 98 of 100 GO identifiers were correct with GPT-3.5-turbo and 97 of 100 with GPT-4-turbo, compared with only 3 correct identifiers from direct GPT-3.5-turbo prompting.For EMAPA, SPIRES returned correct identifiers for all 100 descriptions, while direct prompting produced identifiers from the wrong anatomy ontology.
- Entity Grounding: SPIRES with GPT-3.5-turbo correctly matched 97 of 100 MONDO identifiers, whereas GPT-4-turbo matched 18 of 100.The passage attributes some errors to incorrectly parsing entities before grounding.
- Chemical-Disease Relation Evaluation: On BC5CDR, no-chunking SPIRES achieved F-scores of 36.64 with GPT-3.5-turbo and 43.80 with GPT-4-turbo.The GPT-3.5-turbo no-chunking result had precision 0.63 and recall 0.26; the GPT-4-turbo result had precision 0.69 and recall 0.32.
- Chemical-Disease Relation Evaluation: SPIRES scored just below the average of the 18 original CDR challenge teams while using no task-specific training or fine-tuning.The reported BioGPT comparison was 44.98, while the best CDR relation-extraction result was 0.57 using extensive training examples.
5 Discussion
The discussion presents SPIRES as a flexible, model-agnostic approach for ontology-driven extraction, while emphasizing model-access constraints and the need to validate generated knowledge.
- Comparable Methods: SPIRES is designed as a generally model-agnostic extractor for structured schemas and standardized ontologies.The discussion relates it to approaches that align extracted information with predefined data models and ontology-driven needs.
- Choosing a Model: OpenAI-based OntoGPT runs may be prohibitively expensive, while closed-model training data can be opaque and biased.The discussion notes that smaller and biomedical-adapted language models may complement SPIRES grounding.
- Reliability and Hallucinations: Prompt constraints and low-creativity settings reduced hallucinations, but most observed errors were attributed to incorrect relation extraction.Structured function calling may enforce data structure without resolving hallucinated ontology identifiers or associations.
- Reliability and Hallucinations: The authors recommend careful validation before inserting language-model results into knowledge bases because generated statements may be more specific than the source text.One example replaced a drug class with a specific drug despite the association being supported only at the class level in the source.
- Scope and Contribution: SPIRES can populate complex knowledge schemas from unstructured text using zero-shot learning, ontology grounding, and no model tuning or training data.The authors describe it as customizable and applicable across varied domains.
7.1 Funding
The work was supported by grants from the National Institutes of Health, the U.S. Department of Energy, and Bosch Research.
- Funding: Funding came from the National Institutes of Health, the U.S. Department of Energy, and Bosch Research.
8 Supplementary Data
The supplementary data provide screenshots, schemas, grounding resources, and evaluation results for SPIRES applications.
- Web interface: Figure S1 shows the web-ontogpt form for selecting a schema and entering text, alongside structured results rendered as nested HTML.The displayed input text and results are truncated for brevity.
- Chemical to Disease schema: Figure S2 presents the Chemical to Disease schema used by SPIRES.The schema is available from the OntoGPT web identifier provided in the caption.
- Evaluation resources: Supplementary tables document grounding resources, MeSH value sets, and named entity recognition evaluation on the BC5CDR corpus.The NER evaluation used MeSH grounding, while alternate ontology annotators could provide further accuracy.