PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation of large language models for discovery of gene set function

Mengzhou Hu, Sahar Alkhairy, Ingoo Lee, Rudolf T. Pillich, Dylan Fong, Kevin Smith, Robin Bachelder, Trey Ideker, Dexter Pratt

arXiv:2309.04019v2q-bio.GNcs.AIcs.CLq-bio.MN

TL;DR

Curated functional databases do not cover every gene set, and interpreting unfamiliar sets requires time-consuming synthesis of gene-level evidence and literature. This study evaluates five LLMs for naming gene-set functions with rationale, citations, and confidence; GPT-4 recovered curated concepts, identified functions beyond enrichment, and its supporting text was largely verifiable.

  • Problem

    Curated gene-function databases are incomplete, while interpreting unfamiliar gene sets requires laborious synthesis of gene-level evidence and literature.

    Source: Introduction · pp. 3–4

  • Method

    The study evaluated five LLMs on curated and experimental gene sets, assessing proposed names, supporting analyses, citations, and confidence.

    Source: Introduction · p. 4

  • Results

    GPT-4 often named curated functions, identified functions for ‘omics sets without matching enrichment terms, and produced analysis text that was largely factual.

    Source: Results · p. 9, Discussion · pp. 10–11

  • Takeaways & Limitations

    LLMs can assist gene-set interpretation, but even state-of-the-art models should be coupled to fact-checking or reference validation.

    Source: Discussion · pp. 10, 12

    The GPT-4 evaluation used single queries and prompts developed through informal experimentation.

    Source: Discussion · p. 12

Abstract

from arXiv · show

Gene set analysis is a mainstay of functional genomics, but it relies on curated databases of gene functions that are incomplete. Here we evaluate five Large Language Models (LLMs) for their ability to discover the common biological functions represented by a gene set, substantiated by supporting rationale, citations and a confidence assessment. Benchmarking against canonical gene sets from the Gene Ontology, GPT-4 confidently recovered the curated name or a more general concept (73% of cases), while benchmarking against random gene sets correctly yielded zero confidence. Gemini-Pro and Mixtral-Instruct showed ability in naming but were falsely confident for random sets, whereas Llama2-70b had poor performance overall. In gene sets derived from 'omics data, GPT-4 identified novel functions not reported by classical functional enrichment (32% of cases), which independent review indicated were largely verifiable and not hallucinations. The ability to rapidly synthesize common gene functions positions LLMs as valuable 'omics assistants.

Introduction

‘Omics experiments generate gene sets presumed to share functions, but interpreting unfamiliar sets requires laborious synthesis of gene-level evidence and literature. This study evaluates whether LLMs can automate functional naming with supporting rationale, citations, and confidence scores.

Introduction

  • ‘Omics experiments produce gene sets whose members are presumed to work coherently toward a shared biological process or function.Examples include differentially expressed genes, protein clusters, and genes implicated by knockout screens.Source: Introduction · p. 3
  • Functional enrichment compares an ‘omics gene cluster with predefined function- or pathway-based gene sets, including curated databases and prior experimental annotations.Less-studied clusters may yield no significant enrichment or significant terms with little overlap, motivating further investigation.Source: Introduction · p. 3
  • Interpreting unfamiliar gene sets requires mining gene-level knowledge and synthesizing mechanistic hypotheses, a central genome-scientist task that remains largely unautomated.The study evaluates LLMs for naming gene-set functions and providing supporting text, citations, and confidence assessments.Source: Introduction · p. 4

Results

Across curated gene sets, four models had similar naming performance while Llama2 performed worse; GPT-4 also named some ‘omics sets lacking GO annotations. The study assessed confidence and supporting text, including citation reliability.

Results

  • The pipeline prompted an LLM to generate a concise biological name, supporting analysis essay, and confidence score, with a separate LLM instruction adding literature citations.The evaluation used GO-curated gene sets and experimentally derived ‘omics sets to assess recovery and potential novel insights.Source: Results · p. 5
  • GPT-4, Gemini Pro, GPT-3.5, and Mixtral Instruct had median semantic similarities of 0.45–0.50 to GO names, while Llama2 70b scored 0.40 and performed significantly worse.GPT-4 names exceeded the 95th percentile of background similarities in 60% of cases; in about one-third of the remaining cases, it proposed a broader GO-hierarchy concept.Source: Results · p. 6
  • GPT-4 assigned zero confidence to 87% of fully random gene sets, while 98% of real gene-set analyses received high or medium confidence.Classical enrichment returned significant GO terms for 50/50 contaminated sets, whereas GPT-4 was more conservative about naming.Source: Results · p. 7
  • GPT-4 named 133 of 300 ‘omics gene sets with confidence ≥0.8 (44%) and otherwise deferred with zero confidence.The sets came from drug-treatment and viral-infection transcriptomic profiles and proteomic interaction clusters.Source: Results · p. 8
  • Of 260 ‘omics gene sets lacking GO annotation at Jaccard Index ≥10%, GPT-4 confidently named 97 (37%), synthesizing functions beyond the GO corpus.Overall, GPT-4 confidently named 133 of 300 experimental ‘omics gene sets (44%; confidence ≥0.8).Source: Results · p. 8
  • For NeST 2-123, a 13-protein cluster with no compelling classical enrichment hits, GPT-4 named it “Endosomal Sorting and Trafficking”; the best-matching GO term shared one gene (JI = 0.06, BH-adjusted p = 0.04).Source: Results · p. 8
  • Reviewers found 354 of 403 sentences in analyses of 20 ‘omics gene sets fully verifiable (88%); the remainder included 15 miscategorized and 34 speculative facts.The review counted a sentence as verified only when evidence supported every stated fact.Source: Results · p. 9
  • Of 489 citations added to reviewed essays, 106 (22%) did not verifiably support their statements, and titles and abstracts—not manuscript bodies—were reviewed.The authors regarded most, but not all, citations as reliable guidance rather than unquestioned facts.Source: Results · pp. 9–10

Discussion

LLMs often recover curated gene-set functions, but confidence and factual accuracy vary across models. GPT-4 can also propose functions for experimental gene sets that enrichment misses, although its latent reasoning is opaque and its outputs need verification.

Discussion

  • Four of five LLMs proposed names comparable to GO-curated names, while GPT-4’s analysis text was largely factual but occasionally unverifiable.The authors recommend fact-checking or reference validation for current state-of-the-art models.Source: Discussion · p. 10
  • About one-third of GPT-4’s names that differed from GO terms were conceptually broader, and some mismatches may reflect valid alternate interpretations.The paper gives examples involving lymphocyte homing and immune response, and molecular processes versus a high-level behavioral term.Source: Discussion · p. 10
  • For experimental gene sets, GPT-4 proposed names in about one-third of cases where functional enrichment found no matching term.The authors suggest future research could combine enrichment’s precision with LLMs’ literature knowledge and reasoning.Source: Discussion · p. 11
  • LLM knowledge and output calculations are practically opaque, even though their generated narratives can expose facts and reasoning for review.The paper’s evaluation of GPT-4 used single queries and informally developed prompts, without adding biological or experimental context for gene sets.Source: Discussion · pp. 11–12
  • Using self-confidence scoring, GPT-4 nearly always proposed a confident name for GO-term-affiliated gene sets, named only about 13% of random-gene sets, and was less confident for contaminated sets.Source: Discussion · p. 11

Methods

The methods evaluate five LLMs with a gene-set analysis pipeline that generates names, explanations, and confidence scores. Separate procedures assess semantic similarity, compare against enrichment, and check analysis statements and citations.

Methods

  • Five models were evaluated: GPT-3.5, GPT-4, Gemini Pro, Mixtral Instruct, and Llama2 70B.OpenAI and Gemini models were accessed through APIs; Mixtral and Llama2 were downloaded from Ollama and queried through its API.Source: Methods · p. 13
  • Prompts asked models to analyze gene sets, propose a function name, and assign a 0.00–1.00 confidence score, with zero reserved for unrelated genes.The prompt used a one-shot example and instructed the model to analyze before naming; queries used temperature 0.0, seed 42, and a 1,000-token response limit.Source: Methods · pp. 13–14
  • GO gene sets were built from directly annotated genes and their ontology descendants; ‘omics sets were restricted to 3–100 genes.For L1000, the selected context was MCF7 at 6 hours and 10.0 µm; viral-disease perturbations used a z-score cutoff of 2.Source: Methods · pp. 14–15
  • SapBERT embeddings and cosine similarity measured semantic similarity between generated names and GO names.Scores range from 0 (no similarity) to 1 (identical); the actual GO-name similarity was ranked against similarities to all GO-BP term names to obtain a percentile.Source: Methods · pp. 14–15
  • Enrichment used Enrichr and GO-BP 2023 terms; an ‘omics set counted as annotated when adjusted p-value was ≤0.05 and the largest Jaccard overlap was ≥0.1.When multiple significant GO terms existed, the term with the largest Jaccard Index was selected; GPT-4 annotation required confidence ≥0.8.Source: Methods · p. 15
  • A separate LLM citation workflow extracted gene and function keywords, searched PubMed, prioritized papers mentioning more genes, and assessed evidence in the top three publications.Paragraphs without at least one gene symbol and one functional keyword were skipped as unknown.Source: Methods · p. 16
  • Reviewers checked 403 GPT-4 analysis sentences from 20 ‘omics sets, marking facts unverified when supporting evidence was not found within roughly ten minutes.Checks used NCBI Gene for simple gene statements and PubMed searches for statements not verified there; reviewers also assessed whether citations supported facts and whether references were irrelevant.Source: Methods · pp. 16–17

Figures and Tables

The figures and tables outline the LLM analysis and evaluation pipeline, compare model names with curated GO terms, and present confidence assessments and representative ‘omics analyses.

Figures and Tables

  • Fig. 1 shows gene lists prompting LLMs to generate a name, supporting essay, and confidence score, followed by GO-name benchmarking and assessment of ‘omics findings.GPT-4 is identified as the winning model for the GO-name task; ‘omics outputs are assessed for novelty and accuracy.Source: Figure 1 · p. 18
  • Fig. 3 compares model confidence across real, 50/50 mixed, and random gene sets, with shaded bars denoting confidence categories.The caption defines high confidence as 0.87–1.00, medium as 0.80–0.86, and low as 0.01–0.79.Source: Figure 3 · p. 21
  • Tables 1 and 2 report, respectively, best and worst LLM names for GO terms by semantic similarity and the numbers of ‘omics gene sets annotated by GPT-4 versus functional enrichment.Source: Table 1 · p. 22, Table 2 · p. 23

Author Declarations

The authors disclose that TI has advisory, founding, consulting, and equity relationships with several companies.

Author Declarations

  • TI is a co-founder and advisory-board member with equity in Data4Cure and Serinus Biosciences, and a consultant with equity in Ideaya Biosciences.The University of California San Diego reviewed and approved these arrangements under its conflict-of-interest policies.Source: Author Declarations · p. 32
  • The disclosed arrangements were reviewed and approved by the University of California San Diego under its conflict-of-interest policies.Source: Author Declarations · p. 32
  • TI reports an equity interest in Data4Cure, Serinus Biosciences, and Ideaya Biosciences.Source: Author Declarations · p. 32

Data availability

The study reports using publicly available gene sets and identifies sources for GO, NeST, L1000, and viral-infection data.

Data availability

Extended Data Figures and Tables

The extended data show the citation-module workflow, distributions of gene-set sizes, and additional evaluations of GPT-4’s recovery of GO term names. They also include an example gene-set interpretation centered on pancreatic function and tables describing prompts and language models.

Extended Data Figures and Tables

  • The citation module separately generates gene-symbol and functional keywords, searches PubMed for relevant titles and abstracts, and evaluates abstracts to retain supporting references.Panel b shows the prompts used to query GPT-4.Source: Figure 1 · p. 33
  • The GO size figure shows term-size distributions for 8,910 GO-BP terms with 3–100 genes and for the 1,000 GO terms sampled for Task 1.The ‘omics size figure covers all 300 gene sets considered in the study.Source: Figure 2 · p. 34, Figure 4 · p. 35
  • At the 95% similarity percentile, GPT-4 recovered 642 of 1,000 sampled GO-CC names and 757 of 1,000 sampled GO-MF names.The figure compares semantic similarities with assigned term names against similarities with random term names.Source: Figure 3 · p. 34
  • The Jaccard-index figure plots the percentage of ‘omics gene sets matched to GO terms against overlap, with a marked threshold of 0.1.The extended data also include the engineered gene-set-analysis prompt and an overview of the five language models.Source: Figure 5 · p. 35, Table 1 · p. 36, Table 2 · p. 38
  • The example interpretation describes a gene set related to pancreatic specification, differentiation, growth, and function, especially beta-cell glucose sensing and homeostasis.It discusses GCG, GLP1, and SLC2A2/GLUT2 in relation to glucose metabolism and transport.Source: Extended Data Figures and Tables · p. 37

Supplementary Information

The supplementary information lists complete analyses for the model and gene-set evaluations, alongside reviewer fact-checking of GPT-4’s analysis text and citation relevance.

Supplementary Information

Related papers

Loading 2309.04019v2…