Source-linked AI summary
Benchmarking Large Language Models for Biomedical Relation Extraction
Claudiu Creanga, Teodor Marchitan, Liviu P. Dinu
TL;DR
Identifying SNP–phenotype associations in biomedical literature matters, but specialized terminology and context-sensitive language make extraction challenging. This paper benchmarks language models on three SNPPhenA tasks. OpenAI O1 leads the association-classification tasks, while strength classification remains difficult despite the best result coming from fine-tuned Gemini 2.0 Pro.
Problem
Specialized terminology and context-dependent language make SNP–phenotype relations difficult to extract from biomedical literature.
Source: §1 · pp. 1–2Method
The paper benchmarks MLMs, hybrid architectures, and LLMs on SNPPhenA sentence-level, abstract-level, and association-strength classification tasks.
Source: §1.1 · p. 2, §6 · p. 9Results
OpenAI O1 achieved F1 0.89 for non-fine-tuned sentence-level classification and F1 0.82 for abstract-level classification, while fine-tuned Gemini 2.0 Pro achieved F1 0.60 on association strength.
Source: §6 · p. 9Takeaways & Limitations
Proprietary LLMs outperformed MLMs and hybrid models in the reported benchmark, with O1 and fine-tuned Gemini 2.0 Pro leading different tasks.
Source: §6 · p. 9The authors identify association-strength classification as needing improvement and model explainability as important for clinicians’ trust and verification.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Extracting SNP-phenotype associations from biomedical literature is vital but challenging. We benchmarked diverse NLP models, including MLMs, hybrid architectures, and state-of-the-art LLMs (Gemini 2.0, OpenAI O-series, Qwen, Mistral), on the SNPPhenA corpus across three tasks: sentence-level, abstract-level, and association strength classification. OpenAI O1 achieved state-of-the-art (SOTA) results using few-shot learning for non-finetuned sentence-level classification (F1 0.89) and established a new SOTA for abstract-level classification (F1 0.82). Association strength classification proved difficult, though fine-tuned Gemini 2.0 Pro performed best (F1 0.60) in the first LLM evaluation of this task. Proprietary LLMs, especially in few-shot (O1) or fine-tuned (Gemini 2.0 Pro) settings, significantly outperformed other models. These findings confirm the power of modern LLMs for genomic knowledge extraction.
1. Introduction
SNP–phenotype extraction turns growing biomedical literature into organized genetic-association information, but specialized terminology and context-dependent language make the relationships difficult to identify. The paper benchmarks language models across three SNPPhenA tasks: sentence-level association, abstract-level association, and association-strength classification.
1. Introduction
- SNP–phenotype extraction offers a scalable way to organize genetic relationships in a rapidly growing literature that manual curation cannot feasibly cover.The introduction describes this as transforming unstructured text into genetic insights.Source: §1 · pp. 1–2
- Specialized terminology and context-dependent wording make SNP–phenotype relations difficult to interpret, limiting simple pattern-matching and keyword methods.The introduction frames this research area as relatively unexplored within NLP.Source: §1 · p. 2
1.1. Contribution
- The study benchmarks three tasks: sentence-level and abstract-level positive, negative, or neutral classification, plus strength classification of positive associations as weak, moderate, or strong.The authors describe the strength task as the first study testing LLMs on it.Source: §1.1 · p. 2
2. Related Work
The paper builds on prior SNPPhenA work that reported strong fine-tuned Gemini 1.0 Pro results, and motivates evaluating newer proprietary and open-source models. It frames the comparison as a way to examine performance and accessibility trade-offs.
2. Related Work
- Fine-tuned Gemini 1.0 Pro previously achieved F1 0.89 at sentence level and F1 0.80 at abstract level, including an abstract-level result above the prior 0.645 BioBERTGRU result.The sentence-level result also outperformed BioBERT-GRU and PubMedBERT-LSTM.Source: §2 · p. 2
- The authors motivate comparing newer proprietary and open-source LLMs, including Qwen and Mistral, as a way to assess performance and accessibility trade-offs.They note local open-source models could lower barriers for institutions with limited cloud-computing resources.Source: §2 · pp. 2–3
- Figure 1 depicts SNPPhenA sentence-token distributions split by training and testing sets and colored by positive, negative, and neutral labels.Zoomed views focus on sentences with lower token counts.Source: Figure 1 · p. 3
3. Dataset
SNPPhenA supplies manually annotated SNP–phenotype associations, including polarity and graded confidence labels. The section describes the corpus and its label and token-count distributions, then illustrates how entities and relations are identified in an abstract.
3. Dataset
- The corpus labels associations as positive, negative, or neutral, and grades positive-association confidence as weak, moderate, or strong.Strength labels draw on linguistic cues such as modality markers and adverbs, as well as reported statistical significance.Source: §3 · p. 3
- SNPPhenA contains 360 XML files and 2,625 sentences, with 483 sentences annotated for at least one SNP and phenotype; annotations include 875 SNP names.The corpus is manually curated, and the authors selected it as their primary dataset because it is the sole source they identify as comprehensively annotated, ranked SNP–phenotype associations.Source: §3 · pp. 3–4
- Training sentences are predominantly about 20–100 tokens, while testing has lower counts overall and a more prominent neutral-label share, especially around 100–200 tokens.The training-set distribution has positive associations as the highest-count label in the lower-token range.Source: §3 · pp. 3–4
- An illustrative abstract describes linkage-disequilibrium relations among TOMM40, APOC1, and APOE variants in PPA and bvFTD; the described system extracts entities with NER and detects relations with REL.The example mentions rs1064725, rs429358, and rs157590 among the SNPs.Source: §3 · p. 4
4. System Overview
The section outlines a benchmark of LLMs, masked language models, and a hybrid architecture for SNPPhenA tasks. It describes the model families and evaluation setups.
4. System Overview
- The benchmark evaluates LLMs, MLMs, and a hybrid system; Tables 1–2 report sentence- and abstract-level SOTA, while association-strength SOTA is 0.45 from a BoW model.Source: §4 · p. 4, Table 1 · p. 4, Table 2 · p. 5
4.1. Masked Language Models
- MLMs used a custom binary-classification head and staged training, freezing transformers before updating selected upper layers; ModernBERT performed best.Source: §4.1 · pp. 5–6, Figure 3 · p. 6
4.2. Hybrid Architecture
- The hybrid combines character CNN and word embeddings, processes concatenated features with bidirectional LSTM, and optionally uses CRF for label dependencies.Source: §4.2 · pp. 6–7, Figure 4 · p. 7
- The hybrid neural network underperformed contemporary LLMs and MLMs.Source: §4.2 · p. 7
4.3. Causal Models
- Causal models included open-source and proprietary systems tested zero- and few-shot; fine-tuning used 50 balanced examples per class, with temperature 0 and three runs.Source: §4.3 · pp. 7–8
- Sentence prompts supplied SNP and phenotype and required one positive, neutral, or negative label; neutral covered unremarked or only potential associations.Source: §4.3 · pp. 7–8
5. Results
OpenAI O1 led the sentence- and abstract-level tasks, while association-strength classification remained difficult. Fine-tuned Gemini 2.0 Pro showed the strongest reported balance for the strength task.
5.1. Performance at first task: sentence level
- OpenAI O1 achieved the highest sentence-level performance and a non-finetuned-model SOTA, closely followed by fine-tuned Gemini 2.0 Pro.Table 3 covers Precision, Recall, and F1 across settings and identifies O1's few-shot setting as the highest-F1 result.Source: §5.1 · p. 8, Table 3 · p. 8
5.2. Performance at second task: abstract level
- OpenAI O1 achieved the highest scores across metrics on the abstract-level task, establishing a new SOTA for predicting SNP-phenotype associations from medical abstracts.Qwen had the lowest performance among the compared causal models.Source: §5.2 · p. 8
5.3. Performance at third task: classify strength of associations
- Precision and recall both approached 0.61 for fine-tuned Gemini 2.0 Pro, which achieved the highest performance for association-strength classification.The study presents this as the first systematic investigation of the task and reports limited few-shot F1 scores for Gemini 2.0 Pro (0.34) and OpenAI O1 (0.38).Source: §5.3 · pp. 8–9, Table 5 · p. 9
6. Conclusion and Future Work
The benchmark found strong performance by proprietary LLMs on SNP-phenotype classification, while association-strength classification remained challenging. Future work will refine that task and explore explainability for medical use.
6. Conclusion and Future Work
- Proprietary LLMs excelled across the benchmark: OpenAI O1 achieved sentence-level F1 0.89 and abstract-level F1 0.82, while fine-tuned Gemini 2.0 Pro achieved association-strength F1 0.60.The reported performance gap persisted against MLMs, hybrid models, and tested open-source alternatives in zero-shot and few-shot settings.Source: §6 · p. 9
- Future work targets association-strength classification with refined models, enhanced fine-tuning, and more sophisticated prompting; explainability is important for clinicians to trust and verify medical AI suggestions.The authors connect understanding model reasoning with identifying potential flaws, biases, or reliance on incorrect features to support patient safety.Source: §6 · p. 9