PaperlayerView exact revision on arXiv

Source-linked AI summary

Automating Exploratory Proteomics Research via Language Models

Ning Ding, Shang Qu, Linhai Xie, Yifei Li, Zaoqu Liu, Kaiyan Zhang, Yibai Xiong, Yuxin Zuo, Zhangren Chen, Ermo Hua, Xingtai Lv, Youbang Sun, Yang Li, Dong Li, Fuchu He, Bowen Zhou

arXiv:2411.03743v1cs.AIq-bio.QM

TL;DR

Proteomics research is challenged by complex, high-throughput datasets that require substantial human analysis. PROTEUS uses an LLM to plan and execute analyses with specialized tools, refining its approach as results arrive. Across diverse datasets, it generated hypotheses evaluated through automated scoring and expert review, while identified limitations include incomplete use of LLM knowledge in some analyses.

  • Problem

    Proteomics datasets contain complex relationships among proteins, cells, and pathways, while conventional expert-led analysis can be time-consuming and limited by researchers’ knowledge and habits.

    Source: §1 · p. 1

  • Method

    PROTEUS uses an LLM for hierarchical planning, specialized-tool execution, result interpretation, iterative refinement, and data-grounded hypothesis generation.

    Source: §1 · p. 2, §2.1 · p. 2, §2.1.2 · p. 5

  • Results

    Across 12 diverse proteomics datasets, PROTEUS generated 191 hypotheses assessed with five LLM-scored metrics and human expert review, with the paper reporting reliable, rigorous, and novel outputs.

    Source: §1 · p. 2, §3 · p. 11

  • Takeaways & Limitations

    The authors present PROTEUS as an automated end-to-end approach that integrates proteomics analysis and evaluation and can explore underexamined biological topics.

    Source: §3 · pp. 11–12

    PROTEUS’s framework and prompts do not fully elicit LLM knowledge at key analysis steps, which can leave biological context and nuance unconsidered in hypotheses.

    Source: §2.2.4 · p. 11

Abstract

from arXiv · show

With the development of artificial intelligence, its contribution to science is evolving from simulating a complex problem to automating entire research processes and producing novel discoveries. Achieving this advancement requires both specialized general models grounded in real-world scientific data and iterative, exploratory frameworks that mirror human scientific methodologies. In this paper, we present PROTEUS, a fully automated system for scientific discovery from raw proteomics data. PROTEUS uses large language models (LLMs) to perform hierarchical planning, execute specialized bioinformatics tools, and iteratively refine analysis workflows to generate high-quality scientific hypotheses. The system takes proteomics datasets as input and produces a comprehensive set of research objectives, analysis results, and novel biological hypotheses without human intervention. We evaluated PROTEUS on 12 proteomics datasets collected from various biological samples (e.g. immune cells, tumors) and different sample types (single-cell and bulk), generating 191 scientific hypotheses. These were assessed using both automatic LLM-based scoring on 5 metrics and detailed reviews from human experts. Results demonstrate that PROTEUS consistently produces reliable, logically coherent results that align well with existing literature while also proposing novel, evaluable hypotheses. The system's flexible architecture facilitates seamless integration of diverse analysis tools and adaptation to different proteomics data types. By automating complex proteomics analysis workflows and hypothesis generation, PROTEUS has the potential to considerably accelerate the pace of scientific discovery in proteomics research, enabling researchers to efficiently explore large-scale datasets and uncover biological insights.

1 Introduction

Proteomics datasets offer insights into proteins, cells, and pathways, but their scale and complexity make expert-led analysis time-consuming and potentially narrow in scope. PROTEUS addresses this challenge with an automated, exploratory LLM-based system that analyzes raw data and generates hypotheses, evaluated across diverse datasets and by both automated and human review.

1 Introduction

  • Proteomics data are voluminous and complex, while expert-led analysis can be time-consuming and researchers’ knowledge and habits may bias study design and limit its scope.The paper highlights trial-and-error across many proteins or sample groups as one source of delay.Source: §1 · p. 1
  • LLMs’ ability to generalize across research steps motivates using them to automate proteomics analysis from raw data through hypothesis generation.The authors also describe external information retrieval as a way to reduce hallucinations and improve accuracy in knowledge-intensive or time-sensitive scenarios.Source: §1 · pp. 1–2
  • PROTEUS plans research objectives, workflows, and tool use, executes and interprets analyses, then refines plans using results.Its evaluation covered 12 diverse proteomics datasets and produced 191 high-level scientific hypotheses, assessed with five LLM-scored metrics and human review of a random subset.Source: §1 · p. 2
  • Unlike prior LLM methods focused on isolated tasks, PROTEUS targets end-to-end analysis and hypothesis generation, with both automated metrics and human evaluation of its outputs.The paper says prior full-pipeline approaches still required human intervention or evaluated only whether an initial research question was resolved.Source: §1 · p. 2

2.1 PROTEUS System

PROTEUS combines an LLM coordinator with specialized bioinformatics tools and knowledge sources to analyze proteomics data and generate data-grounded hypotheses. Its hierarchical planning, iterative refinement, and modular workflows support adaptation across datasets and research directions.

2.1 PROTEUS System

  • PROTEUS combines an LLM coordinator with domain-specialized analysis tools and knowledge sources to turn protein-expression data and metadata into data-grounded hypotheses.The system plans research objectives, workflows, and analysis tools, executes the steps, interprets results, and refines plans iteratively.Source: §2.1 · p. 2

2.1.1 Large Language Models

  • The system uses biomedical-instruction fine-tuning of Llama 3.1 70B to strengthen analysis, planning, and biomedical knowledge for proteomics discovery.The fine-tuning uses the UltraMedical dataset; the paper also notes privacy sensitivity and iterative improvement needs as reasons to train a general-purpose biomedical model.Source: §2.1.1 · p. 2

2.1.2 System Framework

  • PROTEUS plans at three levels—research objectives, analysis workflows, and analysis steps—to support flexible and robust analysis planning.The data description informs tailored objectives; workflows sequence dependent tools, while the LLM sets parameters and interprets outputs in relation to the objective.Source: §2.1.2 · pp. 2–4
  • After each workflow and objective, PROTEUS uses new results to update subsequent plans and proposes hypotheses linked to statistical observations from the raw data.The fixed hypothesis format summarizes proteins and cell conditions, statistical tests and numerical results, and a final scientific hypothesis.Source: Figure 1 · p. 3, §2.1.2 · p. 5

2.1.3 Features of PROTEUS

  • PROTEUS’s hierarchical structure supports integrating diverse tools and knowledge sources while adapting to heterogeneous proteomics datasets and research directions.Its feedback loops revise plans after workflow execution and objective analysis based on newly obtained results.Source: §2.1.3 · p. 5

2.2 Evaluation

PROTEUS was evaluated on single-cell and clinical proteomics datasets using automatic scores and expert review. The results assess its hypotheses, evaluation procedure, and performance across distinct biological examples.

2.2.1 Base Language Models

  • On MultiMedQA, the custom model achieved 86.30% average accuracy, exceeding Med42-70B, OpenBioLM-70B, and Med-PaLM 2, and approaching GPT-4-Turbo at 87.00%.Its GPQA accuracy was 45.76%, compared with 49.10% for GPT-4-Turbo and 46.70% for Llama-3.1-70B-Instruct.Source: §2.2.1 · p. 5

2.2.2 Quantitative Evaluation

  • The evaluation covered 10 SPDB single-cell datasets and two clinical cohorts, yielding 191 hypotheses; clinical analyses used individual tools and a distinct tool set.The hypotheses comprised 147 from CyTOF data and 44 from mass spectrometry data.Source: §2.2.2 · pp. 6–7, §2.2.1 · p. 6
  • 2.13 (Paper-Based Alignment), 3.24 (Literature-Based Alignment), 3.29 (Literature-Based Novelty), 3.32 (Logical Coherence), and 3.55 (Evaluability) were PROTEUS’s average scores.More than 78% of hypotheses scored at least 3 on Literature-Based Alignment; all Evaluability scores were 3 or 4.Source: §2.2.2 · pp. 7–8
  • Across five evaluation metrics, GPT-4o achieved a slightly higher average than PROTEUS’s base model, while the average across the four non-paper-alignment metrics improved by more than 0.2 starting from five iterations.Source: §2.2.2 · p. 9

2.2.3 Verifying Evaluation Quality.

  • Human evaluators’ average scores exceeded automatic scores on all metrics except Novelty, with reasonable agreement; three other evaluator LLMs produced scores close to GPT-4o’s.The human comparison covered 30 hypotheses from two SPDB datasets.Source: §2.2.3 · p. 9, Figure 7 · p. 10, Figure 8 · p. 10

2.2.4 Case Studies

  • Experts judged the mesothelioma–HGSC hypothesis plausible, novel, and interesting, while the possible bidirectional interplay means the observed trend does not establish causality.For TmDCs, PROTEUS proposed a CD20-related B- and T-cell interaction hypothesis whose mechanisms had not yet been studied in that setting.Source: §2.2.4 · pp. 10–11
  • On GBM data, PROTEUS integrated differential expression, correlations, and enrichment analysis to propose that SNX32 may suppress tumors through inhibition of VIM expression.Experts noted that SNX32 had not been studied in GBM and that experimental validation is needed to establish the proposed causal relationship.Source: §2.2.4 · p. 11
  • A broad GvHD label led PROTEUS to overlook distinctions between acute and chronic disease; the authors say its framework and prompts did not fully elicit relevant LLM knowledge.The authors identify more effective elicitation of biological knowledge at key analysis steps as necessary for more rigorous hypotheses.Source: §2.2.4 · p. 11

3 Discussion

PROTEUS combines hierarchical planning, bioinformatics tools, iterative refinement, and hypothesis proposal for end-to-end proteomics discovery. Its evaluation found reliable outputs and identified limitations tied to the language model’s context length.

3 Discussion

  • PROTEUS coordinates hierarchical planning, analysis-tool calls, iterative feedback and refinement, and hypothesis proposal through an LLM.The system organizes professional bioinformatics tools into workflows for dataset-specific investigation.Source: §3 · p. 11
  • 191 hypotheses from two proteomics dataset types were evaluated with five LLM-scored metrics and reviews from four human experts, who corroborated the evaluation’s reliability.Experts also identified novel hypotheses pointing to promising research directions.Source: §3 · p. 11
  • PROTEUS performs proteomics research and evaluates its results in an automated, end-to-end manner, unlike assistants limited to single analysis steps.The authors describe it as simulating the full scientific inquiry process.Source: §3 · p. 12
  • Context length limits how many workflows or tools PROTEUS can call because the system provides the LLM with prior analysis records.Most experiments used a maximum of five workflows on SPDB datasets and six to eight on clinical cohort datasets.Source: §3 · p. 12
  • The authors propose extending PROTEUS’s design principles beyond proteomics to multi-omics and other biomedical research.They also anticipate that advances in general LLMs and specialized bioinformatics methods could improve analysis and results.Source: §3 · p. 12

4 Method

The methods combine dataset-specific bioinformatics workflows with LLM-directed choices and interpretations, and use prompt-based criteria to evaluate generated conclusions.

4.1.1 Analyzing CyTOF Data

  • CyTOF workflows cluster cells, annotate and refine cell types, visualize protein abundance, and test differential abundance or expression across selected groups.FlowSOM clustering starts with 30 clusters; the workflow uses the top 10 markers per cluster for cell-type labeling, and can refine an existing cluster further.Source: §4.1.1 · pp. 12–13

4.1.2 Analyzing Clinical Cohort Data

  • Clinical-cohort workflows filter and impute missing data, then support differential expression, subtype clustering, pathway enrichment, survival analysis, and clinical or molecular correlations.Differential expression uses limma; subtype clustering uses NMF and aggregates clustering-quality metrics to select a result.Source: §4.1.2 · pp. 13–14

4.1.3 Accessing External Data

  • External-data workflows use DataChat datasets for molecule correlations and survival analyses, and the Human Protein Atlas API for protein information.The authors intend these workflows to corroborate preliminary observations using other data sources.Source: §4.1.3 · p. 15

4.2 Prompt Engineering in Automatic Evaluation

  • The evaluation prompts assess literature support, novelty, paper alignment, biological and logical coherence, and evaluability of generated conclusions.The criteria include agreement with reported findings, biological plausibility, available validation methods, and potential for further research.Source: §4.2 · pp. 15–17

A List of Datasets

The appendix describes the 12 proteomics datasets used in the main experiments and identifies their data types and sources.

A List of Datasets

B Human Evaluation Instructions

The human evaluation instructions allow experts to provide open-ended reviews of notable aspects of the hypotheses.

B Human Evaluation Instructions

Loading 2411.03743v1…