PaperlayerView exact revision on arXiv

Source-linked AI summary

EMBL AI Librarian: Life-Sciences Knowledge Layer for AI Agents

Luigi Sigillo, Matteo Silvestri, Francesco Tabaro, Rajat Bhatnagar, Syed Irtaza Mubashar, Matt Jeffryes, Daljit Nijjer, Vittorio Perera, Ola Spjuth, Julio Saez-Rodriguez, Melissa Harrison, Fabio Petroni

arXiv:2607.28229v1cs.CLcs.AIcs.IRcs.LG

TL;DR

Life-science literature interfaces were built for human keyword search and whole-paper reading, not agents seeking focused evidence. EMBL AI Librarian uses an LLM to orchestrate complementary Europe PMC searches and extract citable evidence, improving agent performance across four benchmark settings, including a more than 16-point Citation F1 gain on ScholarQABench.

  • Problem

    Europe PMC requires agents to use keyword queries, perform complementary searches, and read whole papers to locate relevant evidence.

  • Method

    EMBL AI Librarian uses a single LLM to generate complementary Europe PMC queries, retrieve papers, and return ranked, citable evidence snippets.

  • Results

    Across literature synthesis, claim verification, open-domain question answering, and biology workflow tasks, Librarian improves agent performance, including Citation F1 gains exceeding 16 points.

  • Takeaways & Limitations

    Literature retrieval can serve as a shared, off-the-shelf knowledge layer for life-science agents rather than being rebuilt for each agent.

  • Takeaways & Limitations

    Librarian’s coverage is bounded by Europe PMC, excluding the full text of some paywalled papers.

Abstract

from arXiv · show

The web is increasingly accessed by AI agents rather than humans. Every agent needs knowledge, especially in the life-sciences, where agentic pipelines are growing fast. Access to the literature is a crucial part of that need, and resources such as Europe PMC, with over 40M indexed records, are widely used to meet it. Yet these resources were not built for AI agents: they take keywords and complex syntax and return whole papers, so every agent must learn the syntax, issue several searches, and read full papers to find the evidence it needs. We introduce EMBL AI Librarian, a knowledge layer that upgrades the Europe PMC interface for AI agents: an agent asks in natural language and receives evidence that answers it. A single LLM orchestrates the whole knowledge retrieval process: it plans complementary subqueries executed by the live Europe PMC search engine, then reads the selected papers and locates the relevant evidence. We evaluate Librarian across four benchmarks: literature synthesis, claim verification, open-domain question answering, and downstream biology tasks such as protocol questions and sequence manipulation. On ScholarQABench, Librarian improves Citation F1 by more than $16$ points over strong recently published baselines. Used as the retrieval layer of an existing claim-verification pipeline, it increases agreement with expert consensus; and on the open-form LitQA2 benchmark, a GPT-5.4 agent scores about $8$ points higher when grounded in Librarian than with web search. Overall, our results show that equipping life-science agents with the Librarian knowledge layer improves performance across a range of tasks. We release our code publicly at https://github.com/petroni-lab/librarian

1 INTRODUCTION

EMBL AI Librarian is an agent-first knowledge layer that lets life-science agents ask Europe PMC questions in natural language and receive compact, citable evidence. It orchestrates live literature retrieval and evidence selection, improving performance across literature synthesis, claim verification, open-domain question answering, and biology workflow tasks.

  • Motivation: Automated-agent traffic now rivals or exceeds human-generated web traffic, motivating agent-first infrastructure in life-science research workflows.The paper frames this shift as extending beyond the consumer web into everyday research.
  • Existing infrastructure: Europe PMC provides open life-science literature access through its website, API, and bulk download, including 11.9 M full-text articles, 40.7 M PubMed abstracts, and 1.2 M preprint records.Its coverage of medicine and health subjects exceeds PubMed.
  • Problem: Europe PMC requires agents to use keyword-based queries and often issue multiple complementary searches, while returning whole papers rather than directly answering natural-language questions.Biological aliases, including gene symbols, protein names, disease synonyms, and MeSH terms, complicate comprehensive retrieval.
  • Contribution: EMBL AI Librarian queries Europe PMC’s live search without maintaining its own index, using one LLM to generate complementary queries, select papers, score evidence, and return ranked citable snippets.The system preserves a natural-language, citable-evidence interface while reusing Europe PMC’s curated, fielded search infrastructure.
  • Architecture: The knowledge layer is model-agnostic, allowing any LLM to serve as its engine and enabling the system to benefit from newer, more capable models.The controller orchestrates the entire retrieval pipeline.
  • Evaluation: Librarian improves agent performance across literature synthesis, claim verification, open-domain question answering, and biology workflow tasks.The evaluated biology tasks include protocol questions, sequence manipulation, and molecular cloning.

2 EMBL AI LIBRARIAN

EMBL AI Librarian converts natural-language queries into ranked, fine-grained evidence items rather than whole-paper search results. Its three-stage pipeline generates validated complementary Europe PMC subqueries, retrieves and ranks paper text, and filters and re-ranks passages into final evidence.

  • Evidence output: LIBRARIAN returns ranked evidence items comprising a short text span, citation metadata, and the Europe PMC query used for retrieval.This finer-grained output lets downstream agents quote, cite, and reason over evidence directly.
  • Stage 1: Subquery generation: The pipeline uses recall-oriented retrieval because life-science concepts appear in multiple surface forms, including gene symbols, protein names, disease names, organisms, and MeSH terms.A single query may miss records indexed under alternative terminology.
  • Stage 1: Subquery generation: Stage 1 maps each user query to complementary keyword or fielded Europe PMC subqueries with an LLM, then validates their syntax deterministically.Subqueries can target titles, abstracts, MeSH terms, genes or proteins, organisms, diseases, chemicals, and other Europe PMC fields.
  • Stage 2: Paper-level retrieval and paragraph ranking: Stage 2 executes validated subqueries in parallel against the live Europe PMC service, pools relevance-ordered records, removes duplicate papers, and ranks their paragraphs.Each query returns up to P records, producing a pool of at most M = N × P records before deduplication.
  • Stage 3: Filtering, re-ranking, and evidence extraction: Stage 3 segments candidate-paper text into sentences, uses an LLM to order sentence identifiers, and filters, re-ranks, and extracts the final evidence.The overview describes this stage as operating on top-ranked paragraphs and abstracts.

3 EXPERIMENTS

The experiments evaluate LIBRARIAN across literature synthesis, claim verification, full-text factoid QA, and foundational biology tasks. LIBRARIAN improves evidence grounding and consensus agreement, while its effects on biology tasks depend on model and task.

  • 3 EXPERIMENTS: The evaluation spans four complementary benchmarks: ScholarQABench, ProClaim-eval, open-form LitQA2, and four LAB-Bench biology tasks.These assess multi-paper synthesis, literature-consensus verdicts, full-text factoid answering, and database, protocol, sequence, and cloning capabilities.
  • ScholarQABench: +11.1 Citation F1 on Bio and +5.4 Citation F1 on Neu over OpenScholar-70B are achieved by the Synthesis Agent using the same OSDS retrieval corpus.Equipping the agent with LIBRARIAN makes it stronger still, according to the reported findings.
  • ProClaim-eval: 0.66 average agreement is reached by the LIBRARIAN-equipped Verifier Agent, +15 points over PubMed and Semantic Scholar retrieval and 9 points below the original ProClaim pipeline.Agreement measures the fraction of claims assigned the correct verdict.
  • Open-form LitQA2: 100 Coverage with 17.6 precision and 17.6 accuracy occurs without retrieval, while Web-Search raises precision to 76.2 and accuracy to 70.3 on open-form LitQA2.The QA Agent uses GPT-5.4 and may abstain when retrieved evidence is insufficient; the benchmark retains 91 full-text-supported questions.
  • LAB-Bench: For GPT-4o on LAB-Bench, LIBRARIAN raises macro precision from 49.0 to 54.0 and lowers macro coverage from 70.2 to 64.8, leaving macro accuracy nearly flat from 35.2 to 35.3.On database lookups, grounding changes accuracy by −2.3 for GPT-5.4 and −1.3 for GPT-4o; human experts reach 73.2 macro accuracy.

4 DISCUSSION

EMBL AI Librarian is a plug-and-play knowledge layer that improves life-science agents across literature-centered benchmarks and biology tasks. Its coverage is limited by Europe PMC and paywalled full texts, while multi-hop requests remain the downstream agent’s responsibility.

  • Capabilities: The same knowledge layer improves literature synthesis, claim verification, open-form question answering, and literature-dependent biology tasks.It achieves the highest agreement in the ProClaim pipeline, performs best with GPT-5.4 on open-form question answering, and helps sequence manipulation and molecular cloning.
  • Capabilities: Tasks focused mainly on structured-database lookup gain little from the literature-based layer.
  • Limitations: Europe PMC coverage excludes the full text of some paywalled papers, and single retrieval leaves multi-hop requests to downstream agents.Planned extensions include EMBL resources such as OpenTargets, UniProt, and ChEMBL.

5 RELATED WORK

Related work spans dense-indexed systems for scientific literature, LLM-controlled scholarly search interfaces, and AI agents that execute multi-step life-science workflows. These efforts motivate strengthening the knowledge interfaces on which scientific agents rely.

  • Dense-indexed scientific literature systems: Dense-indexed literature systems combine large scientific corpora with retrieval, reranking, feedback, citation reasoning, or agentic synthesis.Examples include OpenScholar, SciRAG, PaperQA2, BioSage, and SciSage.
  • LLM-controlled scholarly search interfaces: LLM-controlled scholarly search systems show that language models can effectively use structured lexical-search tools and construct corpus-informed BM25 queries.PI-SERINI combines BM25 with retrieve, browse, and read tools, while SIRA weights BM25 queries using corpus-discriminative terms identified by an LLM.
  • AI Agents in the life sciences: Life-science AI agents increasingly plan and execute multi-step scientific workflows, including general biomedical, domain-specific microbiome, and autonomous scientist systems.Examples include Biomni, Eubiota, Kosmos, and the AI co-scientist.

6 CONCLUSION

EMBL AI Librarian is a life-science knowledge layer that gives AI agents a natural-language interface to Europe PMC and returns focused, citable evidence rather than whole papers. A single LLM orchestrates fielded search and evidence localization without specialized-model training or dense-vector maintenance.

  • Contribution: EMBL AI Librarian gives AI agents a natural-language interface to Europe PMC and returns focused evidence instead of whole papers.It is designed as a life-science knowledge layer for agent access to literature.
  • Contribution: A single LLM generates fielded lexical subqueries executed by the Europe PMC search engine, then localizes relevant evidence within retrieved full-text articles.The orchestration combines complementary search with within-paper evidence finding.
  • Contribution: The system yields compact, citable evidence snippets without training a specialized model or maintaining a dense vector.This describes the implementation approach reported in the conclusion.

A IMPLEMENTATION DETAILS

EMBL AI Librarian’s subquery planning, relevance filtering, and synthesis all run on GLM-5 via a vLLM-compatible OpenAI API endpoint, while other retrieval-stack models are identified separately.

  • Model deployment: All Librarian retrieval and synthesis components use the glm-5-fp8 deployment of GLM-5, served through a vLLM-compatible OpenAI API endpoint.This includes subquery planning, relevance filtering, and every synthesis step.
  • Model deployment: Models outside the retrieval stack are not GLM-5 and are named where used, including LAB-Bench answering models, the LitQA2 judge, and the AutoAIS attribution model.These models are specified in Appendices B.3, B.2, and B.1, respectively.

A.1 SUBQUERY VALIDATION AND FALLBACKS · A.2 XML HANDLING · A.3 RUNTIME CONFIGURATION

EMBL AI LIBRARIAN validates planned subqueries, applies conservative fallbacks, deduplicates overlapping records, and preserves non-empty evidence output when filtering fails. It retrieves and caches JATS XML with focused content selection and uses fixed, parallel retrieval settings across experiments.

  • A.1 SUBQUERY VALIDATION AND FALLBACKS: Planned subqueries are parsed before execution, rejecting invalid field names and malformed Boolean expressions before Europe PMC search.Only surviving subqueries are executed.
  • A.1 SUBQUERY VALIDATION AND FALLBACKS: Unparseable planner JSON is retried once at temperature 0, then replaced by the raw user query if parsing fails again.If every subquery is rejected, the original planned subqueries are searched unmodified rather than substituting the raw query.
  • A.1 SUBQUERY VALIDATION AND FALLBACKS: A final fallback returns the unfiltered candidate pool when both relevance filters reject every candidate, trading precision for non-empty output.This prevents an empty evidence set.
  • A.1 SUBQUERY VALIDATION AND FALLBACKS: Overlapping records are hierarchically deduplicated by PMID, then DOI when PMID is absent, and finally title.This handles records such as preprints or some PMC content that may lack PMIDs, while DOI and title matching capture duplicate metadata representations.
  • A.2 XML HANDLING: For fetchable Europe PMC full text, the system retrieves JATS XML once and caches it by PMCID.Parsing retains body paragraphs while dropping abstracts, references, appendices, front and back matter, and specified administrative or supplementary sections.
  • A.3 RUNTIME CONFIGURATION: 8 search workers and 12 full-text-fetching workers are used in every reported experiment.Abstract and full-text relevance filters use separate batch sizes, with a full-text batch of 3 limiting peak token bursts.
  • A.3 RUNTIME CONFIGURATION: The final evidence set draws from two capped pools of up to 30 full-text papers and up to 30 abstract-only papers.These retrieval settings are identical across all reported experiments.

B BENCHMARK DETAILS · B.1 SCHOLARQA-BENCH · B.2 LITQA2

The benchmarks use Europe PMC-aligned subsets and retrieval settings, with ScholarQABench evaluated on biomedical and neuroscience synthesis splits and LitQA2 restricted to 91 retrievable full-text questions. Evaluation uses Citation F1 for ScholarQABench and a fixed GPT-4o judge for open-form LitQA2 answers.

  • B BENCHMARK DETAILS: Reported dataset choices define the benchmark configurations used for the reported numbers.These choices cover split selection, retrieval sources, evaluation protocol, subset construction, and judging.
  • B.1 SCHOLARQA-BENCH: ScholarQABench reports the biomedicine and neuroscience multi-document synthesis splits, omitting single-document splits and using a 29-question biomedicine subset for Multi.The selected splits align with Europe PMC coverage; single-document splits test extraction from provided documents rather than open-domain synthesis.
  • B.1 SCHOLARQA-BENCH: ScholarQABench compares Europe PMC with OSDS, whose abstracts and full-text chunks are re-indexed into a local Elasticsearch instance for retrieval.Europe PMC is the deployed live source for the other benchmarks, while OSDS is the source used by published baselines.
  • B.1 SCHOLARQA-BENCH: Citation F1 is reported on 1451 Bio questions, 1308 Neu questions, and 29 Multi questions using AttrScore AutoAIS support scoring.Citation F1 is the harmonic mean of corpus-level citation precision and recall, expressed as a percentage.
  • B.2 LITQA2: LitQA2 evaluation retains 91 of the 199 original questions whose supporting papers are retrievable as full text through Europe PMC.Availability is checked using source-DOI matching with HAS_FT:Y and confirmation that the JATS XML endpoint returns the article.
  • B.2 LITQA2: The 91-question LitQA2 subset tests retrieval quality rather than corpus coverage, and all compared systems use it unless stated otherwise.The subset excludes 50 of the 141 questions satisfying HAS_FT:Y because their corresponding articles were not retrievable.
  • B.2 LITQA2: LitQA2 open-form answers are judged by openai/gpt-4o-2024-11-20 at temperature 0 with a constrained JSON schema.The judge receives the question, reference answer, system answer, and—when available—the gold key passage, then returns flags about answer correctness and unsupported commitment.

B.3 LAB-BENCH · C PROMPT TEMPLATES

The paper evaluates Librarian retrieval augmentation on four LAB-Bench biology tasks and specifies prompt templates for query planning, evidence filtering, and relevance ranking. The templates emphasize recall, semantic evidence verification, and ordered selection of papers directly answering the query.

  • B.3 LAB-BENCH: LAB-Bench evaluation covers database lookup, sequence manipulation, protocol troubleshooting, and molecular cloning across DbQA, SeqQA, ProtocolQA, and CloningScenarios.All questions are evaluated using accuracy, precision, and coverage under the LAB-Bench protocol.
  • B.3 LAB-BENCH: 520 DbQA, 600 SeqQA, 108 ProtocolQA, and 33 CloningScenarios questions are evaluated.Each question includes gold answers, distractors, and an “Insufficient information” refusal option.
  • B.3 LAB-BENCH: LIBRARIAN rows keep GPT-4o or GPT-5.4 fixed and add one retrieval step that injects raw passages under a 1500-character budget before a single answer call.There is no synthesis pass or second retrieval round; raw nucleotide and amino-acid sequences are stripped from SEQQA and CLONINGSCENARIOS queries.
  • B.3 LAB-BENCH: When retrieval returns nothing, the empty evidence block biases the model toward refusal, while the question is otherwise answered from the bare multiple-choice prompt.This fallback affects a minority of questions, so LIBRARIAN results mix grounded and parametric answers.
  • C PROMPT TEMPLATES: The prompt templates are runtime-filled with the user query, date, candidate papers, and output-channel settings, with relevance prompts reproduced verbatim and planner and synthesis prompts abridged.The abridged prompts retain behavioral instructions while omitting field catalogs, diversity strategies, worked examples, citation examples, and formatting details.
  • C.1 STAGE 1 (SUBQUERY PLANNER) PROMPT: The subquery planner uses multiple diverse Europe PMC searches and prioritizes recall over precision, with downstream filtering handling precision.It anchors relative date windows to the current date and converts them into explicit PUB_YEAR constraints, including the current partial unit.
  • C.1 STAGE 1 (SUBQUERY PLANNER) PROMPT: The planner requires at least 30-40% plain-text queries, limits queries to two AND operators, and distributes the budget across broad, synonym-expanded, and high-precision tiers.The tier allocation is about 40% broad recall, 40% synonym-expanded field queries, and 20% high-precision structured queries.
  • C.2 STAGE 3 (ABSTRACT RELEVANCE FILTER & EVIDENCE FILTER AND RE-RANKING) PROMPT: The abstract and full-text filters select and rank evidence-bearing papers by semantic relevance, requiring query-specific facts and prioritizing papers that directly answer the full query.The filters cite supporting sentence IDs, respect date and institution constraints, reject keyword-only or mismatched contexts, and output ordered relevant_ids.

D BENCHMARK-SIDE PROMPTS

The benchmark-side prompts define how retrieved evidence is transformed into synthesis, verification, and exact-answer outputs. They enforce source fidelity, structured responses, citation or JSON constraints, and task-specific evaluation rules.

  • D BENCHMARK-SIDE PROMPTS: Benchmark-side prompts surround the Librarian’s internal prompts by specifying each benchmark’s required output and scoring procedure.The internal Librarian prompts are identical across benchmarks, while these prompts define benchmark-specific answer production and evaluation.
  • D.1 SQA-BENCH - SYNTHESIS AGENT PROMPT: The SQA-Bench synthesizer targets expert readers with concise, exhaustive answers focused on answer-changing findings, caveats, and disagreements.It begins with an Executive Summary, avoids generic introductions, and provides the closest evidence when papers do not fully answer the query.
  • D.1 SQA-BENCH - SYNTHESIS AGENT PROMPT: SQA-Bench responses are organized by findings, mechanisms, methods, context, or disagreement rather than by individual paper summaries.The prompt also favors short paragraphs or tight bullets and merged repetition.
  • D.1 SQA-BENCH - SYNTHESIS AGENT PROMPT: SQA-Bench requires using only supplied scientific content, treating each paper’s Evidence field as primary, and applying validated inline numeric citations.It prohibits outside knowledge, embedded paper instructions, author-year citations, bibliographies, and unverified citation numbers.
  • D.1 SQA-BENCH - SYNTHESIS AGENT PROMPT: When available, SQA-Bench prioritizes the Full Text Attention Anchor and treats the Full Text Excerpt as primary for methods, results, datasets, and statistics.The final quality check requires every non-trivial claim to have the correct citation and keeps answers concise and scannable.
  • D.2 PROCLAIM: VERIFIER AGENT PROMPT: The ProClaim verifier makes one temperature-0 call over numbered retrieved passages and returns a JSON verdict of SUPPORT, REFUTE, or UNCERTAIN.Its JSON includes concise reasoning and citation metadata with evidence.
  • D.3.1 QA AGENT RETRIEVAL HINT: LitQA2 retrieval preserves every named entity and prioritizes primary passages containing the exact relation, without changing retrieval parameters.The hint targets questions involving values, genes or proteins, residue ranges, helices, mutations, percentages, fold-changes, or structures.

E EUROPE PMC WEB INTERFACE

Europe PMC offers both keyword-based and structured search interfaces. Its Advanced Search interface generates structured query expressions used programmatically by the retrieval pipeline.

  • Search interfaces: Advanced Search constructs structured Europe PMC query expressions for programmatic retrieval.Figure 3 illustrates the syntax used by the retrieval pipeline.
  • Search interfaces: Europe PMC supports keyword-based retrieval through its standard free-text search interface.The standard interface is contrasted with the structured-query capabilities of Advanced Search.
Loading 2607.28229v1…