Large language model-assisted discovery of cohorts from scientific literature

Background

• Researchers need suitable cohorts with individual-level data and relevant phenotypes to reuse and jointly analyze human data in population-based data science.Identifying such cohorts is a critical first step for these analyses.Source: Introduction · p. 1
• Cohort information is fragmented, and reliance on prior knowledge, web searches, or colleagues can favor well-known cohorts and make findings difficult to audit or reproduce.Catalogues including Maelstrom, Atlas of Longitudinal Datasets, Catalogue of Mental Health Measures, and dbGaP differ in coverage and terminology and are generally organized around resources and measures, not individual research questions.Source: Introduction · p. 2

Introduction

• Although prior work has explored literature-based dataset discovery, the authors knew of no systematic framework for finding suitable cohorts for a specific research question.Existing catalogue-based LLM systems can match query meaning to catalogue metadata, but are limited to resources represented in those catalogues.Source: Introduction · p. 2
• The pipeline systematically searches PubMed from a research question and extracts explicitly named cohorts from titles and abstracts without a predefined cohort list.Its PubMed search and LLM extraction are configurable for other populations, phenotypes, data modalities, and study designs.Source: Introduction · p. 2
• Youth aggression genetics is the use case because genetic analyses of aggressive behaviour in children and adolescents need samples larger than most individual cohorts provide.Cross-cohort research is therefore necessary, but a unified resource of suitable cohorts for this question is missing.Source: Introduction · p. 2
• The use case also validates the pipeline by screening identified cohorts against predefined eligibility criteria and comparing them with established catalogues for overlap and complementary coverage.The workflow searches publications and extracts cohort-level metadata, while targeting human cohorts containing individual-level data.Source: Introduction · p. 2

Study design and framework overview

• The three-stage workflow retrieves question-specific publications through PubMed, extracts explicit cohort mentions with an LLM [24], and outputs deduplicated cohort names, variants, and source records.Its inputs include configurable concept vocabularies, query templates, and extraction instructions.Source: Study design and framework overview · p. 2

Question-specific literature retrieval

• Two configurable query templates broaden or narrow matching across population, domain, phenotype, modality, and study design; the workflow retrieves up to 50 top-ranked titles and abstracts per query.The broad template requires either domain or phenotype to reduce terminology-related misses; the more restrictive template also requires study design, and records are deduplicated by PubMed ID.Source: Question-specific literature retrieval · p. 3

LLM-assisted cohort-name extraction

• GPT-4o screens retrieved titles and abstracts for inclusion and explicit cohort names, repeating extraction three times and leaving final eligibility decisions to human verification.Only titles and abstracts were submitted, not full texts or participant-level data; included records with detected names proceed downstream, with each name linked to its source PMID.Source: LLM-assisted cohort-name extraction · pp. 3–4

Cohort alias and identity resolution

• Near-duplicate names are clustered by string similarity at a 0.8 threshold, then manually reviewed to assign canonical cohort labels, remove non-cohort entities, and link aliases to source PubMed IDs.Human review addresses abbreviations, alternative spellings, and nested substudies that cannot be safely merged without context.Source: Cohort alias and identity resolution · p. 4

Case-study search

• Across 5,400 PubMed queries, retrieval produced 5,254 unique records; extraction identified 188 candidate cohorts, and manual screening retained 44 eligible cohorts with approximately 890,000 reported participants.Query terms spanned population, domain, phenotype, data modality, and study design. Eligibility required documented access, individual-level genetic data, more than 1,000 participants, participants younger than 18, and validated aggression-related measures.Source: Case-study search · p. 4

Validation of automated name extraction

• LLM extraction was within the human-annotator agreement range: F1 = 0.821 vs. R1, 0.648 vs. R2, compared with 0.693 for R1 vs. R2.It achieved perfect precision and recall = 0.769; most false negatives involved boundary or out-of-domain studies, reflecting the LLM’s more conservative, topic-focused interpretation.Source: Table 1 · p. 5, Validation of automated name extraction · p. 5
• Two independent extraction runs across 5,254 title–abstract pairs had Jaccard similarity J(A, B) = 0.85.Agreement was measured between the PubMed ID sets for records in which each run detected explicit names.Source: Validation of automated name extraction · p. 5

Comparison with established cohort catalogues

• The four documented catalogue searches—Atlas [7], CatMH [8], Maelstrom, and dbGaP —returned 27 of 44 eligible cohorts (61.4%); 17 were absent from all four.The catalogue union contained 253 distinct identities, with 237 of 253 (93.7%) appearing in only one catalogue.Source: Figure 2 · p. 6, Comparison with established cohort… · p. 6

Discussion

• Building on prior cohort-mention extraction, the configurable workflow extends this approach to question-driven discovery, reducing 5,254 records to 188 candidates at approximately USD 10.The resulting candidates were screened for cohort eligibility.Source: Discussion · p. 6
• Automated extraction fell within human inter-rater agreement, cohort naming showed interpretive uncertainty, and repeated runs consistently identified the same mentions.GPT-4o was evaluated, but the contribution is the configurable workflow rather than a specific model; human eligibility assessment provided an independent validation setting.Source: Discussion · p. 7
• The workflow complements catalogues by finding cohorts through question-relevant publications; 17 eligible literature-identified cohorts were absent from all four documented searches.The authors propose using the workflow to broaden candidate lists and help identify cohorts for possible addition to catalogues.Source: Discussion · p. 7
• The evaluation covered one application, used PubMed, and depended on cohort names and relevant information appearing in titles and abstracts.Performance with other populations, phenotypes, and terminology remains unestablished; cohorts mainly documented in other disciplines or indexing systems may be less visible.Source: Discussion · p. 7
• Catalogue comparisons were constrained by platform-specific search strategies and unscreened catalogue-only identities, so they describe overlap rather than fully comparable eligibility.PubMed retrieval and manual screening occurred in 2025, while catalogue searches occurred in July 2026; changes during that interval may have affected the observed overlap.Source: Discussion · p. 7

• The configurable, LLM-assisted and human-reviewed workflow combines systematic PubMed retrieval, contextual cohort-name extraction, and canonical identity resolution without relying on a predefined cohort list.It converts a large publication corpus into a screenable cohort inventory.Source: Conclusions · p. 7
• The youth aggression genetics application produced an open catalogue of 44 eligible cohorts, and comparison with four established catalogues showed complementary routes to cohort discovery.The workflow may also help identify candidates for cohort catalogue curation.Source: Conclusions · p. 7
• Further applications should establish performance across other populations, phenotypes, data modalities, and study designs.Source: Conclusions · p. 7

Data provenance statement

• PubMed records were retrieved through generated case-study queries beginning 20 January 2025 and deduplicated by PMID before title and abstract processing.Titles and abstracts were processed in three independent LLM runs.Source: Data provenance statement · p. 8
• Cohort names were combined across LLM runs, clustered by string similarity, manually reconciled to canonical labels, and assessed by two reviewers with disagreements adjudicated.Disagreements were resolved by consensus and adjudication by a third reviewer.Source: Data provenance statement · p. 8
• External comparison searches covered four documented catalogues on 23 July 2026, with identities linked through abbreviations and reviewed name variants before cross-catalogue deduplication.The catalogues were the Atlas of Longitudinal Datasets, Catalogue of Mental Health Measures, Maelstrom Catalogue, and dbGaP.Source: Data provenance statement · p. 8
• Source PMIDs, processing outputs, and configurable code are available in the project repository, and the resulting cohort catalogue is archived on Zenodo.The repository is https://gitlab.rz.uni-frankfurt.de/cap_molgenlab/literature-cohort-discovery; the catalogue DOI is https://doi.org/10.5281/zenodo.18216492.Source: Data provenance statement · p. 8

Ethics statement

• Ethical approval was not required because the study used publicly available publication records and cohort-level metadata, without participants, participant-level data, or identifiable personal information.Source: Ethics statement · p. 8

• GPT-4o processed only publication titles and abstracts to extract candidate cohort names; no participant-level data or personal identifiers were submitted.Its outputs generated candidates rather than final eligibility decisions, so a demographic fairness audit was not applicable.Source: AI disclosure statement · p. 8
• OpenAI Codex and GPT-5 assisted with manuscript editing and code restructuring and development.Source: AI disclosure statement · p. 8
• The authors made all screening and analytical decisions, verified AI-assisted outputs, and take full responsibility for the code, analyses, and manuscript.Potential biases related to PubMed coverage, publication practices, and explicit cohort naming are addressed in the Discussion.Source: AI disclosure statement · p. 8

Supplementary Appendix

• It provides a structured extraction response, an example workflow, extraction-validation details, and documented catalogue search strategies.Source: Supplementary Appendix · p. 9

Supplementary Section S1. Structured extraction response

• Only INCLUDE responses with category 1—one or more explicit cohort names—were retained from the three-field decision, category, and details schema.Category 0 indicated no named resource; category 2 indicated a referenced but unnamed resource.Source: Supplementary Section S1. Structured… · p. 12

Supplementary Section S2. Example extraction workflow

• Titles and abstracts were submitted to GPT-4o in a structured prompt, and extracted cohort names were linked to PMIDs and added to the candidate list.The example prompt, publication record, and model output were based on real instances; the displayed abstract was shortened.Source: Figure S1 · p. 13

Classification metrics

• Recall was defined as TP/(TP + FN), and precision as TP/(TP + FP), using true-positive, false-positive, false-negative, and true-negative counts.Classification performance was calculated from TP, FP, FN, and TN.Source: Classification metrics · p. 13

Examples of false-negative records

• False negatives commonly involved boundary or out-of-domain records, including adult follow-up cohorts recruited in youth, lifespan-happiness studies, and a Marfan syndrome genetic biobank.Examples named VINGO and LONGSCAN; ABCD, Add Health, and UK Biobank; and a non-psychiatric genetic biobank.Source: Examples of false-negative records · p. 13

Atlas of Longitudinal Datasets

• The search used two full Boolean expansions and documented filters for accessible or contact-based access, mental-health or psychological-measure data, biobank/cohort/registry designs, ages 0–18 years, and sample sizes above 1,000.Source: Atlas of Longitudinal Datasets · pp. 13–14

Catalogue of Mental Health Measures

• The Catalogue of Mental Health Measures search documented facets for biomarkers, recruitment from birth through age 18, and sample sizes of at least 1,000.Source: Catalogue of Mental Health Measures · p. 14

Maelstrom Catalogue

• The Maelstrom Catalogue search required a maximum age below 18, more than 1,000 participants, and possible data access.Source: Maelstrom Catalogue · p. 14

dbGaP

• The dbGaP searches paired child, adolescent, or youth terms with aggression, conduct-disorder, behavioral-problem, impulsivity, or callous-unemotional-trait terms.A second documented query also paired mental-health terms with aggression-related terms.Source: dbGaP · p. 14