Source-linked AI summary
Large language model-assisted discovery of cohorts from scientific literature
Moritz Sturm, Lisa M. Berg, Inken Berg, Harishny Sarma, Jasmin Hartmann, Denissa Girschik, Gemma Roig, Christine M. Freitag, Andreas G. Chiocchetti
TL;DR
Researchers lack a unified resource for finding cohorts tailored to specific research questions. This paper introduces a configurable literature-search and LLM extraction workflow that produced 44 eligible cohorts in youth aggression genetics.
Problem
Researchers need suitable cohorts with relevant populations and phenotypes, but a unified resource for question-specific cohort discovery is missing.
Method
The configurable workflow retrieves question-specific PubMed publications, extracts explicit cohort names with an LLM, and deduplicates aliases into cohort inventories.
Results
The youth aggression genetics application produced an open catalogue of 44 eligible cohorts from 188 screening candidates.
Takeaways & Limitations
The workflow provides a literature-based complement to cohort catalogues and can identify candidates for future catalogue curation.
Takeaways & Limitations
The study did not explicitly test whether newer LLM versions improve the extraction step and retained human assessment for eligibility.
Abstract
from arXiv · showhide
Background: Planning multi-study analyses requires identifying cohorts with the relevant participants, phenotypes, and data modalities. This process commonly relies on prior knowledge, cohort catalogues, and manual literature searches. We developed a complementary question-driven framework that searches relevant scientific literature and extracts explicit cohort names. Methods: The framework first generates multiple PubMed queries from configurable vocabularies and templates and retrieves the resulting scientific literature automatically through the PubMed API. A large language model then screens the retrieved titles and abstracts and extracts explicit cohort names using a prompt tailored to the research question. The extracted names are deduplicated with human review. Configurable code, prompts, and example outputs are available at https://gitlab.rz.uni-frankfurt.de/cap_molgenlab/literature-cohort-discovery. Evaluation: As a use case, we applied the framework to youth aggression genetics. From 5,400 generated PubMed queries, the framework retrieved 5,254 unique records and identified 188 candidate cohorts. Manual screening using predefined criteria, including participant age and genetic-data availability, retained 44 eligible cohorts. Automated LLM-based name extraction was within the agreement range of human annotators. We also searched four established cohort catalogues using the same research question. Their combined results contained 27 of the 44 eligible cohorts, while 17 were not returned by any cohort catalogue search. Conclusion: The framework converts research-question-specific vocabulary into screenable cohort inventories via a large, automated literature search. It can be adapted across populations, phenotypes, data modalities, and study designs, and provides a literature-based complement to curated cohort catalogues.
Background · Introduction
Researchers increasingly need to identify cohorts containing relevant populations, individual-level data, and phenotypes for population-based analyses. This study introduces a configurable, LLM-assisted pipeline that systematically searches PubMed and extracts explicitly named cohorts, demonstrated in genetic research on aggressive behaviour in children and adolescents.
- Introduction: Population-based data science increasingly depends on reusing and jointly analyzing existing human data.Suitable cohorts must contain individual-level data from the population of interest and include relevant phenotypes.
- Introduction: Researchers commonly identify cohorts through fragmented internet information, prior knowledge, exploratory web searches, or colleagues’ recommendations.These approaches may favor well-known cohorts and produce sets that are difficult to audit or reproduce.
- Introduction: The pipeline translates research questions into systematic PubMed searches and extracts explicitly named cohorts from titles and abstracts without a predefined cohort list.It is configurable and LLM-assisted.
- Introduction: The approach is demonstrated in genetic research on aggressive behaviour in children and adolescents.This use case applies the cohort-discovery approach to a specific research question.
- Introduction: Genetic analyses of aggressive behaviour typically require samples substantially larger than most individual cohorts can provide.This requirement makes cross-cohort research necessary.
- Introduction: A unified resource of suitable cohorts for genetic research on aggressive behaviour is currently missing.The gap makes aggressive behaviour a suitable use case for niche cohort discovery and pipeline validation.
- Introduction: The identified cohorts are evaluated through human screening against predefined criteria.The supplied passage describes this evaluation as part of validating the automated pipeline.
Methods
The workflow identifies research-question-specific cohorts through PubMed retrieval, LLM-assisted extraction of explicit names, and deduplication with human review. Configurable vocabularies, query templates, and extraction instructions produce deduplicated cohort inventories linked to name variants and source records.
- Workflow: The three-stage workflow retrieves question-specific publications, extracts explicit cohort mentions with an LLM, and deduplicates aliases and alternative names.Its inputs are configurable concept vocabularies, query templates, and extraction instructions; outputs include deduplicated cohort names, observed variants, and PubMed records.
- Publication retrieval: Retrieval vocabularies cover population, domain, phenotype, modality, and study design across two configurable PubMed query templates.One template broadens coverage by requiring domain or phenotype, while the other increases specificity by requiring domain, phenotype, and study design.
- Publication retrieval: 50 highest-ranked titles and abstracts per query were retrieved to balance search coverage and computational feasibility.The resulting queries were submitted through the PubMed API.
- LLM-assisted extraction: Three repeated GPT-4o extraction runs screened unique titles and abstracts for inclusion and explicit database, study, or cohort names.Only titles and abstracts were submitted, so the workflow did not access participant-level data or full-text articles; the union of PubMed ID–cohort name pairs entered deduplication.
- Deduplication and review: 0.8 similarity threshold clustering grouped near-duplicate names, after which human review assigned canonical cohort labels and removed non-cohort entities.Manual review addressed abbreviations, alternative spellings, and nested substudy names, producing an alias map linked to PubMed IDs.
Evaluation & Illustration
The workflow was illustrated in youth aggression genetics, where automated PubMed retrieval and LLM extraction produced a candidate cohort inventory for manual eligibility screening. LLM performance was compared with human annotations, and the inventory was assessed against established cohort catalogues.
- Youth aggression genetics: 5,400 PubMed queries yielded 5,254 unique records after deduplication, from which LLM extraction identified 188 candidate cohorts.The PubMed API returned 261,621 query-level records, and explicit cohort names were extracted from 476 records before alias resolution.
- Youth aggression genetics: Two independent reviewers screened each candidate after alias resolution to assess eligibility.The supplied passage reports that reviewer screening followed identification of 188 candidate cohorts, but its eligibility outcome is truncated.
- LLM validation: Across 199 abstracts, R2 achieved an F1-score of 0.693 and a balanced accuracy of 0.877 when R1 was treated as the reference.The LLM predictions were based on whether an explicit candidate name was extracted and were compared with the gold standard agreement subset (n = 176) and individual raters.
- LLM validation: The LLM operated within the agreement range of human annotators, aligning more closely with R1 (F1 = 0.821) than R1 aligned with R2 (F1 = 0.693).Agreement with R2 was F1 = 0.648, similar to human-rater agreement according to the reported results.
- LLM validation: Most false negatives involved borderline or out-of-domain studies, while the LLM applied a more conservative, topic-focused interpretation than human raters.Examples included adult samples, general well-being, and non-mental-health cohorts that human raters tended to include when an explicit study or cohort name was present.
- Catalogue comparison: Catalogue searches returned 170 Atlas, 22 CatMH, 20 Maelstrom, and 57 unique dbGaP identities for comparison with the literature-derived inventory.Search strategies and inclusion criteria were translated into catalogue-specific strategies because the platforms differed in scope and search functionality.
Discussion
The discussion presents a configurable workflow that combines automated literature retrieval with LLM-based cohort-name extraction for question-driven cohort discovery. In the youth aggression genetics case study, it efficiently narrowed records to screening candidates, showed human-level agreement, and emphasized workflow configurability over any specific model.
- Methodological contribution: The main methodological advance is integrating automated literature retrieval and LLM-based name extraction into a configurable, question-driven cohort-discovery workflow.This extends prior extraction of cohort and dataset mentions toward research-question-specific discovery.
- Case study: 5,254 unique publication records were reduced to 188 cohort-screening candidates without a predefined list of cohort names.The reduction was demonstrated in the youth aggression genetics case study.
- Case study: Approximately USD 10 made LLM processing cost-effective compared with the substantial manual labour otherwise required.The cost assessment refers to the case-study extraction and screening workflow.
- Validation: Automated extraction performed within the range of human inter-rater agreement, while imperfect human agreement indicated genuine interpretive uncertainty in identifying cohort names.The LLM also consistently identified cohort mentions in the same titles and abstracts across the evaluated runs.
- Limitations and future extensions: The contribution is the configurable workflow rather than a specific LLM, allowing the model to be updated and potentially extended with online information aggregation for eligibility assessment.The proposed extension was not explicitly tested.
Conclusions
The study developed a configurable, LLM-assisted and human-reviewed workflow for identifying scientifically relevant cohorts without relying on a predefined cohort list. By combining PubMed retrieval, contextual cohort-name extraction, and canonical identity resolution, it produced a screenable inventory, including 44 eligible cohorts in a youth aggression genetics application.
- Contributions: The workflow identifies cohorts relevant to a scientific question without relying on a predefined cohort list.It was developed as a configurable, LLM-assisted, human-reviewed approach.
- Contributions: The workflow combines systematic PubMed retrieval, contextual cohort-name extraction, and canonical identity resolution to convert publications into a screenable cohort inventory.
- Application: 44 eligible cohorts were catalogued in the youth aggression genetics application, demonstrating practical feasibility.The resulting catalogue was open.
Data provenance statement
Records originated from PubMed and were retrieved through the PubMed API using generated case-study queries beginning 20 January 2025. Titles and abstracts underwent independent LLM processing, followed by cohort-name clustering, manual alias resolution, and independent reviewer assessment.
- PubMed records were retrieved through the PubMed API using generated case-study queries beginning on 20 January 2025.
- Records were deduplicated by PMID before titles and abstracts were processed in three independent LLM runs.
- Extracted cohort names were combined across runs, clustered by string similarity, and manually reviewed to resolve aliases and assign canonical labels.
- Candidate cohorts were then independently assessed by two reviewers.
Ethics statement
Ethical approval was not required because the study analysed publicly available publication records and cohort-level metadata without involving participants, participant-level data, or identifiable personal information.
- Ethics statement: Ethical approval was not required because the study used publicly available publication records and cohort-level metadata without participant involvement or identifiable personal information.The analysis did not use participant-level data.
AI disclosure statement
GPT-4o was used through the OpenAI API to screen PubMed titles and abstracts and extract candidate cohort names, without processing participant-level data or personal identifiers.
- AI disclosure statement: GPT-4o screened publication titles and abstracts to extract explicit cohort names, while final eligibility decisions remained outside the model.Only publication-level metadata were submitted; no participant-level data or personal identifiers were processed.
Supplementary Appendix
The Supplementary Appendix provides structured extraction responses, an example extraction workflow, extraction-validation details, and documented catalogue search strategies.
- The appendix documents the structured extraction response, example workflow, validation details, and catalogue search strategies.
Supplementary Appendix 1 … dbGaP
The supplementary materials specify the structured extraction protocol, illustrate its AI-assisted workflow, define classification metrics, document catalogue search strategies, and describe false-negative examples. Catalogue searches used predefined filters and Boolean queries targeting youth, mental health or aggression, data access, and study characteristics.
- Supplementary Section S1. Structured extraction response: The fixed extraction schema returned a decision, category, and free-text details, retaining only INCLUDE responses with category 1 explicit cohort names.Category 0 indicated no named resource, category 1 explicit names, and category 2 referenced but unnamed resources.
- Supplementary Section S2. Example extraction workflow: Titles and abstracts retrieved through PubMed were submitted to GPT-4o, which returned JSON decisions and cohort names linked to PMIDs and candidate lists.The workflow used a structured prompt and the GPT-4o API.
- Classification metrics: Classification performance was calculated from true positives, false positives, false negatives, and true negatives using recall and precision formulas.The documented formulas were Recall = TP/(TP + FN) and Precision = TP/(TP + FP).
- Examples of false-negative records: Most false negatives came from studies at or beyond the target-domain boundary, including youth-recruited adult follow-ups, lifespan happiness records, and a Marfan syndrome biobank.Examples included VINGO, LONGSCAN, ABCD, Add Health, UK Biobank, and a non-psychiatric genetic biobank.
- Catalogue of Mental Health Measures: The Catalogue of Mental Health Measures search selected biomarkers, recruitment ages from birth to 18 years, and samples of at least 1,000.These documented facets defined the catalogue search criteria.
- Maelstrom Catalogue: The Maelstrom search required maximum age below 18 years, more than 1,000 participants, and possible data access.These were the documented catalogue filters.
- dbGaP: The dbGaP searches combined child, adolescent, or youth terms with aggression, conduct disorder, behavioral problems, impulsive behavior, callous-unemotional traits, or mental health.Two Boolean query forms were documented for dbGaP.